Original: D ISTRIBUTED G RAPH N EURAL N ETWORK T RAINING WITH P ERIODIC S TALE R EPRESENTATION S YNCHRONIZATION
Revised: D ISTRIBUTED G RAPH N EURAL N ETWORK T RAINING WITH P ERIODIC S TALE R EPRESENTATION S YNCHRONIZATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Despite the recent success of Graph Neural Networks (GNNs), it remains challenging to train a GNN on large graphs with over millions of nodes & billions of edges, which are prevalent in many graph-based applications such as social networks, recommender systems, and knowledge graphs.
Revised: Despite the recent success of Graph Neural Networks (GNNs), it remains challenging to train a GNN on large graphs with over millions of nodes & billions of edges, which are prevalent in many graph-based applications such as social networks, recommender systems, and knowledge graphs.

Original: Traditional sampling-based methods accelerate GNN training by dropping edges and nodes, which impairs the graph integrity and model performance.
Revised: Traditional sampling-based methods accelerate GNN training by dropping edges and nodes, which impairs the graph integrity and model performance.

Original: Differently, distributed GNN algorithms accelerate GNN training by utilizing multiple computing devices and can be classified into two types: " partition-based " methods enjoy low communication cost but suffer from information loss due to dropped edges, while " propagation-based " methods avoid information loss but suffer from prohibitive communication overhead caused by neighbor explosion.
Revised: Differently, distributed GNN algorithms accelerate GNN training by utilizing multiple computing devices and can be classified into two types: " partition-based " methods enjoy low communication cost but suffer from information loss due to dropped edges, while " propagation-based " methods avoid information loss but suffer from prohibitive communication overhead caused by neighbor explosion.

Original: To jointly address these problems, this paper proposes DIGEST (DIstributed Graph reprEsentation SynchronizaTion), a novel distributed GNN training framework that synergizes the complementary strength of both categories of existing methods.
Revised: To jointly address these problems, this paper proposes DIGEST (DIstributed Graph reprEsentation SynchronizaTion), a novel distributed GNN training framework that synergizes the complementary strength of both categories of existing methods.

Original: We propose to allow each device utilize the stale representations of its neighbors in other subgraphs during subgraph parallel training.
Revised: We propose to allow each device utilize the stale representations of its neighbors in other subgraphs during subgraph parallel training.

Original: This way, out method preserves global graph information from neighbors to avoid information loss and reduce the communication cost.
Revised: This way, out method preserves global graph information from neighbors to avoid information loss and reduce the communication cost.

Original: Therefore, DIGEST is both computation-efficient and communication-efficient as it does not need to frequently (re-)compute and transfer the massive representation data across the devices, due to neighbor explosion.
Revised: Therefore, DIGEST is both computation-efficient and communication-efficient as it does not need to frequently (re-)compute and transfer the massive representation data across the devices, due to neighbor explosion.

Original: DIGEST provides synchronous and asynchronous training manners for homogeneous and heterogeneous training environment, respectively.
Revised: DIGEST provides synchronous and asynchronous training manners for homogeneous and heterogeneous training environment, respectively.

Original: We proved that the approximation error induced by the staleness of the representations can be upper-bounded.
Revised: We proved that the approximation error induced by the staleness of the representations can be upper-bounded.

Original: More importantly, our convergence analysis demonstrates that DIGEST enjoys the state-of-the-art convergence rate.
Revised: More importantly, our convergence analysis demonstrates that DIGEST enjoys the state-of-the-art convergence rate.

Original: Extensive experimental evaluation on large, real-world graph datasets shows that DIGEST achieves up to 21 .
Revised: Extensive experimental evaluation on large, real-world graph datasets shows that DIGEST achieves up to 21 .

Original: 82 × speedup without compromising the performance compared to state-of-the-art distributed GNN training frameworks.
Revised: 82 × speedup without compromising the performance compared to state-of-the-art distributed GNN training frameworks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Graph Neural Networks (GNNs) have shown impressive success in analyzing non-Euclidean graph data and have achieved promising results in various applications, including social networks, recommender systems and knowledge graphs, etc.
Revised: Graph Neural Networks (GNNs) have shown impressive success in analyzing non-Euclidean graph data and have achieved promising results in various applications, including social networks, recommender systems and knowledge graphs, etc.

Original: Dai et al.
Revised: (Dai et al., 2016;

Original: ; Ying et al. ; Eksombatchai et al.
Revised: Ying et al., 2018; Eksombatchai et al., 2018;

Original: ; Lei et al. Zhu et al.
Revised: Lei et al., 2019; Zhu et al., 2019).

Original: Despite the great promise of GNNs, they meet significant challenges when being applied to large graphs, which are common in real world—the number of nodes of a large graph can be up to millions or even billions.
Revised: Despite the great promise of GNNs, they meet significant challenges when being applied to large graphs, which are common in real world—the number of nodes of a large graph can be up to millions or even billions.

Original: For instance, Facebook social network graph contains over 2.9 billion users and over 400 billion friendship relations among users 1 .
Revised: For instance, Facebook social network graph contains over 2.9 billion users and over 400 billion friendship relations among users 1 .

Original: Amazon provides recommendations over 350 million items to 300 million users 2 .
Revised: Amazon provides recommendations over 350 million items to 300 million users 2 .

Original: Further, natural language processing (NLP) tasks take advantage of knowledge graphs, such as Freebase Chah (2017) with over 1.9 billion triples.
Revised: Further, natural language processing (NLP) tasks take advantage of knowledge graphs, such as Freebase (Chah, 2017) with over 1.9 billion triples.

Original: Training GNNs on large graphs is jointly challenged by the lack of inherent parallelism in the backpropagation optimization and heavy inter-dependencies among graph nodes, rendering existing parallel techniques inefficient.
Revised: Training GNNs on large graphs is jointly challenged by the lack of inherent parallelism in the backpropagation optimization and heavy inter-dependencies among graph nodes, rendering existing parallel techniques inefficient.

Original: To tackle the unique challenges in GNN training, distributed GNN training is a promising open domain that has attracted fast-increasing attention in recent years.
Revised: To tackle the unique challenges in GNN training, distributed GNN training is a promising open domain that has attracted fast-increasing attention in recent years.

Original: A classic and intuitive way is by sampling .
Revised: A classic and intuitive way is by sampling .

Original: Until now, a good number of graph-sampling-based GNN methods have been proposed, including neighbor-sampling-based methods (e.g., GraphSAGE Hamilton et al. (2017), VR-GCN Chen et al. and subgraphsampling-based methods (e.g., Cluster-GCN Chiang et al. (2019), GraphSAINT Zeng et al.
Revised: Until now, a good number of graph-sampling-based GNN methods have been proposed, including neighbor-sampling-based methods (e.g., GraphSAGE (Hamilton et al., 2017), VR-GCN (Chen et al., 2018)) and subgraphsampling-based methods (e.g., Cluster-GCN (Chiang et al., 2019), GraphSAINT (Zeng et al., 2019)).

Original: These methods enable a GNN model to be trained over large graphs on a single machine by sampling a subset of data during forward or backward propagation.
Revised: These methods enable a GNN model to be trained over large graphs on a single machine by sampling a subset of data during forward or backward propagation.

Original: While sampling operations reduce the size of data needed for computation, these methods suffer from degenerated performance due to unnecessary information loss.
Revised: While sampling operations reduce the size of data needed for computation, these methods suffer from degenerated performance due to unnecessary information loss.

Original: To walk around this drawback and also to leverage the increasingly powerful computing capability of modern hardware accelerators, recent solutions propose to train
Revised: To walk around this drawback and also to leverage the increasingly powerful computing capability of modern hardware accelerators, recent solutions propose to train

Original: GNNs on a large number of CPU and GPU devices Thorpe et al. ; Ramezani et al. (2021); Wan et al. and have become the de facto standard for fast and accurate training over large graphs.
Revised: GNNs on a large number of CPU and GPU devices (Thorpe et al., 2021; Ramezani et al., 2021; Wan et al., 2022) and have become the de facto standard for fast and accurate training over large graphs.

Original: Existing methods in distributed training for GNNs can be classified into two categories, namely " partition-based " and " propagation-based ", by how they tackle the trade-off between computation/communication cost and information loss .
Revised: Existing methods in distributed training for GNNs can be classified into two categories, namely " partition-based " and " propagation-based ", by how they tackle the trade-off between computation/communication cost and information loss .

Original: "Partition-based" methods Angerd et al. ; Jia et al. Ramezani et al. (2021) partition the graph into different subgraphs by dropping the edges across subgraphs.
Revised: "Partition-based" methods (Angerd et al., 2020; Jia et al., 2020; Ramezani et al., 2021) partition the graph into different subgraphs by dropping the edges across subgraphs.

Original: This way, the GNN training on a large graph is decomposed into many smaller training tasks, each trained in a siloed subgraphs in parallel, reducing communications among subgraphs, and thus, tasks, due to edge dropping.
Revised: This way, the GNN training on a large graph is decomposed into many smaller training tasks, each trained in a siloed subgraphs in parallel, reducing communications among subgraphs, and thus, tasks, due to edge dropping.

Original: However, this will result in severe information loss due to the ignorance of the dependencies among nodes across subgraphs and cause performance degeneration.
Revised: However, this will result in severe information loss due to the ignorance of the dependencies among nodes across subgraphs and cause performance degeneration.

Original: To alleviate information loss, "propagation-based" methods Ma et al.

Original: Zhu et al.

Original: ; Zheng et al. (2020); Tripathy et al. ; Wan et al. (2022) do not ignore edges across different subgraphs with neighbor communications among subgraphs to satisfy GNN’s neighbor aggregation.
Revised: To alleviate information loss, "propagation-based" methods (Ma et al., 2019; Zhu et al., 2019; Zheng et al., 2020; Tripathy et al., 2020; Wan et al., 2022) do not ignore edges across different subgraphs with neighbor communications among subgraphs to satisfy GNN’s neighbor aggregation.

Original: However, the number of neighbors involved in neighbor aggregation grows exponentially as the GNN goes deeper (i.e., neighborhood explosion Hamilton et al. (2017)), hence inevitably suffering huge communication overhead and plagued training efficiency.
Revised: However, the number of neighbors involved in neighbor aggregation grows exponentially as the GNN goes deeper (i.e., neighborhood explosion (Hamilton et al., 2017)), hence inevitably suffering huge communication overhead and plagued training efficiency.

Original: Therefore, although "partition-based" methods can parallelize a training job among partitioned subgraphs, they suffers from information loss and low accuracy.
Revised: Therefore, although "partition-based" methods can parallelize a training job among partitioned subgraphs, they suffers from information loss and low accuracy.

Original: "Propagation-based" methods, on the other hand, use the entire graph for training without information loss but suffer from huge communication overhead and poor efficiency.
Revised: "Propagation-based" methods, on the other hand, use the entire graph for training without information loss but suffer from huge communication overhead and poor efficiency.

Original: Hence, it is highly imperative to develop a method that can jointly address the problems of high communication cost and severe information loss.
Revised: Hence, it is highly imperative to develop a method that can jointly address the problems of high communication cost and severe information loss.

Original: Moreover, theoretical guarantees (e.g., on convergence, approximation error) are not well explored for distributed
Revised: Moreover, theoretical guarantees (e.g., on convergence, approximation error) are not well explored for distributed

Original: GNN training due to the joint sophistication of graph structure and neural network optimization.
Revised: GNN training due to the joint sophistication of graph structure and neural network optimization.

Original: To address the aforementioned challenges, we propose a novel distributed GNN training framework that synergizes the complementary strengths of both partitioning-based and propagating-based methods, named DI stributed G raph repr E sentation S ynchroniza T ion, or DIGEST .
Revised: To address the aforementioned challenges, we propose a novel distributed GNN training framework that synergizes the complementary strengths of both partitioning-based and propagating-based methods, named DI stributed G raph repr E sentation S ynchroniza T ion, or DIGEST .

Original: DIGEST does not completely discard node information from other subgraphs in order to avoid unnecessary information loss; DIGEST does not frequently update all the node information in order to minimize communication costs.
Revised: DIGEST does not completely discard node information from other subgraphs in order to avoid unnecessary information loss; DIGEST does not frequently update all the node information in order to minimize communication costs.

Original: Instead, DIGEST extends the idea of single-GPU-based GCN training with stale representations Chen et al. (2018); Fey et al. (2021) to a distributed setting, by enabling each device to efficiently exchange a relatively stale version of the neighbor representations from other subgraphs, to achieve scalable and high-performance GNN training.
Revised: Instead, DIGEST extends the idea of single-GPU-based GCN training with stale representations (Chen et al., 2018; Fey et al., 2021) to a distributed setting, by enabling each device to efficiently exchange a relatively stale version of the neighbor representations from other subgraphs, to achieve scalable and high-performance GNN training.

Original: This effectively avoids neighbor updating explosion and reduces communication costs across training devices.
Revised: This effectively avoids neighbor updating explosion and reduces communication costs across training devices.

Original: Considering naive synchronous distributed training that inherently lacks the capability of handling stragglers caused by training environment heterogeneity (e.g. GPU resource heterogeneity), we further design an asynchronous version of DIGEST (DIGEST-A), where each subgraph follows a non-blocking training manner.
Revised: Considering naive synchronous distributed training that inherently lacks the capability of handling stragglers caused by training environment heterogeneity (e.g. GPU resource heterogeneity), we further design an asynchronous version of DIGEST (DIGEST-A), where each subgraph follows a non-blocking training manner.

Original: The synchronous version is a natural generalization of Fey et al.
Revised: The synchronous version is a natural generalization of Fey et al.

Original: (2021) while the asynchronous version can handle the straggler issue Chen et al. ; Zheng et al. (2017) in synchronous version and enjoys even better performance.
Revised: (2021) while the asynchronous version can handle the straggler issue (Chen et al., 2016; Zheng et al., 2017) in synchronous version and enjoys even better performance.

Original: From the system aspect, DIGEST (1) enables efficient, cross-device representation exchanging by using a shared-memory key-value storage (KVS) system, (2) supports both synchronous and asynchronous parameter updating, and (3) overlaps the computation (layer training) with I/Os (pushing/pulling representations to/from the KVS.
Revised: From the system aspect, DIGEST (1) enables efficient, cross-device representation exchanging by using a shared-memory key-value storage (KVS) system, (2) supports both synchronous and asynchronous parameter updating, and (3) overlaps the computation (layer training) with I/Os (pushing/pulling representations to/from the KVS.

Original: Furthermore, we proved that the approximation error induced by the staleness of representation can be bounded.
Revised: Furthermore, we proved that the approximation error induced by the staleness of representation can be bounded.

Original: More importantly, global convergence guarantee is provided, which demonstrates that
Revised: More importantly, global convergence guarantee is provided, which demonstrates that

Original: DIGEST has the state-of-the-art convergence rate.
Revised: DIGEST has the state-of-the-art convergence rate.

Original: Our main contributions can be summarized as:
Revised: Our main contributions can be summarized as:

Original: • Proposing a novel distributed GNN training framework that synergizes the benefits of partition-based and communication-based methods.
Revised: • Proposing a novel distributed GNN training framework that synergizes the benefits of partition-based and communication-based methods.

Original: Existing work in distributed GNN training
Revised: Existing work in distributed GNN training

Original: focus on two contradictory objectives: partition-based methods target minimizing the communication cost while propagation-based methods aim to minimize information loss.
Revised: focus on two contradictory objectives: partition-based methods target minimizing the communication cost while propagation-based methods aim to minimize information loss.

Original: DIGEST drops no edges while avoiding communication overhead by integrating the strengths of both categories.
Revised: DIGEST drops no edges while avoiding communication overhead by integrating the strengths of both categories.

Original: • Developing a periodic stale representation synchronization technique for distributed GNN training.
Revised: • Developing a periodic stale representation synchronization technique for distributed GNN training.

Original: DIGEST utilizes the entire graph for training by separating in- and out-of-subgraph neighbor nodes and approximating the latter with stale representations.
Revised: DIGEST utilizes the entire graph for training by separating in- and out-of-subgraph neighbor nodes and approximating the latter with stale representations.

Original: Instead of making strictly synchronous pull/push operations for the representations of all layers before/after training, DIGEST overlaps pull/push operations with layer training to minimize the overall training time.
Revised: Instead of making strictly synchronous pull/push operations for the representations of all layers before/after training, DIGEST overlaps pull/push operations with layer training to minimize the overall training time.

Original: Furthermore, a shared-memory-based KVS is used among subgraphs for efficiently exchanging representations.
Revised: Furthermore, a shared-memory-based KVS is used among subgraphs for efficiently exchanging representations.

Original: • Providing extensive theoretical guarantee on both performance and convergence of the proposed algorithm.
Revised: • Providing extensive theoretical guarantee on both performance and convergence of the proposed algorithm.

Original: We proved that DIGEST’s convergence rate is O
Revised: We proved that DIGEST’s convergence rate is O

Original: ( T − 2 / 3 M − 1 / 3 ) with T iterations and M subgraphs, which is close to vanilla distributed GNN training without staleness.
Revised: ( T − 2 / 3 M − 1 / 3 ) with T iterations and M subgraphs, which is close to vanilla distributed GNN training without staleness.

Original: Convergence guarantee for both synchronous and asynchronous versions of DIGEST is provided.
Revised: Convergence guarantee for both synchronous and asynchronous versions of DIGEST is provided.

Original: We also showed the upper bound on the approximation error of gradients due to the staleness.
Revised: We also showed the upper bound on the approximation error of gradients due to the staleness.

Original: • Conducting comprehensive empirical results on both performance and speedup.
Revised: • Conducting comprehensive empirical results on both performance and speedup.

Original: We perform extensive evaluation on four benchmark with classic GNNs (e.g., GCN Hamilton et al. (2017) and GAT Veliˇckovi´c et al.
Revised: We perform extensive evaluation on four benchmark with classic GNNs (e.g., GCN (Hamilton et al., 2017) and GAT (Veliˇckovi´c et al., 2017)).

Original: The experimental results show that for the best case DIGEST improves the performance by 33.14 % , and achieves 21.82 × speedup in training time compared to two state-of-the-art distributed GNNs training frameworks.
Revised: The experimental results show that for the best case DIGEST improves the performance by 33.14 % , and achieves 21.82 × speedup in training time compared to two state-of-the-art distributed GNNs training frameworks.

Original:  B ACKGROUND AND PROBLEM FORMULATION
Revised:  B ACKGROUND AND PROBLEM FORMULATION

Original: In this section we first introduce the Graph Neural Network (GNN) and its training on a single machine and then formulate the problem of distributed GNN training.
Revised: In this section we first introduce the Graph Neural Network (GNN) and its training on a single machine and then formulate the problem of distributed GNN training.

Original: Graph Neural Networks.
Revised: Graph Neural Networks.

Original: GNNs aim to learn a function of signals/features on a graph G ( V , E ) with node representations X ∈ R |V|× d , where d denotes the node feature dimension.
Revised: GNNs aim to learn a function of signals/features on a graph G ( V , E ) with node representations X ∈ R |V|× d , where d denotes the node feature dimension.

Original: For typical semisupervised node classification tasks Kipf & Welling (2016), where each node v ∈ V is associated with a label y v , a L -layer GNN f is trained to learn the node representation h v such that y v can be predicted accurately.
Revised: For typical semisupervised node classification tasks (Kipf & Welling, 2016), where each node v ∈ V is associated with a label y v , a L -layer GNN f is trained to learn the node representation h v such that y v can be predicted accurately.

Original: The training process of a GNN can be practically described as the node representation learning based on the message passing mechanism Gilmer et al.
Revised: The training process of a GNN can be practically described as the node representation learning based on the message passing mechanism (Gilmer et al., 2017).

Original: Analytically, given a graph G ( V , E ) and a node v ∈ V , the ( ℓ + 1) -th layer of the GNN is defined as
Revised: Analytically, given a graph G ( V , E ) and a node v ∈ V , the ( ℓ + 1) -th layer of the GNN is defined as

Original: [Equation]
Revised: [Equation]

Original: where h ( ℓ ) v denotes the representation of node v in the ℓ -th layer, and h (0) v being initialized to x v ( v -th row in X ), and N ( v ) represents the set of direct 1 - hop neighbors for node v .
Revised: where h ( ℓ ) v denotes the representation of node v in the ℓ -th layer, and h (0) v being initialized to x v ( v -th row in X ), and N ( v ) represents the set of direct 1 - hop neighbors for node v .

Original: Each layer of the GNN, i.e. f ( ℓ ) , can be further decomposed into two components: 1) Aggregation function Φ ( ℓ ) , which takes the nodes representations of node v ’s neighbors as input, and output the aggregated neighborhood representation.
Revised: Each layer of the GNN, i.e. f ( ℓ ) , can be further decomposed into two components: 1) Aggregation function Φ ( ℓ ) , which takes the nodes representations of node v ’s neighbors as input, and output the aggregated neighborhood representation.

Original: Updating function Ψ ( ℓ ) , which combines the representation of v and the aggregated neighborhood representation to update the representation of node v for the next layer.
Revised: Updating function Ψ ( ℓ ) , which combines the representation of v and the aggregated neighborhood representation to update the representation of node v for the next layer.

Original: Both Φ ( ℓ ) and Ψ ( ℓ ) can choose to use various functions in different types of GNNs.
Revised: Both Φ ( ℓ ) and Ψ ( ℓ ) can choose to use various functions in different types of GNNs.

Original: To train a
Revised: To train a

Original: GNN on a single machine, one can minimize the empirical loss L ( W ) over the entire graph in the training data, i.e., L ( W ) =
Revised: GNN on a single machine, one can minimize the empirical loss L ( W ) over the entire graph in the training data, i.e., L ( W ) =

Original: (1 / |V| ) (cid:80) v ∈V Loss (cid:0)
Revised: (1 / |V| ) (cid:80) v ∈V Loss (cid:0)

Original: h ( L ) v , y v (cid:1) , where Loss ( · , · ) denotes a loss function (e.g., cross entropy loss), and h ( L ) v denotes the representation of node v from the last layer of the
Revised: h ( L ) v , y v (cid:1) , where Loss ( · , · ) denotes a loss function (e.g., cross entropy loss), and h ( L ) v denotes the representation of node v from the last layer of the

Original: GNN and can be calculated by following Eq.
Revised: GNN and can be calculated by following Eq.

Original: 1 recursively.
Revised: 1 recursively.

Original: Distributed Training for GNNs.
Revised: Distributed Training for GNNs.

Original: Distributed GNN training means to first partition the original graph into multiple subgraphs without overlap, which can also be considered as mini batches.
Revised: Distributed GNN training means to first partition the original graph into multiple subgraphs without overlap, which can also be considered as mini batches.

Original: Then different mini-batches are trained in different devices in parallel.
Revised: Then different mini-batches are trained in different devices in parallel.

Original: Here, Eq. 1 can be further reformulated as
Revised: Here, Eq. 1 can be further reformulated as

Original: [Equation]
Revised: [Equation]

Original: where S ( v ) denotes the subgraph that node v belongs to.
Revised: where S ( v ) denotes the subgraph that node v belongs to.

Original: In this paper, we consider the distributed training of GNNs with multiple local machines and a global server.
Revised: In this paper, we consider the distributed training of GNNs with multiple local machines and a global server.

Original: The original input graph G is first partitioned into M subgraphs, where each G m ( V m , E m ) represents the subgraph m .
Revised: The original input graph G is first partitioned into M subgraphs, where each G m ( V m , E m ) represents the subgraph m .

Original: Our goal is to find the optimal set of parameters W in a distributed manner by minimizing each local loss, i.e.,
Revised: Our goal is to find the optimal set of parameters W in a distributed manner by minimizing each local loss, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where W m = { W ( ℓ ) m } Lℓ =1 are local parameters and h ( L ) v follows Eq.
Revised: where W m = { W ( ℓ ) m } Lℓ =1 are local parameters and h ( L ) v follows Eq.

Original: 2 recursively.
Revised: 2 recursively.

Original: Challenges.
Revised: Challenges.

Original: The main challenges for distributed training of GNNs lie in the trade-off between communication cost and information loss .
Revised: The main challenges for distributed training of GNNs lie in the trade-off between communication cost and information loss .

Original: "Partition-based" method generalizes the existing data parallelism techniques of classical distributed training on i.i.d data to graph data and enjoys minimal communication cost.
Revised: "Partition-based" method generalizes the existing data parallelism techniques of classical distributed training on i.i.d data to graph data and enjoys minimal communication cost.

Original: However, directly partitioning a large graph into multiple subgraphs can result in severe information loss due to the ignorance of huge number of cross-subgraph edges and cause performance degeneration Angerd et al.
Revised: However, directly partitioning a large graph into multiple subgraphs can result in severe information loss due to the ignorance of huge number of cross-subgraph edges and cause performance degeneration (Angerd et al., 2020;

Original: ; Jia et al. Ramezani et al.
Revised: Jia et al., 2020; Ramezani et al., 2021).

Original: For these methods, the representation of neighbors out of the current subgraph (second representation set in
Revised: For these methods, the representation of neighbors out of the current subgraph (second representation set in

Original: Eq. 2) are dropped and the connections between subgraphs are thus ignored.
Revised: Eq. 2) are dropped and the connections between subgraphs are thus ignored.

Original: Hence, another line of work Wang et al. (2019), namely "propagation-based" method considers using communication of neighbor nodes for each subgraph to satisfy GNN’s neighbor aggregation, which minimizes the information loss.
Revised: Hence, another line of work (Wang et al., 2019), namely "propagation-based" method considers using communication of neighbor nodes for each subgraph to satisfy GNN’s neighbor aggregation, which minimizes the information loss.

Original: As shown in Eq. 2, the representations for neighbor nodes outside the current subgraph is swapped between different subgraphs.
Revised: As shown in Eq. 2, the representations for neighbor nodes outside the current subgraph is swapped between different subgraphs.

Original: However, the number of neighbors involved in the neighbor aggregation process expands exponentially as the GNN model goes deep, which is known as the neighborhood explosion problem.
Revised: However, the number of neighbors involved in the neighbor aggregation process expands exponentially as the GNN model goes deep, which is known as the neighborhood explosion problem.

Original: Hence, though no edges are dropped in this case, inevitable communication overhead is incurred and plagues the achievable training efficiency Ma et al.
Revised: Hence, though no edges are dropped in this case, inevitable communication overhead is incurred and plagues the achievable training efficiency (Ma et al., 2019;

Original: Zhu et al. ; Zheng et al. (2020); Tripathy et al. ; Wan et al.
Revised: Zhu et al., 2019; Zheng et al., 2020; Tripathy et al., 2020; Wan et al., 2022).

Original: Moreover, theoretical guarantees (e.g., on convergence, approximation error) are not well explored for distributed GNN due to the joint sophistication of graph structure and neural network optimization.
Revised: Moreover, theoretical guarantees (e.g., on convergence, approximation error) are not well explored for distributed GNN due to the joint sophistication of graph structure and neural network optimization.

Original:  P ROPOSED METHOD
Revised:  P ROPOSED METHOD

Original: In this section, we introduce the proposed GNN training framework DIGEST.
Revised: In this section, we introduce the proposed GNN training framework DIGEST.

Original: DIGEST leverages both types of representations in Eq.
Revised: DIGEST leverages both types of representations in Eq.

Original: 2 to address the information loss issue.
Revised: 2 to address the information loss issue.

Original: In addition, instead of exchanging real-time representations during the training process between the subgraphs, DIGEST only pull and push the stale representations before or after each step of training periodically.
Revised: In addition, instead of exchanging real-time representations during the training process between the subgraphs, DIGEST only pull and push the stale representations before or after each step of training periodically.

Original: With this strategy, the communications turn to be more efficient, which are illustrated in Figure 1 and analyzed in more details in Section 3.3.
Revised: With this strategy, the communications turn to be more efficient, which are illustrated in Figure 1 and analyzed in more details in Section 3.3.

Original: Moreover, we prove that the error introduced by the staleness of the stale representation is upper-bounded while the convergence is also guaranteed.
Revised: Moreover, we prove that the error introduced by the staleness of the stale representation is upper-bounded while the convergence is also guaranteed.

Original: 3. D ISTRIBUTED GNN TRAINING WITH FULL - GRAPH AWARENESS
Revised: 3. D ISTRIBUTED GNN TRAINING WITH FULL - GRAPH AWARENESS

Original: In DIGEST, each copy of GNN trained on a local machine will make use of all available graph information, i.e. no edges are dropped in both forward and backward propagation.
Revised: In DIGEST, each copy of GNN trained on a local machine will make use of all available graph information, i.e. no edges are dropped in both forward and backward propagation.

Original: Analytically, calculating each local gradient ∇L Local m as defined in Eq.
Revised: Analytically, calculating each local gradient ∇L Local m as defined in Eq.

Original: 3 will involve out-of-subgraph neighbor information.
Revised: 3 will involve out-of-subgraph neighbor information.

Original: For out-of-subgraph neighbor nodes, we approximate their representations via stale representations acquired in previous training, denoted by ˜h ( ℓ ) v .
Revised: For out-of-subgraph neighbor nodes, we approximate their representations via stale representations acquired in previous training, denoted by ˜h ( ℓ ) v .

Original: Formally, given a node v ∈ G m ( V m , E m ) , the forward propagation for the ( ℓ + 1) -th layer of DIGEST is achieved by modifying Eq. 2 as
Revised: Formally, given a node v ∈ G m ( V m , E m ) , the forward propagation for the ( ℓ + 1) -th layer of DIGEST is achieved by modifying Eq. 2 as

Original: [Equation]
Revised: [Equation]

Original: As can be seen, DIGEST considers ALL neighbor nodes information during forward propagation.
Revised: As can be seen, DIGEST considers ALL neighbor nodes information during forward propagation.

Original: On the other hand, leveraging the entire graph data in forward propagation will in turn improve the estimation of gradient in backpropagation.
Revised: On the other hand, leveraging the entire graph data in forward propagation will in turn improve the estimation of gradient in backpropagation.

Original: To see this, we reformulate Eq.
Revised: To see this, we reformulate Eq.

Original: 4 into the matrix form:
Revised: 4 into the matrix form:

Original: [Equation]
Revised: [Equation]

Original: where H ( ℓ,m ) in and ˜H ( ℓ,m ) out denotes the matrix of in-subgraph node representations and out-of-subgraph stale representations at ℓ -th layer on subgraph G m , respectively.
Revised: where H ( ℓ,m ) in and ˜H ( ℓ,m ) out denotes the matrix of in-subgraph node representations and out-of-subgraph stale representations at ℓ -th layer on subgraph G m , respectively.

Original: F denotes the forward propagation function of one layer of GNN for compact formula.
Revised: F denotes the forward propagation function of one layer of GNN for compact formula.

Original: We consider the GCN model as an example for illustration but our analyses apply to general cases of any GNN models.
Revised: We consider the GCN model as an example for illustration but our analyses apply to general cases of any GNN models.

Original: P ( m ) in and P ( m ) out denotes the propagation matrix for in-subgraph nodes and out-of-subgraph nodes of G m , respectively, and we have P m =
Revised: P ( m ) in and P ( m ) out denotes the propagation matrix for in-subgraph nodes and out-of-subgraph nodes of G m , respectively, and we have P m =

Original: P ( m ) in + P ( m ) out where P m is the original propagation matrix for subgraph G m .
Revised: P ( m ) in + P ( m ) out where P m is the original propagation matrix for subgraph G m .

Original: σ ( · ) is the activation function following GCN’s definition.
Revised: σ ( · ) is the activation function following GCN’s definition.

Original: Hence, the gradient over model parameters is
Revised: Hence, the gradient over model parameters is

Original: [Equation]
Revised: [Equation]

Original: The key observation here is that ALL neighbor nodes are involved in the backpropagation since the gradient above depends on ˜H ( ℓ,m ) out .
Revised: The key observation here is that ALL neighbor nodes are involved in the backpropagation since the gradient above depends on ˜H ( ℓ,m ) out .

Original: The separation of in-subgraph nodes and out-of-subgraph nodes, and their approximation via stale representation form the very foundation of DIGEST.
Revised: The separation of in-subgraph nodes and out-of-subgraph nodes, and their approximation via stale representation form the very foundation of DIGEST.

Original: 3. S YSTEM DESIGN
Revised: 3. S YSTEM DESIGN

Original: This section presents the overall system design of DIGEST as depicted in Figure 1.
Revised: This section presents the overall system design of DIGEST as depicted in Figure 1.

Original: DIGEST maintains a shared-memory-based KVS for storing and retrieving representations.
Revised: DIGEST maintains a shared-memory-based KVS for storing and retrieving representations.

Original: KVS can be easily extended to a truly distributed storage to support large-scale distributed training spanning multiple servers.
Revised: KVS can be easily extended to a truly distributed storage to support large-scale distributed training spanning multiple servers.

Original: We first introduce two operations used by DIGEST to store and retrieve representations.
Revised: We first introduce two operations used by DIGEST to store and retrieve representations.

Original: The stale representations of layer ℓ for all nodes in V can be formulated as ˜H ( ℓ ) = { ˜h ( ℓ ) v : v ∈ V} .
Revised: The stale representations of layer ℓ for all nodes in V can be formulated as ˜H ( ℓ ) = { ˜h ( ℓ ) v : v ∈ V} .

Original: For any subgraph G m to start the forward process of layer ℓ , the necessary stale representations ˜H ( ℓ,m ) out = { ˜h ( ℓ ) u : u ∈ N ( v ) \ V m , ∀ v ∈ V m } are pulled from the KVS that stores representations; this is called a "pull" operation denoted as H ( ℓ,m ) out ← ˜H ( ℓ,m ) out .
Revised: For any subgraph G m to start the forward process of layer ℓ , the necessary stale representations ˜H ( ℓ,m ) out = { ˜h ( ℓ ) u : u ∈ N ( v ) \ V m , ∀ v ∈ V m } are pulled from the KVS that stores representations; this is called a "pull" operation denoted as H ( ℓ,m ) out ← ˜H ( ℓ,m ) out .

Original: See Figure 1(c).
Revised: See Figure 1(c).

Original: After the end of a epoch, the newly-computed representations H ( ℓ,m ) in = { h ( ℓ ) v : ∀ v ∈ V m } are pushed to the KVS, and these newly-stored representations will be fetched as stale representations in future epochs; this is called a "push" operation denoted as H ( ℓ,m ) in → ˜H ( ℓ,m ) in .
Revised: After the end of a epoch, the newly-computed representations H ( ℓ,m ) in = { h ( ℓ ) v : ∀ v ∈ V m } are pushed to the KVS, and these newly-stored representations will be fetched as stale representations in future epochs; this is called a "push" operation denoted as H ( ℓ,m ) in → ˜H ( ℓ,m ) in .

Original: DIGEST features two training modes: (1) DIGEST: a synchronous mode designed ideal for homogeneous training environments.
Revised: DIGEST features two training modes: (1) DIGEST: a synchronous mode designed ideal for homogeneous training environments.

Original: (2) DIGEST-A: an asynchronous mode that better fits for a heterogeneous training environment.
Revised: (2) DIGEST-A: an asynchronous mode that better fits for a heterogeneous training environment.

Original: DIGEST and DIGEST-A follow different parameter and representation updating strategies.
Revised: DIGEST and DIGEST-A follow different parameter and representation updating strategies.

Original: In DIGEST, for each global round, before fetching the aggregated parameters and pulling the stale representations, each subgraph has to wait for other subgraphs to finish updating the latest parameters to the parameter server (PS) and their local representations to the KVS.
Revised: In DIGEST, for each global round, before fetching the aggregated parameters and pulling the stale representations, each subgraph has to wait for other subgraphs to finish updating the latest parameters to the parameter server (PS) and their local representations to the KVS.

Original: However, some subgraphs may have lower computing resource compared to other subgraphs, which we call stragglers.
Revised: However, some subgraphs may have lower computing resource compared to other subgraphs, which we call stragglers.

Original: This may lead to imbalanced local training times.
Revised: This may lead to imbalanced local training times.

Original: In this case, with the synchronous mode, the overall training process can be bottlenecked by the slowest subgraph, therefore suffering from prolonged training time.
Revised: In this case, with the synchronous mode, the overall training process can be bottlenecked by the slowest subgraph, therefore suffering from prolonged training time.

Original: To address this issue, DIGEST-A applies an asynchronous, non-blocking strategy, where each subgraph directly pulls/pushes stale representations of other subgraphs from the shared KVS and downloads/uploads parameters from the PS without blindly waiting for the slowest subgraph to finish.
Revised: To address this issue, DIGEST-A applies an asynchronous, non-blocking strategy, where each subgraph directly pulls/pushes stale representations of other subgraphs from the shared KVS and downloads/uploads parameters from the PS without blindly waiting for the slowest subgraph to finish.

Original: For better scalability, we will explore disaggregated storage techniques Klimovic et al. Nanavati et al. Amaro et al. (2020) as part of our future work, where DIGEST can utilize a network-attached, high-performance far memory storage system for representation storage and retrieval.
Revised: For better scalability, we will explore disaggregated storage techniques (Klimovic et al., 2016; Nanavati et al., 2017; Amaro et al., 2020) as part of our future work, where DIGEST can utilize a network-attached, high-performance far memory storage system for representation storage and retrieval.

Original: We summarize our algorithm in Algorithm 1 in the appendix due to limited space.
Revised: We summarize our algorithm in Algorithm 1 in the appendix due to limited space.

Original: In addition, DIGEST and DIGEST-A use several optimizations to minimize the I/O overhead introduced by pulls and pushes.
Revised: In addition, DIGEST and DIGEST-A use several optimizations to minimize the I/O overhead introduced by pulls and pushes.

Original: First, we observe that there are a large number of node representations involved in both pull and push operations, and more importantly, nodes are independent of each other on these two operations.
Revised: First, we observe that there are a large number of node representations involved in both pull and push operations, and more importantly, nodes are independent of each other on these two operations.

Original: Hence, it is inherently suitable for parallel I/O at the granularity of node level.
Revised: Hence, it is inherently suitable for parallel I/O at the granularity of node level.

Original: For subgraph G m , the total number of stale representations needed to be pulled from the KVS is | ˜H ( ℓ,m ) out | .
Revised: For subgraph G m , the total number of stale representations needed to be pulled from the KVS is | ˜H ( ℓ,m ) out | .

Original: Assume that it takes time t to pull the stale representation of one node, the total time cost should be | ˜ H ( ℓ,m ) out | × t if being pulled in serial.
Revised: Assume that it takes time t to pull the stale representation of one node, the total time cost should be | ˜ H ( ℓ,m ) out | × t if being pulled in serial.

Original: But with parallel I/O where needed representations are pulled in parallel, theoretically we can still keep the pull time for v ∈ V m as t .
Revised: But with parallel I/O where needed representations are pulled in parallel, theoretically we can still keep the pull time for v ∈ V m as t .

Original: Additionally, we observe that the pull operation for ˜H ( ℓ,m ) out can be overlapped with the forward process of layer ℓ − 1 ; similarly, the push operation for H ( ℓ,m ) in can be overlapped with the forward process of layer ℓ + 1 .
Revised: Additionally, we observe that the pull operation for ˜H ( ℓ,m ) out can be overlapped with the forward process of layer ℓ − 1 ; similarly, the push operation for H ( ℓ,m ) in can be overlapped with the forward process of layer ℓ + 1 .

Original: The training process on each subgraph is depicted in Figure 2.
Revised: The training process on each subgraph is depicted in Figure 2.

Original: The cost of pull/push operations is hidden by the layer forward process, therefore, is eliminated.
Revised: The cost of pull/push operations is hidden by the layer forward process, therefore, is eliminated.

Original: Second, to further reduce the I/O overhead,
Revised: Second, to further reduce the I/O overhead,

Original: DIGEST uses a periodic representation synchronization strategy, which pushes updated representations to the KVS once every N epoches.
Revised: DIGEST uses a periodic representation synchronization strategy, which pushes updated representations to the KVS once every N epoches.

Original: This introduces a trade-off in I/O overhead and training performance.
Revised: This introduces a trade-off in I/O overhead and training performance.

Original: Increasing the frequency of the periodic synchronization will benefit performance, but this will introduce more I/O overhead.
Revised: Increasing the frequency of the periodic synchronization will benefit performance, but this will introduce more I/O overhead.

Original: We analyze this trade-off in Section 5.2.
Revised: We analyze this trade-off in Section 5.2.

Original: 3. C OMPLEXITY ANALYSES
Revised: 3. C OMPLEXITY ANALYSES

Original: Here we analyze the memory and communication complexity of DIGEST, compared with propagationbased methods.
Revised: Here we analyze the memory and communication complexity of DIGEST.

Original: Without loss of generality, consider a GNN with L layers and each layer has a fixed width d .

Original: For propagation-based methods, the memory complexity for each local machine grows exponentially with respect to L .

Original: In comparison, DIGEST pulls the required out-of-subgraph node representations and keeps them locally for each local machine.
Revised: DIGEST pulls the required out-of-subgraph node representations and keeps them locally for each local machine.

Original: For the m -th local machine and the corresponding subgraph on it, i.e. G m ( V m , E m ) , the memory complexity per training iteration is O (cid:0) (cid:12)(cid:12)(cid:83) v ∈V m N ( v ) ∪ { v } (cid:12)(cid:12) Ld (cid:1) .
Revised: For the m -th local machine and the corresponding subgraph on it, i.e. G m ( V m , E m ) , the memory complexity per training iteration is O (cid:0) (cid:12) (cid:12) (cid:83) v ∈V m N ( v ) ∪ { v } (cid:12) (cid:12) Ld (cid:1) .

Original: which scales linearly with respect to the number of GNN layers.
Revised: which scales linearly with respect to the number of GNN layers.

Original: Propagation-based methods also suffer from poor scalability in terms of communication since the number of nodes required for each subgraph’s computation again grows exponentially with respect to L .

Original: In contrast, DIGEST’s communication cost per round can be expressed as O (cid:0) MLd 2 + (cid:80)
Revised: DIGEST’s communication cost per round can be expressed as O (cid:0) MLd 2 + (cid:80)

Original: Mm =1 (cid:12)(cid:12)(cid:83) v ∈V m N ( v ) \ V m (cid:12)(cid:12)
Revised: Mm =1 (cid:12)(cid:12)(cid:83) v ∈V m N ( v ) \ V m (cid:12)(cid:12)

Original: Ld + NLd (cid:1) , where the first term represents the cost for pull and push operations of GNN model parameters or gradients, the second term and the third term represents the cost for pull and push operations of stale representations, respectively, and N is the original graph size. The third term is because the partition of subgraphs is non-overlapping.
Revised: Ld + NLd (cid:1) .

Original: Again, the communication cost of DIGEST is only linear with respect to GNN depth L .
Revised: Again, the communication cost of DIGEST is only linear with respect to GNN depth L .

Original:  T HEORETICAL ANALYSES
Revised:  T HEORETICAL ANALYSES

Original: In this section, we provide theoretical analyses of the propose distributed strategy DIGEST, including the bound of error induced by the staleness of node representations, and convergence guarantee for
Revised: In this section, we provide theoretical analyses of the propose distributed strategy DIGEST, including the bound of error induced by the staleness of node representations, and convergence guarantee for

Original: DIGEST under both synchronous and asynchronous settings.
Revised: DIGEST under both synchronous and asynchronous settings.

Original: All proofs can be found in the appendix.
Revised: All proofs can be found in the appendix.

Original: 4. E RROR BOUND ON GLOBAL APPROXIMATED GRADIENTS
Revised: 4. E RROR BOUND ON GLOBAL APPROXIMATED GRADIENTS

Original: Our first theorem shows that under the distributed setting, the approximation error of the global model’s gradients can be upper bounded by the staleness of node representations.
Revised: Our first theorem shows that under the distributed setting, the approximation error of the global model’s gradients can be upper bounded by the staleness of node representations.

Original: Theorem 1.
Revised: Theorem 1.

Original: Given a L -layer GNN f W with r 1 -Lipschitz smooth Φ and r 2 -Lipschitz smooth Ψ .
Revised: Given a L -layer GNN f W with r 1 -Lipschitz smooth Φ and r 2 -Lipschitz smooth Ψ .

Original: Denote ∆( G ) as the maximal node degree for graph G .
Revised: Denote ∆( G ) as the maximal node degree for graph G .

Original: Assume ∀ v ∈ V and ∀ ℓ ∈ { 1 , 2 , · · · , L − 1 } we have ∥ h ( ℓ ) v − ˜h ( ℓ ) v ∥ ≤ ϵ ( ℓ ) , where h ( ℓ ) v and ˜h ( ℓ ) v denotes the node representation computed by DIGEST and the stale one, respectively.
Revised: Assume ∀ v ∈ V and ∀ ℓ ∈ { 1 , 2 , · · · , L − 1 } we have ∥ h ( ℓ ) v − ˜h ( ℓ ) v ∥ ≤ ϵ ( ℓ ) , where h ( ℓ ) v and ˜h ( ℓ ) v denotes the node representation computed by DIGEST and the stale one, respectively.

Original: Further assume each local loss function L Local m is τ -Lipschitz smooth w.r.t the node representation.
Revised: Further assume each local loss function L Local m is τ -Lipschitz smooth w.r.t the node representation.

Original: Then, we have that (cid:13)(cid:13) ∇ W L − ∇ W L ∗ (cid:13)(cid:13) 2 ≤ ( τ/M ) (cid:80) L − 1 ℓ =1 ϵ ( ℓ )
Revised: Then, we have that (cid:13)(cid:13) ∇ W L − ∇ W L ∗ (cid:13)(cid:13) 2 ≤ ( τ/M ) (cid:80) L − 1 ℓ =1 ϵ ( ℓ )

Original: r
Revised: r

Original: L − ℓr L − ℓ(cid:80)
Revised: L − ℓr L − ℓ(cid:80)

Original: Mm =1 | ∆( G m ) | L − ℓ , where ∇ W L and ∇ W L ∗ denotes the global gradient computed by DIGEST and the exact global gradient without any staleness.
Revised: Mm =1 | ∆( G m ) | L − ℓ , where ∇ W L and ∇ W L ∗ denotes the global gradient computed by DIGEST and the exact global gradient without any staleness.

Original: 4. C ONVERGENCE OF S YNCHRONOUS DIGEST
Revised: 4. C ONVERGENCE OF S YNCHRONOUS DIGEST

Original: As both fresh inner-subgraphnode representations and stale out-of-subgraph representations are adopted in our algorithm, its convergence rate is still unknown.
Revised: As both fresh inner-subgraph and stale out-of-subgraph representations are adopted in our algorithm, its convergence rate is still unknown.

Original: We have proved the convergence of DIGEST and present the convergence property in the following theorem:
Revised: We have proved the convergence of DIGEST and present the convergence property in the theorem blow. First, we introduce some assumptions:

Original: Theorem 2.
Revised: Assumption 1.

Revised: The loss function Loss ( · , · ) is C Loss -Lipchitz continuous and L Loss -Lipschitz smooth with respect to the last layer’s node representation, i.e., | Loss ( h ( L ) v , y v ) − Loss

Revised: ( h ( L ) w , y v ) | ≤

Revised: C Loss ∥

Revised: h ( L ) v − h

Revised: ( L ) w ∥ 2 and ∥∇ Loss ( h ( L ) v , y v ) − ∇ Loss ( h ( L ) w , y v )

Revised: ∥ 2 ≤ L Loss ∥ h ( L ) v − h

Revised: ( L ) w ∥ 2 .

Revised: Assumption 2.

Revised: The activation function σ ( · ) is C

Revised: σ -Lipchitz continuous and L σ -Lipschitz smooth, i.e.

Revised: ∥ σ ( Z ( ℓ ) 1 ) − σ ( Z ( ℓ ) 2 ) ∥ 2 ≤ C σ ∥ ( Z ( ℓ ) 1 − Z ( ℓ ) 2 ∥ 2 and ∥ σ ′ ( Z ( ℓ ) 1 ) − σ ′

Revised: ( Z ( ℓ ) 2 ) ∥ 2 ≤ L σ ∥ ( Z ( ℓ ) 1 − Z ( ℓ ) 2 ∥ 2 .

Revised: Assumption 3.

Revised: ∀ ℓ = 1 , 2 , · · · , L , we have ∥ W ( ℓ )

Revised: ∥

Revised: F ≤ K W , ∥ P ( ℓ ) ∥

Revised: P ≤ K W , ∥ X ( ℓ ) ∥ F ≤ K X .

Revised: Theorem 2.

Revised: Consider GCN with L layers that is L f -Lipschitz smooth.

Original: ∀ ϵ > 0 , ∃ constant E > 0 such that, we can choose a learning rate η = √ MϵE and number of training iterationsT
Revised: ∀ ϵ > 0 , ∃ constant E > 0 such that, we can choose a learning rate η = √ MϵE and number of training iterations

Original:  =
Revised: T =

Original: ( L ( W (1) )
Revised: ( L ( W (1) )

Original: − L ( W ∗ ))
Revised: − L ( W ∗ ))

Original: E √ M ϵ − 32 s.t. , T − 1 (cid:80)
Revised: E √ M ϵ − 32 s.t. , T − 1 (cid:80)

Original: Tt =1 ∥∇L ( W ( t ) )
Revised: Tt =1 ∥∇L ( W ( t ) )

Original: ∥ 2 ≤ O ( T − 2 / 3 M − 1 / 3 ) , where W ∗ denotes the optimal parameter.
Revised: ∥ 2 ≤ O ( T − 2 / 3 M − 1 / 3 ) , where W ∗ denotes the optimal parameter.

Original: Our convergence rate of DIGEST is O ( T − 2 / 3 M − 1 / 3 ) , which is better than pipeline-parallelism method O ( T − 2 / 3 ) Wan et al.
Revised: Our convergence rate of DIGEST is O ( T − 2 / 3 M − 1 / 3 ) , which is better than pipeline-parallelism method O ( T − 2 / 3 ) (Wan et al., 2022) and

Original: and

Original: sampling-based method O ( T − 1 / 2 ) Chen et al.
Revised: sampling-based method O ( T − 1 / 2 ) (Chen et al., 2018;

Original: ; Cong et al. (2021), and very close to full-graph training O
Revised: Cong et al., 2021), and very close to full-graph training O

Original: ( T − 1 ) .
Revised: ( T − 1 ) .

Original:  C ONVERGENCE OF A SYNCHRONOUS DIGEST
Revised: 4. C ONVERGENCE OF A SYNCHRONOUS DIGEST

Original: Convergence for asynchronous distributed algorithms could be even harder to obtain due to the delay in parameter’s update (the global model’s parameters may have been updated several times when the slowest local machine finishes its computation.)
Revised: Convergence for asynchronous distributed algorithms could be even harder to obtain due to the delay in parameter’s update (the global model’s parameters may have been updated several times when the slowest local machine finishes its computation.)

Original: Our main result is shown below:
Revised: Our main result is shown below:

Original: Theorem 3.
Revised: Assumption 4.

Revised: ∀ m ∈

Revised: [ M ] , ∥∇ ˜ L

Revised: m ( W ) ∥ 2 ≤ V · ∥∇L ( W ) ∥ 2 , and (cid:10) ∇L ( W ) , ∇

Revised: ˜ L

Revised: m ( W ) (cid:11) ≥ β · ∥∇L ( W ) ∥ 22 , where V and β are positive real numbers, i.e., V, β ∈ R + .

Revised: Theorem 3.

Original: Assume the global model L ( W ) is Lipschitz continuous and theasynchronous delay is bounded.
Revised: Assume the global model L ( W ) is C f -Lipschitz continuous and the delay is bounded, i.e., τ < K .

Original:  There exist constant B and a second-order polynomial of learning rate η , i.e., P ( η ) such that after T global iterations on the server, asynchronous DIGEST converges to the optimal parameter W ∗ by T − 1 (cid:80) Tt =1 (cid:13)(cid:13)
Revised: Further, assume β − V 2 2 > 0 . There exist constant B and a second-order polynomial of learning rate η , i.e., P ( η ) such that after T global iterations on the server, asynchronous DIGEST converges to the optimal parameter W ∗ by T − 1 (cid:80) Tt =1 (cid:13)(cid:13)

Original: ∇L ( W ( t ) )
Revised: ∇L ( W ( t ) )

Original: (cid:13)(cid:13) 2 2 ≤ ( ηTB ) − 1 ( L ( W (1) ) 
Revised: (cid:13)(cid:13) 2 2 ≤ ( ηTB ) − 1 ( L ( W (1) ) −

Original: − L ( W ( ∗ ) ))
Revised: L ( W ( ∗ ) ))

Original: + P ( η ) /B.
Revised: + P ( η ) /B , where B = β − V 2 2 and P ( η ) = 1 2 η 2 K 2 C 2 f L 2 f + (1 + V ) ηKC 2 f L f .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we evaluate DIGEST and compare DIGEST against two state-of-the-art distributed
Revised: In this section, we evaluate DIGEST and compare DIGEST against two state-of-the-art distributed

Original: GNNs training frameworks as baselines in terms of training efficiency and scalability.
Revised: GNNs training frameworks as baselines in terms of training efficiency and scalability.

Original: Considering the distinct training time per epoch between DIGEST and other baselines, we report the F1 scores on validation dataset and training loss over training time, instead of over communication rounds, in the results.
Revised: Considering the distinct training time per epoch between DIGEST and other baselines, we report the F1 scores on validation dataset and training loss over training time, instead of over communication rounds, in the results.

Original: This way it makes a fairer comparison in terms of training performance and efficiency.
Revised: This way it makes a fairer comparison in terms of training performance and efficiency.

Original: 5. E XPERIMENT SETTING
Revised: 5. E XPERIMENT SETTING

Original: Implementation and Setup.
Revised: Implementation and Setup.

Original: We have implemented DIGEST and other comparison GNNs training methods all in PyTorch Paszke et al.
Revised: We have implemented DIGEST and other comparison GNNs training methods all in PyTorch (Paszke et al., 2019).

Original: For all the experiments, we simulate a distributed training environment using an EC2 g4dn.metal virtual machine (VM) instance on AWS, which has 8 NVIDIA T4 GPUs, 96 vCPUs, and 384 GB main memory.
Revised: For all the experiments, we simulate a distributed training environment using an EC2 g4dn.metal virtual machine (VM) instance on AWS, which has 8 NVIDIA T4 GPUs, 96 vCPUs, and 384 GB main memory.

Original: We implemented the shared-memory
Revised: We implemented the shared-memory

Original: KVS using the Plasma in-memory object store 3 for representation storage and retrieval. 
Revised: KVS using the Plasma in-memory object store 3 for representation storage and retrieval. [Figure] [Table]

Original: Baselines.
Revised: Baselines.

Original: Recall in Section 2 we categorize existing distributed GNN training into two types of general methods.
Revised: Recall in Section 2 we categorize existing distributed GNN training into two types of general methods.

Original: In evaluation, we choose two state-of-the-art distributed training frameworks, one from each category as the baseline.
Revised: In evaluation, we choose two state-of-the-art distributed training frameworks, one from each category as the baseline.

Original: For the first category, we choose LLCG Ramezani et al. (2021), which partitions a graph into subgraphs and trains each subgraph strictly independently without incurring any communication among subgraphs.
Revised: For the first category, we choose LLCG (Ramezani et al., 2021), which partitions a graph into subgraphs and trains each subgraph strictly independently without incurring any communication among subgraphs.

Original: LLCG uses a central server to aggregate local models[Figure] [Table] from each device and performs global training using mini-batches with full neighbor information to ensure that the model learns the global structure of the graph.
Revised: LLCG uses a central server to aggregate local models from each device and performs global training using mini-batches with full neighbor information to ensure that the model learns the global structure of the graph.

Original: LLCG uses this additional step to reduce the information loss caused by graph partitioning.
Revised: LLCG uses this additional step to reduce the information loss caused by graph partitioning.

Original: For the second category [Table] we choose to use DGL Wang et al. (2019), which is a commonly-used, distributed GNN training framework.
Revised: For the second category, we choose to use DGL (Wang et al., 2019), which is a commonly-used, distributed GNN training framework.

Original: In contrast to LLCG, DGL requires exchanging node representations among partitioned subgraphs.
Revised: In contrast to LLCG, DGL requires exchanging node representations among partitioned subgraphs.

Original: DGL requires frequent swap operations with other subgraphs for representations during subgraph’s local training in each epoch, and therefore, DGL incurs high communication cost.
Revised: DGL requires frequent swap operations with other subgraphs for representations during subgraph’s local training in each epoch, and therefore, DGL incurs high communication cost.

Original: Models and datasets. We report the experimental results of training GCN Kipf & Welling (2016) and
Revised: [Figure]

Original: GAT Veliˇckovi´c et al. (2017) in a distributed manner spanning multiple GPU devices. Our framework can also be applied, in a straightforward way, to other prominent GNNs, such as PNA Corso et al. GCNII Chen et al.
Revised: [Figure]

Original: (2020a).
Revised: [Figure]

Original: Four benchmark node classification datasets, OGB-Arxiv Hu et al. (2020), Flickr Zeng et al. Reddit Zeng et al. (2019), and
Revised: 5. E XPERIMENTAL RESULTS

Original: OGB-Products Hu et al. (2020), are used in our evaluation. Please refer to the appendix for the summary of the datasets.
Revised: In this section

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: we evaluate both sync & async versions of DIGEST, LLCG

Original: [Figure]
Revised: [Figure]

Original: In this section, we evaluate both sync & async versions of DIGEST, LLCG, and DGL on the four datasets.
Revised: and DGL on the four datasets.

Original: Due to the page limit, we move parts of our evaluation results to the Appendix.
Revised: Due to the page limit, we move parts of our evaluation results to the Appendix.

Original: Efficiency of DIGEST.
Revised: Efficiency of DIGEST.

Original: We first evaluate the training performance of DIGEST and DIGEST-A. As shown in Figure 3, DIGEST outperforms both LLCG and DGL for all the datasets when performing distributed training on a pure GCN.
Revised: We first evaluate the training performance of DIGEST and DIGEST-A. As shown in Figure 3, DIGEST outperforms both LLCG and DGL for all the datasets when performing distributed training on a pure GCN.

Original: LLCG performs worst particularly for the Reddit dataset, because in the global server correction of LLCG, only a mini-batch is trained and it is not sufficient to correct the plain GCN.
Revised: LLCG performs worst particularly for the Reddit dataset, because in the global server correction of LLCG, only a mini-batch is trained and it is not sufficient to correct the plain GCN.

Original: This is also the reason why the authors of LLCG report the performance of a complex model with mixing GCN layers and GraphSAGE layers Ramezani et al.
Revised: This is also the reason why the authors of LLCG report the performance of a complex model with mixing GCN layers and GraphSAGE layers Ramezani et al.

Original: DGL achieves good performance on some dataset (e.g., OGB-products) with uniform node sampling strategy and realtime representation exchanging.
Revised: DGL achieves good performance on some dataset (e.g., OGB-products) with uniform node sampling strategy and realtime representation exchanging.

Original: However, frequent communication also leads to slow performance increasing for dataset Flickr (Figure 3(b)) and poor performance for all four datasets.
Revised: However, frequent communication also leads to slow performance increasing for dataset Flickr (Figure 3(b)) and poor performance for all four datasets.

Original: DIGEST and
Revised: DIGEST and

Original: DIGEST-A avoid these issues and therefore achieve satisfying performance over the training time.
Revised: DIGEST-A avoid these issues and therefore achieve satisfying performance over the training time.

Original: DIGEST-A is slowly catching up DIGEST due to the diverse model parameters used by subgraphs in the early training period.
Revised: DIGEST-A is slowly catching up DIGEST due to the diverse model parameters used by subgraphs in the early training period.

Original: We measure the training time per epoch as shown in Figure 4.
Revised: We measure the training time per epoch as shown in Figure 4.

Original: Since the representation synchronization is only performed before the start or after the end of local training, DIGEST takes significantly shorter training time per epoch than that of LLCG and DGL.
Revised: Since the representation synchronization is only performed before the start or after the end of local training, DIGEST takes significantly shorter training time per epoch than that of LLCG and DGL.

Original: Furthermore, DIGEST performs periodic synchronization instead of per-epoch synchronization, which further shortens the training time.
Revised: Furthermore, DIGEST performs periodic synchronization instead of per-epoch synchronization, which further shortens the training time.

Original: Table 1 presents the detailed numbers for the comparison of three frameworks on the four datasets.
Revised: Table 1 presents the detailed numbers for the comparison of three frameworks on the four datasets.

Original: For all the cases except GAT on OGB-Arxiv, DIGEST achieves leading F1 scores on the validation dataset, demonstrating the efficacy of DIGEST’s design.
Revised: For all the cases except GAT on OGB-Arxiv, DIGEST achieves leading F1 scores on the validation dataset, demonstrating the efficacy of DIGEST’s design.

Original: [Figure]
Revised: [Figure]

Original: Scalability of DIGEST.
Revised: Scalability of DIGEST.

Original: We evaluate the scalability of three frameworks by training a GCN on OGB-Products with varied number of GPUs.
Revised: We evaluate the scalability of three frameworks by training a GCN on OGB-Products with varied number of GPUs.

Original: We use average training time per epoch against that of DGL with a single GPU to calculate the speedup results.
Revised: We use average training time per epoch against that of DGL with a single GPU to calculate the speedup results.

Original: As shown in Figure 5, DIGEST shows the best scalability compared to the other two.
Revised: As shown in Figure 5, DIGEST shows the best scalability compared to the other two.

Original: The speedup rises with the number of GPUs used during training.
Revised: The speedup rises with the number of GPUs used during training.

Original: We observe a similar trend for DGL, but the relative speedup for DGL is significantly smaller than that for DIGEST, due to the using of stale representations instead of real-time representations.
Revised: We observe a similar trend for DGL, but the relative speedup for DGL is significantly smaller than that for DIGEST, due to the using of real-time representations instead of stale representations.

Original: Synchronization frequency.
Revised: Synchronization frequency.

Original: We next perform a sensitivity analysis by varying the synchronization intervals for OGBProducts to study how the synchronization frequency would affect the training performance.
Revised: We next perform a sensitivity analysis by varying the synchronization intervals for OGBProducts to study how the synchronization frequency would affect the training performance.

Original: As shown in Figure 6, DIGEST achieves the highest F1 score over training time when configured to perform synchronization of stale representations every 10 epochs.
Revised: As shown in Figure 6, DIGEST achieves the highest F1 score over training time when configured to perform synchronization of stale representations every 10 epochs.

Original: A large interval ( 20 ) or a small interval ( 1 ) results in performance degradation, due to the long term loss of graph information or additional communication cost.
Revised: A large interval ( 20 ) or a small interval ( 1 ) results in performance degradation, due to the long term loss of graph information or additional communication cost.

Original: Training in heterogeneous environment.
Revised: Training in heterogeneous environment.

Original: Finally, we test DIGEST’s asynchronous training mode.
Revised: Finally, we test DIGEST’s asynchronous training mode.

Original: As stated in Section 3.2, asynchronous training is better suited for GNN training in heterogeneous environments.
Revised: As stated in Section 3.2, asynchronous training is better suited for GNN training in heterogeneous environments.

Original: In this test, we randomly select one subgraph as the straggler before the training starts.
Revised: In this test, we randomly select one subgraph as the straggler before the training starts.

Original: To simulate the straggler lagging caused by limited computing capability, a random delay ranging from 8 to 10 seconds is added to the chosen straggler during the whole training process.
Revised: To simulate the straggler lagging caused by limited computing capability, a random delay ranging from 8 to 10 seconds is added to the chosen straggler during the whole training process.

Original: We can see from Figure 7 that DIGEST-A performs much better than other three synchronous methods and converge to high F1-score at the early stage of the training.
Revised: We can see from Figure 7 that DIGEST-A performs much better than other three synchronous methods and converge to high F1-score at the early stage of the training.

Original: This is because asynchronous mode effectively eliminates GPU’s blocking caused by waiting with significantly improved GPU utilization.
Revised: This is because asynchronous mode effectively eliminates GPU’s blocking caused by waiting with significantly improved GPU utilization.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: There are two general categories in distributed GNN training.
Revised: There are two general categories in distributed GNN training.

Original: Partition-based methods suffer from graph information loss, while propogation-based methods suffer from high communication cost.
Revised: Partition-based methods suffer from graph information loss, while propogation-based methods suffer from high communication cost.

Original: In this work we present DIGEST, a novel distributed GNN training framework that synergizes the complementary strengths of both methods by leveraging stale representations intelligently.
Revised: In this work we present DIGEST, a novel distributed GNN training framework that synergizes the complementary strengths of both methods by leveraging stale representations intelligently.

Original: We provide rigorous theoretical analysis to prove that DIGEST has competitive convergence rate and expressiveness.
Revised: We provide rigorous theoretical analysis to prove that DIGEST has competitive convergence rate and bounded error due to staleness.

Original: Extensive experiments on four benchmark datasets validate our analysis and demonstrate the efficiency and scalability of DIGEST.
Revised: Extensive experiments on four benchmark datasets validate our analysis and demonstrate the efficiency and scalability of DIGEST.

Original: A A PPENDIX
Revised: A A PPENDIX

Original: In this section, we  describe detailed experimental setup, additional experimental results, and complete proofs.
Revised: In this section, we clarify our contributions made in this work and describe detailed experimental setup, additional experimental results, and complete proofs.

Original: We reuse part of code adopted from GNNAutoscale Fey et al.
Revised: We reuse part of code adopted from GNNAutoScale Fey et al.

Original: (2021); our code is available at: https://anonymous.4open.science/r/DIGESTA-78F2/ .
Revised: (2021); our code is available at: https://anonymous.4open. science/r/DIGESTA-78F2/ .

Original: Please note that the code is subjected to reorganization to improve the readability.
Revised: Please note that the code is subjected to reorganization to improve the readability.

Original: A. E XPERIMENTAL S ETUP D ETAILS
Revised: A. C ONTRIBUTIONS AND N OVELTY

Revised: The contributions and novelty of this work are multi-fold.

Revised: In this paper, we propose a new, highlyparallel, and full-graph-aware distributed GNN training method; on top of this new method, we design a novel, compute-and-storage-disaggregated training system to enable better scalability and allow distributed GNN training to potentially benefit from emerging computing paradigms and hardware; finally, we deduce new theoretical guarantees and analyses for the co-designed algorithms and systems.

Revised: Methodology Novelty in Algorithm-System Co-design: Our paper is mainly motivated from a distributed training perspective, where the proposed framework synergizes the best of both partition-based and propagation-based distributed training; GNNAutoScale provides a theoretical foundation, which exposes potential opportunities that can be harnessed by and co-designed with new distributed training system infrastructures to enable highly-parallel GNN training.

Revised: DIGEST goes beyond GNNAutoScale in that we built a novel distributed training framework that effectively decouples the management of state (i.e., representations) and compute (i.e., GNN training).

Revised: System Architecture Novelty: The disaggregated architecture of DIGEST is the result of an algorithm-system co-design as mentioned in the Methodology Novelty, and enables great properties including high scalability and low training time, as demonstrated in our paper.

Revised: More importantly, this disaggregated architecture could enable fundamental opportunities for GNN training systems to take advantage of emerging computing paradigm such as elastic serverless computing as well as emerging hardware such as Zoned Namespace SSD (ZNS) and smart programmable network hardware (SmartNIC); in this work, we have shown the promising scalability and speedup that

Revised: DIGEST offers, which establishes a solid system foundation for further system-level optimizations and innovations.

Revised: This demands/inspires future research along the line, which we plan to do as part of our future work.

Revised: Theoretical Novelty: All of our theoretical analyses are tailored for a distributed training setup, while GNNAutoScale only considers single-GPU training.

Revised: A. E XPERIMENTAL S ETUP D ETAILS

Original: As mentioned in Section 5.1, all the experiments are done on an EC2 g4dn.metal virtual machine (VM) instance on AWS, which has 8 NVIDIA T4 GPUs, 96 vCPUs, and 384 GB main memory.
Revised: As mentioned in Section 5.1, all the experiments are done on an EC2 g4dn.metal virtual machine (VM) instance on AWS, which has 8 NVIDIA T4 GPUs, 96 vCPUs, and 384 GB main memory.

Original: Other important information including operation system version, Linux kernel version, and CUDA version is summarized in Table 2.
Revised: Other important information including operation system version, Linux kernel version, and CUDA version is summarized in Table 2.

Original: For fair comparison, we use the same optimizer (Adam), learning rate, and graph partition algorithm for all the three frameworks, DIGEST, LLCG, and DGL.
Revised: For fair comparison, we use the same optimizer (Adam), learning rate, and graph partition algorithm for all the three frameworks, DIGEST, LLCG, and DGL.

Original: For parameters that are unique to both LLCG and DGL, such as the number of neighbors sampled from each layer for each node, we choose the default value for both LLCG and DGL.
Revised: For parameters that are unique to both LLCG and DGL, such as the number of neighbors sampled from each layer for each node, we choose the default value for both LLCG and DGL.

Original: Each of the three frameworks has a set of parameters that are exclusively unique to that framework; for these exclusive parameters, we tune them in order to achieve the best performance.
Revised: Each of the three frameworks has a set of parameters that are exclusively unique to that framework; for these exclusive parameters, we tune them in order to achieve the best performance.

Original: Please refer to the configuration files under run/conf/model for detailed configuration setups for all the models and datasets.
Revised: Please refer to the configuration files under run/conf/model for detailed configuration setups for all the models and datasets.

Original: [Table]
Revised: [Table]

Original: We use four datasets: OGB-Arxiv Hu et al.
Revised: We use four datasets: OGB-Arxiv Hu et al.

Original: (2020), Flickr Zeng et al.
Revised: (2020), Flickr Zeng et al.

Original: Reddit Zeng et al.
Revised: Reddit Zeng et al.

Original: (2019), and OGB-Products Hu et al.
Revised: (2019), and OGB-Products Hu et al.

Original: (2020) for evaluation.
Revised: (2020) for evaluation.

Original: The detailed information of these datasets is summarized in Table 3. 
Revised: The detailed information of these datasets is summarized in Table 3. [Table]

Original: dataset statistics
Revised: [Figure] [Table]

Original: [Table]
Revised: [Figure]

Original: A. A DDITIONAL E XPERIMENTAL R ESULTS A.2. P ERFORMANCE OF GAT T RAINING
Revised: A. A DDITIONAL E XPERIMENTAL R ESULTS A.3. P ERFORMANCE OF GAT T RAINING

Original:  We first show the learning curves of training GAT with three methods on three different datasets.
Revised: [Figure] We first show the learning curves of training GAT with three methods on three different datasets.

Original: As shown in Figure 8, for dataset Flickr and Reddit, DIGEST acheives the best validation F1 score over training time of all the three frameworks.
Revised: As shown in Figure 8, for dataset Flickr and Reddit, DIGEST acheives the best validation F1 score over training time of all the three frameworks.

Original: For dataset OGB-Arxiv, the performance of DIGEST is slightly worse than DGL but still outperforms LLCG.
Revised: For dataset OGB-Arxiv, the performance of DIGEST is slightly worse than DGL but still outperforms LLCG.

Original: Specifically, LLCG’s training curves are not stable and fluctuate dramatically for both GCN and GAT on Reddit.
Revised: Specifically, LLCG’s training curves are not stable and fluctuate dramatically for both GCN and GAT on Reddit.

Original: This is because Reddit is much denser compared to other datasets, and in this case, the sampling process of the global server correction in LLCG has difficulty capturing all the information loss due to the cut-edges.
Revised: This is because Reddit is much denser compared to other datasets, and in this case, the sampling process of the global server correction in LLCG has difficulty capturing all the information loss due to the cut-edges.

Original: Unlike
Revised: Unlike

Original: LLCG, DIGEST’s training curves are much smoother not only for GCN training but also for GAT training.
Revised: LLCG, DIGEST’s training curves are much smoother not only for GCN training but also for GAT training.

Original: A.2. M EMORY O VERHEAD
Revised: A.3. M EMORY O VERHEAD

Original: In this section we quantify the memory overhead introduced by DIGEST.
Revised: In this section we quantify the memory overhead introduced by DIGEST.

Original:  Figure 9 shows the ratios of the number of out-of-subgraph nodes to the number of in-subgraph nodes across four datasets.
Revised: Ratio of out-of-subgraph nodes and in-subgraph nodes. Figure 9 shows the ratios of the number of out-of-subgraph nodes to the number of in-subgraph nodes across four datasets.

Original: This ratio quantifies additional memory consumption compared to methods that does not use any information of the neighboring nodes during training.
Revised: This ratio quantifies additional memory consumption compared to methods that does not use any information of the neighboring nodes during training.

Original: [Figure]

Original: [Figure]

Original: training a GCN over the four datasets.

Original: Denser graphs like Flickr and Reddit require more memory to store representations of off-subgraph nodes than OGB-Arxiv and OGB-products.
Revised: Denser graphs like Flickr and Reddit require more memory to store representations of off-subgraph nodes than OGB-Arxiv and OGB-products.

Original: This introduces an interesting tradeoff between extra memory storage and gained benefits in reduced communication and preservation of global graph information.
Revised: This introduces an interesting tradeoff between extra memory storage and gained benefits in reduced communication and preservation of global graph information.

Original: We argue that modern GPU servers are equipped with ample GPU memory resources to buffer the out-of-subgraph representations Gandhi & 
Revised: We argue that modern GPU servers are equipped with ample GPU memory resources to buffer the out-of-subgraph representations Gandhi & Iyer (2021);

Original: Iyer (2021); Wang et al. (2021); if indeed more memory is required, our research will benefit from the recent advancement of unified memory Choi et al.
Revised: Wang et al. (2021); if indeed more memory is required, our research will benefit from the recent advancement of unified memory Choi et al.

Original: ; Huang et al.
Revised: ; Huang et al.

Original: ; Peng et al.
Revised: ; Peng et al.

Original: (2020), where DIGEST can use both the host and GPU memory more efficiently.
Revised: (2020), where DIGEST can use both the host and GPU memory more efficiently.

Original: In the worst case, if GPU memory is limited, DIGEST can implement a multi-tier storage system that uses the limited memory as a level-one cache and the host memory as a backing store.
Revised: In the worst case, if GPU memory is limited, DIGEST can implement a multi-tier storage system that uses the limited memory as a level-one cache and the host memory as a backing store.

Original: For large graphs that are sparse (OGB-products), the extra memory cost can be bounded to a relatively lower ratio ( 58 . 
Revised: For large graphs that are sparse (OGB-products), the extra memory cost can be bounded to a relatively lower ratio ( 58 . Host memory cost of stale representations.

Revised: The KVS is responsible for storing the representations of all the nodes in a graph.

Revised: The representations are stored in the memory of the host server instead of the GPUs, the latter of which is rather limited.

Revised: We implemented the in-memory KVS with Apache

Original: A. A LGORITHM
Revised: Plasma, which is a shared memory storage that supports efficient, shared-memory-based inter-process communication (IPC) for multiple training processes located on the same server. However, extending our current KVS implementation to a fully-distributed storage system is trivial. Using off-the-shelf, high-performance distributed in-memory KVSes such as Redis is one option. Alternatively, we could also implement a simple client library, which can be used by the training process for key-value item mapping (e.g., using the commonly-used consistent hashing algorithm) and remote representation retrieval/storage, and with the client library, we could deploy a cluster of Plasma storage processes either on a dedicated storage cluster or on the same training server cluster to support distributed representation storage.

Revised: The overall memory consumption required to store representation data can be calculated with the following equation:

Revised: [Equation]

Revised: where L is the total number of layers of the model, dim denotes the hidden dimension, | V | represents the number of nodes in the graph, and s is the size of data type in Python numpy.

Revised: For the floatdata type, it takes 4 bytes for each single value.

Revised: With the provided formula, for a 3-layer GNN model, training large graph dataset such as OGB-Products (with 2 , 449 , 029 nodes and 128 hidden dimensions), the extra host memory consumption for the representations is around 2 ∗ 128 ∗ 2449029 ∗ 4 / 1024 / 1024 / 1024 = 2 .

Revised: 336 GB.

Revised: DIGEST exhibits an interesting tradeoff: it uses a small amount of extra memory overhead for storing stale representations to enable the disaggregation of the compute and storage for higher scalability and more flexibility.

Revised: We also argue that a small host memory cost of several GBs is negligible considering today’s multi-GPU servers are equipped with hundreds of

Revised: GBs if not more than a few TBs of host memory 4 .

Revised: [Table]

Revised: GPU memory consumption.

Revised: For the concern of GPU memory consumption, we compare DIGEST with GraphSAGE, LLCG, and DGL by including all the information inside a GNN’s receptive field in a single optimization step.

Revised: The comparison results in Table 4 show that DIGEST has the lowest

Revised: GPU memory consumption across all four systems.

Revised: A.3. E MPIRICAL V ALIDATION OF G RADIENT A PPROXIMATION E RROR

Revised: In this section, we empirically evaluate the gradient approximation error due to the usage of stale representation.

Revised: We conduct this experiment to show that the actual approximation error of gradients of DIGEST compared with the ground-truth gradients (i.e., gradients calculated without any stale representation) can be negligible in practice.

Revised: [Figure]

Revised: As can be seen in Figure 10, during the training phase the gradient calculated by DIGEST quickly converge to the ground-truth gradients typically after fewer than 10-20 epochs.

Revised: Hence, for the majority of training epochs the error of gradients is very small and the impact is negligible, which in turn validate our theoretical analyses in Theorem 1.

Revised: A.3. O THER C OMPARISONS

Revised: In this section, we compare DIGEST/DIGEST-A with PipeGCN and GNNAutoScale.

Revised: We train OGBProducts with DIGEST/DIGEST-A, PipeGCN and GNNAutoScale in the heterogeneous environment mentioned in Section 5.2, and report the training time taken to reach the target validation F1 score and time per epoch in Firgure 11, since there is no "epoch" in an asynchronous setting, the value of time per epoch for DIGEST-A is omitted.

Revised: We can see that DIGEST gets slight higher time per epoch than GNNAutoScale but reduces the time per epoch by 24 . 13% compared with PipeGCN.

Revised: Meanwhile, DIGEST-A gets the lowest training time to reach the target F1 score and saves 48 .

Revised: 98% and 19 .

Revised: 12% training time compared with PipeGCN and GNNAutoScale, respectively.

Revised: We further evaluate DIGEST, PipeGCN and GNNAutoScale on a large graph OGB-papers100m which consist of 111 million nodes 1 .

Revised: 6 billion edges to show the efficiency of DIGEST.

Revised: The experiments are done in a homogeneous environment with 32 GPUs.

Revised: Since mini-batches in GNNAutoScale are trained in a serial manner instead of a parallel distributed setting, only one GPU is used.

Revised: DIGEST reduces the time per epoch by 21 .

Revised: 13% compared with distributed GNN training algorithm PipeGCN.

Revised: A. A LGORITHM

Original: Algorithm 1 shows the process of DIGEST’s synchronous mode.
Revised: Algorithm 1 shows the process of DIGEST’s synchronous mode.

Original: At the beginning of training, the original graph is partitioned into several subgraphs with off-the-shelf graph clustering methods; 
Revised: At the beginning of training, the original graph is partitioned into several subgraphs with off-the-shelf graph clustering methods; [Figure] GNNAutoScale

Original:  DIGEST uses the widely-used METIS algorithm Karypis & Kumar (1998).
Revised: [Figure] DIGEST uses the widely-used METIS algorithm Karypis & Kumar (1998).

Original: Then the mini-batches are distributed to distinct workers, each of which handles training on a GPU device.
Revised: Then the mini-batches are distributed to distinct workers, each of which handles training on a GPU device.

Original: Depending on the size of the mini-batch, a single worker can handle one or multiple subgraphs.
Revised: Depending on the size of the mini-batch, a single worker can handle one or multiple subgraphs.

Original: DIGEST has two types of I/
Revised: DIGEST has two types of I/

Original: Os: storing and retrieving the model weights and stale representation as illustrated in Figure 1(c).
Revised: Os: storing and retrieving the model weights and stale representation as illustrated in Figure 1(c).

Original: The former one is performed for each epoch by aggregating all the model weights of other subgraphs in parallel (Line 13).
Revised: The former one is performed for each epoch by aggregating all the model weights of other subgraphs in parallel (Line 13).

Original: For the stale representation synchronization, we synchronize every N epochs, where we empirically tune N to obtain the optimal performance over training time with the defined pull and push operations (Lines 5,6,9,10).
Revised: For the stale representation synchronization, we synchronize every N epochs, where we empirically tune N to obtain the optimal performance over training time with the defined pull and push operations (Lines 5,6,9,10).

Original: To support the asynchronous mode (DIGEST-A), we can simply remove the loop of training epoch and move the parameter aggregation (Line 13) into the subgraph loop.
Revised: To support the asynchronous mode (DIGEST-A), we can simply remove the loop of training epoch and move the parameter aggregation (Line 13) into the subgraph loop.

Original: Input: Graph G ( V , E ) ; GNN depth L ; training epoch R ; global parameters W ( r ) =

Original: { W ( r,ℓ ) }

Original: Lℓ =1 , local parameters W ( r ) m =

Original: { W ( r,ℓ ) m } Lℓ =1 , ∀ m ∈ (cid:2) M (cid:3) , r ∈ (cid:2) R (cid:3) ; non-linearity activation function σ ; neighborhood function N : v → 2 V ; synchronization interval N; learning rate η .

Original: Output: The trained model weights W ( R +1) .

Original: 1 DIGEST() :

Original: Initialize W (1) {G m ( V m , E m ) , m = 1 , 2 , .., M } ← METIS( G ) ▷ graph partition for r = 1 ...

Original: R do for m = 1 , · · · , M in parallel do

Original: W ( r ) m = W ( r ) for ℓ = 1 ...

Original: L do if r % N == 0 and ℓ ̸ =

Original: L then

Original: H ( ℓ,m ) out ← ˜H ( ℓ,m ) out ▷ PULL for v ∈ V m do h ( ℓ ) out = { h ( ℓ ) u : u ∈ N ( v ) \ V m } h ( ℓ ) in = { h ( ℓ ) u : u ∈ N ( v ) ∩ V m } h ( ℓ ) v = σ (cid:16) W ( r,ℓ ) m · CONCAT (cid:0) h ( ℓ ) v , h ( ℓ ) in , h ( ℓ ) out (cid:1)(cid:17) if ( r − 1) % N == 0 and ℓ ̸ =

Original: L then

Original: H ( ℓ,m ) in → ˜H ( ℓ,m ) in ▷ PUSH h ( ℓ ) v ← h ( ℓ ) v / ∥ h ( ℓ ) v ∥ 2 , ∀ v ∈ V m ▷ representation normalization

Original: W ( r,ℓ +1) m

Original: =

Original: W ( r,ℓ ) m − η · ▽ W ( r,ℓ ) m

Original: ▷ update local parameters

Original: W ( r +1)

Original: ← AGG ( W ( r +1)...

Original: W ( r +1) M )

Original: ▷ update global parameters return

Original: A. T HEORETICAL P ROOF
Revised: A. T HEORETICAL P ROOF

Original: In this section, we provide the formal proof for all the theories presented in the main paper.
Revised: In this section, we provide the formal proof for all the theories presented in the main paper.

Original: A.4. P ROOF OF T HEOREM
Revised: A.5. P ROOF OF T HEOREM

Original: Theorem 4 (Formal version of Theorem 1) .
Revised: Theorem 4 (Formal version of Theorem 1) .

Original: Given a L -layer GNN f W with r 1 -Lipschitz smooth Φ and r 2 -Lipschitz smooth Ψ .
Revised: Given a L -layer GNN f W with r 1 -Lipschitz smooth Φ and r 2 -Lipschitz smooth Ψ .

Original: Denote ∆( G ) as the maximal node degree for graph G .
Revised: Denote ∆( G ) as the maximal node degree for graph G .

Original: Assume ∀ v ∈ V and ∀ ℓ ∈ { 1 , 2 , · · · , L − 1 } we have ∥ h ( ℓ ) v − ˜h ( ℓ ) v ∥ ≤ ϵ ( ℓ ) , where h ( ℓ ) v and ˜h ( ℓ ) v denotes the node representation computed by DIGEST and the stale one, respectively.
Revised: Assume ∀ v ∈ V and ∀ ℓ ∈ { 1 , 2 , · · · , L − 1 } we have ∥ h ( ℓ ) v − ˜h ( ℓ ) v ∥ ≤ ϵ ( ℓ ) , where h ( ℓ ) v and ˜h ( ℓ ) v denotes the node representation computed by DIGEST and the stale one, respectively.

Original: Further assume each local loss function L Local m is τ -Lipschitz smooth w.r.t node representation.
Revised: Further assume each local loss function L Local m is τ -Lipschitz smooth w.r.t node representation.

Original: Then the global gradient computed by
Revised: Then the global gradient computed by

Original: DIGEST has the following error bound
Revised: DIGEST has the following error bound

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: where h ∗ ( ℓ ) v denotes the exact output from the ℓ -th layer of GNN without any staleness.
Revised: where h ∗ ( ℓ ) v denotes the exact output from the ℓ -th layer of GNN without any staleness.

Original: Proof.
Revised: Proof.

Original: As stated in Theorem 2 in Fey et al.
Revised: As stated in Theorem 2 in Fey et al.

Original: (2021), under the single-GPU training setup, with
Revised: (2021), under the single-GPU training setup, with

Original: Lipschitz smooth Φ and Ψ as well as not too stale node representations, the GNN last layer’s output can be bounded by
Revised: Lipschitz smooth Φ and Ψ as well as not too stale node representations, the GNN last layer’s output [Equation] ▷ update local parameters W ( r +1)

Revised: ← AGG ( W ( r +1)

Original: [Equation]
Revised: [Equation]

Revised: W

Revised: [Equation]

Revised: +

Revised: [Equation]

Revised: ▷ update global parameters return W ( R +1) can be bounded by

Revised: [Equation]

Original: Now consider the distributed GNN training setting.
Revised: Now consider the distributed GNN training setting.

Original: First, notice that in our distributed setting, the stale node representation ˜h ( ℓ ) v is shared for all subgraphs.
Revised: First, notice that in our distributed setting, the stale node representation ˜h ( ℓ ) v is shared for all subgraphs.

Original: In other words, for m = 1 , 2 , · · · , M we can apply the conclusion above with the Lipschitz smooth asumption and have
Revised: In other words, for m = 1 , 2 , · · · , M we can apply the conclusion above with the Lipschitz smooth asumption and have

Original: [Equation]
Revised: [Equation]

Original: Notice that |N ( v ) | ≤ ∆( G m ) , ∀ v ∈ V m , where ∆( G m ) is defined as the maximal node degree for subgraph G m .
Revised: Notice that |N ( v ) | ≤ ∆( G m ) , ∀ v ∈ V m , where ∆( G m ) is defined as the maximal node degree for subgraph G m .

Original: We can sum over all nodes v ∈ V m and take average on both sides of Eq.
Revised: We can sum over all nodes v ∈ V m and take average on both sides of Eq.

Original: 11 to get 
Revised: 12 to get |V m | (cid:88) [Equation] m

Original: [Equation]
Revised: [Equation]

Original: Finally, since we apply average to aggregate each local subgraph’s gradient to get the global gradients
Revised: Finally, since we apply average to aggregate each local subgraph’s gradient to get the global gradients

Original: [Equation]
Revised: [Equation]

Original: by the triangle inequality
Revised: by the triangle inequality

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: ≤

Original: M

Original: M (cid:88) m

Original: =  |V

Original: [Equation]

Original: 13) which finishes the proof.
Revised: 14) which finishes the proof.

Original: A.4. P ROOF OF T HEOREM
Revised: A.5. P ROOF OF T HEOREM

Original: In this section, we prove the convergence of DIGEST under the synchronous setting.
Revised: In this section, we prove the convergence of DIGEST under the synchronous setting.

Original: First, we introduce some notions, definitions and necessary assumptions.
Revised: First, we introduce some notions, definitions and necessary assumptions.

Original: Preliminaries.
Revised: Preliminaries.

Original: We consider GCN in our proof without loss of generality.
Revised: We consider GCN in our proof without loss of generality.

Original: We denote the input graph as G =
Revised: We denote the input graph as G =

Original: ( V , E ) , L -layer GNN as f , feature matrix as X , weight matrix as W .
Revised: ( V , E ) , L -layer GNN as f , feature matrix as X , weight matrix as W .

Original: The forward propagation of one layer of GCN is
Revised: The forward propagation of one layer of GCN is

Original: [Equation]
Revised: [Equation]

Original: where ℓ is the layer index, σ is the activation function, and P is the propagation matrix following the definition of GCN (Kipf & Welling, 2016).
Revised: where ℓ is the layer index, σ is the activation function, and P is the propagation matrix following the definition of GCN (Kipf & Welling, 2016).

Original: Notice H (0) =
Revised: Notice H (0) =

Original: X .
Revised: X .

Original: We can further define the ( ℓ + 1) -th layer of GCN as:
Revised: We can further define the ( ℓ + 1) -th layer of GCN as:

Original: [Equation]
Revised: [Equation]

Original: The backward propagation of GCN can be expressed as follow:
Revised: The backward propagation of GCN can be expressed as follow:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: and ◦ represents the Hadamard product.
Revised: and ◦ represents the Hadamard product.

Original: Under a distributed training setting, for each subgraph G m =
Revised: Under a distributed training setting, for each subgraph G m =

Original: ( V m , E m ) , m = 1 . 2 , · · · , M , the propagation matrix can be decomposed into two independent matrices, i.e. P = P m,in + P m,out , where P m,in denotes the propagation matrix for nodes inside the subgraph G m while P m,out denotes that for neighbor nodes outside G m .
Revised: ( V m , E m ) , m = 1 . 2 , · · · , M , the propagation matrix can be decomposed into two independent matrices, i.e. P = P m,in + P m,out , where P m,in denotes the propagation matrix for nodes inside the subgraph G m while P m,out denotes that for neighbor nodes outside G m .

Original: If it will not cause confusion, we will use P in and P out in our future proof for simpler notation.
Revised: If it will not cause confusion, we will use P in and P out in our future proof for simpler notation.

Original: For DIGEST, the forward propagation of a single layer of GCN can be expressed as
Revised: For DIGEST, the forward propagation of a single layer of GCN can be expressed as

Original: [Equation]
Revised: [Equation]

Original: where we use ˜ H to differentiate with the counterpart without staleness, i [Equation] e., H (same for other variables).
Revised: where we use ˜ H to differentiate with the counterpart without staleness, i.e., H (same for other variables).

Original: t is the training iteration index.
Revised: t is the training iteration index.

Original: Similarly, we can define each layer as a single function
Revised: Similarly, we can define each layer as a single function

Original: [Equation]
Revised: [Equation]

Original:  Note that ˜ H ( t − 1 ,ℓ − 1) m is not part of the input since it is the stale results from the previous iteration, i [Equation] e., it can be regarded as a constant in the current iteration.
Revised: [Equation] Note that ˜ H ( t − 1 ,ℓ − 1) m is not part of the input since it is the stale results from the previous iteration, i.e., it can be regarded as a constant in the current iteration.

Original: Now we can give the definition of back-propagation in DIGEST:
Revised: Now we can give the definition of back-propagation in DIGEST:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: In our proof [Equation] we use L ( W ( t ) ) to denote the global loss with GCN parameter W after t iterations, and use ˜ L m ( W ( t ) m ) to denotes the local loss for the m -th subgraph with model parameter W ( t ) m after t iterations computed by DIGEST.
Revised: In our proof, we use L ( W ( t ) ) to denote the global loss with GCN parameter W after t iterations, and use ˜ L m ( W ( t ) m ) to denotes the local loss for the m -th subgraph with model parameter W ( t ) m after t iterations computed by DIGEST.

Original: Assumptions.
Revised: Assumptions.

Original: Here we introduce some assumptions about the GCN model and the original input graph.
Revised: Here we introduce some assumptions about the GCN model and the original input graph.

Original: These assumptions are standard ones that are also used in (Chen et al., 2018;
Revised: These assumptions are standard ones that are also used in (Chen et al., 2018;

Original: Cong et al., 2021;
Revised: Cong et al., 2021;

Original: Wan et al., 2022).
Revised: Wan et al., 2022).

Original: Assumption 1.
Revised: Assumption 5.

Original: The loss function Loss ( · , · ) is C Loss -Lipchitz continuous and L Loss -Lipschitz smooth with respect to the last layer’s node representation, i.e.,
Revised: The loss function Loss ( · , · ) is C Loss -Lipchitz continuous and L Loss -Lipschitz smooth with respect to the last layer’s node representation, i.e.,

Original: [Equation]
Revised: [Equation]

Original: and

Original: [Equation]
Revised: [Equation]

Revised: and ∥∇ Loss

Original: [Equation]
Revised: [Equation]

Original: Assumption 2.
Revised: Assumption 6.

Original: The activation function σ ( · ) is C
Revised: The activation function σ ( · ) is C

Original: σ -Lipchitz continuous and L σ -Lipschitz smooth, i.e.
Revised: σ -Lipchitz continuous and L σ -Lipschitz smooth, i.e.

Original: [Equation]
Revised: [Equation]

Original: and ∥ σ
Revised: and ∥ σ

Original: [Equation]
Revised: [Equation]

Original: ( Z (
Revised: ( Z (

Original: [Equation]

Original: 1 ) − σ ′ ( Z ( ℓ ) 2 ) ∥ 2 ≤ L σ ∥ ( Z ( ℓ

Original: [Equation]
Revised: [Equation]

Original: 1 − Z (

Original: [Equation]
Revised: [Equation]

Original: Assumption 3.
Revised: Assumption 7.

Original: ∀ ℓ that ℓ = 1 , 2 , · · · , L , we have
Revised: ∀ ℓ that ℓ = 1 , 2 , · · · , L , we have

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Now we can introduce the proof of our Theorem 2.
Revised: Now we can introduce the proof of our Theorem 2.

Original: We consider a GCN with L layers that is
Revised: We consider a GCN with L layers that is

Original: L 
Revised: L f -Lipschitz smooth, i.e., ∥∇L ( W 1 ) −

Original: [Equation]
Revised: [Equation]

Original: i.e.
Revised: ( W 2 ) ∥ 2 ≤ L f ∥ W 1 − W 2 ∥

Original: [Equation]
Revised: [Equation]

Original: Theorem 5 (Formal version of Theorem 2) .
Revised: Theorem 5 (Formal version of Theorem 2) .

Original: There exists a constant E such that for any arbitrarily small constant ϵ > 0 , we can choose a learning rate η = √ MϵE and number of training iterations
Revised: There exists a constant E such that for any arbitrarily small constant ϵ > 0 , we can choose a learning rate η = √ MϵE and number of training iterations

Original: T =
Revised: T =

Original: [Equation] L [Equation]
Revised: ( L ( W (1) ) − L ( W ∗ ))

Original: ) E √ M ϵ − 32 , such that
Revised: E √ M ϵ − 32 , such that

Original: [Equation]
Revised: [Equation]

Original: where W ( t ) and W ∗ denotes the parameters at iteration t and the optimal one, respectively.
Revised: where W ( t ) and W ∗ denotes the parameters at iteration t and the optimal one, respectively.

Original: Proof.
Revised: Proof.

Original: Beginning from the assumption of smoothness of loss function,
Revised: Beginning from the assumption of smoothness of loss function,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Recall that the update rule of DIGEST is
Revised: Recall that the update rule of DIGEST is

Original: [Equation]
Revised: [Equation]

Original: m ) (30) so we have
Revised: so we have

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Denote δ ( t ) m = ∇ ˜
Revised: Denote δ ( t ) m = ∇ ˜

Original: L m ( W ( t ) m ) − ∇L m ( W ( t ) m ) , we have
Revised: L m ( W ( t ) m ) − ∇L m ( W ( t ) m ) , we have

Original: [Equation]
Revised: [Equation]

Original: Without loss of generality, assume the original graph can be divided evenly into M subgraphs and denote N = |V| as the original graph size, i.e., N = M · S , where S is each subgraph size.
Revised: Without loss of generality, assume the original graph can be divided evenly into M subgraphs and denote N = |V| as the original graph size, i.e., N = M · S , where S is each subgraph size.

Original: Notice that
Revised: Notice that

Original: [Equation]
Revised: [Equation]

Original: which is essentially 
Revised: which is essentially ∇L

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Plugging the equation above into Eq.
Revised: Plugging the equation above into Eq.

Original: 32, we have
Revised: 33, we have

Original: [Equation]
Revised: [Equation]

Original: which after rearranging the terms leads to
Revised: which after rearranging the terms leads to

Original: [Equation]
Revised: [Equation]

Original: By taking η < 1 /L f , using the three assumptions defined earlier and Corollary A.10 in Wan et al.
Revised: By taking η < 1 /L f , using the three assumptions defined earlier and Corollary A.10 in Wan et al.

Original: (2022), and summing up the inequality above over all iterations, i.e., t = 1 , 2 , · · · , T , we have 
Revised: (2022), and summing up the inequality above over all iterations, i.e., t = 1 , 2 , · · · , T , we have [Equation]

Original: [Equation]
Revised: [Equation]

Original: where W ∗ denotes the minima of the loss function and E is a constant depends on E ′ .
Revised: where W ∗ denotes the minima of the loss function and E is a constant depends on E ′ .

Original: Finally, taking η = √ MϵE and T =
Revised: Finally, taking η = √ MϵE and T =

Original: ( L ( W (1) )
Revised: ( L ( W (1) )

Original: − L ( W ∗ ))
Revised: − L ( W ∗ ))

Original: E √ M ϵ − 32 finishes the proof.
Revised: E √ M ϵ − 32 finishes the proof.

Original: A.4. P ROOF OF T HEOREM
Revised: A.5. P ROOF OF T HEOREM

Original: By following Li et al.
Revised: By following Li et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: (2020b); Chai et al.
Revised: (2020); Chai et al.

Original: we make the assumption as below:
Revised: we make the assumption as below:

Original: Assumption 4.
Revised: Assumption 8.

Original: ∀ m ∈
Revised: ∀ m ∈

Original: [ M ] , ∥∇ ˜ L
Revised: [ M ] , ∥∇ ˜ L

Original: m ( W ) ∥ 2 ≤ V · ∥∇L ( W ) ∥ 2 , and (cid:10) ∇L ( W ) , ∇
Revised: m ( W ) ∥ 2 ≤ V · ∥∇L ( W ) ∥ 2 , and (cid:10) ∇L ( W ) , ∇

Original: ˜ L
Revised: ˜ L

Original: m ( W ) (cid:11) ≥ β · ∥∇L ( W ) ∥ 22 , where V and β are positive real numbers, i.e., V, β ∈ R + .
Revised: m ( W ) (cid:11) ≥ β · ∥∇L ( W ) ∥ 2 2 , where V and β are positive real numbers, i.e., V, β ∈ R + .

Original: We naturally assume that each local copy of the global GCN model is also L f -Lipschitz smooth, i.e.,∥∇ ˜
Revised: We naturally assume that each local copy of the global GCN model is also L f -Lipschitz smooth, i.e.,

Original: L m ( W 1 ) − ∇ ˜ L m ( W 2 ) ∥ 2 ≤ L f ∥ W 1 − W 2 ∥ 2 , ∀ m = 1 , 2 , · · · , M .
Revised: [Equation]

Original: Now we can give the proof of Theorem 3.
Revised: Now we can give the proof of Theorem 3.

Original: Theorem 6 (Formal version of Theorem 3) .
Revised: Theorem 6 (Formal version of Theorem 3) .

Original: Assume the global model L ( W ) is C f -Lipschitz continuous and the delay is bounded, i.e., τ < K .
Revised: Assume the global model L ( W ) is C f -Lipschitz continuous and the delay is bounded, i.e., τ < K .

Original: Further, assume the constants defined in Assumptionsatisfy β − V 2 2 > 0 .
Revised: Further, assume the constants defined in Assumptionsatisfy β − V 2 2 > 0 .

Original: Then, after T global iterations on the server, asynchronous DIGEST converges to the optimal parameter W ∗ by
Revised: Then, after T global iterations on the server, asynchronous DIGEST converges to the optimal parameter W ∗ by

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where B = β − V 2 2 and P ( η ) = 12 η 2 K 2 C 2 f L 2 f + (1 + V ) ηKC 2 f L f .
Revised: where B = β − V 2 2 and P ( η ) = 12 η 2 K 2 C 2 f L 2 f + (1 + V ) ηKC 2 f L f .

Original: Proof.
Revised: Proof.

Original: By the smoothness assumption of global model,
Revised: By the smoothness assumption of global model,

Original: [Equation]
Revised: [Equation]

Original: Suppose at global iteration t + 1 , the server receives an update from subgraph m , where m could be any value from 1 up to M .
Revised: Suppose at global iteration t + 1 , the server receives an update from subgraph m , where m could be any value from 1 up to M .

Original: Then,
Revised: Then,

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where τ is the delay for subgraph m when sending server its update.
Revised: where τ is the delay for subgraph m when sending server its update.

Original: Denote r m := ∇ ˜
Revised: Denote r m := ∇ ˜

Original: L m ( W ( t − τ ) )
Revised: L m ( W ( t − τ ) )

Original: − ∇ ˜ L
Revised: − ∇ ˜ L

Original: m ( W ( t ) ) .
Revised: m ( W ( t ) ) .

Original: Since τ ≤ K , by the Lipschitz smoothness of ˜ L m ,
Revised: Since τ ≤ K , by the Lipschitz smoothness of ˜ L m ,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where g i is the gradient or update the global server receives at iteration i .
Revised: where g i is the gradient or update the global server receives at iteration i .

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Now we want to find bounds for (I - V) above.
Revised: Now we want to find bounds for (I - V) above [Equation]

Original: By Assumption 4, we have
Revised: By Assumption 8, we have

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: By our previous result on ∥ r m ∥ 2 , we have
Revised: By our previous result on ∥ r m ∥ 2 , we have

Original: [Equation]
Revised: [Equation]

Original: Taking η ≤ 1 /L f [Equation] we have
Revised: Taking η ≤ 1 /L f , we have

Original: [Equation]
Revised: [Equation]

Original: By Cauchy-Schwartz inequality, triangle inequality and Assumption 
Revised: By Cauchy-Schwartz inequality, triangle inequality and Assumption 8,

Original: [Equation]
Revised: [Equation]

Original: Put everything together, we have
Revised: Put everything together, we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Hence
Revised: Hence

Original: [Equation]
Revised: [Equation]

Original: Summing up from t = 1 to T and taking the average,
Revised: Summing up from t = 1 to T and taking the average,

Original: [Equation]
Revised: [Equation]

Original: where B = β − V 2 2 and P ( η ) = 12 η 2 K 2 C 2 f L 2 f + [Equation] 1 + V 
Revised: where B = β − V 2 2 and P ( η ) = 12 η 2 K 2 C 2 f L 2 f + (1 + V ) ηKC 2 f L f .

Original: [Equation]

Original: ηKC 2 f L

Original: [Equation]