Original: L EARN L OCALLY , C ORRECT G LOBALLY : A D ISTRIBUTED A LGORITHM FOR T RAINING G RAPH N EURAL N ETWORKS
Revised: L EARN L OCALLY , C ORRECT G LOBALLY : A D ISTRIBUTED A LGORITHM FOR T RAINING G RAPH N EURAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Despite the recent success of Graph Neural Networks (GNNs), training GNNs on large graphs remains challenging.
Revised: Despite the recent success of Graph Neural Networks (GNNs), training GNNs on large graphs remains challenging.

Original: The limited resource capacities of the existing servers, the dependency between nodes in a graph, and the privacy concern due to the centralized storage and model learning have spurred the need to design an effective distributed algorithm for GNN training.
Revised: The limited resource capacities of the existing servers, the dependency between nodes in a graph, and the privacy concern due to the centralized storage and model learning have spurred the need to design an effective distributed algorithm for GNN training.

Original: However, existing distributed
Revised: However, existing distributed

Original: GNN training methods impose either excessive communication costs or large memory overheads that hinders their scalability.
Revised: GNN training methods impose either excessive communication costs or large memory overheads that hinders their scalability.

Original: To overcome these issues, we propose a communication-efﬁcient distributed GNN training technique named
Revised: To overcome these issues, we propose a communication-efﬁcient distributed GNN training technique named

Original: Learn Locally, Correct Globally (LLCG).
Revised: Learn Locally, Correct Globally (LLCG).

Original: To reduce the communication and memory overhead, each local machine in LLCG ﬁrst trains a GNN on its local data by ignoring the dependency between nodes among different machines, then sends the locally trained model to the server for periodic model averaging.
Revised: To reduce the communication and memory overhead, each local machine in LLCG ﬁrst trains a GNN on its local data by ignoring the dependency between nodes among different machines, then sends the locally trained model to the server for periodic model averaging.

Original: However, ignoring node dependency could result in signiﬁcant performance degradation.
Revised: However, ignoring node dependency could result in signiﬁcant performance degradation.

Original: To solve the performance degradation, we propose to apply Global Server Corrections on the server to reﬁne the locally learned models.
Revised: To solve the performance degradation, we propose to apply Global Server Corrections on the server to reﬁne the locally learned models.

Original: We rigorously analyze the convergence of distributed methods with periodic model averaging for training GNNs and show that naively applying periodic model averaging but ignoring the dependency between nodes will suffer from an irreducible residual error.
Revised: We rigorously analyze the convergence of distributed methods with periodic model averaging for training GNNs and show that naively applying periodic model averaging but ignoring the dependency between nodes will suffer from an irreducible residual error.

Original: However, this residual error can be eliminated by utilizing the proposed global corrections to entail fast convergence rate.
Revised: However, this residual error can be eliminated by utilizing the proposed global corrections to entail fast convergence rate.

Original: Extensive experiments on real-world datasets show that LLCG can signiﬁcantly improve the efﬁciency without hurting the performance.
Revised: Extensive experiments on real-world datasets show that LLCG can signiﬁcantly improve the efﬁciency without hurting the performance.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In recent years, Graph Neural Networks (GNNs) have achieved impressive results across numerous graph-based applications, including social networks (Hamilton et al., 2017;
Revised: In recent years, Graph Neural Networks (GNNs) have achieved impressive results across numerous graph-based applications, including social networks (Hamilton et al., 2017;

Original: Deng et al., 2019), recommendation systems (Ying et al., 2018; Wang et al., 2018), and drug discovery (Fout, 2017; Do et al., 2019).
Revised: Deng et al., 2019), recommendation systems (Ying et al., 2018; Wang et al., 2018), and drug discovery (Fout et al., 2017; Do et al., 2019).

Original: Despite their recent success, effective training of GNNs onthe large-scale real-world graphs, such as Facebook social network (Boldi & Vigna, 2004), remains challenging.
Revised: Despite their recent success, effective training of GNNs on large-scale real-world graphs, such as Facebook social network (Boldi & Vigna, 2004), remains challenging.

Original: [Figure]
Revised: [Figure]

Original: Although several attempts have been made to scale GNN training by sampling techniques (Hamilton et al., 2017; Zou et al., 2019; Zeng et al., 2019; Chiang et al., 2019; Chen et al., 2017; Zhang et al., 2021), they are still inefﬁcient for training on extremely large graphs, due to the unique structure of GNNs and the limited memory capacity/bandwidth of current servers.
Revised: Although several attempts have been made to scale GNN training by sampling techniques (Hamilton et al., 2017; Zou et al., 2019; Zeng et al., 2020; Chiang et al., 2019; Chen et al., 2018; Zhang et al., 2021), they are still inefﬁcient for training on extremely large graphs, due to the unique structure of GNNs and the limited memory capacity/bandwidth of current servers.

Original: One potential solution to tackle these limitations is employing distributed training with data parallelism, which have become almost a de facto standard for fast and accurate training for natural language processing (Lin et al., 2021;
Revised: One potential solution to tackle these limitations is employing distributed training with data parallelism, which have become almost a de facto standard for fast and accurate training for natural language processing (Lin et al., 2021;

Original: Hard et al., 2018) and computer vision (Bonawitz et al.; Koneˇcn`y et al., 2018).
Revised: Hard et al., 2018) and computer vision (Bonawitz et al., 2019; Koneˇcn`y et al., 2018).

Original: For example, as shown in Figure 1, moving from single machine to multiple machines reduces the training time and alleviate the memory burden on each machine.
Revised: For example, as shown in Figure 1, moving from single machine to multiple machines reduces the training time and alleviates the memory burden on each machine.

Original: Besides, scaling the training of GNNs with sampling techniques can result in privacy concerns: existing sampling-based methods require centralized data storage and model learning, which could result in privacy concerns in real-world scenario (Shin et al., 2018; Wu et al., 2021).
Revised: Besides, scaling the training of GNNs with sampling techniques can result in privacy concerns: existing samplingbased methods require centralized data storage and model learning, which could result in privacy concerns in real-world scenarios (Shin et al., 2018; Wu et al., 2021).

Original: Fortunately, the privacy in distributed learning can be preserved by avoiding mutual access to data between different local machines, and using only a trusted third party server to access the entire data.
Revised: Fortunately, the privacy in distributed learning can be preserved by avoiding mutual access to data between different local machines, and using only a trusted third party server to access the entire data.

Original: Nonetheless, generalizing the existing data parallelism techniques of classical distributed training to the graph domain is non-trivial, which is mainly due to the dependency between nodes in a graph.
Revised: Nonetheless, generalizing the existing data parallelism techniques of classical distributed training to the graph domain is non-trivial, which is mainly due to the dependency between nodes in a graph.

Original: For example, unlike solving image classiﬁcation problems where images are mutually independent, such that we can divide the image dataset into several partitions without worrying about the dependency between images; GNNs are heavily relying on the information inherent to a node and its neighboring nodes.
Revised: For example, unlike solving image classiﬁcation problems where images are mutually independent, such that we can divide the image dataset into several partitions without worrying about the dependency between images; GNNs are heavily relying on the information inherent to a node and its neighboring nodes.

Original: As a result, partitioning graph leads to subgraphs with edges spanning subgraphs ( cut-edges ), which will cause information loss and hinder the performance of the model (Angerd et al., 2020).
Revised: As a result, partitioning graph leads to subgraphs with edges spanning subgraphs ( cut-edges ), which will cause information loss and hinder the performance of the model (Angerd et al., 2020).

Original: To cope with this problem, (Md et al., 2021; Jiang & Rumi, 2021; Angerd et al., 2020) propose to transfer node features and (Zheng et al., 2020; Tripathy et al., 2020; Scardapane et al., 2020) propose to transfer both the node feature and its hidden embeddings between local machines, both of which can cause signiﬁcant storage/communication overhead and privacy concerns (Shin et al., 2018; Wu et al., 2021).
Revised: To cope with this problem, (Md et al., 2021; Jiang & Rumi, 2021; Angerd et al., 2020) propose to transfer node features and (Zheng et al., 2020; Tripathy et al., 2020; Scardapane et al., 2020) propose to transfer both the node feature and its hidden embeddings between local machines, both of which can cause signiﬁcant storage/communication overhead and privacy concerns (Shin et al., 2018; Wu et al., 2021).

Original: [Figure]
Revised: [Figure]

Original:  cut-edges are ignored and only the model parameters are transferred and Global
Revised: the cut-edges are ignored and only the model parameters are transferred and Global

Original: Graph Sampling (GGS), where the cut-edges are considered and the node features of the cut-edges are transferred to the corresponding local machine, on the Reddit dataset using 8 machines.
Revised: Graph Sampling (GGS), where the cut-edges are considered and the node features of the cut-edges are transferred to the corresponding local machine, on the Reddit dataset using 8 machines.

Original: To better understand the challenge of distributed
Revised: To better understand the challenge of distributed

Original: GNN training, we compare the validation F1score in Figure 2 (a) and the average data communicated per round in Figure 2 (b) for two different distributed GNN training methods on the
Revised: GNN training, we compare the validation F1score in Figure 2 (a) and the average data communicated per round in Figure 2 (b) for two different distributed GNN training methods on the

Original: Reddit dataset.
Revised: Reddit dataset.

Original: On the one hand, we can observe that when ignoring the cut-edges, Parallel
Revised: On the one hand, we can observe that when ignoring the cut-edges, Parallel

Original: SGD with Periodic Averaging (PSGD-PA (Dean et al., 2012; Li et al., 2019b)) suffers from signiﬁcant accuracy drop and cannot achieve the same accuracy as the single machine training, even by increasing the number of communication.
Revised: SGD with Periodic Averaging (PSGD-PA (Dean et al., 2012; Li et al., 2020b)) suffers from signiﬁcant accuracy drop and cannot achieve the same accuracy as the single machine training, even by increasing the number of communication.

Original: However, Global Graph Sampling (GGS) can successfully reach the baseline by considering the cut-edges and allowing feature transfer, at the cost of signiﬁcant communication overhead, and potential violation of privacy.
Revised: However, Global Graph Sampling (GGS) can successfully reach the baseline by considering the cut-edges and allowing feature transfer, at the cost of signiﬁcant communication overhead, and potential violation of privacy.

Original: In this paper, we propose a communication-efﬁcient distributed GNN training method, called Learn
Revised: In this paper, we propose a communication-efﬁcient distributed GNN training method, called Learn

Original: Locally, Correct Globally (LLCG).
Revised: Locally, Correct Globally (LLCG).

Original: To reduce the communication overhead, inspired by the recent success of the distributed optimization with periodic averaging (Stich, 2018; Yu et al., 2019), we propose Local Training with Periodic Averaging : where each local machine ﬁrst locally trains a
Revised: To reduce the communication overhead, inspired by the recent success of the distributed optimization with periodic averaging (Stich, 2019; Yu et al., 2019), we propose Local Training with Periodic Averaging : where each local machine ﬁrst locally trains a

Original: GNN model by ignoring the cut-edges, then sends the trained model to the server for periodic model averaging, and receive the averaged model from server to continue the training.
Revised: GNN model by ignoring the cut-edges, then sends the trained model to the server for periodic model averaging, and receive the averaged model from server to continue the training.

Original: By doing so we eliminate the features exchange phase between server and local machines, but it can result in a signiﬁcant performance degradation due to the lack of the global graph structure and the dependency between nodes among different machines.
Revised: By doing so we eliminate the features exchange phase between server and local machines, but it can result in a signiﬁcant performance degradation due to the lack of the global graph structure and the dependency between nodes among different machines.

Original: To compensate for this error, we propose a Global Server
Revised: To compensate for this error, we propose a Global Server

Original: Correction scheme to take advantage of the available global graph structure on the server and reﬁne the averaged locally learned models before sending it back to each local machine.
Revised: Correction scheme to take advantage of the available global graph structure on the server and reﬁne the averaged locally learned models before sending it back to each local machine.

Original: Notice that without
Revised: Notice that without

Original: Global Server Correction , LLCG is similar to PSGD-PA as introduced in Figure 2.
Revised: Global Server Correction , LLCG is similar to PSGD-PA as introduced in Figure 2.

Original: To get a deeper understanding on the necessity of Global Server Correction , we provide the ﬁrst theoretical analysis on the convergence of distributed training for GNNs with periodic averaging.
Revised: To get a deeper understanding on the necessity of Global Server Correction , we provide the ﬁrst theoretical analysis on the convergence of distributed training for GNNs with periodic averaging.

Original: In particular, we show that solely averaging the local machine models and ignoring the global graph structure will suffer from an irreducible residual error, which provides sufﬁcient explanation on why Parallel SGD with Periodic Averaging can never achieve the same performance as the model trained on a single machine in Figure 2 (a).
Revised: In particular, we show that solely averaging the local machine models and ignoring the global graph structure will suffer from an irreducible residual error, which provides sufﬁcient explanation on why Parallel SGD with Periodic Averaging can never achieve the same performance as the model trained on a single machine in Figure 2 (a).

Original: Then, we theoretically analysis the convergence of our proposal LLCG .
Revised: Then, we theoretically analyze the convergence of our proposal LLCG .

Original: We show that by carefully choosing the number of global correction steps,
Revised: We show that by carefully choosing the number of global correction steps,

Original: LLCG can overcome the aforementioned residual error and enjoys O
Revised: LLCG can overcome the aforementioned residual error and enjoys O

Original: [Equation]
Revised: [Equation]

Original: convergence rate with P local machines and T iterations of gradient updates, which matches the rate of (Yu et al., 2019) on a general (not speciﬁc for GNN training) non-convex optimization setting.
Revised: with P local machines and T iterations of gradient updates, which matches the rate of (Yu et al., 2019) on a general (not speciﬁc for GNN training) non-convex optimization setting.

Original: Finally, we conduct comprehensive evaluations on real-world graph datasets with ablation study to validate the effectiveness of LLCG and its improvements over the existing distributed methods.
Revised: Finally, we conduct comprehensive evaluations on real-world graph datasets with ablation study to validate the effectiveness of LLCG and its improvements over the existing distributed methods.

Original: Before moving onto the problem formulation and our results, we summarize related work below.
Revised: Before moving onto the problem formulation and our results, we summarize related work below.

Original: Related works.
Revised: Related works.

Original: Recently, several attempts have been made on distributed GNN training.
Revised: Recently, several attempts have been made on distributed GNN training.

Original: According to how they deal with the input/hidden feature of nodes that are associated with the cut-edges (i.e., the edges spanning subgraphs of each local machine), existing methods can be classiﬁed into two main categories : ( 1 ) Input feature only communication-based methods : In these methods, each local machine receives the input features of all nodes required for the gradient computation from other machines, and train individually.
Revised: According to how they deal with the input/hidden feature of nodes that are associated with the cut-edges (i.e., the edges spanning subgraphs of each local machine), existing methods can be classiﬁed into two main categories : ( 1 ) Input feature only communication-based methods : In these methods, each local machine receives the input features of all nodes required for the gradient computation from other machines, and trains individually.

Original: However, since the number of required nodes grows exponentially with the number of layers, these methods suffer from a signiﬁcant communication and storage overhead.
Revised: However, since the number of required nodes grows exponentially with the number of layers, these methods suffer from a signiﬁcant communication and storage overhead.

Original: To alleviate these issues, (Md et al., 2021) proposes to split the original graph using a min-cut graph partition algorithm that can minimize the number of cut-edges.
Revised: To alleviate these issues, (Md et al., 2021) proposes to split the original graph using a min-cut graph partition algorithm that can minimize the number of cut-edges.

Original: (Jiang & Rumi, 2021) proposes to use importance sampling to assign nodes on the local machine with a higher probability.
Revised: (Jiang & Rumi, 2021) proposes to use importance sampling to assign nodes on the local machine with a higher probability.

Original: (Angerd et al., 2020) proposes to sample and save a small subgraph from other local machines as an approximation of the original graph structure.
Revised: (Angerd et al., 2020) proposes to sample and save a small subgraph from other local machines as an approximation of the original graph structure.

Original: Nonetheless, these methods are limited to a very shallow GNN structure and suffer from signiﬁcant performance degradation when the original graph is dense.
Revised: Nonetheless, these methods are limited to a very shallow GNN structure and suffer from signiﬁcant performance degradation when the original graph is dense.

Original: ( 2 ) Input and hidden feature communication-based methods : These methods propose to communicate hidden features in addition to the input node features.
Revised: ( 2 ) Input and hidden feature communication-based methods : These methods propose to communicate hidden features in addition to the input node features.

Original: Although these methods reduce the number of transferred bytes during each communication round (due to the smaller size of hidden embedding and less required nodes features), the number of communication rounds grows linearly as the number of layers, and are prone to more communication delay.
Revised: Although these methods reduce the number of transferred bytes during each communication round (due to the smaller size of hidden embedding and less required nodes features), the number of communication rounds grows linearly as the number of layers, and are prone to more communication delay.

Original: To address these issues, in addition to optimal partitioning of the graph, (Zheng et al., 2020) proposes to use sparse embedding to reduce the number of bytes to communicate and (Tripathy et al., 2020) proposes several graph partitioning techniques to diminish the communication overhead.
Revised: To address these issues, in addition to optimal partitioning of the graph, (Zheng et al., 2020) proposes to use sparse embedding to reduce the number of bytes to communicate and (Tripathy et al., 2020) proposes several graph partitioning techniques to diminish the communication overhead.

Original:  B ACKGROUND AND PROBLEM FORMULATION
Revised:  B ACKGROUND AND PROBLEM FORMULATION

Original: In this section, we start by describing Graph Convolutional Network (GCN) and its training algorithm on a single machine, then formulate the problem of distributed GCN training.
Revised: In this section, we start by describing Graph Convolutional Network (GCN) and its training algorithm on a single machine, then formulate the problem of distributed GCN training.

Original: Note that we use
Revised: Note that we use

Original: GCN with mean aggregation for simplicity, however, our discussion is also applicable to other GNN architectures, such as SAGE (Hamilton et al., 2017), GAT (Veliˇckovi´c et al., 2017), ResGCN (Li et al., 2019a) and APPNP (Klicpera et al., 2018).
Revised: GCN with mean aggregation for simplicity, however, our discussion is also applicable to other GNN architectures, such as SAGE (Hamilton et al., 2017), GAT (Velickovic et al., 2018), ResGCN (Li et al., 2019) and APPNP (Klicpera et al., 2019).

Original: Training GCN on a single machine.
Revised: Training GCN on a single machine.

Original: Here, we consider the semi-supervised node classiﬁcation in an undirected graph G ( V , E ) with N = |V| nodes and |E| edges.
Revised: Here, we consider the semi-supervised node classiﬁcation in an undirected graph G ( V , E ) with N = |V| nodes and |E| edges.

Original: Each node v i ∈ V is associated with a pair ( x i , y i ) , where x i ∈ R
Revised: Each node v i ∈ V is associated with a pair ( x i , y i ) , where x i ∈ R

Original: d is the input feature vector, y i ∈ R |C| is the ground truth label, and C is the candidate labels in the multi-class classiﬁcations.
Revised: d is the input feature vector, y i ∈ R |C| is the ground truth label, and C is the candidate labels in the multi-class classiﬁcations.

Original: Besides, let X =
Revised: Besides, let X =

Original: [ x 1 , . . .
Revised: [ x 1 , . . .

Original: , x N ] ∈ R N × d denote the input node feature matrix.
Revised: , x N ] ∈ R N × d denote the input node feature matrix.

Original: Our goal is to ﬁnd a set of parameters θ = { W ( (cid:96) ) } L(cid:96)
Revised: Our goal is to ﬁnd a set of parameters θ = { W ( (cid:96) ) } L(cid:96)

Original: =1 by minimizing the empirical loss L ( θ ) over all nodes in the training set, i.e.,
Revised: =1 by minimizing the empirical loss L ( θ ) over all nodes in the training set, i.e.,

Original: [Equation]
Revised: [Equation]

Original: where φ ( · , · ) is the loss function (e.g., cross entropy loss), σ ( · ) is the activation function (e.g., ReLU), and N ( v i ) is the neighborhood of node v i .
Revised: where φ ( · , · ) is the loss function (e.g., cross entropy loss), σ ( · ) is the activation function (e.g., ReLU), and N ( v i ) is the neighborhood of node v i .

Original: In practice, we can update the model parameters by the stochastic gradient computed on a sampled mini-batch (using full-neighbors) by
Revised: In practice, we can update the model parameters by the stochastic gradient computed on a sampled mini-batch (using full-neighbors) by

Original: [Equation]
Revised: [Equation]

Original: where ξ denotes an i.i.d.
Revised: where ξ denotes an i.i.d.

Original: sampled mini-batch of size B and we have E [ ˜ ∇L ( θ , ξ )]
Revised: sampled mini-batch of size B and we have E [ ˜ ∇L ( θ , ξ )]

Original: =
Revised: =

Original: ∇L ( θ ) .
Revised: ∇L ( θ ) .

Original: Distributed GCN training with periodic averaging.
Revised: Distributed GCN training with periodic averaging.

Original: In this paper, we consider the distributed learning setting with P local machines and a single parameter server.
Revised: In this paper, we consider the distributed learning setting with P local machines and a single parameter server.

Original: The original input graph G is partitioned into P subgraphs, where G p ( V p , E p ) denotes the subgraph on the p -th local machine with N p =
Revised: The original input graph G is partitioned into P subgraphs, where G p ( V p , E p ) denotes the subgraph on the p -th local machine with N p =

Original: |V p | nodes, and X p ∈ R N p
Revised: |V p | nodes, and X p ∈ R N p

Original: ×
Revised: ×

Original: d as the input feature of all nodes in V p located on the p -th machine.
Revised: d as the input feature of all nodes in V p located on the p -th machine.

Original: Then, the full-batch local gradient ∇L local p ( θ p ) is computed as ∇L
Revised: Then, the full-batch local gradient ∇L local p ( θ p ) is computed as ∇L

Original: [Equation]
Revised: [Equation]

Original: where θ p = { W ( (cid:96) ) p } L(cid:96)
Revised: where θ p = { W ( (cid:96) ) p } L(cid:96)

Original: =1 is the model parameters on the p -th local machine, N p ( v i ) =
Revised: =1 is the model parameters on the p -th local machine, N p ( v i ) =

Original: { v j | ( v i , v j ) ∈
Revised: { v j | ( v i , v j ) ∈

Original: E p } is the local neighbors of node v i on the p -th local machine.
Revised: E p } is the local neighbors of node v i on the p -th local machine.

Original: When the graph is large, the computational complexity of forward and backward propagation could be very high.
Revised: When the graph is large, the computational complexity of forward and backward propagation could be very high.

Original: One practical
Revised: One practical

Original: Input: Global parameters ¯ θ 0 , local parameters θ 0 p = ¯ θ 0 , time-step t = 0 , learning rate η .  ξ
Revised: Input: Global parameters ¯ θ 0 , local parameters θ 0 p = ¯ θ 0 , time-step t = 0 , learning rate η . t . ξ

Original: ˜ ∇L local p ( θ tp , ξ tp ) .
Revised: ˜ ∇L local p ( θ tp , ξ tp ) .

Original:  +1 p 
Revised: t +1 p =

Original: = θ tp − η ˜ ∇L local p ( θ tp , ξ tp ) . 
Revised: θ tp − η ˜ ∇L local p ( θ tp , ξ tp ) . +

Original:  solution is to compute the stochastic gradient on a sampled mini-batch with neighbor sampling, i.e., ˜ ∇L
Revised: 1 p . solution is to compute the stochastic gradient on a sampled mini-batch with neighbor sampling, i.e., ˜ ∇L

Original: [Equation]
Revised: [Equation]

Original: where ξ p is an i.i.d.
Revised: where ξ p is an i.i.d.

Original: sampled mini-batch of B p nodes, ˜ N p ( v i ) ⊂
Revised: sampled mini-batch of B p nodes, ˜ N p ( v i ) ⊂

Original: N ( v i ) is the sampled neighbors.
Revised: N ( v i ) is the sampled neighbors.

Original: An illustration of distributed GCN training with Parallel SGD with Periodic Averaging (PSGD-PA) is summarized in Algorithm 1.
Revised: An illustration of distributed GCN training with Parallel SGD with Periodic Averaging (PSGD-PA) is summarized in Algorithm 1.

Original: Before training, the server maintains a global model ¯ θ 

Revised: Before training, the server maintains a global model ¯ θ

Original: and each local machine keeps a local copy of the same model θ 0 p .
Revised: and each local machine keeps a local copy of the same model θ 0 p .

Original: During training, the local machine ﬁrst updates the local model θ tp using the stochastic gradient ˜ ∇L local p ( θ tp , ξ tp ) computed by Eq.
Revised: During training, the local machine ﬁrst updates the local model θ tp using the stochastic gradient ˜ ∇L local p ( θ tp , ξ tp ) computed by Eq.

Original: 4 for K iterations (line 8), then sends the local model θ tp to the server (line 10).
Revised: 4 for K iterations (line 8), then sends the local model θ tp to the server (line 10).

Original: At each communication step, the server collects and averages the model parameters from the local machines (line 12) and send the averaged model θ t +1 p back to each local machine.
Revised: At each communication step, the server collects and averages the model parameters from the local machines (line 12) and send the averaged model θ t +1 p back to each local machine.

Original: Limitations.
Revised: Limitations.

Original: Although PSGD-PA can signiﬁcantly reduce the communication overhead by transferring the locally trained models instead of node feature/embeddings (refer to Figure 2 (b)), it suffers from performance degeneration due to ignorance of the cut-edges (refer to Figure 2 (a)).
Revised: Although PSGD-PA can signiﬁcantly reduce the communication overhead by transferring the locally trained models instead of node feature/embeddings (refer to Figure 2 (b)), it suffers from performance degeneration due to ignorance of the cut-edges (refer to Figure 2 (a)).

Original: In the next section, we introduce a communication-efﬁcient algorithm LLCG that does not suffer from this issue, and can achieve almost the same performance as training the model on a single machine.
Revised: In the next section, we introduce a communication-efﬁcient algorithm LLCG that does not suffer from this issue, and can achieve almost the same performance as training the model on a single machine.

Original:  P ROPOSED A LGORITHM : L EARN L OCALLY C ORRECT G LOBALLY
Revised:  P ROPOSED A LGORITHM : L EARN L OCALLY C ORRECT G LOBALLY

Original: In this section, we describe Learn Locally, Correct Globally (LLCG) for distributed GNN training.
Revised: In this section, we describe Learn Locally, Correct Globally (LLCG) for distributed GNN training.

Original: LLCG includes two main phases, local training with periodic model averaging and global server correction , to help reduce both the number of required communications and size of transferred data, without compromising the predictive accuracy.
Revised: LLCG includes two main phases, local training with periodic model averaging and global server correction , to help reduce both the number of required communications and size of transferred data, without compromising the predictive accuracy.

Original: We summarize the details of LLCG in Algorithm 2.
Revised: We summarize the details of LLCG in Algorithm 2.

Original: 3. LOCAL TRAINING WITH PERIODIC MODEL AVERAGING
Revised: 3. LOCAL TRAINING WITH PERIODIC MODEL AVERAGING

Original: At the beginning of a local epoch, each local machine receives the latest global model parameters from the server (line 3).
Revised: At the beginning of a local epoch, each local machine receives the latest global model parameters from the server (line 3).

Original: Next, each local machine runs Kρ r iterations to update the local model (line 4 to 9), where K and ρ are the hyper-parameters that control the local epoch size.
Revised: Next, each local machine runs Kρ r iterations to update the local model (line 4 to 9), where K and ρ are the hyper-parameters that control the local epoch size.

Original: Note that instead of using a ﬁxed local epoch size as Algorithm 1, we choose to use exponentially increasing local epoch size in LLCG with ρ > 1 .
Revised: Note that instead of using a ﬁxed local epoch size as Algorithm 1, we choose to use exponentially increasing local epoch size in LLCG with ρ > 1 .

Original: The reason are as follows.
Revised: The reasons are as follows.

Original: At the beginning of the training phase, all local models θ t p are far from the optimal solution and will receive a gradient ˜ ∇L local p ( θ tp , ξ tp ) computed by Eq. 4.
Revised: At the beginning of the training phase, all local models θ t p are far from the optimal solution and will receive a gradient ˜ ∇L local p ( θ tp , ξ tp ) computed by Eq. 4.

Original: Using a smaller local update steps at the early stage guarantees each local models does not diverge too much from each other before the model averaging step at the server side (line 12).
Revised: Using a smaller local update step at the early stage guarantees each local model does not diverge too much from each other before the model averaging step at the server side (line 12).

Original: However, towards the end of the training, all local models θ tp will receive relatively smaller gradient ˜
Revised: However, towards the end of the training, all local models θ tp will receive relatively smaller gradient ˜

Original: ∇L local p ( θ tp , ξ tp ) , such that we can chose a larger local epoch size to reduce the number of communications, without worrying about the divergence of local models.
Revised: ∇L local p ( θ tp , ξ tp ) , such that we can chose a larger local epoch size to reduce the number of communications, without worrying about the divergence of local models.

Original: By doing so, after total number of T = (cid:80)
Revised: By doing so, after total number of T = (cid:80)

Original: [Equation]
Revised: [Equation]

Original: iterations, LLCG only requires
Revised: iterations, LLCG only requires

Original: Input: Global parameters ¯ θ 0 , local parameters θ 0 p , time-step t = 0 , local step size hyperparameters K, ρ ,  learning rate γ, η 1:  do 2: do parallel  machine 3:
Revised: Input: Global parameters ¯ θ 0 , local parameters θ 0 p , time-step t = 0 , local step size hyperparameters K, ρ , and learning rate γ, η 1: for do 2: do parallel training local machine 3:

Original: Local machine  t
Revised: Local machine parameters ← t

Original:  4: for k  do 5: t 
Revised: Communication 4: for k ← do 5: t ←

Original: Local machine p constructs mini batch ξ tp neighbor sampling 7:
Revised: Local machine p constructs mini-batch ξ tp neighbor sampling 7:

Original: Local machine p computes stochastic gradients ˜
Revised: Local machine p computes stochastic gradients ˜

Original: ∇L local p ( θ tp , ξ tp ) 8:
Revised: ∇L local p ( θ tp , ξ tp ) 8:

Original: Local machine p updates model parameter by θ t +1 p
Revised: Local machine p updates model parameter by θ t +1 p

Original: = θ tp − η ˜ ∇L local p ( θ tp , ξ tp ) 9: end for 10:
Revised: = θ tp − η ˜ ∇L local p ( θ tp , ξ tp ) 9: end for 10:

Original: Local machine p sends the local parameters θ t +1 p to the server (cid:46) Communication 11: end 
Revised: Local machine p sends the local parameters θ t +1 p to the server (cid:46) Communication 11: end for 12:

Original: Server updates the global parameters using parameter averaging ¯ θ t +1 = 1 P (cid:80)
Revised: Server updates the global parameters using parameter averaging ¯ θ t +1 = 1 P (cid:80)

Original: Pp =1 θ t +1 p 13: for s ← 1 to S do (cid:46) Server Correction 14: t ← t
Revised: Pp =1 θ t +1 p 13: for s ← 1 to S do (cid:46) Server Correction 14: t ← t

Original: Server constructs a mini batch ξ t with full neighbors 16:
Revised: Server constructs a mini-batch ξ t with full-neighbors 16:

Original: Server computes the stochastic gradient ˜ ∇L (¯ θ t , ξ t ) 17:
Revised: Server computes the stochastic gradient ˜ ∇L (¯ θ t , ξ t ) 17:

Original: Server updates the  parameters by θ t
Revised: Server updates the global parameters by θ t

Original: 1 = ¯ θ t − γ ˜ ∇L
Revised: 1 = ¯ θ t − γ ˜ ∇L

Original: ¯ θ t , ξ t ) 18: end for 19: end for Output: Server θ 
Revised: ¯ θ t , ξ t ) 18: end for 19: end for Output: Server θ t )

Original: R = log ρ TK rounds of communications.
Revised: R = log ρ TK rounds of communications.

Original: Therefore, compared to the fully-synchronous method, we can signiﬁcantly reduce the total number of communications from O ( T ) to O (log ρ TK ) .
Revised: Therefore, compared to the fully-synchronous method, we can signiﬁcantly reduce the total number of communications from O ( T ) to O (log ρ TK ) .

Original: 3. G LOBAL SERVER CORRECTION
Revised: 3. G LOBAL SERVER CORRECTION

Original: The design of the global server correction is to ensure that the trained model not only learns from the data on each local machine, but also learns the global structure of the graph, thus reducing the information loss causing by graph partitioning and avoiding cut-edges.
Revised: The design of the global server correction is to ensure that the trained model not only learns from the data on each local machine, but also learns the global structure of the graph, thus reducing the information loss caused by graph partitioning and avoiding cut-edges.

Original: Before the correction, the server receives the locally trained models from all local machines (line 10) and applies model parameter averaging (line 12).
Revised: Before the correction, the server receives the locally trained models from all local machines (line 10) and applies model parameter averaging (line 12).

Original: Next, S server correction steps are applied on top of the averaged model (line 13 to 18).
Revised: Next, S server correction steps are applied on top of the averaged model (line 13 to 18).

Original: During the correction, the server ﬁrst construct a mini-batch ξ t using fullneighbors 1 (line 15), compute the stochastic gradient ˜ ∇L (¯ θ t , ξ t ) on the constructed mini-batch by
Revised: During the correction, the server ﬁrst constructs a mini-batch ξ t using fullneighbors 1 (line 15), compute the stochastic gradient ˜ ∇L (¯ θ t , ξ t ) on the constructed mini-batch by

Original: Eq. 2 (line 16) and update the averaged model ¯ θ t for S iterations (line 17).
Revised: Eq. 2 (line 16) and update the averaged model ¯ θ t for S iterations (line 17).

Original: In practice, the number of correction steps S  depends on the heterogeneity among the subgraph on each local machine: the more heterogeneous the subgraphs are, the more correction steps is required to better reﬁne the averaged model and reduce the divergence across the local models.
Revised: The number of correction steps S 2 depends on the heterogeneity among the subgraphs on each local machine: the more heterogeneous the subgraphs are, the more correction steps are required to better reﬁne the averaged model and reduce the divergence across the local models.

Original: Note that, the heterogeneity is minimized when employing GGS (Figure 2) with the local machines having access to the full graph, as a result.
Revised: Note that, the heterogeneity is minimized when employing GGS (Figure 2) with the local machines having access to the full graph, as a result.

Original: However, GGS requires sampling from the global graph and communication at every iterations, which results in additional overhead and lower efﬁciency.
Revised: However, GGS requires sampling from the global graph and communication at every iteration, which results in additional overhead and lower efﬁciency.

Original: Instead, in LLCG we are trading computation on the server for the costly communication between local machines and the server.
Revised: Instead, in LLCG we are trading computation on the server for the costly feature communication, and only requires periodic communication.

Original:  T HEORETICAL A NALYSIS
Revised:  T HEORETICAL A NALYSIS

Original: In this section, we provide the convergence analysis on the distributed training of GCN under two different settings, i.e., with and without server correction.
Revised: In this section, we provide the convergence analysis on the distributed training of GCN under two different settings, i.e., with and without server correction.

Original: In the following, we ﬁrst introduce the notations and assumptions for the analysis (Section 4.1).
Revised: In the following, we ﬁrst introduce the notations and assumptions for the analysis (Section 4.1).

Original: Then, we show that periodic averaging of local machine models alone and ignoring the global graph structure will suffers from an irreducible residual error (Section 4.2).
Revised: Then, we show that periodic averaging of local machine models alone and ignoring the global graph structure will suffer from an irreducible residual error (Section 4.2).

Original: Finally, we show that this residual error can be eliminated by running server correction steps after each periodic averaging step on the server (Section 4.3).[Figure] \mathcal{N}(v_
Revised: Finally, we show that this residual error can be eliminated by running server correction steps after each periodic averaging step on the server (Section 4.3).

Original: [Figure]
Revised: [Figure]

Original: 4. N OTATIONS AND ASSUMPTIONS
Revised: 4. N OTATIONS AND ASSUMPTIONS

Original: Let us ﬁrst recall the notations deﬁned in Section 2, where L ( θ ) denotes the global objective function computed using the all node features X and the original graph G , L p ( θ ) denotes the local objective function computed using the local node features X p and local graph G p , θ tp denotes the model parameters on the p -th local machine at the t -th step, and ¯ θ t = 1 P (cid:80)
Revised: Let us ﬁrst recall the notations deﬁned in Section 2, where L ( θ ) denotes the global objective function computed using the all node features X and the original graph G , L p ( θ ) denotes the local objective function computed using the local node features X p and local graph G p , θ tp denotes the model parameters on the p -th local machine at the t -th step, and ¯ θ t = 1 P (cid:80)

Original: P p =1 θ tp denotes the virtual averaged model at the t -th step.
Revised: P p =1 θ tp denotes the virtual averaged model at the t -th step.

Original: In the non-convex optimization, our goal is to show the expected gradient of the global objective on the virtual averaged model parameters E [ (cid:107)∇L (¯ θ t ) (cid:107) 2 ] decreases as the number of local machine P and the number of training steps T increase.
Revised: In the non-convex optimization, our goal is to show the expected gradient of the global objective on the virtual averaged model parameters E [ [Equation] (¯ θ t ) [Equation]

Revised: 2 ] decreases as the number of local machines P and the number of training steps T increase.

Original: Besides, we introduce ∇L full p ( θ ) as the gradient computed on the p -th local machine but have access the full node features
Revised: Besides, we introduce ∇L full p ( θ ) as the gradient computed on the p -th local machine but have access the full node features

Original: X and the original graph structure G as ∇L
Revised: X and the original graph structure G as ∇L

Original: [Equation]
Revised: [Equation]

Original: Please refer to Figure 3 for an illustration of different gradient computations.
Revised: Please refer to Figure 3 for an illustration of different gradient computations.

Original: Besides, we introduce local-global gradient discrepancy as κ 2 = κ 2 A
Revised: Besides, we introduce local-global gradient discrepancy as κ 2 = κ 2 A

Original: + κ 2 X , where κ 2 A
Revised: + κ 2 X , where κ 2 A

Original: = max p ∈
Revised: = max p ∈

Original: [ P ] {(cid:107)∇L local p ( θ ) − ∇L full p ( θ ) (cid:107) 2 } is the maximum difference between the gradient computed on the local machine with and without having access to the global graph structure, which is mainly due to fact that the local machines are oblivious to the full graph information; and κ 2 X = max p ∈
Revised: [ P ] { [Equation] local p ( θ ) − ∇L full p ( θ ) (cid:107) 2 } is the maximum difference between the gradient computed on the local machine with and without having access to the global graph structure, which is mainly due to fact that the local machines are oblivious to the full graph information; and κ 2 X = max p ∈

Original: [ P ] {
Revised: [ P ] {

Original: [Equation]
Revised: [Equation]

Original: p ( θ ) − ∇L ( θ ) (cid:107) 2 } is the maximum difference between the gradient computed using the local node and all nodes, which is mainly due to the heterogeneity of the node features on each local machine, and we have κ 2 X
Revised: p ( θ ) − [Equation] θ ) (cid:107) 2 } is the maximum difference between the gradient computed using the local node and all nodes, which is mainly due to the heterogeneity of the node features on each local machine, and we have κ 2 X

Original: =if the nodes are i.i.d.
Revised: =if the nodes are i.i.d.

Original: sampled to each local machine.
Revised: sampled to each local machine.

Original: Notice that local-global gradient discrepancy κ 2 plays an important role in our theoretical results.
Revised: Notice that local-global gradient discrepancy κ 2 plays an important role in our theoretical results.

Original: For the convergence analysis, we make the following standard assumptions.
Revised: For the convergence analysis, we make the following standard assumptions.

Original: Assumption 1 The stochastic gradient on the p -th local machine (with neighbor sampling) has stochastic gradient variance bounded by σ 2 var and stochastic gradient bias bounded by σ 2 bias , i.e.,
Revised: Assumption 1 The stochastic gradient on the p -th local machine (with neighbor sampling) has stochastic gradient variance bounded by σ 2 var and stochastic gradient bias bounded by σ 2 bias , i.e.,

Original: E [ (cid:107) ˜
Revised: E [ (cid:107) ˜

Revised: ∇L local

Original: [Equation]
Revised: [Equation]

Original: Assumption 2 The stochastic gradient for global server correction (with full neighbors) has stochastic gradient variance bounded by σ 2 global , i.e., E [ (cid:107) ˜ ∇L full p ( [Equation] ξ ) − ∇L full p ( θ 
Revised: Assumption 2 The stochastic gradient for global server correction (with full neighbors) has stochastic gradient variance bounded by σ 2 global , i.e., E [ (cid:107) ˜ ∇L full p ( θ ; ξ ) − ∇L full p ( θ )] (cid:107) 2 ] ≤ σ 2 global .

Original: [Equation] ] (cid:107) 2 ] ≤ σ 2 global . The existence of stochastic gradient bias and variance in sampling-based GNN training have been studied in (Cong et al., 2020; 2021), where (Cong et al., 2021) further quantify the stochastic gradient bias and variance as a function of the number of GCN layers.
Revised: The existence of stochastic gradient bias and variance in sampling-based GNN training have been studied in (Cong et al., 2020; 2021), where (Cong et al., 2021) further quantify the stochastic gradient bias and variance as a function of the number of GCN layers.

Original: In particular, they show that the existence of σ 2 bias is due to neighbor sampling and non-linear activation, and we have σ 2 bias = 0 if all neighbors are used or the non-linear activation is removed.
Revised: In particular, they show that the existence of σ 2 bias is due to neighbor sampling and non-linear activation, and we have σ 2 bias = 0 if all neighbors are used or the non-linear activation is removed.

Original: The existence of σ 2 var is because we are sampling mini-batches to compute the stochastic gradient on each local machine during training.
Revised: The existence of σ 2 var is because we are sampling mini-batches to compute the stochastic gradient on each local machine during training.

Original: As the mini-batch size increases, σ 2 var will be decreasing, and we have σ 2 var = 0 when using full-batch.
Revised: As the mini-batch size increases, σ 2 var will be decreasing, and we have σ 2 var = 0 when using full-batch.

Original: 4. D ISTRIBUTED GNN VIA P ARAMETER A VERAGING
Revised: 4. D ISTRIBUTED GNN VIA P ARAMETER A VERAGING

Original: In the following, we provide the ﬁrst convergence analysis on distributed training of GCN.
Revised: In the following, we provide the ﬁrst convergence analysis on distributed training of GCN.

Original: We show that solely periodic averaging of the local machine models and ignoring the global graph structure suffers from a residual error that is irreducible.
Revised: We show that solely periodic averaging of the local machine models and ignoring the global graph structure suffers from a residual error that is irreducible.

Original: Comparing to the traditional distributed training (e.g., distributed training Convolutional Neural Network for image classiﬁcation (Dean et al., 2012;
Revised: Comparing to the traditional distributed training (e.g., distributed training Convolutional Neural Network for image classiﬁcation (Dean et al., 2012;

Original: Li et al., 2019b)), the key challenges in the distributed GCN training is the two different types of gradient bias: ( 1 ) The expectation of the local full-batch gradient is a biased estimation of the global full-batch gradient, i.e., [Equation] P
Revised: Li et al., 2020b)), the key challenges in the distributed GCN training is the two different types of gradient bias: ( 1 ) The expectation of the local full-batch gradient is a biased estimation of the global full-batch gradient, i.e., 1 P

Original: [Equation]
Revised: [Equation]

Original: θ ) .
Revised: θ ) .

Original: This is because each local machine does not have access to the original input graph and full node feature matrix.
Revised: This is because each local machine does not have access to the original input graph and full node feature matrix.

Original: Note that the aforementioned equivalence is important for the classifcal distributed training analysis Dean et al.
Revised: Note that the aforementioned equivalence is important for the classifcal distributed training analysis Dean et al.

Original: Yu et al.
Revised: Yu et al.

Original: The expectation of the local stochastic gradient is a biased estimation of the local fullbatch gradient i.e., E [ ˜ ∇L local p ( θ , ξ p )] (cid:54) =
Revised: The expectation of the local stochastic gradient is a biased estimation of the local fullbatch gradient i.e., E [ ˜ ∇L local p ( θ , ξ p )] (cid:54) =

Original: ∇L local p ( θ ) .
Revised: ∇L local p ( θ ) .

Original: This is because the stochastic gradient on each local machine is computed by using neighbor sampling, which has been studied in (Cong et al., 2021).
Revised: This is because the stochastic gradient on each local machine is computed by using neighbor sampling, which has been studied in (Cong et al., 2021).

Original: Theorem 1 (Distributed GCN via Parameter Averaging) Consider applying model averaging for
Revised: Theorem 1 (Distributed GCN via Parameter Averaging) Consider applying model averaging for

Original: GNN training under Assumption 1 and 2.
Revised: GNN training under Assumption 1 and 2.

Original: If we choose learning rate η = √ P √ T and the local step size
Revised: If we choose learning rate η = √ P √ T and the local step size

Original: K ≤ √ 2 T 1 / 4 8 LP 3 / 4 , then for any T ≥ L 2 P steps of gradient updates we have
Revised: K ≤ √ 2 T 1 / 4 8 LP 3 / 4 , then for any T ≥ L 2 P steps of gradient updates we have

Original: [Equation]
Revised: [Equation]

Original: Theorem 1 implies that, by carefully choosing the learning rate η and the local step size K , the gradient norm computed on the virtual averaged model is bounded by O (1 / √ PT ) after R = T/K
Revised: Theorem 1 implies that, by carefully choosing the learning rate η and the local step size K , the gradient norm computed on the virtual averaged model is bounded by O (1 / √ PT ) after R = T/K

Original: = O
Revised: = O

Original: ( P 3 / 4 T 3 / 4 ) communication rounds, but suffers from an irreducible residual error O ( κ 2 + σ 2 bias ) .
Revised: ( P 3 / 4 T 3 / 4 ) communication rounds, but suffers from an irreducible residual error O ( κ 2 + σ 2 bias ) .

Original: In the next section, we show that this residual error can be eliminated by applying server correction.
Revised: In the next section, we show that this residual error can be eliminated by applying server correction.

Original: 4. D ISTRIBUTED GCN VIA S ERVER C ORRECTION
Revised: 4. D ISTRIBUTED GCN VIA S ERVER C ORRECTION

Original: Before processing to our result, in order to simplify the presentation, let us ﬁrst deﬁne the notationG
Revised: Before proceeding to our result, in order to simplify the presentation, let us ﬁrst deﬁne the notation

Original: [Equation]
Revised: [Equation]

Original: (¯ θ t ) (cid:107) 2 ] and G r local =
Revised: (¯ θ t ) (cid:107) 2 ] and G r local =

Original: min t ∈T local ( r ) E
Revised: min t ∈T local ( r ) E

Original: (cid:2)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: (cid:80) Pp =1 ∇L localp ( 
Revised: as the minimum gradient computed at the r -th round global and local step, where T global ( r ) and T local ( r ) are the number of iteration run after the r -th communication round on server and local machine, respectively.

Original: [Equation]
Revised: Please refer to Eq.

Original: tp ) (cid:13)(cid:13) (cid:3) as the minimum gradient computed at the r -th round global and local step.
Revised: 42 in Appendix C.2 for a formal deﬁnition.

Original: Theorem 2 Consider applying model averaging for GCN training under Assumption 1 and 2.
Revised: Theorem 2 Consider applying model averaging for GCN training under Assumption 1 and 2.

Original: If we choose learning rate γ = η = √ P √ T , the local step size K, ρ such that
Revised: If we choose learning rate γ = η = √ P √ T , the local step size K, ρ such that

Original: [Equation]
Revised: [Equation]

Original: K [Equation] ρ
Revised: and server correction step size

Original: [Equation]
Revised: [Equation]

Original: r ≤ RT [Equation] 2 , and server correction step size S = maxr ∈ [ R ]
Revised: = max

Original: [Equation]
Revised: [Equation]

Original: Kρ r G r local , then for any T ≥ L 2 P steps of gradient updates we have
Revised: , then for any T ≥ L 2 P steps of gradient updates we have: 1 T

Original: [Equation]
Revised: [Equation]

Original: Theorem 2 implies that, by carefully choosing the learning rates γ and η , the local step size hyperparameters K, ρ , and the number of global correction steps S , after T steps ( R rounds of communication), employing parameter averaging with Global Server Correction , we have the norm of gradient bounded by O (1 / √ PT ) , without suffering the residual error that exists in the naive parameter averaging (in Theorem 1).
Revised: Theorem 2 implies that, by carefully choosing the learning rates γ and η , the local step size hyperparameters K, ρ , and the number of global correction steps S , after T steps ( R rounds of communication), employing parameter averaging with Global Server Correction , we have the norm of gradient bounded by O (1 / √ PT ) , without suffering the residual error that exists in the naive parameter averaging (in Theorem 1).

Original: Besides, the server correction step size is proportional to the scale of κ 2 and local stochastic gradient bias σ 2 bias .
Revised: Besides, the server correction step size is proportional to the scale of κand local stochastic gradient bias σ 2 bias .

Original: The larger κ 2 and σ 2 bias , the more corrections are required to eliminate the residual error.
Revised: The larger κ 2 and σ 2 bias , the more corrections are required to eliminate the residual error.

Original: However, in practice, we observe that  very small number of correction steps (e.g., S = 1 ) performs well, which minimize the computation overhead on the server.
Revised: However, in practice, we observe that a very small number of correction steps (e.g., S = 1 ) performs well, which minimizes the computation overhead on the server.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Real-world simulation.
Revised: Real-world simulation.

Original: In a real-world distributed setting, the server and local machines  located on different machines, connected through the network (Li et al., 2020).
Revised: In a real-world distributed setting, the server and local machines are located on different machines, connected through the network (Li et al., 2020a).

Original: However, for our experiments, we only have access to a single machine with multiple GPUs.
Revised: However, for our experiments, we only have access to a single machine with multiple GPUs.

Original: As a result, we simulate a real-world distributed learning scenario, such that each GPU is responsible for the computation of two local machines (8 in total) and the CPU acts as the server.
Revised: As a result, we simulate a real-world distributed learning scenario, such that each GPU is responsible for the computation of two local machines (8 in total) and the CPU acts as the server.

Original: For these reasons, in our evaluations, we opted to report the communication size and number of communication rounds, instead of the wall-clock time, which can show the beneﬁt of distributed training.
Revised: For these reasons, in our evaluations, we opted to report the communication size and number of communication rounds, instead of the wall-clock time, which can show the beneﬁt of distributed training.

Original: We argue that these are acceptable measures in real-world scenarios as well since the two main factors in distributed training are initializing connection overhead and bandwidth (Tripathy et al., 2020).
Revised: We argue that these are acceptable measures in real-world scenarios as well since the two main factors in distributed training are initializing connection overhead and bandwidth (Tripathy et al., 2020).

Original: Baselines.
Revised: Baselines.

Original: To illustrate the effectiveness of LLCG, we setup two general synchronized distributed training techniques as the our baseline methods, namely “ Parallel SGD with Parameter Averaging ” (PSGD-PA) and “ Global Graph Sampling ” (GGS), as introduced in Figure 2. Note that we choose GGS as a reasonable representation for most existing proposals (Md et al., 2021;Zheng et al., 2020; Tripathy et al., 2020) for distributed GNN training, since these methods have very close communication cost and also require a large cluster of machines to truly show their performance
Revised: To illustrate the effectiveness of LLCG, we setup two general synchronized distributed training techniques as the our baseline methods, namely “ Parallel SGD with Parameter Averaging ” (PSGD-PA) and “ Global Graph Sampling ” (GGS), as introduced in Figure 2, where the cut-edges in PSGD-PA are ignored and only the model parameters are transferred, but the cut-edges in GGS are considered and the node features of the cut-edges are transferred to the corresponding machine. Note that we choose GGS as a reasonable representation for most existing proposals (Md et al., 2021;

Original: [Table]
Revised: [Table]

Original:  improvement.
Revised: Zheng et al., 2020; Tripathy et al., 2020) for distributed GNN training, since these methods have very close communication cost and also require a large cluster of machines to truly show their performance improvement.

Original: We also use PSGD-PA as a lower bound for communication size, which is widely used in traditional distributed training and similar to the one used in (Angerd et al., 2020; Jiang &
Revised: We also use PSGD-PA as a lower bound for communication size, which is widely used in traditional distributed training and similar to the one used in (Angerd et al., 2020; Jiang &

Original: Rumi, 2021).
Revised: Rumi, 2021).

Original: However, we did not speciﬁcally include these methods in our results since we could not reproduce their results in our settings.
Revised: However, we did not speciﬁcally include these methods in our results since we could not reproduce their results in our settings.

Original: Please refer to Appendix A for a detailed description of implementation, hardware speciﬁcation and link to our source code.
Revised: Please refer to Appendix A for a detailed description of implementation, hardware speciﬁcation and link to our source code.

Original: Datasets and evaluation metric.
Revised: Datasets and evaluation metric.

Original: We compare LLCG and other baselines onsix real-world semisupervised node classiﬁcation datasets, detail of which are summarized in Table 2.
Revised: We compare LLCG and other baselines on real-world semisupervised node classiﬁcation datasets, detail of which are summarized in Table 2 in the Appendix.

Revised: The input graphs are splitted into multiple subgraphs using METIS before training, then the same set of subgraphs are used for all baselines.

Original: For training, we use neighborhood sampling (Hamilton et al., 2017) with 10 neighbors sampled per node and ρ = 1.for LLCG . 
Revised: For training, we use neighborhood sampling (Hamilton et al., 2017) with 10 neighbors sampled per node and ρ = 1 . 1 for LLCG.

Revised: For a fair comparison, we chose the base local update step K such that LLCG has the same number of local update steps as

Original:  During evaluation, we use full-batch without sampling, and report the performance on the full graph using AUC ROC and F1 Micro as the evaluation metric.
Revised: PSGD-PA. During evaluation, we use full-batch without sampling, and report the performance on the full graph using AUC ROC and F1 Micro as the evaluation metric.

Original: Unless otherwise stated, we conducted each experiment ﬁve times and report the mean and standard deviation.
Revised: Unless otherwise stated, we conduct each experiment ﬁve times and report the mean and standard deviation.

Original: 5. P RIMARY R ESULTS
Revised: 5. P RIMARY R ESULTS

Original: In this section we summarize the evaluation of our proposed LLCG algorithm against baselines on four datasets.
Revised: In this section, we compare our proposed LLCG algorithm with baselines on four datasets.

Original: We also evaluate these methods on Yelp and OGB-Products , however, due to space limitations we differ the detailed discussion on these datasets to the Appendix A.3.
Revised: We also evaluate these methods on Yelp , OGB-Products , and OGB-MAG240M . However, due to space limitations we defer the detailed discussion on these datasets to the Appendix A.4.

Original: [Figure]
Revised: [Figure]

Original:  requires 
Revised: round of communications in ( e,f ), and the global validation score per bytes of exchanged data in ( g,h ). LLCG requires same number of communications.

Original: Figure 4 (a) through 4 (d) illustrate the validation accuracy per communication rounds on four different datasets.
Revised: Figure 4 (a) through 4 (d) illustrate the validation accuracy per communication rounds on four different datasets.

Original: We run a ﬁxed number of communication rounds and plot the global validation score (the validation score computed using the full-graph on the server) at the end of each communication step.
Revised: We run a ﬁxed number of communication rounds and plot the global validation score (the validation score computed using the full-graph on the server) at the end of each communication step.

Original: For PSGD-PA and GGS, the score is  calculated on the averaged model, whereas for LLCG the validation is calculated after the correction step.
Revised: For PSGD-PA and GGS, the score is [Figure] [Figure] calculated on the averaged model, whereas for LLCG the validation is calculated after the correction step.

Original: It can be seen that PSGD-PA suffers from accuracy loss compared to other two methods, due to the residual error we discussed in Section 4, while both GGS and LLCG perform well and can achieve the expected accuracy.
Revised: It can be seen that PSGD-PA suffers from performance drop compared to other two methods, due to the residual error we discussed in Section 4, while both GGS and LLCG perform well and can achieve the expected accuracy.

Original: Note that the accuracy loss of PSGD-PA can vary across different datasets; in some cases such as Reddit , PSGD-PA can signiﬁcantly hurt the accuracy, while on other datasets the gap is smaller.
Revised: Note that the performance drop of PSGD-PA can vary across different datasets; in some cases such as Reddit , PSGD-PA can signiﬁcantly hurt the accuracy, while on other datasets the gap is smaller.

Original: Nevertheless, LLCG can always close the gap between PSGD-PA [Figure] [Figure]
Revised: Nevertheless, LLCG can always close the gap between PSGD-PA and GGS with minimal overhead.

Original: [Figure]

Original: LLCG convergences as fast as GGS.
Revised: LLCG convergences as fast as GGS.

Original: To represent the effect of communication on the real-time convergence, in Figure 4 (e) and 4 (f), we plot the global training loss (training loss computed on the full-graph on the server) after each communication round.
Revised: To represent the effect of communication on the real-time convergence, in Figure 4 (e) and 4 (f), we plot the global training loss (training loss computed on the full-graph on the server) after each communication round.

Original: Similar to the accuracy score, the training loss is also computed on the server averaged (and corrected, in case of LLCG) global model.
Revised: Similar to the accuracy score, the training loss is also computed on the server averaged (and corrected, in case of LLCG) global model.

Original: These results clearly indicate that LLCG can improve the convergence compared to PSGD-PA, while it shows  similar performance per communication rounds to GGS.
Revised: These results clearly indicate that LLCG can improve the convergence over PSGD-PA, while it shows a similar convergence speed to GGS.

Original: LLCG exchanges data as little as PSGD-PA.
Revised: LLCG exchanges data as little as PSGD-PA.

Original: Figure 4 (g) and 4 (h) show the relation between global validation accuracy with the average size (volume) of communication in bytes.
Revised: Figure 4 (g) and 4 (h) show the relation between global validation accuracy with the average size (volume) of communication in bytes.

Original: As expected, this ﬁgure clearly shows the effectiveness of LLCG .
Revised: As expected, this ﬁgure clearly shows the effectiveness of LLCG .

Original: On the one hand, LLCG has a similar amount of communication volume as PSGD-PA but can achieve a higher accuracy.
Revised: On the one hand, LLCG has a similar amount of communication volume as PSGD-PA but can achieve a higher accuracy.

Original: On the other hand,
Revised: On the other hand,

Original: LLCG requires signiﬁcantly less amount of communication volume than GGS to achieve the same accuracy, which leads to slower training time in real world setting.
Revised: LLCG requires signiﬁcantly less amount of communication volume than GGS to achieve the same accuracy, which leads to slower training time in real world settings.

Original: LLCG works with various GNN models and aggregations.
Revised: LLCG works with various GNN models and aggregations.

Original: We evaluate four popular GNN models, used in recent graph learning literature: GCN Kipf & Welling (2016), SAGE Hamilton et al.
Revised: We evaluate four popular GNN models, used in recent graph learning literature: GCN Kipf & Welling (2017), SAGE Hamilton et al.

Original:  GAT Veli ˇ ckovi ´ c et al.
Revised: (2017), GAT Velickovic et al.

Original: (2017) and APPNP Klicpera et al.
Revised: and APPNP Klicpera et al.

Original: In Table 1, we summarize the test score and average communication size (in MB) on different datasets for  ﬁxed number of communication rounds.
Revised: In Table 1, we summarize the test score and average communication size (in MB) on different datasets for a ﬁxed number of communication rounds.

Original: Note that we only include the results for the aggregation methods (GCN or SAGE) that has higher accuracy for the speciﬁc datasets, details of which can be found in
Revised: Note that we only include the results for the aggregation methods (GCN or SAGE) that have higher accuracy for the speciﬁc datasets, details of which can be found in

Original: Appendix A.2.
Revised: Appendix A.2.

Original: As shown here, LLCG can consistently improve the test accuracy for all different models comparing to PSGD-PA, while the communication size is signiﬁcantly lower than GGS, since
Revised: As shown here, LLCG can consistently improve the test accuracy for all different models compared to PSGD-PA, while the communication size is signiﬁcantly lower than GGS, since

Original: LLCG only need to exchange the model parameters.
Revised: LLCG only needs to exchange the model parameters.

Original: Effect of local epoch size.
Revised: Effect of local epoch size.

Original: Figure 5 compares the effect of various values of local epoch size
Revised: Figure 5 compares the effect of various values of local epoch size

Original: K ∈ { 1 , 4 , 16 , 64 , 128 } for ﬁxed ρ and S on the OGB-Arxiv dataset.
Revised: K ∈ { 1 , 4 , 16 , 64 , 128 } for ﬁxed ρ and S on the OGB-Arxiv dataset.

Original: When using fully synchronous with K = 1 , the model suffers from very slow convergence and need more communications.
Revised: When using fully synchronous with K = 1 , the model suffers from very slow convergence and needs more communications.

Original: Further increasing the K to larger values can speed up the training; however, we found a diminishing return point for K > 128 in this dataset and extremely large K in general.
Revised: Further increasing the K to larger values can speed up the training; however, we found a diminishing return point for K > 128 in this dataset and extremely large K in general.

Original: Effect of sampling at correction.

Original: Recall that LLCG requires full-neighbors for the global server correction step for the convergence analysis, however, we ﬁnd that server correction with neighbor sampling also works well in practice.

Original: As shown in Figure 6 (and 9), although server correction with neighbor sampling can introduce some randomness at the beginning of the training phase, the ﬁnal accuracy of training is very close to the server correction with full-neighbors.

Original: Effect of sampling in local machines.
Revised: Effect of sampling in local machines.

Original: In Figure 7, we report the validation scores per round of communication to compare the effect of neighborhood sampling at local machines.
Revised: In Figure 6, we report the validation scores per round of communication to compare the effect of neighborhood sampling at local machines.

Original: We can observe that when the neighborhood sampling size is reasonably large (i.e., 20% ), the performance is very similar to full neighborhood training.
Revised: We can observe that when the neighborhood sampling size is reasonably large (i.e., 20% ), the performance is very similar to full neighborhood training.

Original: However, reducing the neighbor sampling ratio to 5% could result in a larger local stochastic gradient bias σ 2 bias , which requires using more correction steps ( S ).
Revised: However, reducing the neighbor sampling ratio to 5% could result in a larger local stochastic gradient bias σ 2 bias , which requires using more correction steps ( S ).

Original:  C ONCLUDING R EMARKS
Revised:  C ONCLUDING R EMARKS

Original: In this paper, we propose a novel distributed algorithm for training Graph Neural Networks (GNNs).
Revised: In this paper, we propose a novel distributed algorithm for training Graph Neural Networks (GNNs).

Original: We theoretically analyze various GNN models and discover that, unlike the traditional deep neural networks, due to inherent data samples dependency in GNNs,  applying naive parameter averaging leads to a residual error and current solutions to this issue impose huge communication overheads.
Revised: We theoretically analyze various GNN models and discover that, unlike the traditional deep neural networks, due to inherent data samples dependency in GNNs, naively applying periodic parameter averaging leads to a residual error and current solutions to this issue impose huge communication overheads.

Original: Instead, our proposed method, LLCG tackles these problems by applying correction on top of locally learned models, to infuse the global structure of the graph back into the network and avoid any costly communication.
Revised: Instead, our proposal tackles these problems by applying correction on top of locally learned models, to infuse the global structure of the graph back into the network and avoid any costly communication.

Original: In addition, through extensive empirical analysis, we support our theoretical ﬁndings and demonstrate that LLCG can achieve high accuracy without additional communication costs.
Revised: In addition, through extensive empirical analysis, we support our theoretical ﬁndings and demonstrate that LLCG can achieve high accuracy without additional communication costs.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: We provide an anonymous git repository in Appendix A including all code and scripts used in our experimental studies.
Revised: We provide an anonymous git repository in Appendix A including all code and scripts used in our experimental studies.

Original: This repository includes a README.md ﬁle, explaining how to install and prepare the code and required packages.
Revised: This repository includes a README.md ﬁle, explaining how to install and prepare the code and required packages.

Original: We also provide detailed instruction on how to use the partitioning scripts for various datasets.
Revised: We also provide detailed instruction on how to use the partitioning scripts for various datasets.

Original: In addition, we provide several conﬁguration ﬁles (under scripts/configs ) folder for different hyper-parameters on each individual dataset and a general script ( scripts/run-config.py ) to run and reproduce the results which these conﬁgurations.
Revised: In addition, we provide several conﬁguration ﬁles (under scripts/configs ) folder for different hyper-parameters on each individual dataset and a general script ( scripts/run-config.py ) to run and reproduce the results which these conﬁgurations.

Original: Details of various models and parameters used in our evaluation studies can also be found in
Revised: Details of various models and parameters used in our evaluation studies can also be found in

Original: Appendix A
Revised: Appendix A

Original: Appendix Table of Contents A
Revised: Appendix Table of Contents A

Original:  A.1 Hardware speciﬁcation and environment . . . . . . . . . . . . . . . . . . . . .
Revised: studies A.1 Hardware speciﬁcation and environment . . . . . . . . . . . . . . . . . . . . .

Original: A.2 Dataset and models details . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.2 Dataset and models details . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A.3  Effectiveness of correction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 
Revised: A.3 Minibatch selection for correction step . . . . . . . . . . . . . . . . . . . . . . A.4 Effectiveness of correction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.5 Effectiveness in large-scale settings . . . . . . . . . . . . . . . . . . . . . . . .

Original: B Proof of Theorem
Revised: B Proof of Theorem

Original: B.1 Useful lemmas .
Revised: B.1 Useful lemmas .

Original: B.2 Main proof of Theorem 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: B.2 Main proof of Theorem 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Revised: B.3 Discussion on the irreducible error . . . . . . . . . . . . . . . . . . . . . . . .

Original: B.3 Proof of Lemma 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B.4 Proof of Lemma 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: B.4 Proof of Lemma 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B.5 Proof of Lemma 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: B.5 Proof of Lemma 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: B.6 Proof of Lemma 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: C Proof of Theorem
Revised: C Proof of Theorem

Original: C.1 Useful lemma . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: C.1 Useful lemma . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Revised: [Table]

Original: C.2 Main proof of Theorem 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 
Revised: Main proof of Theorem 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C.

Original: Proof of Lemma 
Revised: Proof of Lemma 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A A DDITIONAL EXPERIMENTAL SETUP AND STUDIES
Revised: A A DDITIONAL EXPERIMENTAL SETUP AND STUDIES

Original: We provide detailed experimental setup and additional evaluations in this section.
Revised: We provide detailed experimental setup and additional evaluations in this section.

Original: The link to the anonymized repository is available: https://anonymous.4open.science/r/llcg-2733/.
Revised: The link to the anonymized repository is available: https://anonymous.4open.science/r/llcg-2733/.

Original: A. H ARDWARE SPECIFICATION AND ENVIRONMENT
Revised: A. H ARDWARE SPECIFICATION AND ENVIRONMENT

Original: For all our experiments we use a single server equipped with 4 NVIDIA QUADRO RTX
Revised: For all our experiments we use a single server equipped with 4 NVIDIA QUADRO RTX

Original: GPUs with driver version 460.80, two Intel Xeon 6230 CPU and 768GB of main memory using Ubuntu 18.04 running kernel 5.4.0.
Revised: GPUs with driver version 460.80, two Intel Xeon 6230 CPU and 768GB of main memory using Ubuntu 18.04 running kernel 5.4.0.

Original: For fair comparison between all methods, we developed a uniﬁed framework for performing the experiments, using Pytorch 1.7 compiled for CUDA 11.2.
Revised: For fair comparison between all methods, we developed a uniﬁed framework for performing the experiments, using Pytorch 1.7 compiled for CUDA 11.2.

Original: We also used Pytorch Geometric 1.7.0 and Pytorch Sparse 0.6.8 for various GNN and sparse operations.
Revised: We also used Pytorch Geometric 1.7.0 and Pytorch Sparse 0.6.8 for various GNN and sparse operations.

Original: Please refer to README.md for detailed instruction on how to install and run the code, and also dataset speciﬁc conﬁgurations
Revised: Please refer to README.md for detailed instruction on how to install and run the code, and also dataset speciﬁc conﬁgurations

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A. D ATASET AND MODELS DETAILS
Revised: A. D ATASET AND MODELS DETAILS

Original: In all experiments, we used ADAM optimizer, and the learning rate, alongside the local machine mini-batch size ( B L ) and server mini-batch size ( B S ) are included in config ﬁles under scripts/configs folder.
Revised: In all experiments, we used ADAM optimizer, and the learning rate, alongside the local machine mini-batch size ( B L ) and server mini-batch size ( B S ) are included in config ﬁles under scripts/configs folder.

Original: We used three datasets Flickr, Reddit, Yelp from Zeng et al.
Revised: We used three datasets Flickr, Reddit, Yelp from Zeng et al.

Original: (2019) and OGB-Proteins, OGB-Arxiv, OGB-Products from Hu et al.
Revised: (2020) and OGB-Proteins, OGB-Arxiv, OGB-Products from Hu et al.

Original: For each dataset, we choose the aggregation method that has been shown to achieve the highest accuracy, and reported the results in Table 1.
Revised: For each dataset, we choose the aggregation method that has been shown to achieve the highest accuracy, and reported the results in Table 1.

Original: In addition, we also evaluate two frequently used architectures,
Revised: In addition, we also evaluate two frequently used architectures,

Original: GAT and APPNP to show the ﬂexibility of our proposed method to other models.
Revised: GAT and APPNP to show the ﬂexibility of our proposed method to other models.

Original: Note that some of
Revised: Note that some of

Original: OGB datasets can use more complicated models Sun & Wu (2021; 2020), however for the sake of simplicity, we only used popular GNN operators which show competitive results in most cases.
Revised: OGB datasets can use more complicated models Sun & Wu (2021; 2020), however for the sake of simplicity, we only used popular GNN operators which show competitive results in most cases.

Original: Next we formally introduce various layers and operators used in our experiments, followed by the base architecture for each dataset in Table 2.
Revised: Next we formally introduce various layers and operators used in our experiments, followed by the base architecture for each dataset in Table 2.

Original: • GCN (G) : Originally introduced by Kipf et.
Revised: • GCN (G) : Originally introduced by Kipf et.

Original: al Kipf & Welling (2016), the representation for node v i is calculated using:
Revised: al Kipf & Welling (2017), the representation for node v i is calculated using:

Original: [Equation]
Revised: [Equation]

Original: where α i,j = 1 √ deg( v i )deg( v j ) for symmetric normalized Laplacian and α i,j =
Revised: where α i,j = 1 √ deg( v i )deg( v j ) for symmetric normalized Laplacian and α i,j =

Original: [Equation]
Revised: [Equation]

Original: ( v i ) for row normalized Laplacian.
Revised: ( v i ) for row normalized Laplacian.

Original: • SAGE (S) : First introduce in Hamilton et al.
Revised: • SAGE (S) : First introduce in Hamilton et al.

Original: (2017), SAGE learn different weights for the node and its neighbors and the representation is calculated by:
Revised: (2017), SAGE learn different weights for the node and its neighbors and the representation is calculated by:

Original: [Equation]
Revised: [Equation]

Original: Note that different operator such as concatenation can be used instead of addition in the above formula.
Revised: Note that different operator such as concatenation can be used instead of addition in the above formula.

Original: However in our experiments we used addition for all SAGE layers.
Revised: However in our experiments we used addition for all SAGE layers.

Original: • Linear (L) : Applies a linear transformation to the node features.
Revised: • Linear (L) : Applies a linear transformation to the node features.

Original: In other word, the graph structured is completely ignored in this layer and the representation is computed as follows.
Revised: In other word, the graph structured is completely ignored in this layer and the representation is computed as follows.

Original: [Equation]
Revised: [Equation]

Original: [Figure] • BatchNorm (B) : Applies batch normalization according to Ioffe & Szegedy (2015) using the following where γ and β are learnable parameters and (cid:15) is added for numerical stability.
Revised: • BatchNorm (B) : Applies batch normalization according to Ioffe & Szegedy (2015) using the following where γ and β are learnable parameters and (cid:15) is added for numerical stability.

Original: [Equation]
Revised: [Equation]

Original: • GAT : The Graph Attention layer proposed by (Veliˇckovi´c et al., 2017) as follows.
Revised: • GAT : The Graph Attention layer proposed by (Velickovic et al., 2018) as follows.

Original: [Equation]
Revised: [Equation]

Original: where α i,j is the attention between node i and j and calculated by:
Revised: where α i,j is the attention between node i and j and calculated by:

Original: [Equation]
Revised: [Equation]

Original: where a is the learnable weight vector and || indicates concatenation.
Revised: where a is the learnable weight vector and || indicates concatenation.

Original: • APPNP :
Revised: • APPNP :

Original: This network is proposed by (Klicpera et al., 2018), which is a combination of graph-agnostic prediction and label propagation:
Revised: This network is proposed by (Klicpera et al., 2019), which is a combination of graph-agnostic prediction and label propagation:

Original: [Equation]
Revised: [Equation]

Original: where α i,j is same as deﬁned in GCN, β is the teleport probability and h (0) is the output of ﬁrst linear layer.
Revised: where α i,j is same as deﬁned in GCN, β is the teleport probability and h (0) is the output of ﬁrst linear layer.

Revised: Effect of sampling at correction.

Revised: Recall that LLCG requires full-neighbors for the server correction step for the convergence analysis, however, we ﬁnd that server correction with neighbor sampling also works well in practice.

Revised: As shown in Figure 7 and Figure 8 (in Appendix ??

Revised: ), although server correction with neighbor sampling can introduce some randomness at the beginning of the training phase, the ﬁnal accuracy of training is very close to the server correction with full-neighbors.

Original:  A. E FFECTIVENESS OF CORRECTION 
Revised: [Figure] [Figure] A. M INIBATCH SELECTION FOR CORRECTION STEP

Revised: Recall that in a server correction step, the mini-batch is selected by sampling from the entire graph, uniformly at random to estimate an unbiased stochastic gradient of the global loss function.

Revised: However, one might suggest to include more cut-edges, which are missing from the local machines, in the correction minibatch instead of uniform random sampling, to improve the performance of the global model.

Revised: To this end, we conduct an experiments on two datasets ( Reddit and Arxiv ), where we compare the random minibatch (default setting in LLCG ) and minibatch with higher number of cut-edges, by selecting the nodes on the ends of cut-edges and building the minibatch from there.

Revised: As shown in Figure 9, including more cut-edges nodes in the mini-batch does not make signiﬁcant improvement when comparing to selecting mini-batchs using uniform sampling.

Revised: This potentially happens due to the fact that sampling more cut-edges make the gradient of server correction step biased,

Revised: [Figure]

Revised: G l ob a l V a l .

Revised: F 1 S c

Revised: o r e

Revised: Reddit

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: G l ob a l V a l .

Revised: F 1 S c

Revised: o r e

Original:  To illustrate the effectiveness of correction steps and relate it to the beneﬁts of global graph structure, we conduct the same experiments on Yelp and OGB-Products with 700K and 2.4M nodes, respectively and report the results in Figure 8.
Revised: Arxiv [Figure] A. E FFECTIVENESS OF CORRECTION To illustrate the effectiveness of correction steps and relate it to the beneﬁts of global graph structure, we conduct the same experiments on Yelp and OGB-Products with 700K and 2.4M nodes, respectively and report the results in Figure 10.

Original:  In case of Yelp in Figure 8 (a), as we can see the PSGDPA and GGS are performing quite similarly.
Revised: Yelp dataset. In case of Yelp in Figure 10 (a), as we can see the PSGD-PA and GGS are performing quite similarly.

Original: To further investigate this, we compared the validation accuracy of Yelp using GCN against MLP (i.e., without utilizing the graph structure in training), where the GCN layers replaced by Linear layer (i.e LLL instead of GGG architecture) and plot the validation accuracy per iterations in Figure 8 (b).
Revised: To further investigate this, we compared the validation accuracy of Yelp using GCN against MLP (i.e., without utilizing the graph structure in training), where the GCN layers replaced by Linear layer (i.e LLL instead of GGG architecture) and plot the validation accuracy per iterations in Figure 10 (b).

Original: As it can be deducted from this ﬁgure, for this dataset MLP is performing as good as GCN, which means this dataset does not depend on the global structure of the graph, further explaining why there is no performance gap between PSGD-PA and GGS.
Revised: As it can be deducted from this ﬁgure, for this dataset MLP is performing as good as GCN, which means this dataset does not depend on the global structure of the graph, further explaining why there is no performance gap between PSGD-PA and GGS.

Original: In other words, no server correction is necessary and we can use S = 0 in this case for LLCG.
Revised: In other words, no server correction is necessary and we can use S = 0 in this case for LLCG.

Original: [Figure] Similarly, in case of OGB-Products in Figure 8 (c), we cannot see any noticeable accuracy drop due to distributed training.
Revised: OGB-Products. Similarly, in case of OGB-Products in Figure 10 (c), we cannot see any noticeable accuracy drop due to distributed training.

Original: However, unlike Yelp , this is mainly due to very small κ which is caused by two main factors: (1) very small ratio of training nodes for this dataset, only 8% of the nodes are used for training, and (2) very small number of cut-edges (less than 7% ) after applying METIS on this dataset.
Revised: However, unlike Yelp , this is mainly due to very small κ which is caused by two main factors: (1) very small ratio of training nodes for this dataset, only 8% of the nodes are used for training, and (2) very small number of cut-edges (less than 7% ) after applying METIS on this dataset.

Original: It is also worth mentioning that due to shallow structure of the model required for this dataset, we barely need multi-hop neighbors and consequently we cannot see any noticeable performance drop. 
Revised: It is also worth mentioning that due to shallow structure of the model required for this dataset, we barely need multi-hop neighbors and consequently we cannot see any noticeable performance drop. [Figure]

Revised: A. E FFECTIVENESS IN LARGE - SCALE SETTINGS

Revised: In this section, we further investigate the effect of large number of local machines and large graphs on the performance of proposed algorithm, under the simulated environment with up to 16 local machines on OGB-Products and OGB-MAG240M Hu et al.

Original: We would like to compare larger datasets as more GNN datasets are released in the future.
Revised: (2021) datasets for node classiﬁcation. Besides, we also compare against subgraph approximation distributed GNN training algorithm Angerd et al. (2020) using 10% extra storage overhead and fully synchronous distributed GNN training. Node classiﬁcation tasks.

Revised: For the OGB-Products dataset, we train a 3 -layer GraphSAGE model with learning rate 0 .

Original: B P ROOF OF T HEOREM
Revised: 003 with 50 rounds of communications , which has the same hyper-parameter conﬁguration and model architecture as OGB-Products ’s leaderboard implementation . For the OGB-MAG240M dataset, we train a 2 -layer skip-connected GraphSAGE model with learning rate 0 . 001 with 400 rounds of communications, which has the same hyper-parameter selection and model architecture as OGB-MAG240M ’s leaderboard implementation.

Revised: As shown in Figure 11, we can observe that

Revised: • PSGD-PA suffers from performance degeneration and has a large gap to the full sync training accuracy.

Revised: Subgraph approximation might alleviate the issue to some extent, but requires signiﬁcant storage overhead.

Revised: Our proposal LLCG can bridge the gap between PSGD-PA and full sync training.

Revised: • The pure computation time 4 is negligible when ignoring the communication time but just consider the computation time.

Revised: Besides, the server correction step does not introduce signiﬁcant computation overhead.

Revised: It is worth mentioning that we do not observe the performance gap between PSGD-PA and fully sync distributed training in Figure 10 (c) because of different number of layers (2-layers) and limited number of local machine (8 local machines) are used in the previous experiments.

Revised: However, as the number of layers and number of local machines increase, the effect of ignoring cut-edges and data heterogeneity is becoming more serious, which is an interesting observation and worth exploring as a potential future direction.

Original: In this section, we ﬁrst introduce the useful lemmas in Section B.1, then process our proof of theorem in Section B.2.
Revised: In this section, we ﬁrst introduce the useful lemmas in Section B.1, then process our proof of theorem in Section B.2.

Original: In particular, we show that solely averaging the local machine models and ignoring the global graph structure will suffer from an irreducible residual error.
Revised: In particular, we show that solely averaging the local machine models and ignoring the global graph structure will suffer from an irreducible residual error.

Original: B. U SEFUL LEMMAS
Revised: B. U SEFUL LEMMAS

Original: The following lemma gives the upper bound for the norm of the stochastic gradient on each local machine as the norm of the expectation of local gradient and the stochastic gradient variance scaled by the number of local machines P .
Revised: The following lemma gives the upper bound for the norm of the stochastic gradient on each local machine as the norm of the expectation of local gradient and the stochastic gradient variance scaled by the number of local machines P .

Original: Lemma 1 Let ˜ ∇L local p
Revised: Lemma 1 Let ˜ ∇L local p

Original: [Equation]
Revised: [Equation]

Revised: θ p

Revised: [Equation]

Original: ξ p ) be stochastic gradients such that
Revised: ξ p ) be stochastic gradients such that

Original: [Equation]
Revised: [Equation]

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: The following lemma provides an upper bound on the difference of each local gradient ∇L local p ( θ p ) compared to 1 P 
Revised: The following lemma provides an upper bound on the difference of each local gradient ∇L local p ( θ p ) compared to 1 P (cid:80)

Original: [Equation]
Revised: [Equation]

Original: ( θ q ) , which is a function of:
Revised: ( θ q ) , which is a function of:

Original: • The deviation of each local model to the virtual averaged model, i.e., 1 P
Revised: • The deviation of each local model to the virtual averaged model, i.e., 1 P

Original: [Equation]
Revised: [Equation]

Original: • The difference of gradient computed with and without having access to the full graph information, i.e., κ 2 . 
Revised: • The difference of gradient computed with and without having access to the full graph information, i.e., κ 2 . [Figure]

Revised: Comparing PSGD-PA, periodic model averaging with subgraph approximation, full sync distributed GNN training, and LLCG on OGB-Products and OGB-MAG240M datasets on a simulated enviroment with 16 local machines.

Original: Lemma 

Revised: Lemma

Original: For any P machines θ p , p ∈ { 1 , . . .
Revised: For any P machines θ p , p ∈ { 1 , . . .

Original: , P } , if we deﬁne ¯ θ = 1 P (cid:80)
Revised: , P } , if we deﬁne ¯ θ = 1 P (cid:80)

Original: Pi =1 θ p and let κ > 0 such that
Revised: Pi =1 θ p and let κ > 0 such that

Original: [Equation]
Revised: [Equation]

Original:  , we have
Revised: κ 2 , we have

Original: [Equation]
Revised: [Equation]

Original: The following lemma provides an upper bound on the deviation of each local model to virtual averaged model, which plays an important role in the previous lemma.
Revised: The following lemma provides an upper bound on the deviation of each local model to virtual averaged model, which plays an important role in the previous lemma.

Original: Lemma 

Revised: Lemma

Original: For all P machines with parameters θ p , p ∈ { 1 , . . .
Revised: For all P machines with parameters θ p , p ∈ { 1 , . . .

Original: , P } , if we deﬁne ¯ θ = 1 P (cid:80)
Revised: , P } , if we deﬁne ¯ θ = 1 P (cid:80)

Original: Pi =1 θ p and let κ > 0 such that (cid:107)L local p
Revised: P i =1 θ p and let κ > 0 such that (cid:107)L local p

Original: [Equation]
Revised: [Equation]

Original: , we have
Revised: , we have

Original: [Equation]
Revised: [Equation]

Original: B. M AIN PROOF OF T HEOREM
Revised: B. M AIN PROOF OF T HEOREM

Original: Equipped with the above lemmas, we are now ready to present the proof of Theorem 1.
Revised: Equipped with the above lemmas, we are now ready to present the proof of Theorem 1.

Original: From the smoothness assumption, we have
Revised: From the smoothness assumption, we have

Original: [Equation]
Revised: [Equation]

Original:  Recall that ¯ θ t is deﬁned as ¯ θ t = 1 P (cid:80)
Revised: [Equation] Recall that ¯ θ t is deﬁned as ¯ θ t = 1 P [Equation]

Original: Pi =1 θ tp for any t .
Revised: Pi =1 θ tp for any t .

Original: Then, according to the update rule
Revised: Then, according to the update rule

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: We can upper bound the right hand side of Eq. 19 by
Revised: We can upper bound the right hand side of Eq. 19 by

Original: [Equation]
Revised: [Equation]

Original: where ( a ) is due to 2 (cid:104)
Revised: where ( a ) is due to 2 (cid:104)

Original: x , y
Revised: x , y

Original: [Equation]
Revised: [Equation]

Original: = (cid:107) x (cid:107) 2 +
Revised: = (cid:107) x (cid:107) 2 +

Original: [Equation]
Revised: [Equation]

Original: y (cid:107) 2 − (cid:107) x − y (cid:107) 2 and ( b ) is due to
Revised: y (cid:107) 2 − (cid:107) x − y (cid:107) 2 and ( b ) is due to

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where ( c ) follows from smoothness assumption and the deﬁnition of κ 2 , σ 2 bias in Theorem
Revised: where ( c ) follows from smoothness assumption and the deﬁnition of κ 2 , σ 2 bias in Theorem

Original: [Equation]
Revised: [Equation]

Original: Combining Eq.
Revised: Combining Eq.

Original: 19 and Eq.
Revised: 19 and Eq.

Original: 20 gives us
Revised: 20 gives us

Original: [Equation]
Revised: [Equation]

Original: According to the update rule
Revised: According to the update rule

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: we have
Revised: we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Substituting Eq.
Revised: Substituting Eq.

Original: 22 and Eq.
Revised: 22 and Eq.

Original: 24 to Eq.
Revised: 24 to Eq.

Original: 17, we have
Revised: 17, we have

Original: [Equation]
Revised: [Equation]

Original: Dividing both sides by η 2 and rearranging the terms yields
Revised: Dividing both sides by η 2 and rearranging the terms yields

Original: [Equation]
Revised: [Equation]

Original: P P
Revised: P P

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: ∇L
Revised: ∇L

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: ) (cid:13)(cid:13)(cid:13) (cid:105)
Revised: ) (cid:13)(cid:13)(cid:13) (cid:105)

Original: [Equation]
Revised: [Equation]

Original: (cid:104)(cid:13)(cid:13)(cid:13)
Revised: (cid:104)(cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: (cid:104) ˜
Revised: (cid:104) ˜

Original: [Equation]
Revised: [Equation]

Original: pt ) (cid:105) (cid:13)(cid:13)(cid:13) (cid:105)
Revised: pt ) (cid:105) (cid:13)(cid:13)(cid:13) (cid:105)

Original: [Equation]
Revised: [Equation]

Original: P (cid:88)
Revised: P (cid:88)

Original: [Equation]
Revised: [Equation]

Original: (cid:107)
Revised: (cid:107)

Original: [Equation]
Revised: [Equation]

Original: (cid:107)
Revised: (cid:107)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:16)
Revised: (cid:16)

Original: [Equation]
Revised: [Equation]

Original: (cid:104)(cid:13)(cid:13)(cid:13)
Revised: (cid:104)(cid:13)(cid:13)(cid:13)

Original: P
Revised: P

Original: P (cid:88) p
Revised: P (cid:88) p

Original: [Equation]
Revised: [Equation]

Original: E (cid:104) ˜
Revised: E (cid:104) ˜

Original: ∇L
Revised: ∇L

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: ) (cid:105) (cid:13)(cid:13)(cid:13)
Revised: ) (cid:105) (cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: (cid:105)
Revised: (cid:105)

Original: [Equation]
Revised: [Equation]

Original: var
Revised: var

Original: [Equation]
Revised: [Equation]

Original: P P
Revised: P P

Original: [Equation]
Revised: [Equation]

Original: E (cid:104) ˜
Revised: E (cid:104) ˜

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: tp
Revised: tp

Original: [Equation]
Revised: [Equation]

Original: tp ) (cid:105) (cid:13)(cid:13)(cid:13)
Revised: tp ) (cid:105) (cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: where ( a ) is due to Lemma 1.
Revised: where ( a ) is due to Lemma 1.

Original: Summing over t ∈ { 0 , . .
Revised: Summing over t ∈ { 0 , . .

Original: , T − 1 } and dividing both side by T , we get
Revised: , T − 1 } and dividing both side by T , we get

Original: [Equation]
Revised: [Equation]

Original: where ( a ) is due to Lemma 3, ( b ) is due to 0 < η < 1 L , and ( c ) is due to K ≤ √
Revised: where ( a ) is due to Lemma 3, ( b ) is due to 0 < η < 1 L , and ( c ) is due to K ≤ √

Original: 2 8 Lη is selected to satisfy 1 − 16 η 2 L 2 K 2 ≥ 12 .
Revised: 2 8 Lη is selected to satisfy 1 − 16 η 2 L 2 K 2 ≥ 12 .

Original: If we choose η = √ P √ T and K ≤ √ 2 T 1 / 4 8 LP 3 / 4 =
Revised: If we choose η = √ P √ T and K ≤ √ 2 T 1 / 4 8 LP 3 / 4 =

Original: O ( T 1 / 4 P 3 / 4 ) , then for any T ≥ L 2 P we have
Revised: O ( T 1 / 4 P 3 / 4 ) , then for any T ≥ L 2 P we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: with R =
Revised: with R =

Original: O ( P 3 / 4 T 3 / 4 ) rounds of communication.
Revised: O ( P 3 / 4 T 3 / 4 ) rounds of communication.

Revised: B. D ISCUSSION ON THE IRREDUCIBLE ERROR

Revised: To better understand why the error in the upper bound might be irreducible (i.e., the second term in

Revised: RHS of Theorem 1 is independent of T), let’s ﬁrst recall the key sources that cause gradient diversity (cid:13)(cid:13)(cid:13) ∇L (¯ θ t ) − 1 P

Revised: [Equation]

Original: B. P ROOF OF L EMMA
Revised: P [Equation] ( θ t p [Equation]

Revised: (cid:13)(cid:13)(cid:13) 2 (in Eq. 21), in our setting: • The κ 2 term, which is the upper bound of 1 P

Revised: [Equation]

Revised: Pp =

Revised: [Equation]

Revised: ¯ θ

Revised: [Equation]

Revised: [Equation]

Revised: p

Revised: [Equation]

Revised: ¯ θ t )

Revised: [Equation]

Revised: and is caused by ignoring the cut-edges for local gradient computation; • The σ 2 bias term, which is the upper bound of 1 P

Revised: [Equation]

Revised: and is caused by using neighbor sampling; • The model divergence term 1 P

Revised: [Equation]

Revised: Pp

Revised: [Equation]

Revised: θ

Revised: [Equation]

Revised: θ tp

Revised: [Equation]

Revised: 2 which is caused by the difference between the model parameters and the virtual average model due to periodic averaging.

Revised: Notice that the model divergence term also exists in distributed learning regardless of whether the dataset is graph or not and caused by infrequent synchronization.

Revised: Now

Revised: [Equation]

Revised: let’s have a closer look at the model divergence term.

Revised: As shown in Lemma 3 the model divergence term 1 P

Revised: [Equation]

Revised: is further caused by three factors:

Revised: • The σ 2 var term (in Eq. 64), which is the upper bound of mini-batch sampling variance 1 P

Revised: [Equation]

Revised: θ

Revised: [Equation]

Revised: (cid:105)

Revised: [Figure]

Revised: • The κ 2 term, which is the upper bound of 1 P

Revised: [Figure]

Revised: is caused by ignoring the cut-edges for local gradient computation; • The σ 2 bias term, which is the upper bound of 1 P

Revised: [Figure]

Revised: [Equation]

Revised: and is caused by using neighbor sampling.

Revised: Fortunately, the model divergence term can be controlled by the number of local gradient update steps and learning rate, which is reducing with respect to the number of total training steps T and the number of local machine P , and it leads to the ﬁrst term in our upper bound in Theorem 1, i.e.,

Revised: O ( σ 2 var + σ 2 bias +

Revised: [Equation]

Revised: 2 √ PT )

Revised: =

Revised: O ( 1 √ PT ) .

Revised: However, unfortunately, κ 2 and σ 2 bias in gradient diversity are isolated from the model diversity part, therefore are irreducible and results in O ( σ 2 bias + κ 2 ) in the second term in our upper bound in Theorem 1.

Original: By the deﬁnition of L local p ( θ p ; ξ p ) , we have
Revised: Please refer to Figure 12 for an illustration. Intuitively, these theoretical results make sense. During training, we are minimizing the loss without cut-edges 1 P [Equation]

Original: E (cid:104)(cid:13)(cid:13) (cid:13) P P (cid:88)
Revised: Pp =1 L local ( θ ) , which is a different objective to the loss deﬁned on the full-graph by taking the cut-edges into consideration L

Original: [Equation]
Revised: ( θ ) .

Original: ˜ [Equation] local
Revised: Therefore, solely by adding the number of training iterations, we cannot guarantee a small gradient of L ( θ ) by minimizing 1 P

Original: [Equation]
Revised: [Equation]

Original: p ) (cid:13)(cid:13) (cid:13) (cid:105) =

Original: E (cid:34)(cid:13) (cid:13)(cid:13)

Original:  P P (cid:88) p =1 (cid:16) ˜ ∇L local p ( θ p ; ξ p ) − E (cid:104) ˜ ∇L local p ( θ p , ξ tp ) (cid:105)(cid:17) (cid:13) (cid:13)(cid:13) (cid:35) + E (cid:104) (cid:13) (cid:13)(cid:13)
Revised: B. P ROOF OF L EMMA

Original: P P (cid:88) p = E (cid:104) ˜ ∇L local p ( θ p , ξ tp )(cid:105) (cid:13) (cid:13)(cid:13) (cid:105)
Revised: By the deﬁnition of L local p ( θ p ; ξ p )

Original: [Table]
Revised: [Equation]

Original: P

Original: P (cid:88) p =

Original: E (cid:20) (cid:13) (cid:13) (cid:13) ˜

Original: ∇L local p ( θ p ; ξ p ) − E (cid:104) ˜ ∇L local p ( θ p ; ξ p ) (cid:105) (cid:13)(cid:13) (cid:13) (cid:21) + E   (cid:13)(cid:13)(cid:13) (cid:13)(cid:13) P P (cid:88) p =
Revised: we have

Original: E (cid:104) ˜

Original: ∇L local p ( θ p , ξ tp ) (cid:105) (cid:13)(cid:13) (cid:13) (cid:13)(cid:13)  

Original: [Table]
Revised: [Equation]

Original: var P + E (cid:104)(cid:13)(cid:13)(cid:13)
Revised: (cid:105)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 

Original: P P (cid:88) p = E (cid:104) ˜ ∇L
Revised: 

Original: [Equation]
Revised: [Equation]

Original: (cid:105) (cid:13)(cid:13)(cid:13) (cid:105) (29) where ( a ) is due to fact that each ˜ ∇L local p ( θ p ; ξ p ) − E [ ˜ ∇L [Equation] p ( θ p 
Revised: (29) where ( a ) is due to fact that each ˜ ∇L local p ( θ p ; ξ p ) − E [ ˜ ∇L local p ( θ p ; ξ p )] is independent random vectors with zero mean and ( b ) is from Assumption

Original: [Equation]
Revised: [Equation]

Original: ξ p )] is independent random vectors with zero mean and ( b ) is from Assumption 1.

Original: B. P ROOF OF L EMMA
Revised: B. P ROOF OF L EMMA

Original: By the deﬁnition of local gradient ∇L local p ( θ )
Revised: By the deﬁnition of local gradient ∇L local p ( θ )

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Revised: [Equation]

Revised: P

Original: [Equation]
Revised: [Equation]

Original:  where ( a ) is due to (cid:107)
Revised: 30) where ( a ) is due to (cid:107)

Original: [Equation]
Revised: [Equation]

Original: ni =1 n (cid:107) x i (cid:107) and ( b ) is due to the deﬁnition of κ .
Revised: ni =1 n (cid:107) x i (cid:107) and ( b ) is due to the deﬁnition of κ .

Original: B. P ROOF OF L EMMA
Revised: B. P ROOF OF L EMMA

Original: When ( t mod K ) = 

Revised: When ( t mod K ) =

Original: we have ¯ θ t = θ tp .
Revised: we have ¯ θ t = θ tp .

Original: When ( t mod K ) (cid:54) = 0 and t ≥ 1 , let t 0 < t be the largest iteration index that t 0 mod K = 0 .
Revised: When ( t mod K ) (cid:54) = 0 and t ≥ 1 , let t 0 < t be the largest iteration index that t 0 mod K = 0 .

Original: For any τ ∈ { t 0 + 1 , . . .
Revised: For any τ ∈ { t 0 + 1 , . . .

Original: , t } , we have
Revised: , t } , we have

Original: [Equation]
Revised: [Equation]

Original: Summing over τ ∈ { t 0 + 1 , . .
Revised: Summing over τ ∈ { t 0 + 1 , . .

Original: . , t } , we have
Revised: . , t } , we have

Original: [Equation]
Revised: [Equation]

Original: 32) where ( a ) is due to ¯ θ t = θ tp when ( t mod K )
Revised: where ( a ) is due to ¯ θ t = θ tp when ( t mod K )

Original: = 0 .
Revised: = 0 .

Original: Similarly, we have
Revised: Similarly

Original: [Equation]
Revised: [Equation]

Original: Combining Eq. 32 and Eq. 33, we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation] [Table] [Equation] [Table]
Revised: Combining Eq.

Original: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: 32 and Eq.

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: we have

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: +

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table] (cid:17)(cid:13)(cid:13)(cid:13) (cid:105) (cid:124) (cid:123)(cid:122) (cid:125) ( B ) , (34) where ( a ) is due to addition and subtraction of ˜ ∇L local p ( θ τ p ; ξ τ p ) and ∇L local p ( θ τ p ) and using (cid:107) x 
Revised: 34) where ( a ) is due to addition and subtraction of ˜ ∇L local p ( θ τ p ; ξ τ p ) and ∇L local p ( θ τ p ) and using (cid:107) x + y (cid:107) 2 ≤ (cid:107)

Original: + y (cid:107) 2 ≤ (cid:107) x
Revised: x

Original: (cid:107) 

Revised: (cid:107)

Original: + (cid:107) y (cid:107) 2 .
Revised: + (cid:107) y (cid:107) 2 .

Original: We can upper bound ( A ) in Eq. 63 by
Revised: We can upper bound ( A ) in Eq. 63 by

Original: [Equation]
Revised: [Equation]

Original: where ( a ) is due to
Revised: where ( a ) is due to

Original: [Equation]
Revised: [Equation]

Original: We can further bound ( B ) in Eq. 63 by
Revised: We can further bound ( B ) in Eq. 63 by

Original: [Equation]
Revised: [Equation]

Original:  where ( a ) is due to Lemma 2 and the deﬁnition of κ 
Revised: ) where ( a ) is due to Lemma 2 and the deﬁnition of κ .

Original: [Equation]

Original: By plugging Eq.
Revised: By plugging Eq.

Original: 64 and Eq. 66 into Eq.
Revised: 64 and Eq. 66 into Eq.

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: By summing over t ∈ {
Revised: By summing over t ∈ {

Original: [Equation]
Revised: [Equation]

Original: . , T }
Revised: . , T }

Original: [Equation]
Revised: [Equation]

Original: we haveT −
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Collecting common terms and dividing both sides by (1 − 16 η 2 L 2 K 2 ) gives
Revised: Collecting common terms and dividing both sides by (1 − 16 η 2 L 2 K 2 ) gives

Original: [Equation]
Revised: [Equation]

Original: C P ROOF OF T HEOREM
Revised: C P ROOF OF T HEOREM

Original: In the following, we ﬁrst introduce the useful lemmas in Section C.1, then process our proof of theorem in Section C.2.
Revised: In the following, we ﬁrst introduce the useful lemmas in Section C.1, then process our proof of theorem in Section C.2.

Original: In particular, we show that this residual error (which is cause by ignoring the cut-edges) can eliminated by running server correction steps after the parameter averaging on the server.
Revised: In particular, we show that this residual error (which is cause by ignoring the cut-edges) can eliminated by running server correction steps after the parameter averaging on the server.

Original: C. U SEFUL LEMMA
Revised: C. U SEFUL LEMMA

Original: The following lemma provides an upper bound on the deviation of each local model to virtual averaged model, which is important to upper bound on the difference of each local gradient ∇L local p ( θ p ) compared to 1 P
Revised: The following lemma provides an upper bound on the deviation of each local model to virtual averaged model, which is important to upper bound on the difference of each local gradient ∇L local p ( θ p ) compared to 1 P

Original: [Equation]
Revised: [Equation]

Original: Lemma 

Revised: Lemma

Original: For all P machines with θ p , p ∈ { 1 , . . .
Revised: For all P machines with θ p , p ∈ { 1 , . . .

Original: , P } , if we deﬁne ¯ θ = 1 P (cid:80)
Revised: , P } , if we deﬁne ¯ θ = 1 P (cid:80)

Original: Pp =1 θ p and let κ >such that (cid:107)L local p
Revised: Pp =1 θ p and let κ >such that (cid:107)L local p

Original: [Equation]
Revised: [Equation]

Original: , we have
Revised: , we have

Original: [Equation]
Revised: [Equation]

Original: C. M AIN PROOF OF T HEOREM
Revised: C. M AIN PROOF OF T HEOREM

Original: Equipped with the above lemma and results from Appendix B.1 , we are now ready to present the proof of Theorem 2.
Revised: Equipped with the above lemma and results from Appendix B.1 , we are now ready to present the proof of Theorem 2.

Original: Let T local ( r ) and T server ( r ) as the iteration indicies that a local machine and server run on, which is deﬁned as
Revised: Let T local ( r ) and T server ( r ) as the iteration indicies that a local machine and server run on, which is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: 42) and let deﬁne T local =
Revised: 42) and let deﬁne T local =

Original: T local (1) ∪ . . .
Revised: T local (1) ∪ . . .

Original: ∪ T local ( R ) and T global =
Revised: ∪ T local ( R ) and T global =

Original: T global (1) ∪ . . .
Revised: T global (1) ∪ . . .

Original: ∪ T global
Revised: ∪ T global

Original: ( R ) .
Revised: ( R ) .

Original: By the smoothness assumption, we have
Revised: By the smoothness assumption, we have

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Let ﬁrst consider t ∈ T server ( r ) , with the following update ¯
Revised: Let ﬁrst consider t ∈ T server ( r ) , with the following update ¯

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: 45) where the equality is due to 1 P
Revised: 45) where the equality is due to 1 P

Original: [Equation]
Revised: [Equation]

Original: ¯ 
Revised: ¯ θ

Original: [Equation]
Revised: [Equation]

Original: ) since all neighbors are used for the server correction steps.
Revised: ) since all neighbors are used for the server correction steps.

Original: Besides, by taking the norm on the both side of Eq. 44, we have the following equality
Revised: Besides, by taking the norm on the both side of Eq. 44, we have the following equality

Original: [Equation]
Revised: [Equation]

Original: where inequality ( b ) hold due to Lemma 1.
Revised: where inequality ( b ) hold due to Lemma 1.

Original: Notice that the result in Lemma 1 holds for both the “local” and the “global” setting since 1 P
Revised: Notice that the result in Lemma 1 holds for both the “local” and the “global” setting since 1 P

Original: [Equation]
Revised: [Equation]

Original: Substituting Eq. 45, 46 into Eq.
Revised: Substituting Eq. 45, 46 into Eq.

Original: [Equation]
Revised: [Equation]

Original: we know that for t ∈ T server ( r )
Revised: we know that for t ∈ T server ( r )

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Dividing both sides by γ 2 and rearranging terms yields
Revised: Dividing both sides by γ 2 and rearranging terms yields

Original: [Equation]
Revised: [Equation]

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: let consider the local update steps where t ∈ T local ( r ) .
Revised: let consider the local update steps where t ∈ T local ( r ) .

Original: According to Eq.
Revised: According to Eq.

Original: in proof of
Revised: in proof of

Original: Theorem
Revised: Theorem

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Let T =
Revised: Let T =

Original: (cid:16)(cid:80)
Revised: (cid:16)(cid:80)

Original: Rr
Revised: Rr

Original: [Equation]
Revised: [Equation]

Original: + SR and summing over t ∈ { 1 , . .
Revised: + SR and summing over t ∈ { 1 , . .

Original: , T }
Revised: , T }

Original: [Equation]
Revised: [Equation]

Original: combining Eq.
Revised: combining Eq.

Original: 48 and Eq.
Revised: 48 and Eq.

Original: 49 we have
Revised: 49 we have

Original: [Equation]
Revised: [Equation]

Original: Rearrange the above equation, we have T
Revised: Rearrange the above equation, we have T

Original: [Equation]
Revised: [Equation]

Original: Let deﬁne
Revised: Let deﬁne

Original: [Equation]
Revised: [Equation]

Original: Our goal is to select the size of T local ( r ) and T global ( r ) such that the following inequality holds
Revised: Our goal is to select the size of T local ( r ) and T global ( r ) such that the following inequality holds

Original: [Equation]
Revised: [Equation]

Original: After rearranging it
Revised: After rearranging it

Original: [Equation]
Revised: [Equation]

Original: we will have
Revised: we will have

Original: [Equation]
Revised: [Equation]

Original: Suppose Eq.
Revised: Suppose Eq.

Original: 54 holds
Revised: 54 holds

Original: [Equation]
Revised: [Equation]

Original: we have T
Revised: we have T

Original: [Equation]
Revised: [Equation]

Original: where ( a ) is due to Lemma
Revised: where ( a ) is due to Lemma

Original: [Equation]
Revised: [Equation]

Original: By selecting Kρ r such that 1 − 16 η 2 L 2 K 2 ρ 2 r ≥ 12 , we have Kρ r ≤ √ 2 8 Lη and
Revised: By selecting Kρ r such that 1 − 16 η 2 L 2 K 2 ρ 2 r ≥ 12 , we have Kρ r ≤ √ 2 8 Lη and

Original: [Equation]
Revised: [Equation]

Original: Notice that the condition Kρ r ≤ √ 2 8 Lη implies
Revised: Notice that the condition Kρ r ≤ √ 2 8 Lη implies

Original: R
Revised: R

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: To this end, by selecting η = γ = √ P √ T ,
Revised: To this end, by selecting η = γ = √ P √ T ,

Original: [Equation]
Revised: [Equation]

Original: , and σ 2 = max { σ 2 var , σ 2 global } , we have
Revised: , and σ 2 = max { σ 2 var , σ 2 global } , we have

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: C. P ROOF OF L EMMA
Revised: C. P ROOF OF L EMMA

Original: Recall that when t ∈ T global ( r ) we have ¯ θ t
Revised: Recall that when t ∈ T global ( r ) we have ¯ θ t

Original: = θ tp .
Revised: = θ tp .

Original: For any t ∈ T local ( r ) , let t 0 + 1 denote as the ﬁrst indices of T local ( r ) and |T local ( r ) |
Revised: For any t ∈ T local ( r ) , let t 0 + 1 denote as the ﬁrst indices of T local ( r ) and |T local ( r ) |

Original: =
Revised: =

Original: Kρ r .
Revised: Kρ r .

Original: Then, for any τ ∈ { t 0 + 1 , . . .
Revised: Then, for any τ ∈ { t 0 + 1 , . . .

Original: , t } , we have
Revised: , t } , we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Summing over τ ∈ { t 0 + 1 , . .
Revised: Summing over τ ∈ { t 0 + 1 , . .

Original: . , t } , we have
Revised: . , t } , we have

Original: [Equation]
Revised: [Equation]

Original: where ( a ) is due to ¯ θ t
Revised: where ( a ) is due to ¯ θ t

Original: = θ tp when t ∈ T global ( r ) .
Revised: = θ tp when t ∈ T global ( r ) .

Original: Similarly, we have
Revised: Similarly, we have

Original: [Equation]
Revised: [Equation]

Original: By combining Eq.
Revised: By combining Eq.

Original: 61 and Eq.
Revised: 61 and Eq.

Original: 62, we have
Revised: 62, we have

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation] [Table] 
Revised: [Equation] [Table] [Equation]

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: (cid:17)(cid:13)(cid:13)(cid:13) (cid:105) (cid:124) (cid:123)(cid:122) (cid:125) ( B ) , (63) where ( a ) follows by adding and substracting ˜
Revised: (cid:17)(cid:13)(cid:13)(cid:13) (cid:105) (cid:124) (cid:123)(cid:122) (cid:125) ( B ) , (63) where ( a ) follows by adding and substracting ˜

Original: ∇L local p ( θ τ p ; ξ τ p ) , ∇L local p ( θ τ p ) and using (cid:107) x + y (cid:107) 2 ≤ (cid:107) x
Revised: ∇L local p ( θ τ p ; ξ τ p ) , ∇L local p ( θ τ p ) and using (cid:107) x + y (cid:107) 2 ≤ (cid:107) x

Original: (cid:107) 

Revised: (cid:107)

Original: + (cid:107) y (cid:107) 2 .
Revised: + (cid:107) y (cid:107) 2 .

Original: We can upper bound ( A ) in Eq. 63 by
Revised: We can upper bound ( A ) in Eq. 63 by

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where ( a ) is due to
Revised: where ( a ) is due to

Original: [Equation]
Revised: [Equation]

Original: We can further bound ( B ) in Eq. 63 by
Revised: We can further bound ( B ) in Eq. 63 by

Original: [Equation]
Revised: [Equation]

Original: where ( a ) is due to Lemma 2 and the deﬁnition of κ
Revised: where ( a ) is due to Lemma 2 and the deﬁnition of κ

Original: [Equation]
Revised: [Equation]

Original: By plugging Eq.
Revised: By plugging Eq.

Original: 64 and Eq. 66 into Eq.
Revised: 64 and Eq. 66 into Eq.

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Let us deﬁne T local =
Revised: Let us deﬁne T local =

Original: T local (1) ∪ . . .
Revised: T local (1) ∪ . . .

Original: ∪ T local ( R ) .
Revised: ∪ T local ( R ) .

Original: By summing over t ∈ T local , we have R
Revised: By summing over t ∈ T local , we have R

Original: [Equation]
Revised: [Equation]

Original: Rearranging the terms gives R
Revised: Rearranging the terms gives R

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: we conclude that
Revised: we conclude that

Original: [Equation]
Revised: [Equation]

Original: which completes the proof.
Revised: which completes the proof.