H YBRID
L OCAL
SGD
FOR
F
EDERATED L
EARNING
WITH H ETEROGENEOUS C OMMUNICATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Communication is a key bottleneck in federated learning where a large number of edge devices collaboratively learn a model under the orchestration of a central server without sharing their own training data.
While local SGD has been proposed to reduce the number of FL rounds and become the algorithm of choice for FL, its total communication cost is still prohibitive when each device needs to communicate with the remote server repeatedly for many times over bandwidthlimited networks.
In light of both device-to-device (D2D) and device-to-server (D2S) cooperation opportunities in modern communication networks, this paper proposes a new federated optimization algorithm dubbed hybrid local SGD (HLSGD) in FL settings where devices are grouped into a set of disjoint clusters with high D2D communication bandwidth.
HL-SGD subsumes previous proposed algorithms such as local SGD and gossip SGD and enables us to strike the best balance between model accuracy and runtime.
We analyze the convergence of
HL-SGD in the presence of heterogeneous data for general nonconvex settings.
We also perform extensive experiments and show that the use of hybrid model aggregation via D2D and D2S communications in HL-SGD can largely speed up the training time of federated learning.
I NTRODUCTION
Federated learning (FL) is a distributed machine learning paradigm in which multiple edge devices or clients cooperate to learn a machine learning model under the orchestration of a central server, and enables a wide range of applications such as autonomous driving, extended reality, and smart manufacturing (Kairouz et al., 2019).
Communication is a critical bottleneck in FL as the clients are typically connected to the central server over bandwidth-limited networks.
Standard optimization methods such as distributed SGD are often not suitable in FL and can cause high communication costs due to the frequent exchange of large-size model parameters or gradients.
To tackle this issue, local SGD, in which clients update their models by running multiple SGD iterations on their local datasets before communicating with the server, has emerged as the de facto optimization method in
FL and can largely reduce the number of communication rounds required to train a model (McMahan et al., 2017; Stich, 2019).
However, the communication beneﬁt of local SGD is highly sensitive to non-iid data distribution as observed in prior work (Rothchild et al., 2020; Karimireddy et al., 2020).
Intuitively, taking many local iterations of SGD on local dataset that is not representative of the overall data distribution will lead to local over-ﬁtting, which will hinder convergence.
In particular, it is shown in (Zhao et al., 2018) that the convergence of local SGD on non-iid data could slow down as much as proportionally to the number of local iteration steps taken.
Therefore, local SGD with a large aggregation period can converge very slow on non-iid data distribution, and this may nullify its communication beneﬁt (Rothchild et al., 2020).
Local SGD assumes a star network topology where each device connects to the central server for model aggregation.
In modern communication networks, rather than only communicating with the server over slow communication links, devices are increasingly connected to others over fast communication links.
For instance, in 5G-and-beyond mobile networks, mobile devices can directly communicate with their nearby devices via device-to-device links of high data rate (Asadi et al., 2014).
Also, edge devices within the same local-area network (LAN) domain can communicate with each other rapidly without traversing through slow wide-area network (WAN)
(Yuan et al., 2020).
This gives the potential to accelerate the FL convergence under non-iid data distribution by leveraging fast D2D cooperation so that the total training time can be reduced in FL over bandwidthlimited networks.
Motivated by the above observation, this paper proposes hybrid local SGD (HL-SGD), a new distributed learning algorithm for FL with heterogeneous communications, to speed up the learning process and reduce the training time.
HL-SGD extends local SGD with fast gossip-style D2D communication after local iterations to mitigate the local over-ﬁtting issue under non-iid data distribution and accelerate convergence.
A hybrid model aggregation scheme is designed in HL-SGD to integrate both fast device-to-device (D2D) and slow device-to-server (D2S) cooperations.
We analyze the convergence of HL-SGD in the presence of heterogeneous data for general nonconvex settings, and characterize the relationship between the optimality error bound and algorithm parameters.
Our algorithm and analysis are general enough and subsume previously proposed SGD variations such as distributed SGD, local SGD and gossip SGD.
Speciﬁcally, we consider the FL setting in which all devices are partitioned into disjoint clusters, each of which includes a group of connected devices capable of communicating with each other using fast D2D links.
The clustering can be a natural result of devices belonging to different LAN domains so that those devices connected to the same LAN domain are considered as one cluster.
In another example, clustering is based on the geographic locations of mobile devices so that devices in a cluster are connected to each other through D2D communication links.
In summary, the paper makes the following main contributions:
• We propose a novel distributed learning algorithm for FL called HL-SGD to address the communication challenge of FL over bandwidth-limited networks by leveraging the availability of fast D2D links to accelerate convergence under non-iid data distribution and reduce training time.
• We provide the convergence analysis of HL-SGD under general assumptions about the loss function, data distribution, and network topology, generalizing previous results on distributed SGD, local SGD, and gossip SGD.
• We conduct extensive empirical experiments on two common benchmarks under realistic network settings to validate the established theoretical results of HL-SGD.
Our experimental results show that HL-SGD can largely accelerate the learning process and speed up the runtime.
B ACKGROUND AND R ELATED W ORK
Large-scale machine learning based on distributed SGD has been well studied in the past decade, but often suffers from large network delays and bandwidth limits (Bottou et al., 2018).
Considering that communication is a major bottleneck in federated settings, local SGD has been proposed recently to reduce the communication frequency by running SGD independently in parallel on different devices and averaging the sequences only once in a while (Stich, 2019; Lin et al., 2019; Haddadpour et al., 2019; Yu et al., 2018; Wang et al., 2021).
However, they all assume the client-server architecture and do not leverage the fast D2D communication capability in modern communication networks.
Some studies (Liu et al., 2020; Abad et al., 2020; Castiglia et al., 2020) develop hierarchical FL algorithms that ﬁrst aggregate client models at local edge servers before aggregating them at the cloud server or with neighboring edge servers, but they still rely on D2S communication links only and suffer from the scalability and fault-tolerance issues of centralized setting.
On the other hand, while existing works on decentralized or gossip SGD consider D2D communications (Tsitsiklis, 1984; Boyd et al., 2006), they assume a connected cluster with homogeneous communication links and will converge very slow on the large and sparse network topology that is typically found in FL settings.
Unlike previous works, HL-SGD leverages both D2S and D2D communications in the system.
Some recent studies aim to encapsulate variants of SGD under a uniﬁed framework.
Speciﬁcally, a cooperative SGD framework is introduced in (Wang & Joshi, 2018) that includes communication reduction through local SGD steps and decentralized mixing between clients under iid data distribution.
A general framework for topology-changing gossip SGD under both iid and non-iid data distributions is proposed in (Koloskova et al., 2020).
Note that all of the above works assume undirected network topology for communications in every iteration.
In comparison, our proposed HL-SGD is different: the D2S communication is asymmetric due to the use of device sampling and model broadcasting in each global aggregation round and cannot be modeled in an undirected graph.
Therefore, the convergence analysis of HL-SGD does not ﬁt into the prior frameworks and is much more challenging.
Moreover, our major focus is on the runtime of the algorithm rather than its convergence speed in iterations.
S YSTEM M ODEL
In this section, we introduce the FL system model, problem formulation, and assumptions we made.
Notation.
All vectors in this paper are column vectors by default.
For convenience, we use 1 to denote the all-ones vector of appropriate dimension, 0 to denote the all-zeros vector of appropriate dimension, and [ n ] to denote the set of integers { 1 , 2 , . . . , n } with any positive integer n .
Let (cid:107)·(cid:107) denote the (cid:96) 2 vector norm and Frobenius matrix norm and (cid:107)·(cid:107) 2 denote the spectral norm of a matrix.
We consider a FL system consisting of a central server and K disjoint clusters of edge devices.
Devices in each cluster k ∈ [ K ] can communicate with others across an undirected and connected graph G k =
( V , E k ) , where V k denotes the set of edge devices in the cluster, and edge ( i,
j ) ∈
E k denotes that the pair of devices i, j ∈ V k can communicate directly using D2D as determined by the communication range of D2D links.
Besides, each device can directly communicate with the central server using D2S links.
Denote the set of all devices in the system as V :
= (cid:83)
k ∈
[ K ] V k , the number of devices in each cluster k ∈
[ K ] as n :=
|V k | , and the total number of devices in the system as
N := (cid:80) k ∈ [ K ] n 1 .
The FL goal of the system is to solve an optimization problem of the form:
[Equation]
where f i ( x ) :
=
E z ∼D i [ (cid:96) i ( x ; z )] is the local objective function of device i , ¯ f k ( x ) :=
(1 /n ) (cid:80) i ∈V k f i ( x ) is the local objective function of cluster k , and D i is the data distribution of device i .
Here (cid:96) i is the (non-convex) loss function deﬁned by the learning model and z represents a data sample from data distribution D i .
When applying local SGD to (1) in FL with heterogeneous communications, the communications between the server and devices in FL are all through D2S links that are bandwidth-limited, particularly for the uplink transmissions.
Therefore, the incurred communication delay is high.
Due to the existing of high-bandwidth D2D links that are much more efﬁcient than low-bandwidth D2S links, it would be highly beneﬁcial if we can leverage D2D links to reduce the usage of D2S links such that the total training time can be reduced.
This motivates us to design a new learning algorithm for
FL with heterogeneous communications.
H YBRID L OCAL SGD
In this section, we present our HL-SGD algorithm suitable for the FL setting with heterogeneous communications.
Algorithm 1 provides pseudo-code for our algorithm.
At the beginning of r -th global communication round, the server broadcasts the current global model x r to all devices in the system via cellular links (Line 4).
Note that in typical FL systems, the downlink communication is much more efﬁcient than uplink communication due to the larger bandwidth allocation and higher data rate.
Therefore, devices only consume a smaller amount of energy when receiving data from the server compared with transmitting data to the server.
After that, devices in each cluster initialize their local models to be the received global model and run τ iterations of gossip-based SGD via
D2D links to update their local models in parallel (lines 5– 9).
Let x r,si denote the local model of device i at the r -th local iteration of s -th round.
Here
Compute x
,s + 12 i =
x r,si − ηg i x r si )
8: x s +
i
=
j ∈N ki ( W k ) i,j x r,s + 12 j do 13:
S
←
15: end
18: return each gossip-based SGD iteration consists of two steps: (i) SGD update, performed locally on each device (lines 6–7), followed by a (ii) gossip averaging, where devices average their models with their neighbors (line 8).
In the gossip averaging protocol, N ki denotes the neighbors of device i , including itself, on the D2D communication graph G k of cluster k , and W k ∈ [0 , 1] n × n denotes the mixing matrix of cluster k with each element ( W k ) i,j being the weight assigned by device i to device j .
Note that ( W k ) i,j > 0 only if devices i and j are directly connected via D2D links.
Next, a set S rk of m devices are sampled uniformly at random (u.a.r.) with probability p without replacement from each cluster k ∈
[ K ] by the server (lines 13–14), and their ﬁnal updated local models { x r,τi , ∀ i ∈ S rk } are sent to the server via D2S links.
After that, the server updates the global model x r +1 by averaging the received local models from all sampled devices (line 16).
Note that only m devices per cluster will upload their models to the server in each round to save the usage of expensive D2S uplink transmissions.
The intuition is that after multiple iterations of gossip-based
SGD, devices have already reached approximate consensus within each cluster, and the sampled average can well represent the true average.
By trading D2D local aggregation for D2S global aggregation, the total communication cost can be reduced.
We will empirically validate such beneﬁts later in the experiments.
It is worth noting that HL-SGD inherits the privacy beneﬁts of classic FL schemes by keeping the raw data on device and sharing only model parameters.
Moreover, HL-SGD is compatible with existing privacy-preserving techniques in FL such as secure aggregation (Bonawitz et al., 2017), differential privacy (McMahan et al., 2018), and shufﬂing (Girgis et al., 2021) since only the sum rather than individual values is needed for the local and global model aggregation steps.
4. R UNTIME ANALYSIS OF HL-SGD
We now present a runtime analysis of HL-SGD.
Here we ignore the communication time of downloading models from the server by each device since the download bandwidth is often much larger than upload bandwidth for the D2S communication in practice (Kairouz et al., 2019).
In each round of HL-SGD, we denote the average time taken by a device to compute a local update, perform one round of D2D communication and one round of D2S communication as c cp , c d2d and c d2s , respectively.
Assume the uplink bandwidth between the server and devices is ﬁxed and evenly shared among the sampled devices in each round, then c d2s is linearly proportional to the sampling ratio p .
Similarly, c cp depends on the D2D network topology G k and typically increases with the maximum node degree ∆( G k ) .
The total runtime of HL-SGD after R communication rounds is
R × [
[Equation]
The speciﬁc values of c cp , c d2d and c d2s depend on the system conﬁgurations and applications.
In comparison, the total runtime of local SGD after R communication rounds is R × [ τ × c cp + c d2s ] .
P REVIOUS ALGORITHMS AS SPECIAL CASES
When devices do not communicate with each other, i.e., W k
= I, ∀ k ∈
[ K ] , and sampling ratio p = 1 , HL-SGD reduces to distributed SGD (when τ = 1 ) or local SGD (
when τ > 1 ) where each device only directly communicates with the server with D2S links.
Also, when τ → ∞ , HLSGD reduces to gossip SGD where devices only cooperate with their neighboring devices through a gossip-based communication protocol with D2D links to update their models without relying on the server.
Therefore, HL-SGD subsumes existing algorithms and enables us to strike the best balance between runtime and model accuracy by tuning τ , W k , and p .
However, due to the generality of
HL-SGD, there exist signiﬁcantly new challenges in its convergence analysis, which constitutes one of the main contributions of this paper as elaborated in the following section.
C ONVERGENCE A NALYSIS OF HL-SGD
In this section, we analyze the convergence of HL-SGD with respect to the gradient norm of the objective function f ( · ) , speciﬁcally highlighting the effects of τ and p .
Before stating our results, we make the following assumptions:
Assumption 1 (Smoothness) .
Each local objective function f
i : R d → R is L -smooth for all i ∈ V , i.e., for all x, y ∈ R d , (cid:107)∇ f i ( x ) − ∇ f i ( y ) (cid:107) ≤ L (cid:107) x − y (cid:107) , ∀ i ∈ V .
Assumption 2 (Unbiased Gradient and Bounded Variance) .
The local mini-batch stochastic gradient in Algorithm 1 is unbiased, i.e., E ξ i
[ g i ( x )]
= ∇ f i ( x ) , and has bounded variance, i.e.,
E ξ i (cid:107) g i ( x ) − ∇ f i ( x ) (cid:107) 2 ≤ σ 2 , ∀ x ∈ R d , i ∈ V , where the expectation is over all the local minibatches.
Assumption 3 (Mixing Matrix) .
For any cluster k ∈ [ K ] , the D2D network is strongly connected and the mixing matrix W k ∈ [0 , 1] n × n satisﬁes W k 1 = 1 , 1 (cid:62) W k = 1 (cid:62) , null ( I − W k )
=
span ( 1 ) .
We also assume || W k − (1 /n )
(cid:62) ||
2 ≤ ρ k for some ρ k ∈ [0 , 1) .
Assumption 4 (Bounded Intra-Cluster Dissimilarity) .
There exists a constant (cid:15) k ≥ 0 such that (1 /n ) (cid:80) i ∈V k
(cid:107)∇ f i ( x ) − ∇ ¯ f k ( x ) (cid:107) 2 ≤ (cid:15) 2 k for any x ∈ R d and k ∈ [ K ] .
If local functions are identical to each other within a cluster, then we have (cid:15) k = 0 .
Assumption 5 (Bounded Inter-Cluster Dissimilarity) .
There exist constants α ≥ 1 , (cid:15) ≥ 0 such that (1 /K )
(cid:80)
k ∈
[ K ] (cid:107)∇ ¯ f k ( x ) (cid:107) 2 ≤ α 2 (cid:107)∇ f ( x ) (cid:107) 2 + (cid:15) 2 g for any x ∈ R d .
If local functions are identical to each other across all clusters, then we have α = 1 , (cid:15) g = 0 .
Assumptions 1–3 are standard in the analysis of SGD and decentralized optimization (Bottou et al., 2018; Koloskova et al., 2019).
Assumptions 4–5 are commonly used in the federated optimization literature to capture the dissimilarities of local objectives (Koloskova et al., 2020; Wang et al., 2020).
5. M AIN R ESULTS
We now provide the main theoretical results of the paper in Theorem 1 and Theorem 2.
The detailed proofs are provided in the appendices.
Deﬁne the following constants:
[Equation]
and let
[Equation] [Equation]
Theorem 1 (Full device participation) .
Let Assumptions 1–5 hold, and let L , σ , ¯ (cid:15) L , (cid:15) g , D τ,ρ , ρ max , r 0 , r 1 , and r 2 be as deﬁned therein.
If the learning rate η satisﬁes
[Equation]
then for any R > 0 , the iterates of Algorithm 1 with full device participation for HL-SGD satisfy
[Equation]
where ¯ x r,s = 1 N (cid:80) Ni =
1 x r,si .
In the following, we analyze the iteration complexity of HL-SGD and compare it with those of some classic and state-of-the-art algorithms relevant to our setting in Table 1.
First, we consider two extreme cases of HL-SGD where ρ max = 0 and ρ k = 1 , ∀ k ∈
[ K ] , and show that our analysis recovers the best known rate of local SGD.
Fully Connected D2D networks.
In this case, ρ max = 0 , and each cluster can be viewed as a single device, and thus HL-SGD reduces to local SGD with K devices.
Substuting ρ max = 0 into (6), the iteration complexity of HL-SGD reduces to O ( σ/ √ NτR + (cid:0) τ 2 (cid:15) 2 g + τ · ( σ 2 /n )
(cid:1) 1 / 3 / ( τR ) 2 / 3 + 1 /R ) .
This coincides with the complexity of local SGD provided in Table 1 with device number K and stochastic gradient variance σ 2 /n thanks to the fully intra-cluster averaging.
Disconnected D2D networks.
In this case, HL-SGD reduces to local SGD with N devices.
Substituting ρ max = 1 into (6), the iteration complexity of HL-SGD becomes O ( σ/ √ NτR + (cid:0) τ 2 ( (cid:15) 2 g + ¯ (cid:15) 2 L ) + τσ 2 (cid:1) 1 / 3 / ( τR ) 2 / 3 + 1 /R ) . This coincides with the complexity of local SGD with N devices, stochastic gradient variance σ 2 , and gradient heterogeneity of order (cid:15) 2 g + ¯ (cid:15) 2 L . [Table]
O
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
O
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] Gossip PGA (Chen et al., 2021a)
O
[Equation]
[Equation]
[Table]
[Equation]
HL-SGD (this work) O (cid:18) [Equation] Next, we compare the complexities of HL-SGD, local SGD, gossip SGD and gossip PGA.
Comparison to Local SGD.
Comparing (6) and the complexity of local SGD, we can see the intracluster D2D communication provably improves the iteration complexity by reducing the transient iterations.
This is reﬂected in the smaller coefﬁcient associated with the O (( τR ) − 2 / 3 ) term.
In particular, improving D2D communication connectivity will lead to a smaller ρ max and consequently, mitigate the impact of both local data heterogeneity and stochastic noise on the convergence rate.
Comparison to Gossip SGD.
Under the condition that ρ = ρ max , i.e., the connectivity of D2D network in gossip SGD is the same as that of HL-SGD, Table 1 shows HL-SGD outperforms gossip
SGD when τ/n ≤ ρ 2 / (1 − ρ ) .
In other words, HL-SGD is beneﬁcial for weakly connected networks, which is the case in FL settings where a large number of devices are often loosely connected or disconnected into several disjoint clusters via D2D communications only.
Comparison to Gossip PGA.
Gossip PGA improves local SGD by integrating gossiping among all devices in one round using a connected network. Compared to gossip SGD, gossip PGA has one extra full averaging step with period τ . The complexity of gossip PGA improves both by reducing the transient iterations. HL-SGD (full participation) differs from gossip PGA in the sense that gossiping is performed within multiple clusters instead of a single one.
The beneﬁt comes from the fact that for many commonly used D2D network topologies, the spectral gap 1 − ρ decreases as the network size decreases, see Table 2.
Therefore, when employing the same D2D network topology,
HL-SGD enjoys a smaller connectivity number ρ max than ρ .
Considering the scenario where τ and n are ﬁxed while the cluster number K grows, the total device number N = nK grows and hence ρ → 1 for gossip PGA.
In the case when τ = D τ,ρ (cid:48)
≈ C τ,ρ , the fastest decaying O (1 /τR )
terms are comparable for both algorithms.
However, the O (( τR ) − 2 / 3 ) term of gossip GPA can be larger than that of HL-SGD since ρ increases with N .
This observation shows for large-scale networks, it is advantageous to use HL-SGD with multiple connected clusters instead of gossip GPA with a single cluster under the D2D network topology.
Our next result shows the iteration complexity of HL-SGD with partial device participation.
We assume the devices participate in synchronizing their models at the end of each FL round following the sampling rule given by Assumption 6.
Assumption 6 (Sampling strategy) .
Each S r k contains a subset of m indices uniformly sampled from { 1 , . .
. , n } without replacement.
Furthermore, S rk is independent of S r (cid:48) k (cid:48) for all ( k, r ) (cid:54) =
( k (cid:48) , r (cid:48) ) .
Theorem 2 (Partial device participation) .
Let Assumptions 1–6 hold, and let L , σ , ¯ (cid:15) L , (cid:15) g , D τ,ρ , ρ max , r 0 , r 1 , and r 2 be as deﬁned therein.
If the network connectivity satisﬁes
[Equation]
then for suitably chosen learning rate η , the iterates of Algorithm 1 with partial device participation for HL-SGD satisfy
[Equation]
where ¯ x r,s = 1 N (cid:80) Ni =
1 x r,si ,
E
[Equation]
and
[Equation]
Compared to Theorem 1, Theorem 2 shows partial device participation deteriorates the rate by O ( E ( (cid:15) g ,
¯ (cid:15) L , σ, ρ max ) / √ NτR ) .
From the expression of E , we observe that as ρ max → 0 ,
E ( (cid:15) g , ¯ (cid:15) L , σ, ρ max ) vanishes, which indicates that the loss caused by device sampling can be compensated by increasing network connectivity uniformly for all clusters.
The next corollary ﬁnds the critial ρ max so that E
=
O (1) , and the order of convergence rate of partial device participation matches that of the full participation case.
Corollary 1.
Under the same assumptions as Theorem 2, if the network connectivity satisﬁes
[Equation]
then
[Equation]
Corollary 1 reveals the tradeoff between sampling intensity and network connectivity.
More connected D2D networks result in smaller ρ max , and thus (11) can be satisﬁed by a smaller m .
This means we can sample fewer devices at the end of each round and reduce the D2S communication delay when the D2D network is more connected.
E XPERIMENTAL E VALUATION 6. E XPERIMENTAL S ETTINGS
We use two common datasets in FL literature (McMahan et al., 2017; Reddi et al., 2020; Wang et al., 2020):
Federated Extended MNIST (Caldas et al., 2018) (FEMNIST) and CIFAR-10 (Krizhevsky et al., 2009).
The 62-class FEMNIST is built by partitioning the data in Extended MNIST (Cohen et al., 2017) based on the writer of the digit/character and has a naturally-arising device partitioning.
CIFAR-10 is partitioned across all devices using a Dirichlet distribution Dir (0 . 1) as done in (Hsu et al., 2019; Yurochkin et al., 2019;
Reddi et al., 2020; Wang et al., 2020).
We evaluate our algorithms by training CNNs on both datasets, and the CNN models for FEMNIST and CIFARwere taken from (Caldas et al., 2018) and (McMahan et al., 2017) with around 6 . 5 and 1 million parameters, respectively.
For each dataset, the original testing set (without partitioning) is used to evaluate the generalization performances of the trained global model.
We consider a FL system consisting of a central server and 32 devices.
The devices are evenly divided into four clusters, and each cluster has a ring topology by default, which provides a conservative estimation for the cluster connectivity and convergence speed.
In our experiments, the mixing matrix of each cluster W k is set according to the Metropolis-Hastings weights (Nedi´c et al., 2018).
According to the real-world measurements in (Yuan et al., 2020; Yang et al., 2021), we set the average time for a device to perform a local update, a round of D2D communication under ring topology, and a round of D2S communication with one device sampled per cluster to be c cp = 0 .
01 h, c d2d (∆ = 2) = 0 .
005 h and c d2s ( p = 1 / 8) = 0 .
05 h, respectively, in the runtime model (2).
For arbitrary device sampling ratio and D2D network topology, we consider a linear-scaling rule (Wang et al., 2019) and let c d2d (∆) =
(∆ / 2) × 0 .
005 h and c d2s ( p ) = 8 p × 0 .
h.
We compare HL-SGD with local SGD in the experiments.
For local SGD, devices will only communicate with the central server periodically.
In all experiments, we let the local iteration period τ to be the same for both local SGD and HL-SGD to have a fair comparison.
On the FEMNIST dataset, we ﬁx the batch size as 30 and tune the learning rate η from { 0 . 005 , 0 . 01 , 0 . 02 , 0 .
08 } for each algorithm separately.
On the CIFAR-10 dataset, we ﬁx the batch size as 50 and tune η from { 0 . 01 , 0 . 02 , 0 .
1 } for each algorithm separately.
We run each experiment with 3 random seeds and report the average.
All experiments in this paper are conducted on a Linux server with NVIDIA RTX 8000 GPUs.
The algorithms are implemented by PyTorch.
More details are provided in Appendix F.
6. E XPERIMENTAL R ESULTS
[Figure]
We ﬁrst compare the convergence speed and runtime of HL-SGD and local SGD while ﬁxing τ =and p = 1 .
We measure the test accuracy of the global model on the server in every FL round.
Figure 1 shows the convergence process.
From the ﬁgure, we can observe that HL-SGD can largely accelerate the model convergence while improving model accuracy in FL.
On FEMNIST, the best accuracy of HL-SGD achieved over 100 rounds is 4 .
78% higher than that of local SGD (i.e., 83 .
76% vs. 79 .
94% ), and its runtime necessary to achieve a target test-set accuracy of 75% is only 17 .
64% of that of the baseline (i.e., 5 .
67 × speedup).
On CIFAR-10, the best accuracy of HL-SGD achieved over 100 rounds is 9 .
32% higher than that of local SGD (i.e., 70 .
13% vs. 64 . 15% ), and its runtime necessary to achieve a target test-set accuracy of 60% is 18 .
36% less than that of local SGD (i.e., 1 .
× speedup).
[Figure]
Next, we investigate the effect of D2D network speed on the runtime beneﬁt of HL-SGD.
We consider the same FL setting as before and choose c d2d from { 0 . 003 , 0 .
02 } h. As illustrated in
Figure 2, the training speed of HL-SGD can be further improved by using faster D2D links.
For instance, it takes HL-SGD 4.2 hours to reach 75% test accuracy on FEMNIST with c d2d = 0 .
003 h, which is 1.5 × faster than HL-SGD with c d2d = 0 .
01 h.
On CIFAR-10, to reach a target level of 60% test accuracy, HL-SGD with c d2d = 0 .
003 h needs 21 hours, which is 1.14 × faster than HL-SGD with c d2d = 0 .
Note that the runtime cannot proportionally decrease with the increase of D2D network speed because local update also takes substantial computation time.
Also, when c d2d is relatively large compared with c d2s , the convergence acceleration beneﬁt of gossip averaging could be offset by the increased communication time in each round of HL-SGD, leading to an inferior total runtime as shown in Figure 2b.
[Figure]
Finally, we investigate how the sampling ratio p affects the performance of HL-SGD.
We select p from { 0 . 125 , 0 . 25 , 0 . 5 , 1 } , corresponding to sampling { 1 , 2 , 4 , 8 } devices from each cluster to upload models to the server.
Figure 3 depicts the best value of test-set accuracy achieved over all prior rounds.
As can be observed from the ﬁgures, sampling one device per cluster only results in slightly lower model accuracy, e.g., 1 . 92% drop compared to full participation on CIFAR-10.
This matches the theoretical result in Corollary 1 that device sampling does not affect the order of convergence rate under certain conditions.
However, decreasing p can lead to faster training speed due to its shorter D2S communication delay as observed in Figure 3d.
In practice, the optimal value of p needs to be tuned to strike a good balance between model accuracy and runtime.
C ONCLUSION
In this paper, we have proposed a new optimization algorithm called HL-SGD for FL with heterogeneous communications.
Our algorithm leverages the D2D communication capabilities among edge device to accelerate the model convergence while improving model accuracy in FL.
We have provided the theoretical convergence analysis of HL-SGD and conducted experiments to demonstrate the beneﬁts of HL-SGD.
In the future, we plan to extend HL-SGD to handle straggler issues under device heterogeneity and provide rigorous privacy protection for HL-SGD.
A P RELIMINARIES
Intra-cluster dynamics.
To facilitate the analysis, we introduce matrices X k ∈ R n × d and G k ∈ R n × d constructed by stacking respectively
x i and g i for i
∈
V k row-wise.
Similarly, we deﬁne the pseudo-gradient ∇ F k ( X k ) ∈ R n × d associated to cluster k by stacking ∇ f i ( x i ) for i ∈ V k row-wise.
In addition, deﬁne the following intra-cluster averages for each cluster k :
[Equation]
(cid:44)
[Equation]
and
[Equation]
The update within each cluster then can be written compactly in matrix form as
[Equation]
Since each W k is bi-stochastic, we obtain the following update of the intra-cluster average
[Equation]
We proceed to derive the update of the intra-cluster consensus error.
Deﬁne the averaging matrix
[Equation]
with 1 = [1 , . . .
[Equation]
[Equation]
[Equation]
Multiplying both sides of (14) from the left by ( I − J ) leads to the following update of the consensus error:
[Equation]
Global average dynamics.
Deﬁne the global average among all x i ’s as
[Equation]
Then accordingly to (15) we have the following update of ¯ x for all s = 0 , . .
, τ − 1 :
[Equation]
Filtration.
Let G =
[ G 1 ; . . .
; G K ] ∈ R N × d be the matrix constructed by stacking all the stochastic gradients.
We introduce the following ﬁltration
[Equation]
Therefore we have x r, 0 i = x r ∈ F r for r ≥ 1 , and x r,si ∈ F r,s − 1 for 1 ≤ s ≤ τ .
For simplicity the conditional expectation E ( · |F r,s ) is denoted as E r,s , and we deﬁne the noise in the stochastic gradient as
[Equation]
Since at the end of round r all nodes are picked with equal probability, the sampling procedure preserves average in expectation:
[Equation]
where the last equality holds since P (cid:0)
i ∈
S rk | i ∈ V k (cid:1)
=
mn .
B C ONVERGENCE A NALYSIS
To prove the convergence we ﬁrst establish in Sec.
B.1 that the objective value E f ( x r ) is descending at each round r , up to some consensus error terms.
Subsequently, bounds on the error terms are provided in Sec.
B.2-B.4.
Based on these results, the proof of convergence of Algorithm 1 with full and partial device participation are given in Sec.
B.5 and B.6, respectively.
The proofs of the main propositions are given in Sec.
C and that of the supporting lemmas are deferred to Sec.
D.
B. O BJECTIVE D ESCENT
Lemma 1.
Let { x r,si } be the sequence generated by Algorithm 1 under Assumptions 1-6.
If η > 0 , then the following inequality holds for all r ∈ N + :
[Equation]
Proof.
The proof is a standard application of the descent lemma and the sampling rule applied at iteration τ to obtain x r +1 .
See Appendix D.1.
Lemma 1 shows the objective value f ( x r ) is descending in expectation up to the following error terms:
[Equation]
In the sequel
[Equation]
we will show these quantities can be bounded by the optimality gap measured in terms of the gradient norms (cid:107)∇ f
[Equation]
(cid:107) 2 , and (cid:107) (1 /N
[Equation]
(cid:80)
[Equation]
x
[Equation]
) (cid:107)
[Equation]
B. B OUNDING T 1 , T 2 AND T
[Equation]
Deﬁne
[Equation]
Therefore it holds 0 ≤ ρ max ≤ 1 by Assumption 3.
Since each f i is L -smooth by Assumption 1, we have ¯ f k and f are also L -smooth.
Using this fact and the convexity of (cid:107) ·
[Equation]
2 we can bound T 1 , T 2 and T 3 as
[Equation]
Clearly, in order to bound T 1 , 2 , 3 we ﬁrst need to bound the inter-cluster consensus error (cid:107) ¯ x r,s − ¯ x r,sk (cid:107) and the intra-cluster consensus error (cid:107) ¯ x r,sk − x r,si (cid:107) .
Lemma 2 (Inter-Cluster Consensus Error Bound) .
Let { x r,si } be the sequence generated by Algorithm 1 under Assumptions 1, 2, 3, and 5.
If the learning rate η > 0 satisﬁes
[Equation]
then for all s = 0 , . .
. , τ − 1 it holds
K
[Equation]
where
[Equation]
Proof.
See Appendix D.2.
Lemma 3 (Intra-Cluster Consensus Error Bound) .
Let { x r,si } be the sequence generated by Algorithm 1 under Assumptions 1-5.
If η > 0 , then for all s = 0 , . .
. , τ − 1 it holds
[Equation]
where ρ max is deﬁned in (25) and
[Equation]
with ζ k > 0 being a free parameter to be chosen properly for all k = 1 , . . .
, K .
Proof.
See Appendix D.3.
Combining Lemma 2 and 3 we can obtain the following bound on the sum of intra- and interconsensus errors using gradient (cid:107)∇ f (¯ x r,s ) (cid:107) 2 .
Proposition 1.
Let { x r,si } be the sequence generated by Algorithm 1 under Assumptions 1-5.
If the learning rate η > 0 satisﬁes
[Equation]
then for all s = 0 , . .
. , τ − 1 it holds
[Equation]
where
D
[Equation]
and C 1 > 0 is some universal constant.
Proof.
See Appendix C.1.
Notice that according to (26) the gradient difference terms in Lemma 1 can be bounded as
[Equation] [Equation]
for all s = 1 , . . .
, τ .
Therefore Proposition 1 immediately leads to the following result.
Corollary 2.
Under the same setting as Proposition
[Equation]
it holds
[Equation]
We conclude this section by providing a separate bound on the consensus error
N
[Equation]
− 1 k, ⊥ (cid:107) 2 that will be useful in bounding T 5 .
Proposition 2.
Under the same setting as Proposition 1, if ρ max ≤ 1 − 1 τ , then we have
[Equation]
for some universal constant C 2 > 0 .
Proof.
See Appendix C.2.
Proposition 2 shows that the average intra-cluster consensus error 1 N (cid:80) Kk =1 (cid:107) X r,τ − 1 k, ⊥ (cid:107) 2 decreases as the network connectivity improves, and vanishes if ρ max goes to zero.
B. B OUNDING T
Proposition 3.
Under the same setting as Lemma 1, we have
[Equation]
for s = 0 , . . .
, τ − 1 and r ∈ N + .
Proof.
Recall the algorithmic update at iteration s for all s = 0 , . .
, τ − 1 :
[Equation]
Therefore
[Equation]
it holds under Assumption 2 that
[Equation]
B. B OUNDING T
We provide the bound on T 5 separately for the full device participation and partial participation cases.
Full participation.
When the sampling probability p = 1 , we have
[Equation]
In this case, it follows from Proposition 3 that
[Equation]
Partial participation.
We proceed to bound T 5 for
[Equation] [Equation]
Deﬁne p = m/n .
Recall the algorithmic update at iteration τ − 1 :
[Equation]
43) and
[Equation] [Equation]
Therefore, with ( W k ) i,j being the ij -th element of matrix W k we have under Assumption 2:
[Equation]
1 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   = E (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
Np
K (cid:88) k
= (cid:88) i ∈
S rk (cid:0) (cid:88)
j ∈V k ( W k ) i,j ( x r,τ − 1 j − ηg r,τ − 1 j ) (cid:1) − ¯ x r,τ − 1 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) =
E (cid:13)(cid:13)(cid:13) (cid:13) (cid:13)(cid:13)
Np
K (cid:88) k
= (cid:88) i ∈V k
I ( i ∈ S rk ) (cid:0) (cid:88)
j ∈V k ( W k ) i,j ( x r,τ − 1 j − η ∇ f j ( x r,τ − 1 j ) (cid:1) − ¯ x r,τ − 1 (cid:13)(cid:13)(cid:13) (cid:13) (cid:13)(cid:13) (cid:124) (cid:123)(cid:122) (cid:125)
A 2 ,+ η 2 E (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 1 Np
K (cid:88) k
= (cid:88) i ∈V k
I ( i ∈ S rk ) (cid:0) (cid:88)
j ∈V k ( W k ) i,j ( ∇ f j ( x r,τ − 1 j ) − g r,τ − 1 j ) (cid:1) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
Proposition 4.
Let { x r,si } be the sequence generated by Algorithm 1 under Assumptions 1-6.
If the learning rate η > 0 satisﬁes
[Equation]
then we have the following bounds on A 2 , 1 :
[Equation]
where
[Equation]
Proof.
See Appendix C.3.
Proposition 5.
Under the same setting as Proposition 4, A 2 , 2 can be bounded as
[Equation]
Proof.
See Appendix C.
B. P ROOF OF T HEOREM 1 ( FULL PARTICIPATION )
We ﬁrst prove the descent of the objective value under suitable choice of η .
Proposition 6.
If the learning rate satisﬁes
[Equation]
then we have
[Equation]
where
[Equation]
C 1 > 0 is some universal constant.
Proof.
See Appendix C.5.
To attain the expression of the convergence rate
[Equation]
we sum (51) over r = 0 , . . .
, R :
[Equation]
The ﬁrst two terms of (53) corresponds to the impact of stochastic noise and is of the same order as the centralized SGD algorithm.
The last term is of order η 2 and corresponds to the deterioration of convergence rate due to the fact that we are not computing the average gradients of all devices at each iteration.
Denote
[Equation]
The rest of the proof follows the same argument as (Chen et al., 2021b, Appendix B.5) and thus we omit the details.
B. P ROOF OF T HEOREM 2 AND C OROLLARY 1 ( PARTIAL PARTICIPATION )
Proposition 7.
Let { x r,si } be the sequence generated by Algorithm 1 under Assumption 1-5.
If the learning rate η and the network connectivity satisﬁes
[Equation]
55) then
[Equation]
56) with
[Equation]
C 1 , C 3 > 0 are some universal constants
[Equation]
and
[Equation]
with G p = n − m
[Equation]
Proof.
See Appendix C.
Comparing (56) to (51) we can see that R (1)part ( η ) is of the same order as R (1)full ( η ) , while R (2)part ( η ) is an extra loss term introduced by sampling.
Following the same steps as the proof of Theorem 1 gives
[Equation]
where
[Equation]
Our last step simpliﬁes the overall conditions on ρ max so that E 2 ( (cid:15) g , ¯ (cid:15) L , σ, ρ max )
=
O (1) :
[Equation]
We claim to fulﬁll (61) it sufﬁces to require
[Equation]
When τ = 1 , the condition trivially requires ρ max = 0 .
We then consider the case for τ ≥ 2 .
By deﬁnition, it can be veriﬁed that
[Equation]
First notice that
[Equation]
and
[Equation] [Equation]
Therefore, it remains to prove (62) implies
[Equation]
Using the fact that under (62) ρ max ≤ 1 / 4 we have
[Equation] [Equation]
This proves the claim.
C P ROOF OF M AIN P ROPOSITIONS C. P ROOF OF P ROPOSITION
Denote for short
[Equation]
Invoking Lemma 2 and Lemma 3 we obtain that under the condition that the learning rate η >satisﬁes
[Equation]
[Equation]
the following inequality is satisﬁed for all s = 0 , . .
, τ −
[Equation]
where
[Equation]
The inequality in (69) is deﬁned elementwise.
Unrolling (69) yields
[Equation]
where we have used the fact that M r, 0 =
due to full synchronization of the x i ’s at the beginning of each round r .
We ﬁrst provide a bound on the sum of the two elements of G (cid:96) B r,s − (cid:96) .
For simplicity we omit the round index r in the superscript for the rest of this section.
Lemma 4.
Let b s − (cid:96)and
b s − (cid:96)be the ﬁrst and second element of B s − (cid:96) , respectively.
Suppose the learning rate η > 0 then
[Equation]
where λ 1 ≤ λ 2 are the eigenvalues of G ; and ρ L is deﬁned in (31).
Proof.
See Appendix D.4.
From Lemma 4 we immediately get
[Equation]
Since λ 2 ≥ C τ >
[Equation]
we have
[Equation]
75) and thus s
[Equation]
Recall the deﬁnition of ρ L given by (31):
[Equation]
By the Gershgorin’s theorem, since η > 0 , we can upperbound λ 2 as
[Equation]
where the last inequality is due to the bound on η :
[Equation]
Deﬁne constant
[Equation]
We consider two cases.
• Case 1: ρ
[Equation]
Thus D τ,ρ = 1 / (1 − ρ max ) .
We let ζ k = ρ k / (1 − ρ k ) and it gives max
[Equation]
Substituting into the bound of λ 2 [cf.
(78)] gives
[Equation]
where in the second inequality we used the condition (81).
Since s ≤ τ and λ 2 ≥ 1 , we obtain the following bound
[Equation]
Moreover, since
[Equation]
we can bound λ 2 − λ 1 as
[Equation]
Collecting (84) and (86) we can bound (76) as
[Equation]
[Equation]
Substituting the expression of b (cid:96) 1 and b (cid:96) 2 gives
[Equation]
where C 1 is some universal constant.
The last inequality holds since ρ L = ρ 2max D τ,ρ and D τ,ρ ≤ τ .
[Equation]
In such a case
[Equation]
we let ζ k =
(4 τ − 1) and thus
[Equation]
Substituting into the bound of λ 2 given in (78), applying again the learning rate condition (79) and using the fact that D τ,ρ = τ :
[Equation]
Therefore by (76)
[Equation]
[Equation] [Equation]
and the fact that
[Equation] [Equation]
we obtain
[Equation]
Substituting the expression of b 1 and b 2 and using the fact that
[Equation]
we arrive at the same bound as in Case 1, possibly with a different constant C 1 .
C. P ROOF OF P ROPOSITION
We are in Case 1 described in the proof of Proposition 1.
By letting ζ k = ρ k / (1 − ρ k ) we have
[Equation]
and the following bound on the difference of the eigenvalues of G :
[Equation]
Notice that according to (72) and (149)
[Equation]
Therefore
N
[Equation] [Equation]
+
[Equation]
≤
[Equation]
In the following, we bound λ
and λ 2 − C τ as a function of ρ max .
For notation simplicity we omit the subscript of ρ max in the rest of the proof.
Further, we introduce the following shorthand notation for the elements of G :
[Equation]
The eigenvalues of G then can be written in closed form as
[Equation]
Since we have proved in Proposition 1 that C τ ≥ f ( ρ ) [cf. (85)]
[Equation]
this gives us the following upperbounds on λ 2 as:
[Equation]
Furthermore, applying the Gershgorin’s theorem we obtain
[Equation]
(100) under the learning rate condition (79).
Substituting the bounds into (96) gives
[Equation]
where we have used the bound λ 1 ≤ f ( ρ ) < 1 , λ 2 > 1 and λ τ 2 < 3 .
Plug in the expression of b 1 and b 2 and using the fact that C τ < 2 , ρ L = ρ 2 D τ,ρ gives
N
[Equation]
+
[Equation] [Equation]
where the second inequality holds since
[Equation]
and the last inequality holds since D τ,ρ ≤ τ .
Tidy up the expression gives
[Equation]
for some C 2 > 0 .
The last inequality holds since
[Equation]
C. P ROOF OF P ROPOSITION
We prove (47) by splitting the terms A 2 , 1 follows:
[Equation]
where r ik (cid:44) (cid:88)
[Equation]
Equality (a) holds since
K (cid:88) k
= (cid:88)
[Equation]
(cid:88)
[Equation] [Equation]
and similarly, (cid:88) i ∈ S r k (cid:88)
j
[Equation]
Since samples are taken according to the rule speciﬁed by Assumption
[Equation]
the following probabilities hold:
[Equation]
Consequently
[Equation]
we can evaluate the second term in (107) and obtain
[Equation]
+ ( Np ) 2 E (cid:32) p · np −n − 1 (cid:13)(cid:13)(cid:13)
K (cid:88) k
= (cid:88) i ∈V k r ik (cid:13)(cid:13)(cid:13) 2 + p (1 − p ) n n −
K (cid:88) k
= (cid:88) i ∈V k (cid:107) r ik (cid:107)(cid:33)
+ ( Np )p (1 − p ) n − 1 E  (cid:88)
k (cid:54) =
(cid:96) (cid:88)
i ∈V k (cid:88)
j ∈V (cid:96) r (cid:62) ik r j(cid:96) 
 ≤ 2 E (cid:13)(cid:13)(cid:13) (cid:13)(cid:13) 1 K
K (cid:88) k
= η ∇ ¯ f k (¯ x r,τ − 1 k ) (cid:13)(cid:13)(cid:13) (cid:13)(cid:13)
+ ( Np ) 2 E (cid:32) p · np −n − 1 (cid:13)(cid:13)(cid:13)
K (cid:88) k
= (cid:88) i ∈V k r ik (cid:13)(cid:13)(cid:13) 2 + p (1 − p ) n n −
K (cid:88) k
= (cid:88) i ∈V k (cid:107) r ik (cid:107)(cid:33)
+ ( Np )p (1 − p ) n − 1 E  (cid:88)
k (cid:54) =
(cid:96) (cid:88)
i ∈V k (cid:88)
j ∈V
[Equation]
[Table]
k )) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Table]
N
K (cid:88) k
= (cid:88) i ∈V k (cid:88)
j ∈V k ( W k ) i,j L 2 (cid:107) x r,τ − 1 j − ¯ x r,τ − 1 k (cid:107)
[Table]
N
[Table]
(cid:88) k
= (cid:88) i ∈V k
L 2 (cid:107) x r,τ − 1 i − ¯ x r,τ − 1 k (cid:107) 2 = η 2 · NL
K (cid:88) k
= (cid:107)
[Table] [Table]
[Table]
(cid:107)(114) and
[Table]
(cid:107) r
[Table]
=
[Table]
i ∈V
[Table]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Table]
j ∈V k ( W k ) i,j ( x r,τ − 1 j − ¯ x r,τ − 1 k − η ( ∇ f j ( x r,τ − 1 j ) − ∇ ¯ f k (¯ x r,
[Table]
− 1 k ))
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Table]
( W k ) i,j (cid:13)(cid:13)(cid:13) x r,
[Table]
j − ¯ x r,τ − 1 k − η ( ∇ f j ( x r,τ − 1 j ) − ∇ f j (¯ x r,
[Equation]
Tidy up the expression leads to the following bound of A 2 , 1 :
[Equation] [Equation]
where the last inequality holds under the learning rate condition
[Equation]
This completes the proof of (47).
C. P ROOF OF P ROPOSITION
We bound A 2 , 2 in following the same rationale as Proposition 4.
[Equation]
j ∈V k ( W k ) i,j (
[Equation]
f
[Equation]
( x r,sj )
[Equation]
r,sj ) (cid:1) (cid:124) (cid:123)(cid:122) (cid:125)
e ik (cid:13)(cid:13)(cid:13)=
[Equation]
(cid:88)
i
[Equation]
k e
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
K (cid:88) k
= (cid:88) i ∈V k
E (cid:107) e ik (cid:107)(cid:33)
[Equation]
(cid:88)
i
[Equation]
k (cid:88)
j
[Equation]
(cid:96) e (cid:62)
[Equation]
e
[Equation]

[Equation]
(cid:88) i ∈V k e ik (cid:13)(cid:13)(cid:13)
[Equation]
K (cid:88)
[Equation]
= (cid:88)
i
[Equation]
k
E (cid:107) e ik (cid:107)(cid:33) , (118) where the last equality is due to fact that the inter-cluster stochastic noise is zero mean and independent.
Recall the deﬁnition ξ r,s i
[Equation]
Using again the independence of the ξ i ’s we get
E (cid:13)(cid:13)(cid:13) K (cid:88) k
= (cid:88) i ∈V k e ik (cid:13)(cid:13)(cid:13) 2 = E (cid:13)(cid:13)(cid:13)
K (cid:88) k
= (cid:88) i ∈V k (cid:88)
j ∈V k ( W k ) i,j ξ r,sj (cid:13)(cid:13)(cid:13)
= E (cid:13)(cid:13)(cid:13)
N (cid:88) i = ξ r,si (cid:13)(cid:13)(cid:13) 2 = Nσ 2 , (119) and
K (cid:88) k
= (cid:88) i ∈V k
E (cid:107)
[Equation]
(cid:107)
[Equation]
=
K (cid:88) k
= (cid:88) i ∈V k
E (cid:13)(cid:13)(cid:13) (cid:88)
j ∈V k ( W k ) i,j ξ r,sj (cid:13)(cid:13)(cid:13) 2 =
K (cid:88) k
= (cid:107) W k (cid:107) 2 σ=
K (cid:88) k
= σ 2 n (cid:88)
j = d 2 j ≤
K (cid:88) k
= σ 2 (cid:16)
[Equation]
(cid:0)
[Equation]
(cid:1)
[Equation]
k (cid:17)
[Equation]
[Equation] [Equation]
= (cid:0)
[Equation]
( n
[Equation]
1) ρ 2 max (cid:1) Kσ 2 .
where d 1 ≤ d 2 ≤ · · · ≤ d n = 1 are the singular values of W k .
Therefore,
[Equation]
The last inequality is due to p ≥ 1 /n .
C. P ROOF OF P ROPOSITION
Invoking the descent inequality Lemma 1 and the error bound for T 1 - T 5 given by Corollary 2,
Proposition 3 and Eq.
[Equation]
The last inequality holds under the condition that
[Equation]
If we further enforce
[Equation]
then
[Equation]
the last inequality is due to D τ,ρ ≤ τ and ρ max ≤
[Equation]
C. P ROOF OF P ROPOSITION
Invoking the descent inequality Lemma 1 and the error bound for T 1 - T 5 given by Corollary 2,
Proposition
[Equation]
and 4:
[Equation]
2 (cid:88) s =
E (cid:18)
L 2 (cid:107) ¯ x r,s +
− ¯ x r,s (cid:107)
(cid:19)
+ E (cid:18)
L 2 (cid:107) x
r +1 − ¯ x r,τ − 1 (cid:107) 2 (cid:19) ≤
E f ( x r ) − ητ − 1 (cid:88) s =
E (cid:107)∇ f (¯ x r,s ) (cid:107) 2 − ητ − 1 (cid:88) s =
E (cid:13)(cid:13)(cid:13)
N
N (cid:88) i = ∇ f i ( x r,si ) (cid:13)(cid:13)(cid:13)− η τ − 1 (cid:88) s =
E (cid:13)(cid:13)(cid:13)
K
K (cid:88) k
= ∇ ¯ f k (¯ x r,sk ) (cid:13)(cid:13)(cid:13)+ C 1 L 2 η 3 τ (cid:0) τ
+ ρ 2max D τ,ρ (cid:1) τ − 1 (cid:88) s =
( α 2 (cid:107)∇ f (¯ x r,s ) (cid:107) 2 + (cid:15) 2 g )
+ C 1 L 2 η 3 τ 2 ρ 2max D τ,
ρ ¯ (cid:15) 2 L + C 1 τ 2 L 2 η
ρ 2max σ 2 + C 1 L 2 ( τ 2 + D 2 τ,ρ ρ 2max ) η 3 σ 2 n + L 2 ητ − 2 (cid:88) s =
E (cid:13)(cid:13)(cid:13)(cid:13) (cid:13)
N
N (cid:88) i = ∇ f i ( x r,si ) (cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
[Equation]
Denote for short
[Equation]
Further applying the bounds on the consensus error derived in Proposition 2:
[Equation] [Equation]
Rearranging terms and tidy up the expression we have
[Equation]
s =
E (cid:107)∇ f (¯ x r,s ) (cid:107) 2 − ητ − 1 (cid:88) s =
E (cid:13)(cid:13)(cid:13)
N
N (cid:88) i = ∇ f i ( x r,si ) (cid:13)(cid:13)(cid:13)− η τ − 1 (cid:88) s =
E (cid:13)(cid:13)(cid:13)
K
K (cid:88) k
= ∇ ¯ f k (¯ x r,sk ) (cid:13)(cid:13)(cid:13) 2 + L 2 ητ − 2 (cid:88) s =
E (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
N
N (cid:88) i = ∇ f i ( x r,si ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
+ Lη 2 E (cid:13)(cid:13)(cid:13)(cid:13) (cid:13)
K
K (cid:88) k
= ∇ ¯ f k (¯ x r,τ − 1 k ) (cid:13)(cid:13)(cid:13)(cid:13) (cid:13)
+ C 1 L 2 η 3 τ (cid:0) τ
+ ρ 2max D τ,ρ (cid:1) τ − 1 (cid:88) s =
( α 2 (cid:107)∇ f (¯ x r,s ) (cid:107) 2 + (cid:15) 2 g ) (129) + 8 C 2 G (cid:48)
p LD 2 τ,ρ ρ max η(cid:32) τ − 2 (cid:88) s =
( α 2 (cid:107)∇ f (¯ x r,s ) (cid:107) 2 + (cid:15) 2 g ) (cid:33)
[Equation]
Notice that if the following conditions on the learning rate are satisﬁed
[Equation]
then the terms associated to the gradients will be negative and
[Equation]
In the last step we clean the condition on the learning rate η .
Collecting all the conditions on η :
[Equation]
[Equation]
Clearly, (131) implies (132).
To ensure (133) it sufﬁces to require
[Equation] [Equation]
Recall the deﬁnition of G (cid:48) p :
[Equation]
and G p = n
[Equation]
m
[Equation]
It can be veriﬁed that
[Equation]
Using the fact that D τ,ρ ≤ τ and ρ max ≤ 1 we can see that it sufﬁces to require
[Equation]
(137) for some C 3 > 0 large enough to satisfy both (131) and (133).
D S UPPORTING LEMMAS D. P ROOF OF L EMMA
Since the global average of the local copies follows the update [cf.
[Equation]
Under Assumption
[Equation]
we can apply the descent lemma at points ¯ x r,s +1 and ¯ x r,s for s = 0 , . .
, τ − 2 , conditioned on F r,s − 1 :
[Equation]
[Table]
[Equation]
where (a) is due to Assumption 2, (b) is due to 2 ab = (cid:107) a (cid:107) 2 + (cid:107) b (cid:107) 2 − (cid:107) a − b (cid:107) 2 and ab ≤ (cid:107) a (cid:107)(cid:107) b (cid:107) , and (c) is due to (cid:107) a (cid:107)(cid:107) b (cid:107) ≤ 12 (cid:107) a (cid:107)
+ 12 (cid:107) b (cid:107) 2 .
Notation a ± b stands for adding and subtracting, i.e., a ± b = a + b − b .
For the pair (¯ x r,τ − 1 , x r +1 ) we have according to (19) and (22):
[Equation]
Applying the descent lemma in the same way as before yields
[Equation] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Taking expectation, summing over the iterations in round r over s = 0 , . .
, τ − 1 and using the fact that x r = ¯ x r, 0 completes the proof.
D. P ROOF OF L EMMA
Recall the average update of the k -th cluster and that of the global average given by (15) and (19), respectively, for s = 0 , . .
, τ − 1 :
[Equation]
Taking the difference gives
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) , (141) where (cid:15) > 0 is some constant to be chosen.
Averaging over k = 1 , . .
, K :
[Equation]
+
[Equation] [Equation]
K K
[Equation]
k =
E (cid:107) ¯ x r,s
[Equation]
¯ x r,sk (cid:107)+ (1 + (cid:15) − 1 ) η(cid:32)
K
K (cid:88) k
=
E (cid:13)(cid:13)(cid:13)n (cid:88) i ∈V k ∇ f i ( x r,
[Equation]
i ) (cid:13)(cid:13)(cid:13) 2 −
[Equation]
(cid:13)(cid:13)(cid:13) 1 N
N (cid:88) i =
[Equation]
f
[Equation]
( x
[Equation]
(cid:13)(cid:13)(cid:13)(cid:33)
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
2 + (1 + (cid:15) − 1 ) η 2 3 K
K (cid:88) k
=
E (cid:13)(cid:13)(cid:13) ∇ ¯ f k (¯ x r,s ) (cid:13)(cid:13)(cid:13)
+ η 2 K − 1 N σ= (cid:0) 1 + (cid:15)
+ 3 L 2 η 2 (1 + (cid:15) − 1 ) (cid:1) 1 K
K (cid:88) k
=
E (cid:107) ¯ x r,s − ¯ x r,sk (cid:107)
+ (1 + (cid:15) − 1 ) η(cid:32)
N
K (cid:88)
K =
L 2 E (cid:13)(cid:13)(cid:13)
X r,sk, ⊥ (cid:13)(cid:13)(cid:13)(cid:33) + (1 + (cid:15) − 1 ) η 2 3 K
K (cid:88) k
=
E (cid:13)(cid:13)(cid:13) ∇ ¯ f k (¯ x r,s ) (cid:13)(cid:13)(cid:13)
+ η 2 K − 1 N σ 2 .
In (a) we used the fact that
K
K (cid:88) k
=  n (cid:88) i ∈V k ∇ f i ( x r,si ) = 1 N
N (cid:88) i = ∇ f i ( x r,si ) ,
K
K (cid:88) k
=  n (cid:88)
i ∈V k ξ r,si =
N
N (cid:88) i = ξ r,si .
(143) and
K (cid:88) i
=
(cid:107) x
i − ¯ x (cid:107) 2 =
K (cid:88) i =
[Equation]
with ¯ x = 1 K
K (cid:88) k
= x i .
In (b) we applied the L -smoothness of f i and ¯ f k .
Choosing (cid:15) =
1 4 τ − 1 and using the condition that
[Equation]
we have
[Equation] [Equation]
In the last inequality we applied Assumption 5 on the inter-cluster heterogeneity
[Equation]
D. P ROOF OF L EMMA
We follow the perturbed average consensus analysis.
Recall the update equation of the consensus error given in (17):
[Equation]
Squaring both sides and conditioning:
[Equation]
where ζ k > 0 is some free parameter to be properly chosen.
Next
[Equation]
we bound the norm of the pseudo-gradient ∇ F k ( X r,sk ) .
[Equation]
(cid:107) 2 +
(cid:107)∇ ¯ f k (¯
[Equation]
r,s )
[Equation]
k .
The last inequality is due to Assumption 4 on the intra-cluster heterogeneity.
Averaging over k = 1 , . .
, K clusters:
[Equation] [Equation]
D. P ROOF OF L EMMA
To simplify the notation we omit the superscript in B r,s −
[Equation]
in this section
[Equation]
Let Λ = diag ( λ 1 , λ 2 ) and the eigendecomposition of G = T Λ T −
[Equation]
we can obtain the closed form expression of T as
[Equation]
and
[Equation]
where
[Equation]
Consequently
[Equation]
) with
[Equation]
Therefore
[Equation]
Substituting the expression of det( T ) and dividing both sides of the equality by 12 τη 2 ( λ 1 − λ 2 ) we have
[Equation]
[Equation]
where in the last inequality we used the fact that λ 1 ≤ C τ ≤ λ
[Equation]
Note that
[Equation]
4 L 4 .
[Equation]
Therefore, we further obtain
[Equation]
Dividing both sides by L 2 completes the proof.
E N ETWORK C ONNECTIVITY C ONDITIONS IN T HEOREMS AND C OROLLARY
Both Theorem 2 and Corollary 1 impose some sufﬁcient conditions on the network connectivity ρ max for convergence. This can be satisﬁed in practice as follows. For Theorem 2, as long as ρ max < 1 , we can choose τ large enough so that (7) is fulﬁlled.
Corollary 1 strengthens the result of
Theorem 2 by requiring no loss in the order of convergence rate compared to full device participation.
This naturally leads to a more stringent condition on ρ max given by (11).
For any given D2D network topology, this can be satisﬁed by running multiple
D2D gossip averaging steps per SGD update in Algorithm 1.
Since the right hand side of (11) depends only on the algorithmic parameters, we can choose the suitable gossip averaging steps to fulﬁll this condition before launching the algorithm.
F M ORE E XPERIMENTS D ETAILS
In this section, we provide additional experimental results on CIFAR-10 dataset.
We follow the same CNN model and non-iid data partition strategy as before and run each experiments for 3 times with different random seeds to report the mean values of best test accuracy.
Instead of using a constant learning rate, we decay the local learning rate η by half after ﬁnishing 50% and 75% of the communication rounds and tune the initial learning rate from { 0 . 01 , 0 . 02 , 0 .
1 } for each algorithm.
[Figure]
First, we evaluate the convergence processes of HL-SGD and local SGD under varying D2D network topologies in Figure 4.
We generate random network topologies by Erd ˝ os-R ´ enyi model with edge probability from { 0 . 2 , 0 , 5 , 0 . 8 , 1 } and use Metropolis-Hastings weights to set W k , corresponding to spectral norm ρ max = { 0 . 9394 , 0 . 844 , 0 .
As observed in Figure 4a, a more connected D2D network topology (i.e., a smaller value of ρ max ) generally accelerates the convergence and leads to a higher model accuracy achieved over 100 communication rounds in HL-SGD.
However, in terms of runtime, a more connected D2D network topology corresponds to a larger D2D communication delay c d2d per round, and hence the total runtime is larger as well, which can be clearly observed in Figure 4b.
Therefore, to achieve a target level of model accuracy within the shortest time in HLSGD, a sparse D2D network topology could work better than the fully connected one in practice.
Second, to give a more comprehensive analysis on the runtime beneﬁts of HL-SGD, we vary τ from { 5 , 10 , 20 , 50 } and compare the performances of HL-SGD and local SGD in Figure 5.
From the ﬁgure, we can observe that HL-SGD can consistently outperform local SGD across a wide range of τ .
In particular, on CIFAR-10, the best accuracy of HL-SGD achieved over 100 rounds is 2.49%, 3.99%, 4.05%, and 7% higher than that of local SGD, respectively, as τ increases from 5 to 50.
At the same time, the runtime of HL-SGD needed to achieve a target test-set accuracy of 60% is 9.66%, 19.76%, 33.46%, and 45.88% less than that of local SGD, respectively.
Third, we consider an extension of HL-SGD by allowing each device to perform multiple SGD updates before the gossip averaging step in Algorithm 1 and empirically evaluate its performance.
Speciﬁcally, each device performs l = { 1 , 5 , 10 } steps of SGD update before aggregating models with their neighbors in the same cluster.
Note that l = 1 corresponds to the original version of HLSGD in Algorithm 1.
As observed in Figure 6a, when communicating and aggregating models with neighbors more frequently, HL-SGD with l = 1 has the best convergence speed and will converge to the highest level of test accuracy.
In terms of runtime, choosing a value of l > 1 might be favorable in some cases due to the reduced D2D communication delay per round.
For instance, to achieve a target level of 60% test accuracy, HL-SGD with l = 5 needs 5.22% less amount of time than l = 1 .
It is an interesting direction to rigorously analyze the convergence properties of HL- [Figure] SGD with arbitrary l and ﬁnd the best hyperparameter tuning method for minimizing the runtime to achieve a target level of model accuracy in the future.
[Figure]
G R ELATIONSHIP BETWEEN S PECTRAL G AP AND N ETWORK T OPOLOGY
ring/path 2D-grid 2-D torus
Erd ˝ os [Equation] ´ enyi exponential
[Equation]
n ))