A CHIEVING L INEAR S PEEDUP WITH P ARTIAL W ORKER P ARTICIPATION IN N ON -IID F EDERATED L EARNING
A BSTRACT
Federated learning (FL) is a distributed machine learning architecture that leverages a large number of workers to jointly learn a model with decentralized data.
FL has received increasing attention in recent years thanks to its data privacy protection, communication efﬁciency and a linear speedup for convergence in training (i.e., convergence performance increases linearly with respect to the number of workers).
However, existing studies on linear speedup for convergence are only limited to the assumptions of i.i.d.
datasets across workers and/or full worker participation, both of which rarely hold in practice.
So far, it remains an open question whether or not the linear speedup for convergence is achievable under non-i.i.d.
datasets with partial worker participation in FL.
In this paper, we show that the answer is afﬁrmative.
Speciﬁcally, we show that the federated averaging (FedAvg) algorithm (with two-sided learning rates) on non-i.i.d.
datasets in non-convex settings achieves a convergence rate
O ( 1 √ mKT + 1 T ) for full worker participation and a convergence rate O ( 1 √ nKT + 1 T ) for partial worker participation, where K is the number of local steps, T is the number of total communication rounds, m is the total worker number and n is the worker number in one communication round if for partial worker participation.
Our results also reveal that the local steps in FL could help the convergence and show that the maximum number of local steps can be improved to T/m .
We conduct extensive experiments on MNIST and CIFARto verify our theoretical results.
I NTRODUCTION
Federated Learning (FL) is a distributed machine learning paradigm that leverages a large number of workers to collaboratively learn a model with decentralized data under the coordination of a centralized server.
Formally, the goal of FL is to solve an optimization problem, which can be decomposed as:
[Equation]
where F i ( x ) (cid:44) E ξ i ∼ D i [ F i ( x, ξ i )] is the local (non-convex) loss function associated with a local data distribution D i and m is the number of workers.
FL allows a large number of workers (such as edge devices) to participate ﬂexibly without sharing data, which helps protect data privacy.
However, it also introduces two unique challenges unseen in traditional distributed learning algorithms that are used typically for large data centers:
• Non-independent-identically-distributed (non-i.i.d.)
datasets across workers (data heterogeneity): In conventional distributed learning in data centers, the distribution for each worker’s local dataset can usually be assumed to be i.i
d., i.e., D i
=
D, ∀ i ∈ { 1 , ..., m } .
Unfortunately, this assumption rarely holds for FL since data are generated locally at the workers based on their circumstances, i.e., D i (cid:54) =
D j , for i (cid:54) =
j .
It will be seen later that the non-i.i.d assumption imposes signiﬁcant challenges in algorithm design for FL and their performance analysis.
• Time-varying partial worker participation (systems non-stationarity): With the ﬂexibility for workers’ participation in many scenarios (particularly in mobile edge computing), workers may randomly join or leave the FL system at will, thus rendering the active worker set stochastic and time-varying across communication rounds.
Hence, it is often infeasible to wait for all workers’ responses as in traditional distributed learning, since inactive workers or stragglers will signiﬁcantly slow down the whole training process.
As a result, only a subset of the workers may be chosen by the server in each communication round, i e., partial worker participation.
In recent years, the Federated Averaging method (FedAvg) and its variants (McMahan et al., 2016;
Li et al., 2018; Hsu et al., 2019; Karimireddy et al., 2019; Wang et al., 2019a) have emerged as a prevailing approach for FL.
Similar to the traditional distributed learning, FedAvg leverages local computation at each worker and employs a centralized parameter server to aggregate and update the model parameters.
The unique feature of FedAvg is that each worker runs multiple local stochastic gradient descent (SGD) steps rather than just one step as in traditional distributed learning between two consecutive communication rounds.
For i.i.d.
datasets and the full worker participation setting,
Stich (2018) and Yu et al.
(2019b) proposed two variants of FedAvg that achieve a convergence rate of O ( mKT + 1 √ mKT ) with a bounded gradient assumption for both strongly convex and nonconvex problems, where m is the number of workers, K is the local update steps, and T is the total communication rounds.
Wang & Joshi (2018) and Stich & Karimireddy (2019) further proposed improved FedAvg algorithms to achieve an O ( mT + 1 √ mKT ) convergence rate without bounded gradient assumption.
Notably, for a sufﬁciently large T , the above rates become O ( 1 √ mKT ) 1 , which implies a linear speedup with respect to the number of workers.
This linear speedup is highly desirable for an FL algorithm because the algorithm is able to effectively leverage the massive parallelism in a large FL system.
However, with non-i.i.d.
datasets and partial worker participation in FL, a fundamental open question arises: Can we still achieve the same linear speedup for convergence, i e., O ( 1 √ mKT ) , with non-i.i d. datasets and under either full or partial worker participation?
In this paper we show the answer to the above question is afﬁrmative.
Speciﬁcally, we show that a generalized FedAvg with two-sided learning rates achieves linear convergence speedup with non-i i d. datasets and under full/partial worker participation.
We highlight our contributions as follows:
• For non-convex problems, we show that the convergence rate of the FedAvg algorithm on noni.i.d.
dataset are O ( 1 √ mKT + 1 T ) and O ( 1 √ nKT + 1 T ) for full and partial worker participation, respectively, where n is the size of the partially participating worker set.
This indicates that our proposed algorithm achieves a linear speedup for convergence rate for a sufﬁciently large T .
When reduced to the i.i.d.
case, our convergence rate is O ( 1 TK + 1 √ mKT ) , which is also better than previous works.
We summarize the convergence rate comparisons for both i.i.d.
and non-i.i.d.
cases in Table 1.
It is worth noting that our proof does not require the bounded gradient assumption.
We note that the SCAFFOLD algorithm (Karimireddy et al., 2019) also achieves the same rate but extra variance reduction operations are required, which lead to higher communication costs and implementation complexity.
By contrast, we do not have such extra requirements in this paper.
• In order to achieve a linear speedup, i.e., a convergence rate O ( 1 √ mKT ) , we show that the number of local updates K can be as large as T/m , which improves the T 1 / 3 /m result previously shown in Yu et al. (2019a) and Karimireddy et al.
As shown later in the communication complexity comparison in Table 1, a larger number of local steps implies relatively fewer communication rounds, thus less communication overhead.
Interestingly, our results also indicate that the number of local updates K does not hurt but rather help the convergence with a proper learning rates choice.
This overcomes the limitation as suggested in Li et al.
(2019b) that local SGD steps might slow down the convergence ( O ( KT ) for strongly convex case).
This result also reveals new insights on the relationship between the number of local steps and learning rate.
[Table]
is used at each worker.
A FedAvg algorithm with two-sided learning rates.
M 2 =
O (
+ O ( KS (1 − Sm ))
S
= m ( S = n ) for full (partial) worker participation.
The SCAFFOLD algorithm in Karimireddy et al.
for non-convex case.
The convergence rate becomes O ( 1 T + 1 √ nKT ) under partial worker participation.
Shorthand notation for convexity: SC: Strongly Convex, C: Convex, and NC: Non-Convex.
Notation.
In this paper, we let m be the total number of workers and S t be the set of active workers for the t -th communication round with size | S t | = n for some n ∈ (0 , m ] .
We use K to denote the number of local steps per communication round at each worker.
We let T be the number of total communication rounds.
In addition, we use boldface to denote matrices/vectors.
We let [ · ] it,k represent the parameter of k -th local step in the i -th worker after the t -th communication.
We use (cid:107)·(cid:107) 2 to denote the (cid:96) 2 -norm.
For a natural number m , we use [ m ] to represent the set { 1 , · · · , m } .
The rest of the paper is organized as follows.
In Section 2, we review the literature to put our work in comparative perspectives.
Section 3 presents the convergence analysis for our proposed algorithm.
Section 4 discusses the implication of the convergence rate analysis.
Section 5 presents numerical results and Section 6 concludes this paper.
Due to space limitation, the details of all proofs and some experiments are provided in the supplementary material.
R ELATED WORK
The federated averaging (FedAvg) algorithm was ﬁrst proposed by McMahan et al.
(2016) for FL as a heuristic to improve communication efﬁciency and data privacy.
Since then, this work has sparked many follow-ups that focus on FL with i.i.d.
datasets and full worker participation (also known as
LocalSGD (Stich, 2018; Yu et al., 2019b; Wang & Joshi, 2018; Stich & Karimireddy, 2019; Lin et al., 2018; Khaled et al., 2019a; Zhou & Cong, 2017)).
Under these two assumptions, most of the theoretical works can achieve a linear speedup for convergence, i.e., O ( 1 √ mKT ) for a sufﬁciently large T , matching the rate of the parallel SGD.
In addition, LocalSGD is empirically shown to be communication-efﬁcient and enjoys better generalization performance (Lin et al., 2018).
For a
Algorithm 1 A
Initialize xfor t = 0 , · · · , T − 1 do
The server samples a subset S t of workers with | S t | = n .
for each worker i ∈ S t in parallel do x it,
= x t for k = 0 , · · · , K − 1 do
Compute an unbiased estimate g it,k = ∇ F i ( x it,k , ξ it,k ) of ∇ F i ( x it,k ) .
Local worker update: x it,k
+
= x
it,k − η L g it,k . end for
Let ∆ it =
x it,K − x it, 0 = − η L (cid:80) K − 1 k =
g it,k .
Send ∆ it to the server.
end for
At Server:
Receive ∆ it , i ∈ S .
Let ∆ t = 1 | S | (cid:80) i ∈ S ∆ it .
Server Update: x t +1 = x t + η ∆ t .
Broadcasting x t +1 to workers.
end for
For non-i.i.d. datasets, many works (Sattler et al., 2019; Zhao et al., 2018; Li et al., 2018; Wang et al., 2019a; Karimireddy et al., 2019; Huang et al., 2018; Jeong et al., 2018) heuristically demonstrated the performance of FedAvg and its variants. On convergence rate with full worker participation, many works (Stich et al., 2018; Yu et al., 2019a; Wang & Joshi, 2018; Karimireddy et al., 2019; Reddi et al., 2020) can achieve linear speedup, but their convergence rate bounds could be improved as shown in this paper.
On convergence rate with partial worker participation, Li et al.
(2019b) showed that the original FedAvg can achieve O ( K/T ) for strongly convex functions, which suggests that local
SGD steps slow down the convergence in the original FedAvg.
Karimireddy et al.
(2019) analyzed a generalized FedAvg with two-sided learning rates under strongly convex, convex and non-convex cases.
However, as shown in Table 1, none of them indicates that linear speedup is achievable with non-i.i.d.
datasets under partial worker participation.
Note that the SCAFFOLD algorithm (Karimireddy et al., 2019) can achieve linear speedup but extra variance reduction operations are required, which lead to higher communication costs and implementation complexity.
In this paper, we show that this linear speedup can be achieved without any extra requirements.
For more detailed comparisons and other algorithmic variants in FL and decentralized settings, we refer readers to Kairouz et al.
L INEAR S PEEDUP OF THE G ENERALIZED F ED A VG WITH T WO -S IDED L EARNING R ATES FOR N ON -IID D ATASETS
In this paper, we consider a FedAvg algorithm with two-sided learning rates as shown in Algorithm 1, which is generalized from previous works (Karimireddy et al., 2019; Reddi et al., 2020).
Here, workers perform multiple SGD steps using a worker optimizer to minimize the local loss on its own dataset, while the server aggregates and updates the global model using another gradient-based server optimizer based on the returned parameters.
Speciﬁcally, between two consecutive communication rounds, each worker performs K SGD steps with the worker’s local learning rate η L .
We assume an unbiased estimator in each step, which is denoted by g it,k = ∇ F i ( x it,k , ξ it,k ) , where ξ it,k is a random local data sample for k -th steps after t -th communication round at worker i .
Then, each worker sends the accumulative parameter difference ∆ it to the server.
On the server side, the server aggregates all available ∆ it -values and updates the model parameters with a global learning rate η .
The FedAvg algorithm with two-sided learning rates provides a natural way to decouple the learning of workers and server, thus utilizing different learning rate schedules for workers and the server.
The original
FedAvg can be viewed as a special case of this framework with server-side learning rate being one.
In what follows, we show that a linear speedup for convergence is achievable by the generalized
FedAvg for non-convex functions on non-i.i.d.
datasets.
We ﬁrst state our assumptions as follows.
Assumption 1.
( L -Lipschitz Continuous Gradient)
There exists a constant L > 0 , such that
[Equation]
Assumption 2.
(Unbiased Local Gradient Estimator) Let ξ it be a random local data sample in the t -th step at the i -th worker.
The local gradient estimator is unbiased, i.e., E [ ∇ F i ( x t , ξ it )]
= ∇ F i ( x t ) , ∀ i ∈ [ m ] , where the expectation is over all local datasets samples.
Assumption 3.
(Bounded Local and Global Variance)
There exist two constants σ L > 0 and σ G > 0 , such that the variance of each local gradient estimator is bounded by E
[ (cid:107)∇ F i ( x t , ξ it [Equation] −∇ F i ( x t ) (cid:107) 2 ] ≤ σ 2 L , ∀ i ∈ [ m ] , and the global variability of the local gradient of the cost function is bounded by (cid:107)∇
[Equation]
The ﬁrst two assumptions are standard in non-convex optimization (Ghadimi & Lan, 2013; Bottou et al., 2018).
For Assumption 3, the bounded local variance is also a standard assumption.
We use a universal bound σ G to quantify the heterogeneity of the non-i.i.d.
datasets among different workers.
In particular, σ G = 0 corresponds to i.i.d.
datasets.
This assumption is also used in other works for
FL under non-i.i.d.
datasets (Reddi et al., 2020; Yu et al., 2019b; Wang et al., 2019b) as well as in decentralized optimization (Kairouz et al., 2019).
It is worth noting that we do not require a bounded gradient assumption, which is often assumed in FL optimization analysis.
3. C ONVERGENCE ANALYSIS FOR FULL WORKER PARTICIPATION
In this subsection, we ﬁrst analyze the convergence rate of the generalized FedAvg with two-sided learning rates under full worker participation, for which we have the following result:
Theorem 1.
Let constant local and global learning rates η L and η be chosen as such that η L ≤ 1 8 LK and ηη L ≤ 1 KL .
Under Assumptions 1–3 and with full worker participation, the sequence of outputs { x k } generated by Algorithm 1 satisﬁes:
[Equation]
where Φ (cid:44) 1 c [ Lηη
[Equation]
m σ
[Equation]
L + 5 Kη 2 L L
[Equation]
( σ 2 L + 6 Kσ 2 G )] , c is a constant, f 0 (cid:44) f ( x 0 ) , f ∗ (cid:44) f
( x ∗ ) and the expectation is over the local dataset samples among workers.
Remark 1.
The convergence bound contains two parts: a vanishing term f 0 − f ∗ cηη L KT as T increases and a constant term Φ whose size depends on the problem instance parameters and is independent of T .
The vanishing term’s decay rate matches that of the typical SGD methods.
Remark 2.
The ﬁrst part of Φ (i.e., Lηη L 2 m σ 2 L ) is due to the local stochastic gradients at each worker, which shrinks at rate 1 m as m increases.
The cumulative variance of the K local steps contributes to the second term in Φ (i.e., 5 Kη 2 L L 2( σ 2 L + 6 Kσ 2 G )) , which is independent of m and largely affected by the data heterogeneity.
To make the second part small, an inverse relationship between the local learning rate and local steps should be satisﬁed, i.e., η L
= O ( 1 K ) .
Speciﬁcally, note that the global and local variances are linearly ampliﬁed by K .
This requires a sufﬁciently small η L to offset the variance between two successive communication rounds to make the second term in Φ small.
This is consistent with the observation in strongly convex FL that a decaying learning rate is needed for FL to converge under non-i.i.d.
datasets even if full gradients used in each worker (Li et al., 2019b).
However, we note that our explicit inverse relationship between η L and K in the above is new.
Intuitively, the K local steps with a sufﬁciently small η L can be viewed as one SGD step with a large learning rate.
With Theorem 1, we immediately have the following convergence rate for the generalized FedAvg algorithm with a proper choice of two-sided learning rates:
Corollary 1.
Let η L = [Equation] TKL and η = √ Km .
The convergence rate of the generalized FedAvg algorithm under full worker participation is min
[Equation]
Remark 3.
The generalized FedAvg algorithm with two-sided learning rates can achieve a linear speedup for non-i.i.d.
datasets, i.e., a O ( 1 √ mKT ) convergence rate as long as T ≥ mK .
Although many works have achieved this convergence rate asymptotically, we improve the maximum number of local steps K to T/m , which is signiﬁcantly better than the state-of-art bounds such as T 1 / 3 /m shown in (Karimireddy et al., 2019; Yu et al., 2019a; Kairouz et al., 2019).
Note that a larger number of local steps implies relatively fewer communication rounds, thus less communication overhead.
See also the communication complexity comparison in Table 1.
For example, when T = 10 6 and m = 100 (as used in (Kairouz et al., 2019)), the local steps in our algorithm is K ≤ T/m = 10 4 .
However, K ≤ T 1 / 3 m = 1 means that no extra local steps can be taken to reduce communication costs.
Remark 4.
When degenerated to the i.i.d.
case ( σ G = 0 ), the convergence rate becomes O ( 1 TK +√ mKT ) , which has a better ﬁrst term in the bound compared with previous work as shown in Table 1.
3. C ONVERGENCE ANALYSIS FOR PARTIAL WORKER PARTICIPATION
Partial worker participation in each communication round may be more practical than full worker participation due to many physical limitations of FL in practice (e.g., excessive delays because of too many devices to poll, malfunctioning devices, etc.).
Partial worker participation can also accelerate the training by neglecting stragglers.
We consider two sampling strategies proposed by Li et al.
(2018) and Li et al.
(2019b).
Let S t be the participating worker index set at communication round t with | S t | = n , ∀ t , for some n ∈ (0 , m ] .
S t is randomly and independently selected either with replacement (Strategy 1) or without replacement (Strategy 2) sequentially according to the sampling probabilities p i , ∀ i ∈ [ m ] .
For each member in S t , we pick a worker from the entire set [ m ] uniformly at random with probability p
i
= 1 m , ∀ i ∈ [ m ] .
That is, selection likelihood for anyone worker
i ∈
S t is p = nm .
Then we have the following results:
Theorem 2.
Under Assumptions 1–3 with partial worker participation, the sequence of outputs { x k } generated by Algorithm 1 with constant learning rates η and η L satisﬁes:
[Equation]
where f 0 = f ( x 0 ) , f ∗ = f ( x ∗ ) , and the expectation is over the local dataset samples among workers.
For sampling Strategy 1, let η and η L be chosen as such that η L ≤ 1 8 LK , ηη L KL < n − 1 n and 30 K 2 η 2 L L 2 − Lηη L n
[Equation]
90 K 2 L 2 η 2 L + 3) < 1 .
It then holds that:
[Equation]
For sampling Strategy 2, let η and η L be chosen as such that η L ≤ 1 8 LK , ηη L KL ≤ n [Equation]
[Equation]
and
[Equation]
2 + [Equation] [Equation]
It then holds that:
[Equation]
From Theorem 2, we immediately have the following convergence rate for the generalized FedAvg algorithm with a proper choice of two-sided learning rates:
Corollary 2.
Let η L = 1 √ TKL and η = √ Kn .
The convergence rate of the generalized FedAvg algorithm under partial worker participation and both sampling strategies are:
[Equation]
Remark 5.
The convergence rate bound for partial worker participation has the same structure but with a larger variance term.
This implies that the partial worker participation through the uniform sampling does not result in fundamental changes in convergence (in order sense) except for an ampliﬁed variance due to fewer workers participating and random sampling.
The intuition is that the uniform sampling (with/without replacement) for worker selection yields a good approximation of the entire worker distribution in expectation, which reduces the risk of distribution deviation due to the partial worker participation.
As shown in Section 5, the distribution deviation due to fewer worker participation could render the training unstable, especially in highly non-i.i.d.
cases.
Remark 6.
The generalized FedAvg with partial worker participation under non-i.i.d.
datasets can still achieve a linear speedup O ( 1 √ nKT ) with proper learning rate settings as shown in Corollary 2.
In addition, when degenerated to i.i.d.
case ( σ G = 0 ), the convergence rate becomes O ( 1 TK + 1 √ nKT ) .
Remark 7.
Here, we let | S t | = n only for ease of presentation and better readability.
We note that this is not a restrictive condition.
We can show that | S t | = n can be relaxed to | S t | ≥ n, ∀ t ∈
[ T ] and the same convergence rate still holds.
In fact, our full proof in Appendix A.2 is for | S t | ≥ n .
D ISCUSSION
In light of above results, in what follows, we discuss several insights from the convergence analysis:
Convergence Rate: We show that the generalized FedAvg algorithm with two-sided learning rates can achieve a linear speedup, i.e., an O ( 1 √ mKT ) convergence rate with a proper choice of hyperparameters.
Thus, it works well in large FL systems, where massive parallelism can be leveraged to accelerate training.
The key challenge in convergence analysis stems from the different local loss functions (also called “model drift” in the literature) among workers due to the non-i.i.d.
datasets and local steps.
As shown above, we obtain a convergence bound for the generalized FedAvg method containing a vanishing term and a constant term (the constant term is similar to that of SGD).
In contrast, the constant term in SGD is only due to the local variance.
Note that, similar to SGD, the iterations do not diminish the constant term.
The local variance σ 2 L (randomness of stochastic gradients), global variability σ 2 G (non-i.i.d.
datasets), and the number of local steps K (ampliﬁcation factor) all contribute to the constant term, but the total global variability in K local steps dominates the term.
When the local learning rate η L is set to an inverse relationship with respect to the number of local steps K , the constant term is controllable.
An intuitive explanation is that the K small local steps can be approximately viewed as one large step in conventional SGD.
So this speedup and the more allowed local steps can be largely attributed to the two-sided learning rates setting.
Number of Local Steps: Besides the result that the maximum number of local steps is improved to K ≤ T/m , we also show that the local steps could help the convergence with the proper hyperparameter choices, which supports previous numerical results (McMahan et al., 2016; Stich, 2018;
Lin et al., 2018) and is veriﬁed in different models with different non-i.i.d.
degree datasets in Section 5.
However, there are other results showing the local steps slow down the convergence (Li et al., 2019b).
We believe that whether local steps help or hurt the convergence in FL worths further investigations.
Number of Workers: We show that the convergence rate improves substantially as the the number of workers in each communication round increases.
This is consistent with the results for i.i.d.
cases in Stich (2018).
For i.i.d.
datasets, more workers means more data samples and thus less variance and better performance.
For non-i.i.d.
datasets, having more workers implies that the distribution of the sampled workers is a better approximation for the distribution of all workers.
This is also empirically observed in Section 5.
On the other hand, the sampling strategy plays an important role in non-i.i.d. case as well.
Here, we adopt the uniform sampling (with/without replacement) to enlist workers to participate in FL.
Intuitively, the distribution of the sampled workers’ collective datasets under uniform sampling yields a good approximation of the overall data distribution in expectation.
Note that, in this paper, we assume that every worker is available to participate once being enlisted.
However, this may not always be feasible.
In practice, the workers need to be in certain states in order to be able to participate in FL (e.g., in charging or idle states, etc.
(Eichner et al., 2019)).
Therefore, care must be taken in sampling and enlisting workers in practice.
We believe that the joint design of sampling schemes and the generalized FedAvg algorithm will have a signiﬁcant impact on the convergence, which needs further investigations.
N UMERICAL R ESULTS
We perform extensive experiments to verify our theoretical results.
We use three models: logistic regression (LR), a fully-connected neural network with two hidden layers (2NN) and a convolution neural network (CNN) with the non-i.i.d.
version of MNIST (LeCun et al., 1998) and one ResNet model with CIFAR-10 (Krizhevsky et al., 2009).
Due to space limitation, we relegate some experimental results in the supplementary material.
[Figure]
[Figure] [Figure]
[Figure]
In this section, we elaborate the results under non-i.i.d.
MNIST datasets for the 2NN.
We distribute the MNIST dataset among m = 100 workers randomly and evenly in a digit-based manner such that the local dataset for each worker contains only a certain class of digits.
The number of digits in each worker’s dataset represents the non-i.i.d.
degree.
For digits _ 10 , each worker has training/testing samples with ten digits from 0 to 9 , which is essentially an i.i.d. case.
For digits _ 1 , each worker has samples only associated with one digit, which leads to highly non-i.i.d.
datasets among workers.
For partial worker participation, we set the number of workers n = 10 in each communication round.
Impact of non-i.i.d.
datasets: As shown in Figure 1(a), for the 2NN model with full worker participation, the top-row ﬁgures are for training loss versus communication round and the bottomrow are for test accuracy versus communication round.
We can see that the generalized FedAvg algorithm converges under non-i.i.d.
datasets with a proper learning rate choice in both cases.
For ﬁve digits ( digits _ 5 ) in each worker’s dataset with full (partial) worker participation in Figure 1(a), the generalized FedAvg algorithm achieves a convergence speed comparable to that of the i.i.d.
case ( digits _ 10 ).
Another key observation is that non-i.i.d.
datasets slow down the convergence under the same learning rate settings for both cases.
The higher the non-i.i.d.
degree, the slower the convergence speed.
As the non-i.i.d.
degree increases (from case digits _ 10 to case digits _ 1 ), it is obvious that the training loss is increasing and test accuracy is decreasing.
This trend is more obvious from the zigzagging curves for partial worker participation.
These two observations can also be veriﬁed for other models as shown in the supplementary material, which conﬁrms our theoretical analysis.
Impact of worker number: As shown in Figure 1(b), we compare the training loss and test accuracy between full worker participation n = 100 and partial worker participation n = 10 with the same hyper-parameters.
Compared with full worker participation, partial worker participation introduces another source of randomness, which leads to zigzagging convergence curves and slower convergence.
This problem is more prominent for highly non-i.i.d.
datasets.
For full worker participation, it can neutralize the the system heterogeneity in each communication round.
However, it might not be able to neutralize the gaps among different workers for partial worker participation.
That is, the datasets’ distribution does not approximate the overall distribution well.
Speciﬁcally, it is not unlikely that the digits in these datasets among all active workers are only a proper subset of the totaldigits in the original MNIST dataset, especially with highly non-i.i.d.
datasets.
This trend is also obvious for complex models and complicated datasets as shown in the supplementary material.
The sampling strategy here is random sampling with equal probability without replacement.
In practice, however, the actual sampling of the workers in FL could be more complex, which requires further investigations.
Impact of local steps: One open question of FL is that whether the local steps help the convergence or not.
In Figure 1(c)
[Table]
we show that the local steps could help the convergence for both full and partial worker participation.
These results verify our theoretical analysis.
However, Li et al.
(2019b) showed that the local steps may hurt the convergence, which was demonstrated under unbalanced non-i.i.d.
MNIST datasets.
We believe that this may be due to the combined effect of unbalanced datasets and local steps rather than just the use of local steps only.
[Table]
Comparison with SCAFFOLD: Lastly, we compare with the SCAFFOLD algorithm (Karimireddy et al., 2019) since it also achieves the same linear speedup effect under non-i.i.d.
datasets.
We compare communication rounds, total communication load, and estimated wall-clock time under the same settings to achieve certain test accuracy, and the results are reported in Table 2.
The non-i.i.d.
dataset is digits _ 2 and the i.i.d.
dataset is digits _ 10 .
The learning rates are η L = 0 .
1 , η = 1 . 0 , and number of local steps K is 5 epochs.
We set the target accuracy (cid:15) =
95% for MNIST and (cid:15) =
75% for CIFAR-10.
Note that the total training time contains two parts: i) the computation time for training the local model at each worker and ii) the communication time for information exchanges between the workers and the server.
We assume the bandwidth 20 MB/s for both uplink and downlink connections.
For MNIST datasets, we can see that our algorithm is similar to or outperforms SCAFFOLD.
This is because the numbers of communication rounds of both algorithms are relatively small for such simple tasks.
For non-i.i.d.
CIFAR-10, the SCAFFOLD algorithm takes slightly fewer number of communication rounds than our FedAvg algorithm to achieve (cid:15)
= 75% thanks to its variance reduction.
However, it takes more than 1.5 times of communication cost and wall-clock time compared to those of our FedAvg algorithm.
Due to space limitation, we relegate the results of time proportions for computation and communication to Appendix B (see Figure 7).
C ONCLUSIONS AND FUTURE WORK
In this paper, we analyzed the convergence of a generlized FedAvg algorithm with two-sided learning rates on non-i.i.d.
datasets for general non-convex optimization.
We proved that the generalized
FedAvg algorithm achieves a linear speedup for convergence under full and partial worker participation.
We showed that the local steps in FL could help the convergence and we improve the maximum number of local steps to T/m .
While our work sheds light on theoretical understanding of FL, it also opens the doors to many new interesting questions in FL, such as how to sample optimally in partial worker participation, and how to deal with active participant sets that are both time-varying and size-varying across communication rounds.
We hope that the insights and proof techniques in this paper can pave the way for many new research directions in the aforementioned areas.
A CKNOWLEDGEMENTS
This work is supported in part by NSF grants CAREER CNS-1943226, CIF-2110252, ECCS-1818791,
CCF-1934884, ONR grant ONR N00014-17-1-2417, and a Google Faculty Research Award.
A A PPENDIX I: P ROOFS
In this section, we give the proofs in detail for full and partial worker participation in Section A.1 and
Section A.2, respectively.
A. P ROOF OF T HEOREM
Theorem 1.
Let constant local and global learning rates η L and η be chosen as such that η L ≤ 1 8 LK and ηη L ≤ 1 KL .
Under Assumptions 1–3 and with full worker participation, the sequence of outputs { x k } generated by Algorithm 1 satisﬁes:
[Equation]
where Φ (cid:44) 1 c
[ Lηη L 2 m σ 2 L + 5 Kη 2 L L 2( σ 2 L + 6 Kσ 2 G )] , c is a constant, f 0 (cid:44) f
( x 0 ) , f ∗ (cid:44) f
( x ∗ ) and the expectation is over the local dataset samples among workers.
Proof.
For convenience, we deﬁne ¯ ∆ t (cid:44) 1 | S t | (cid:80) m i =1 ∆ i t .
Under full device participation (i.e., S t = [ m ] ), it is clear that ∆ t = 1 m (cid:80) mi =1 ∆ it =
¯∆ t .
Due to the smoothness in Assumption 1, taking expectation of f ( x t +1 ) over the randomness at communication round t , we have:
[Equation]
Note that the term A 1 in (1) can be bounded as follows:
[Equation]
m (cid:88) i =
K − 1 (cid:88) k
= ∇ F i ( x it,k ) −∇
F i ( x t ) (cid:13)(cid:13)(cid:13)(cid:13) − η L 2 Km 2 E t (cid:13)(cid:13)(cid:13)(cid:13) m (cid:88) i =
K − 1 (cid:88) k
= ∇ F i ( x it,k ) (cid:13)(cid:13)(cid:13)(cid:13) ( a 2) ≤ η L K 2 (cid:107)∇ f ( x t ) (cid:107) 2 + η L 2 m m (cid:88) i =
K − 1 (cid:88) k
=
E t (cid:107)∇ F i ( x it,k ) −∇ F i ( x t ) (cid:107) 2 − η L 2 Km 2 E t (cid:13)(cid:13)(cid:13)(cid:13) m (cid:88) i =
K − 1 (cid:88) k
= ∇ F i ( x it,k ) (cid:13)(cid:13)(cid:13)(cid:13) ( a 3) ≤ η L K 2 (cid:107)∇ f ( x t ) (cid:107) 2 + η L L 2 2 m m (cid:88) i =
K − 1 (cid:88) k
=
E t (cid:107) x it,k − x t (cid:107) 2 − η L 2 Km 2 E t (cid:13)(cid:13)(cid:13)(cid:13) m (cid:88) i =
K − 1 (cid:88) k
= ∇ F i ( x it,k ) (cid:13) (cid:13) (cid:13)(cid:13)
[Equation]
where ( a 1) follows from that (cid:10) x , y (cid:11)
= 12 [ (cid:107) x (cid:107) 2 + (cid:107) y (cid:107) 2 − (cid:107) x − y (cid:107) 2 ] for x = √ η L K ∇ f ( x t ) and y = − [Equation]
K [Equation] mi =1 (cid:80) K − 1 k =0 ( ∇ F i ( x it,k ) − ∇ F i ( x t )) , ( a 2) is due to that E [ (cid:107) x 1 + · · · + x n (cid:107) 2 ] ≤ n E [ (cid:107) x 1 (cid:107) 2 + · · · + (cid:107) x n (cid:107) 2 ] , ( a 3) is due to Assumption 1 and ( a 4) follows from Lemma 2.
The term A 2 in (1) can be bounded as:
[Equation]
E
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:88)
[Equation]
=
[Equation]
(cid:88)
[Equation]
=
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:88) [Equation] (3) where ( a 5) follows from the fact that E [ (cid:107) x (cid:107) 2 ] =
E [ (cid:107) x − E
[ x ] (cid:107) 2 ] + (cid:107) E
[ x ] (cid:107) 2 ] and ( a 6) is due to the bounded variance assumption in Assumption 3 and the fact that E [ (cid:107) x 1 + · · · + x n (cid:107) 2 ] =
E [ (cid:107) x 1 (cid:107) 2 + · · · + (cid:107) x n (cid:107) 2 ] if x (cid:48)
i s are independent with zero mean and E
[ g it,j ] = ∇ F i ( x it,j ) .
Substituting the inequalities in (2) of A 1 and (3) of A 2 into inequality (1), we have:
E t [ f ( x t +1 )]
≤ f ( x
[Equation]
where ( a 7) follows from ( ηη L 2 Km 2 − Lη 2 η [-
 L-]2 L 2 m 2 ) ≥ 0 if ηη L ≤ 1 KL , ( a 8) holds because there exists a constant c > 0 satisfying
[Equation]
Rearranging and summing from t = 0 , · · · , T − 1 , we have: T
[Equation]
(cid:88)
[Equation]
which implies,
[Equation]
where Φ = 1 c [ Lηη L 2 m σ 2 L + 5 Kη 2 L L 2( σ 2 L + 6 Kσ 2 G )] .
This completes the proof.
A. P ROOF OF T HEOREM
Theorem 2.
Under Assumptions 1–3 with partial worker participation, the sequence of outputs { x k } generated by Algorithm 1 with constant learning rates η and η L satisﬁes:
[Equation]
where f 0 = f ( x 0 ) , f ∗ = f ( x ∗ ) , and the expectation is over the local dataset samples among workers.
For sampling Strategy 1, let η and η L be chosen as such that η L ≤ 1 8 LK , ηη L KL < n − 1 n and 30 K 2 η [Equation] L L 2 − Lηη L n (90 K 2 L 2 η 2 L + 3) < 1 .
It then holds that:
[Equation]
For sampling Strategy 2, let η and η L be chosen as such that η L ≤ 1 8 LK , ηη L KL ≤ n ( m − 1) m ( n − 1) and 10 K
[Equation]
L
[Equation]
[Equation]
[Equation]
) < 1 .
It then holds that:
[Equation]
Proof.
Let ¯∆ t be deﬁned the same as in the proof of Theorem 1.
Under partial device participation, note that ¯∆ t (cid:54) =
∆ t (recall that ¯∆ t (cid:44) 1 m (cid:80) mi =1 ∆ it , ∆ t = 1 n (cid:80) i ∈
S t ∆ it , and | S t | = n ).
The randomness for partial worker participation contains two parts: the random sampling and the stochastic gradient.
We still use E t [ · ] to represent the expectation with respect to both types of randomness.
Due to the smoothness assumption in Assumption 1, taking expectation of f ( x t +1 ) over the randomness at communication round t:
[Equation]
The term A (cid:48) 1 in (4) can be bounded as follows: Since E S t [ A (cid:48) 1 ]
=
A 1 due to Lemma 1 for both sampling strategies, we have the same bound as in inequality 2 for A (cid:48) 1 :
[Equation]
For strategy 1: We can bound A (cid:48) 2 in (4) as follows.
Note S t is an index set (multiset) for independent sampling (equal probability) with replacement in which some elements may have the same value.
Suppose S t = { l 1 , . . .
, l n } .
[Equation]
[Equation]
where ( b 1) follows from the fact that E [ (cid:107) x (cid:107) 2 ] =
E [ (cid:107) x − E
[ x ] (cid:107) 2 ] + (cid:107) E
[ x ] (cid:107) 2 ] and ( b 2) is due to the bounded variance assumption 3 and E [ (cid:107) x 1 + · · · + x n (cid:107) 2 ] ≤ n E [ (cid:107) x 1 (cid:107) 2 + · · · + (cid:107) x n (cid:107) 2 ] .
By letting t i = (cid:80) K − 1 j =0 ∇ F i ( x it,j ) , we have: E t (cid:20)(cid:13) (cid:13)(cid:13)(cid:13) n (cid:88) z = K − 1 (cid:88)
j = ∇ F
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13) =
[Equation]
(cid:20)(cid:13)(cid:13)(cid:13)(cid:13) n (cid:88) z
= t l
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13) 2 (cid:21) =
E t (cid:20) n (cid:88)
z
= (cid:107) t l z (cid:107)
[Equation]
+ (cid:88) i (cid:54)
=
j ; l i
[Equation]
l j ∈ S t (cid:10) t l i , t l j (cid:11)(cid:21) ( b 3) =
E t (cid:20) n (cid:107) t l 1 (cid:107) 2 + n ( n − 1) (cid:10) t l 1 , t l 2 (cid:11)(cid:21) =
n m m (cid:88)
i =
(cid:107) t
i (cid:107) 2 + n ( n − 1) m(cid:88) i,j ∈ [ m ] (cid:10) t i , t j (cid:11)
[Equation]
where ( b 3) is due to the independent sampling with replacement
[Equation]
So we can bound A (cid:48) 2 as follows.
[Equation]
For t
i
[Equation]
we have: m
[Equation]
where ( b 4) is due to the fact that E [ (cid:107) x 1 + · · · + x n (cid:107) 2 ] ≤ n E [ (cid:107) x 1 (cid:107) 2 + · · · + (cid:107) x n (cid:107) 2 ] , Assumptionsand 1, and ( b 5) follows from Lemma 2.
Substituting the inequalities in ( 5) of A (cid:48) 1 and ( 6) of A (cid:48) 2 into inequality (4), we have:
[Table]
[Equation]
where ( b 6) follows from ( n − 1)
Lη 2 η 2 L 2 m 2 n − ηη
[Equation]
2 Km 2 ≤ 0 if ηη L KL ≤ n − 1 n is due to inequality (7) and ( b 7) holds since there exists a constant c [-≥ 
-]> 0 such that [ 12 − 15 K 2 η 2 L L 2 − Lηη L 2 n ( [Equation] 2 L 2 η
[Equation]
L +3)] > c >if 30 K 2 η 2 L L 2 − Lηη L n (90 K 2 L 2 η 2 L + 3) < 1 .
Note that the requirement of | S t | = n can be relaxed to | S t | ≥ n .
With p t ≥ n workers in t -th communication round, 8 is
[Equation]
That is, the same convergence rate can be guaranteed if at least n workers in each communication round (no need to be exactly n ).
Rearranging and summing from t = 0 , · · · , T − 1 , we have the convergence for partial device participation with sampling strategy 1 as follows:
[Equation]
where Φ = 1 c (cid:2) Lηη L 2 n σ 2 L + 3 Lηη L 2 n σ 2 G + ( 5 Kη 2 L L 2+ 15 Kηη 3 L L 3 2 n )( σ 2 L + 6 Kσ 2 G ) (cid:3) and c is a constant.
For strategy 2: Under the strategy of independent sampling with equal probability without replacement.
We bound A (cid:48) 2 as follows. A
[Equation]
[Table]
m (cid:88) i =
I { i ∈ S t }
K − 1 (cid:88)
j = ∇ F i ( x it,j ) (cid:13)(cid:13)(cid:13)(cid:13) 2 (cid:21) ( b 9) =
η 2 L nm E t (cid:20)
[Equation]
i =
K
[Equation]
(cid:88) j = (cid:13)(cid:13)(cid:13)(cid:13)
g
[Equation]
j
[Equation]
F i ( x
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13) 2 (cid:21) + η 2 L n 2 E t
(cid:20)(cid:13)(cid:13)(cid:13)(cid:13) m (cid:88) i =
I { i ∈ S t }
K − 1 (cid:88)
j = ∇ F i ( x it,j ) (cid:13)(cid:13)(cid:13)(cid:13) 2 (cid:21) ( b 10) ≤ Kη 2 L
[Equation]
(cid:13)(cid:13) (cid:13) (cid:13) m (cid:88) i =
[Equation]
{ i
[Equation]
t
[Equation]
K − 1 (cid:88)
j
=
[Equation] F i ( [Equation] (cid:13)(cid:13) (cid:13) (cid:13) , (9) where ( b 9) is due to the fact that E [ (cid:107) x 1 + · · · + x n (cid:107) 2 ]
=
E [ (cid:107) x 1 (cid:107) 2 + · · · + (cid:107) x n (cid:107) 2 ] if x (cid:48)
i s are independent with zero mean, x i
=
g
it,j − ∇ F i ( x it,j ) is independent random variable with mean zero, and P { i ∈ S t }
= nm .
( b 10) is due to bounded variance assumption in Assumption
Substituting the inequalities in (5) of A (cid:48) 1 and (9) of A (cid:48) 2 into inequality (4), we have:
[Equation]
Then we bound A (cid:48) 3 as follows.
By letting t i = (cid:80) K − 1 j =0 ∇ F i ( x it,j ) , we have: m (cid:88) i = E [Equation] mK
[Equation]
It then follows that (cid:107) [Equation] [Equation] (cid:54)
[Equation]
( b 11)
[Equation]
(cid:88)
i
[Equation]
m ]
[Equation]
i (cid:54)
[Equation]
j (cid:107) t
i
[Equation]
j (cid:107)
[Equation]
(cid:107) m (cid:88) i =
P { i ∈ S t } t i
[Equation]
i
[Equation]
m ]
P { i
[Equation]
t
[Equation]
t i (cid:107)
[Equation]
(cid:88) i (cid:54) =
j
P { i
[Equation]
j ∈ S t } <
t i [Equation] t j > ( b 12) = n m (cid:88) i ∈ [ m ] (cid:107) t i (cid:107)
[Equation]
+ n ( n − 1) m ( m − 1) (cid:88) i (cid:54) =
j < t i
[Equation]
t j >
[Equation]
=
[Equation]
where ( b 11) and ( b 13) are due to the fact that (cid:10) x , y (cid:11) =
12 [ (cid:107) x (cid:107) 2 + (cid:107) y (cid:107) 2 −(cid:107)
x − y (cid:107) 2 ] ≤ 12 [ (cid:107) x (cid:107) 2 + (cid:107)
y (cid:107) 2 ] , ( b 12) follows from the fact that P { i ∈ S t } = nm and P { i, j ∈ S t } = n ( n − 1) m ( m − 1) .
Therefore, we have
[Equation]
[Equation] , where ( b 14) follows from the fact that (cid:107) (cid:80) i ∈ [ m ] t i (cid:107) {+
+}
= (cid:80) i ∈ [ m ] m (cid:107) t
i (cid:107) 2 − 12 (cid:80) i (cid:54) =
j (cid:107) t i − t j (cid:107) 2 , and ( b 15) is due to the fact that ( ηη L 2 Km[-2 − Lη 2 η 2 L ( n − 1) 2 mn ( m − 1) ) ≥ 
-]
[Equation]
Then we have
E
[Equation]
where ( b 16) holds because there exists a constant c > 0 satisfying ( 12 − 5 K 2 η 2 L L 2 −
Lηη L m − n 2 n ( m − 1) (90 K 2 η 2 L L 2 + 3)) >
c > 0 if 10 K 2 η 2 L L 2 − Lηη L m − n n ( m − 1) (90 K 2 η 2 L L 2 + 3) < 1 .
Note that the requirement of | S t | = n can be relaxed to | S t | ≥ n .
With p t ≥ n workers in t -th communication round, 10 is
[Equation]
That is, the same convergence rate can be guaranteed if at least n workers in each communication round (no need to be exactly n ).
Rearranging and summing from t = 0 , · · · , T − 1 , we have the convergence for partial device participation with sampling strategy 2 as follows:
[Equation]
where Φ = 1 c (cid:2) Lηη L 2 n σ 2 L +3 Lηη L m − n 2 n ( m − 1) σ 2 G +( 5 Kη 2 L L 2+15 Kηη 3 L L 3 m − n 2 n ( m − 1) )
( σ 2 L +6 Kσ 2 G ) (cid:3) and c is a constant.
This completes the proof.
A.2. K EY L EMMAS
Lemma 1 (Unbiased Sampling) .
For strategies 1 and 2, the estimator ∆ t is unbiased, i.e.,
[Equation]
Proof of Lemma 1.
Let S t = { t 1 , · · · , t n } with size n .
Both for sampling strategies 1 and 2, each sampling distribution is identical.
Then we have:
[Equation]
A. A UXILIARY L EMMAS
Lemma 2 (Lemma 4 in Reddi et al.
For any step-size satisfying η L ≤ 1 8 LK , we can have the following results:
[Equation]
Proof.
In order for this paper to be self-contained, we restate the proof of Lemma 4 in (Reddi et al., 2020) here [Equation]
For any worker i ∈ [ m ] and k ∈ [ K ] [Equation] we have:
[Equation]
=
[Equation]
+
[Equation]
≤
[Equation]
Unrolling the recursion, we get:
[Equation]
This completes the proof
[Equation]
B A PPENDIX II: E XPERIMENTS
We provide the full detail of the experiments.
We uses non-i.i.d.
versions for MNIST and CIFAR-10, which are described as follows:
B. MNIST
We study image classiﬁcation of handwritten digits 0-9 in MNIST and modify the MNIST dataset to a non-i.i.d.
version.
To impose statistical heterogeneity, we split the data based on the digits ( p ) they contain in their dataset.
We distribute the data to m = 100 workers such that each worker contains only a certain class of digits with the same number of training/test samples.
For example, for p = 1 , each worker only has training/testing samples with one digit, which causes heterogeneity among different workers.
For p = 10 , each worker has samples with 10 digits, which is essentially i.i.d. case.
In this way, we can use the digits in worker’s local dataset to represent the non-i.i.d.
degree qualitatively.
In each communication round, 100 workers run K epochs locally in parallel and then the server samples n workers for aggregation and update.
We make a grid-search experiments for the hyper-parameters as shown in Table 3.
[Table]
We run three models: multinomial logistic regression, fully-connected network with two hidden layers (2NN) (two 200 neurons hidden layers with ReLU followed by an output layer), convolutional neural network (CNN), as shown in Table 4.
The results are shown in Figures 2, 3 and 4.
[Table]
B. CIFAR-
Unless stated otherwise, we use the following default parameter setting: the server learning rate and client learning rate are set to η = 1 .
and η L = 0 .
1 , respectively.
The local epochs is set to K = 10 .
The total number of clients is set to 100, and the clients partition number is set to n = 10 .
We use the same strategy to distribute the data over clients as suggested in McMahan et al.
For the i.i.d.
setting, we evenly partition all the training data among all clients, i.e., each client observes 500 data; for the non-i.i.d.
setting, we ﬁrst sort the training data by label, then divide all the training data into 200 shards of size 250, and randomly assign two shards to each client.
For the CIFAR-10 dataset, we train our classiﬁer with the ResNet model.
The results are shown in Figure 5 and Figure 6.
B. D ISCUSSION
Impact of non-i.i.d.
datasets: Figure 2 shows the results of training loss (top) and test accuracy (bottom) for three models under different non-i.i.d.
datasets with full and partial worker participation [Figure] [Figure] u r a c
[Figure]
on MNIST.
We can see that the FedAvg algorithm converges under non-i.i [Figure]
d. datasets with a proper learning rate choice in these cases.
We believe that the major challenge in FL is the non-i.i [Figure]
d. datasets.
For these datasets with a lower degree of non-i.i.d., the FedAvg algorithm can achieve a good result compared with the i.i.d. case.
For example, when the local dataset in each worker has ﬁve digits ( p = 5 ) with full (partial) worker participation, the FedAvg algorithm achieves a convergence speed comparable with that of the i.i.d.
case ( p = 10 ).
This result can be observed in Figure 2 for all three models.
As the degree of non-i.i.d.
datasets increases, its negative impact on the convergence is becoming more obvious.
The higher the degree of non-i.i.d., the slower the convergence speed.
As the non-i.i.d.
degree increases (from case p = 10 to case p = 1 ), it is obvious that the training loss is increasing and test accuracy is decreasing.
For these with high degree of non-i.i.d., the convergence curves oscillate and are highly unstable.
This trend is more obvious for complex models such for
CNN in Figure 2(c).
Impact of worker number: For full worker participation, the server can have an accurate estimation of the system heterogeneity after receiving the updates for all workers and neutralize this heterogeneity in each communication round.
However, partial worker participation introduces another source of randomness, which leads to zigzagging convergence curves and slower convergence. In each [Figure]
[Figure] [Figure] communication round, the server can only receive a subset of workers based on the sampling strategy.
So the server could only have a coarse estimation of the system heterogeneity and might not be able to neutralize the heterogeneity among different workers for partial worker participation.
This problem is more prominent for highly non-i [Figure] i [Figure]
d. datasets.
It is not unlikely that the digits in these datasets among all active workers are only a proper subset of the total 10 digits in the original MNIST dataset, especially with highly non-i [Figure] i [Figure]
d. datasets.
For example, for p = 1 with 10 workers in each communication round, it is highly likely that the datasets formed by these ten workers only includes certain small number of digits (say, 4 or 5 ) rather than total 10 digits.
But for p = 5 , it is the opposite, that is, the digits in these datasets among these 10 workers are highly likely to be 10 .
So in each communication round, the server can mitigate system heterogeneity since it covers the training samples with all 10 digits.
This trend is more obvious for complex models and datasets given the dramatic drop of test accuracy in the result of CIFAR-10 in Figure 5.
The sample strategy here is random sampling with equal probability without replacement.
In practice, the workers need to be in certain states in order to be able to participate in FL (e.g., in charging or idle states, etc.(Eichner et al., 2019)).
Therefore, care must be taken in sampling and enlisting workers in practice.
We believe that the joint design of sampling schemes, number of workers and the FedAvg algorithm will have a signiﬁcant impact on the convergence, which needs further investigations.
Impact of local steps: Figure 3 and Figure 4 shows the results of training loss (top) and test accuracy (bottom) for three models under different local steps with full and partial worker participation respectively.
Figure 6 shows the impact of local steps in CIFAR-10.
One open question of FL is that whether the local steps help the convergence or not.
Li et al.
(2019b) showed a convergence rate O ( KT ) , i.e., the local steps may hurt the convergence for full and partial worker participation.
In this two ﬁgures, we can see that local steps could help the convergence for both full and partial worker participation.
However, it only has a slight effect on the convergence compared to the effects of non-i.i.d.
datasets and number of workers. Comparison with SCAFFOLD: We compare SCAFFOLD (Karimireddy et al., 2019) with the generalized FedAVg algorithm in this paper in terms of communication rounds, total communication overloads and estimated wall-clock time to achieve certain test accuracy in Table 2.
We run the experiments using the same GPU (NVIDIA V100) to ensure the same conditions.
Here, we give a speciﬁc comparison for these two algorithms under exact condition.
Note that we divide the total training time to two parts: the computation time when the worker trains the local model and the communication time when information exchanges between the worker and server.
We only compare the computation time and communication time with a ﬁxed bandwidth 20 MB/s for both uploading and downloading connections.
As shown in Figure 7, to achieve (cid:15) = 75% , SCAFFOLD performs less communication round due to the variance reduction techniques.
That is, it spends less time on computation.
However, it needs to communicates as twice as the FedAvg since the control variate to perform variance reduction in each worker needs to update in each round.
In this way, the communication time would be largely prolonged. Time
[Equation]
This paper
SCAFFOLD
Computation time Communication time