C LIENT S ELECTION IN F EDERATED L EARNING : C ONVERGENCE A NALYSIS AND P OWER - OF -C HOICE S ELECTION S TRATEGIES
Anonymous authors
Paper under double-blind review
A BSTRACT
Federated learning is a distributed optimization paradigm that enables a large number of resource-limited client nodes to cooperatively train a model without data sharing.
Several works have analyzed the convergence of federated learning by accounting of data heterogeneity, communication and computation limitations, and partial client participation.
However, they assume unbiased client participation, where clients are selected at random or in proportion of their data sizes.
In this paper, we present the ﬁrst convergence analysis of federated optimization for biased client selection strategies, and quantify how the selection skew affects convergence speed.
We reveal that biasing client selection towards clients with higher local loss achieves faster error convergence.
Using this insight, we propose P OWER - OF - CHOICE , a communication- and computation-efﬁcient client selection framework that can ﬂexibly span the trade-off between convergence speed and solution bias.
We also propose an extension of P OWER - OF - CHOICE that is able to maintain convergence speed improvement while diminishing the selection skew.
Our experiments demonstrate that P OWER - OF - CHOICE strategies can converge up to 3 × faster and give 10% higher test accuracy than the baseline random selection.
I NTRODUCTION
Until recently, machine learning models were largely trained in the data center setting (Dean et al., 2012) using powerful computing nodes, fast inter-node communication links, and large centrally available training datasets.
The future of machine learning lies in moving both data collection as well as model training to the edge.
The emerging paradigm of federated learning (McMahan et al., 2017;
Kairouz et al., 2019; Bonawitz et al., 2019) considers a large number of resource-constrained mobile devices that collect training data from their environment.
Due to limited communication capabilities and privacy concerns, these data cannot be directly sent over to the cloud.
Instead, the nodes locally perform a few iterations of training using local-update stochastic gradient descent (SGD) (Yu et al., 2018; Stich, 2018; Wang & Joshi, 2018; 2019), and only send model updates periodically to the aggregating cloud server.
Besides communication limitations, the key scalability challenge faced by the federated learning framework is that the client nodes can have highly heterogeneous local datasets and computation speeds.
The effect of data heterogeneity on the convergence of local-update
SGD is analyzed in several recent works (Reddi et al., 2020; Haddadpour & Mahdavi, 2019; Khaled et al., 2020; Stich & Karimireddy, 2019;
Woodworth et al., 2020; Koloskova et al., 2020; Huo et al., 2020; Zhang et al., 2020; Pathak & Wainwright, 2020; Malinovsky et al., 2020; Sahu et al., 2019) and methods to overcome the adverse effects of data and computational heterogeneity are proposed in (Sahu et al., 2019; Wang et al., 2020; Karimireddy et al., 2019), among others.
Partial Client Participation.
Most of the recent works described above assume full client participation, that is, all nodes participate in every training round.
In practice, only a small fraction of client nodes participate in each training round, which can exacerbate the adverse effects of data heterogeneity.
While some existing convergence guarantees for full client participation and methods to tackle heterogeneity can be generalized to partial client participation (Li et al., 2020), these generalizations are limited to unbiased client participation, where each client’s contribution to the expected global objective optimized in each round is proportional to its dataset size.
In Ruan et al.
, the authors analyze the convergence with ﬂexible device participation, where devices can freely join or leave the training process or send incomplete updates to the server.
However, adaptive client selection that is cognizant of the training progress at each client has not been understood yet.
It is important to analyze and understand biased client selection strategies since they can sharply accelerate error convergence, and hence boost communication efﬁciency in heterogeneous environments by preferentially selecting clients with higher local loss values, as we show in our paper.
This idea has been explored in recent empirical studies (Goetz et al., 2019; Laguel et al., 2020; Ribero &
Vikalo, 2020).
Nishio & Yonetani (2019) proposed grouping clients based on hardware and wireless resources in order to save communication resources.
Goetz et al.
(2019) (which we include as a benchmark in our experiments) proposed client selection with local loss, and Ribero & Vikalo (2020) proposed utilizing the progression of clients’ weights.
But these schemes are limited to empirical demonstration without a rigorous analysis of how selection skew affects convergence speed.
Another relevant line of work (Jiang et al., 2019; Katharopoulos & Fleuret, 2018; Shah et al., 2020;
Salehi et al., 2018) employs biased selection or importance sampling of data to speed-up convergence of classic centralized SGD – they propose preferentially selecting samples with highest loss or highest gradient norm to perform the next SGD iteration.
In contrast, Shah et al.
(2020) proposes biased selection of lower loss samples to improve robustness to outliers.
Generalizing such strategies to the federated learning setting is a non-trivial and open problem because of the large-scale distributed and heterogeneous nature of the training data.
Our Contributions.
In this paper, we present the ﬁrst (to the best of our knowledge) convergence analysis of federated learning with biased client selection that is cognizant of the training progress at each client.
We discover that biasing the client selection towards clients with higher local losses increases the rate of convergence compared to unbiased client selection.
Using this insight, we propose the P OWER - OF - CHOICE client selection strategy and show by extensive experiments that
P OWER - OF - CHOICE yields up to 3 × faster convergence with 10% higher test performance than the standard federated averaging with random selection.
P OWER - OF - CHOICE is designed to incur minimal communication and computation overhead, enhancing resource efﬁciency in federated learning.
In fact, we show that even with 3 × less clients participating in each round as compared to random selection, P OWER - OF - CHOICE gives 2 × faster convergence and 5% higher test accuracy.
P ROBLEM F ORMULATION
Consider a cross-device federated learning setup with total K clients, where client k has a local dataset B k consisting |B k | =
D k data samples.
The clients are connected via a central aggregating server, and seek to collectively ﬁnd the model parameter w that minimizes the empirical risk:
[Equation]
where f ( w , ξ ) is the composite loss function for sample ξ and parameter vector w .
The term p k
= D k / (cid:80)
Kk =1 D k is the fraction of data at the k -th client, and F k ( w ) =
|B k | (cid:80)
ξ ∈B k f ( w , ξ ) is the local objective function of client k .
In federated learning, the vectors w ∗ , and w ∗ k for k = 1 , . . .
, K that minimize F ( w ) and F k ( w ) respectively can be very different from each other.
We deﬁne
F ∗ = min w F ( w ) =
F ( w ∗ ) and F ∗ k = min w F k ( w ) =
F k ( w ∗ k ) .
Federated Averaging with Partial Client Participation.
The most common algorithm to solve (1) is federated averaging (FedAvg) proposed in McMahan et al.
The algorithm divides the training into communication rounds.
At each round, to save communication cost at the central server, the global server only selects a fraction C of m =
CK clients to participate in the training.
Each selected/active client performs τ iterations of local SGD (Stich, 2018; Wang & Joshi, 2018; Yu et al., 2018) and sends its locally updated model back to the server.
Then, the server updates the global model using the local models and broadcasts the global model to a new set of active clients.
Formally, we index the local SGD iterations with t ≥ 0 .
The set of active clients at iteration t is denoted by S ( t ) .
Since active clients performs τ steps of local update, the active set S ( t ) also remains constant for every τ iterations.
That is, if ( t + 1) mod τ = 0 , then S ( t +1) =
S ( t +2) = · · · = S ( t + τ ) .
[Table]
</latexit>
[Figure]
Accordingly, the update rule of FedAvg can be written as follows:
[Equation]
where w ( t +1)
k denotes the local model parameters of client k at iteration t , η t is the learning rate, and g k ( w ( t ) k , ξ ( t ) k ) = 1 b (cid:80) ξ ∈ ξ ( t ) k ∇ f ( w ( t ) k , ξ ) is the stochastic gradient over mini-batch ξ ( t ) k of size b that is randomly sampled from client k ’s local dataset B k .
Moreover, w ( t +1) denotes the global model at server.
Although w ( t ) is only updated after every τ iterations, for the purpose of convergence analysis we consider a virtual sequence of w ( t ) that is updated at each iteration as follows
[Equation]
with g ( t ) = 1 m (cid:80) k ∈S ( t ) g k ( w ( t ) k , ξ ( t ) k ) .
Note that in (2) and (3) we do not weight the client models by their dataset fractions p k because p k is considered in the client selection scheme used to decide the set S ( t ) .
Our convergence analysis can be generalized to when the global model is a weighted average instead of a simple average of client models, and we show in Appendix E that our convergence analysis also covers the sampling uniformly at random without replacement scheme proposed by Li et al.
The set S ( t ) can be sampled either with or without replacement.
For sampling with replacement, we assume that multiple copies of the same client in the set S ( t ) behave as different clients, that is, they perform local updates independently.
Client Selection Strategy.
To guarantee FedAvg converges to the stationary points of the objective function (1), most current analysis frameworks (Li et al., 2020; Karimireddy et al., 2019; Wang et al., 2020) consider a strategy that selects the set S ( t ) by sampling m clients at random (with replacement) such that client k is selected with probability p k , the fraction of data at that client.
This sampling scheme is unbiased since it ensures that in expectation, the update rule (3) is the same as full client participation.
Hence, it enjoys the same convergence properties as local-update SGD methods (Stich, 2018; Wang & Joshi, 2018).
We denote this unbiased random client selection strategy as π rand .
In this paper, we consider a class of biased client selection strategies that is cognizant of the global training progress which (to the best of our knowledge) has not been worked on before.
Note that for any aggregation scheme and sampling scheme with partial client participation, if the expectation over the sampling scheme for the update rule of the global model is equal to the case of the update rule for full client participation, we distinguish this as an unbiased client participation scheme.
For example in
Horváth & Richtárik (2020), even with a biased sampling scheme, with the normalizing aggregation the update rule is unbiased.
Henceforth, we state that our paper encompasses both biased and unbiased update rules.
In the two-client example in Figure 1, we set S ( t +1) =
arg max k ∈ [ K ] F k ( w ( t ) ) , a single client with the highest local loss at the current global model.
In this toy example, the selection strategy cannot guarantee the updates (3) equals to the full client participation case in expectation.
Nevertheless, it gives faster convergence to the global minimum than the random one.
Motivated by this observation, we deﬁne a client selection strategy π as a function that maps the current global model w to a selected set of clients S ( π, w ) .
C ONVERGENCE A NALYSIS
In this section we analyze the convergence of federated averaging with partial device participation for any client selection strategy π as deﬁned above.
This analysis reveals that biased client selection can give faster convergence, albeit at the risk of having a non-vanishing gap between the true optimum w ∗ =
arg min F ( w ) and lim t →∞ w ( t ) .
We use this insight in Section 4 to design client selection strategies that strike a balance between convergence speed and bias.
3. A SSUMPTIONS AND D EFINITIONS
First we introduce the assumptions and deﬁnitions utilized for our convergence analysis.
Assumption 3.1.
F 1 , ..., F k are all L − smooth, i.e., for all v and w , F k ( v ) ≤ F k ( w )
+ ( v − w ) T ∇ F k ( w ) + L 2 (cid:107) v − w (cid:107) 22 .
Assumption 3.2.
F 1 , ..., F k are all µ − strongly convex, i.e., for all v and w , F k ( v ) ≥ F k ( w )
+ ( v − w ) T ∇ F k ( w ) + µ 2 (cid:107) v − w (cid:107) 22 .
Assumption 3.3.
For the mini-batch ξ k uniformly sampled at random from B k from user k , the resulting stochastic gradient is unbiased, that is, E [ g k ( w k , ξ k )]
= ∇ F k ( w k ) .
Also, the variance of stochastic gradients is bounded: E (cid:107) g k ( w k , ξ k ) − ∇ F k ( w k ) (cid:107) 2 ≤ σ 2 for all k
= 1 , ..., K .
Assumption 3.4.
The stochastic gradient’s expected squared norm is uniformly bounded, i.e., E (cid:107) g k
( w k , ξ k ) (cid:107) 2 ≤ G 2 for k = 1 , ..., K .
The above assumptions are common in related literature, see (Stich, 2018;
Basu et al., 2019; Li et al., 2020; Ruan et al., 2020).
Next, we introduce two metrics, the local-global objective gap and the selection skew , which feature prominently in the convergence analysis presented in Theorem 3.1.
Deﬁnition 3.1 ( Local-Global Objective Gap ) .
For the global optimum w ∗ = arg min w F ( w ) and local optimum w ∗ k = arg
min w F k ( w ) we deﬁne the local-global objective gap as
[Equation]
Note that Γ is an inherent property of the local and global objective functions, and it is independent of the client selection strategy.
This deﬁnition was introduced in previous literature by Li et al.
A larger Γ implies higher data heterogeneity.
If Γ =
then it implies that the local and global optimal values are consistent, and there is no solution bias due to the client selection strategy (see Theorem 3.1).
Next, we deﬁne another metric called selection skew, which captures the effect of the client selection strategy on the local-global objective gap.
Deﬁnition 3.2 ( Selection Skew ) .
For any k ∈ S ( π, w ) we deﬁne,
[Equation]
which reﬂects the skew of a client selection strategy π .
The ﬁrst w in ρ ( S ( π, w ) , w (cid:48) ) is the parameter vector that governs the client selection and w (cid:48) is the point at which F k and F in the numerator and denominator respectively are evaluated.
Note, E S ( π, w ) [ · ] is the expectation over the randomness from the selection strategy π , since there can be multiple sets S that π can map from a speciﬁc w .
Since ρ ( S ( π, w ) , w (cid:48) ) is a function of versions of the global model w and w (cid:48) , which change during training, we deﬁne two related metrics that are independent of w and w (cid:48) .
These metrics enable us to obtain a conservative error bound in the convergence analysis.
[Equation]
where w ∗ = arg min w F ( w ) .
From (6) , we have ρ ≤ (cid:101) ρ for any client selection strategy π .
Effect of the Client Selection Strategy on ρ and (cid:101) ρ .
For the unbiased client selection strategy π rand we have ρ ( S ( π rand , w ) , w (cid:48) )
= 1 for all w and w (cid:48) since the numerator and denominator of (5) become equal, and ρ = (cid:101) ρ = 1 .
For a client selection strategy π that chooses clients with higher F k ( w ) more often, ρ and (cid:101) ρ will be larger (and ≥ 1 ).
In the convergence analysis we show that a larger ρ implies faster convergence, albeit with a potential error gap, which is proportional to ( (cid:101) ρ/ρ − 1) .
Motivated by this, in Section 4 we present an adaptive client selection strategy that prefers selecting clients with higher loss F k ( w ) and achieves faster convergence speed with low solution bias.
3. M AIN C ONVERGENCE R ESULT
Here, we present the convergence results for any client selection strategy π for federated averaging with partial device participation in terms of local-global objective gap Γ , and selection skew ρ, (cid:101) ρ .
Theorem 3.1 (Convergence with Decaying Learning Rate) .
Under Assumptions 3.1 to 3.4, for learning rate η t = 1 µ ( t + γ ) with γ = 4 Lµ , and any client selection strategy π , the error after T iterations of federated averaging with partial device participation satisﬁes
[Equation]
To the best of our knowledge, Theorem 3.1 provides the ﬁrst convergence analysis of federated averaging with a biased client selection strategy π .
We also show the results for ﬁxed learning rate in Appendix A.
The proof is presented in Appendix C.
The ﬁrst part of our proof follows techniques presented by Li et al.
Then we introduce the novel concept of selection skew to the proof, and analyze the effect of biased client selection strategies that has not been seen before in previous literature.
We highlight that our convergence result is a general analysis that is applicable for any selection strategy π that is cognizant of the training progress.
In the following paragraphs, we discuss the effects of the two terms in (7) in detail.
Large ρ and Faster Convergence.
A key insight from Theorem 3.1 is that a larger selection skew
ρ results in faster convergence at the rate O ( 1 Tρ ) .
Note that since we obtain ρ (deﬁned in (6)) by taking a minimum of the selection skew ρ ( S ( π, w ) , w (cid:48) ) over w , w (cid:48) , this is a conservative bound on the true convergence rate.
In practice, since the selection skew ρ ( S ( π, w ) , w (cid:48) ) changes during training depending on the current global model w and the local models w (cid:48) , the true convergence rate can be improved by a factor larger than and at least equal to ρ .
Non-vanishing Bias Term.
The second term Q ( ρ, (cid:101) ρ ) =
8 L Γ 3 µ (cid:16) (cid:101) ρρ − 1 (cid:17) in (7) denotes the solution bias, which is dependent on the selection strategy.
By the deﬁnitions of ρ and (cid:101) ρ , it follows that (cid:101) ρ ≥ ρ , which implies that Q ( ρ, (cid:101) ρ ) ≥ 0 .
For an unbiased selection strategy, we have ρ = (cid:101) ρ
= 1 , Q ( ρ, (cid:101) ρ ) = 0 , and hence (7) recovers previous bound for unbiased selection strategy as (Li et al., 2020).
For ρ > 1 , while we gain faster convergence rate by a factor of ρ , we cannot guarantee Q ( ρ, (cid:101) ρ ) = 0 .
Thus, there is a trade-off between the convergence speed and the solution bias.
Later in the experimental results, we show that even with biased selection strategies, the term (cid:101) ρρ − 1 in Q ( ρ, (cid:101) ρ ) can be close to 0 , and hence Q ( ρ, (cid:101) ρ ) has a negligible effect on the ﬁnal error ﬂoor.
P ROPOSED P OWER - OF - CHOICE C LIENT S ELECTION S TRATEGY
From (5) and (6) we discover that a selection strategy π that prefers clients with larger F k ( w ) − F ∗ k will result in a larger ρ , yielding faster convergence.
Using this insight, a naive client selection strategy can be choosing the clients with highest local loss F k ( w ) .
However, a larger selection skew ρ may result in a larger ρ/ (cid:101) ρ , i.e., a larger non-vanishing error term.
This naive selection strategy has another drawback – to ﬁnd the current local loss F k ( w ) , it requires sending the current global model to all K clients and having them evaluate F k and sending it back.
This additional communication and computation cost can be prohibitively high because the number of clients K is typically very large, and these clients have limited communication and computation capabilities.
In this section, we use these insights regarding the trade-off between convergence speed, solution bias and communication/computation overhead to propose the P OWER - OF - CHOICE client selection strategy.
P OWER - OF - CHOICE is based on the power of d choices load balancing strategy (Mitzenmacher, 1996), which is extensively used in queueing systems.
In the P OWER - OF - CHOICE client selection strategy (denoted by π pow-d ), the central server chooses the active client set S ( t ) as follows:
Sample the Candidate Client Set.
The central server samples a candidate set A of d ( m ≤ d ≤ K ) clients without replacement such that client k is chosen with probability p k , the fraction of data at the k -th client for k = 1 , . . .
K .
Estimate Local Losses.
The server sends the current global model w ( t ) to the clients in set A , and these clients compute and send back to the central server their local loss F k ( w ( t ) ) .
Select Highest Loss Clients. From the candidate set A , the central server constructs the active client set S ( t ) by selecting m = max( CK, 1) clients with the largest values F k ( w ) , with ties broken at random. These S ( t ) clients participate in the training during the next round, consisting of iterations t + 1 , t + 2 , ...
t + τ .
Variations of π pow d .
The three steps of π pow-d can be ﬂexibly modiﬁed to take into account practical considerations.
For example, intermittent client availability can be accounted for in step 1 by constructing set
A only from the set of available clients in that round.
We demonstrate the performance of π pow-d with intermittent client availability in Appendix G.3.
The local computation cost and server-client communication cost in step 2 can be reduced or eliminated by the following proposed variants of π pow-d (see Appendix F for their pseudo-codes).
• Computation-efﬁcient Variant π cpow-d : To save local computation cost, instead of evaluating the F k ( w ) by going through the entire local dataset B k , we use an estimate (cid:80) ξ ∈ (cid:98) ξ k f ( w , ξ ) /
| (cid:98) ξ k | , where (cid:98) ξ k is the mini-batch of b samples sampled uniformly at random from B k .
• Communication- and Computation-efﬁcient Variant π rpow-d : To save both local computation and communication cost, the selected clients for each round sends their accumulated averaged loss over local iterations, i.e.,
[Equation]
(cid:80)
[Equation]
f ( w ( l ) k , ξ ) when they send their local models to the server.
The server uses the latest received value from each client as a proxy for F k ( w ) to select the clients.
For the clients that have not been selected yet, the latest value is set to ∞ .
• Adaptive Selection Skew Variant
π adapow-d : To minimize the non-vanishing bias term in Theorem 3.1 while simultaneously gaining the beneﬁt of convergence speed from ρ , we gradually reduce d until d = m 1 .
This enables convergence speed up in the initial training phase, while eventually diminishing the non-vanishing bias term when d = m .
Which d to start with and how gradually we decrease d to m is ﬂexible, analogous to setting the environment and
hyper-parameters.
Selection Skew of P OWER - OF - CHOICE Strategy.
The size d of the candidate client set A is an important parameter which controls the trade-off between convergence speed and solution bias.
With d = m we have random sampling without replacement in proportion of p k .
As d increases, the selection skew ρ increases, giving faster error convergence at the risk of a higher error ﬂoor.
However, note that the convergence analysis replaces ρ ( w , w (cid:48) ) with ρ to get a conservative error bound. In d makes sampling
[Figure] convergence speed increases, solution bias also increases for , but to eliminate this bias while gaining nearly identical speed to π practice, the convergence speed and the solution bias is dictated by ρ ( w ( τ (cid:98) t/τ (cid:99) ) , w ( t ) ) which changes during training.
With π pow-d which is biased towards higher local losses, we expect the selection skew ρ ( w , w (cid:48) ) to reduce through the course of training.
We conjecture that this is why π pow-d gives faster convergence as well as little or no solution bias in our experiments presented in Section 5.
E XPERIMENTAL R ESULTS
We evaluate our proposed π pow-d and its practical variants π cpow-d , π rpow-d , and π adapow-d by three sets of experiments: (1) quadratic optimization, (2) logistic regression on a synthetic federated dataset,
Synthetic (1,1) (Sahu et al., 2019), and (3) DNN trained on a non-iid partitioned FMNIST dataset (Xiao et al., 2017).
We also benchmark the selection strategy proposed by Goetz et al.
(2019), active federated learning, denoted as π aﬂ .
Details of the experimental setup are provided in Appendix F, and the code for all experiments are shared in the supplementary material.
To validate consistency in our results, we present additional experiments with DNN trained on a non-iid partitioned EMNIST (Cohen et al., 2017) dataset sorted by digits with K = 500 clients. We present the results in Appendix G.4.
[Figure]
Quadratic and Synthetic Simulation Results.
In Figure 2(a), even with few clients ( K = 30 ), π pow-d converges faster than π rand with nearly negligible solution bias for small d .
The convergence speed increases with the increase in d , at the cost of higher error ﬂoor due to the solution bias.
For K = 100 in Figure 2(b), π pow-d shows convergence speed-up as with K = 30 , but the bias is smaller.
Figure 3 shows the theoretical values ρ and (cid:101) ρ/ρ which represents the convergence speed and the solution bias respectively in our convergence analysis.
Compared to π rand , π pow-d has higher ρ for all d implying higher convergence speed than π rand .
By varying d we can span different points on the trade-off between the convergence speed and bias.
For d = 15 and K = 100 , (cid:101) ρ/ρ of π pow-d and π rand are approximately identical, but π pow-d has higher ρ , implying that π pow-d can yield higher convergence speed with negligible solution bias.
In Appendix G.1, we present the clients’ selected frequency ratio for π pow-d and π rand which gives novel insights regarding the difference between the two strategies.
For the synthetic dataset simulations, we present the global losses in Figure 4 for π rand and π pow-d for different d and m .
We show that π pow-d converges approximately 3 × faster to the global loss ≈ 0 .
7 than π rand when d = 10 m , with a slightly higher error ﬂoor.
Even with d = 2 m , we get 2 × faster convergence to global loss ≈ 0 .
7 than π rand .
Elimination of Selection Skew with π adapow-d .
For π pow-d , the selection skew is the trade-off for the convergence speed gain in Figure 2 and Figure 4.
For both simulations, π pow-d converges slightly above the global minimum value due to the selection skew.
We eliminate this selection skew while maintaining the beneﬁt of convergence speed with π adapow-d .
In Figure 2(a)-(b), π adapow-d shows a
[Figure] [Figure] convergence speed similar to π pow-d , d = K , but has no selection skew, converging to the same minimum as π rand (see Figure 2(c)). In Figure 4, π adapow-d again shows a convergence speed similar to π pow-d , d = 10 m , but has no adversarial selection skew.
As a matter of fact, π adapow-d converges to the minimum global loss value at least 3 × faster than π rand .
Hence π adapow-d gains the beneﬁt of both worlds from biased client selection: convergence speed and elimination of selection skew.
Experiments with Heterogeneously Distributed FMNIST.
As elaborated in Appendix F, α determines the data heterogeneity across clients.
Smaller α indicates larger data heterogeneity.
In Figure 5, we present the test accuracy and training losses for the different sampling strategies from the FMNIST experiments with α = 0 .
3 and α = 2 .
Observe that π pow-d achieves approximately 10% and 5% higher test accuracy than π rand and π aﬂ respectively for both α = 2 and α = 0 .
For higher α (less data heterogeneity) larger d (more selection skew) performs better than smaller d .
Figure 5(a) shows that this performance improvement due to the increase of d eventually converges.
For smaller α , as in Figure 5(b), smaller d = 6 performs better than larger d which shows that too much solution bias is adversarial to the performance in the presence of large data heterogeneity.
The observations on training loss are consistent with the test accuracy results.
Performance of the Communication- and Computation-Efﬁcient variants.
Next, we evaluate π cpow-d and π rpow-d which were introduced in Section 4.
In Figure 6, for α = 2 , π rpow-d and π cpow-d each yields approximately 5% and 6% higher accuracy than π rand , but both yield lower accuracy than π pow-d that utilizes the highest computation and communication resources.
For α = 0 .
3 , π cpow-d and π rpow-d perform as well as π pow-d and give a 10% accuracy improvement over π rand .
Moreover, π pow-d , π rpow-d and π cpow-d all have higher accuracy and faster convergence than π aﬂ .
We evaluate the communication and computation efﬁciency of P OWER - OF - CHOICE by comparing different strategies in terms of R 60 , the number of communication rounds required to reach test accuracy 60%, and t comp , the average computation time (in seconds) spent per round.
The computation time includes the the time taken by the central server to select the clients (including the computation [Figure] [Table] time for the d clients to compute their local loss values) and the time taken by selected clients to perform local updates.
In Table [Table] with only C = 0 .
03 fraction of clients, π pow-d , π cpow-d , and π rpow-d have about 5% higher test accuracy than ( π rand , C = 0 . 1) .
The R 60 for π pow-d , π cpow-d , π rpow-d is 0 .
57 times that of ( π rand , C = 0 . 1) respectively.
This implies that even for π rpow-d which does not incur any additional communication cost for client selection, we can get a 2 × reduction in the number of communication rounds using 1 / 3 of clients compared to ( π rand , C = 0 . 1) and still get higher test accuracy performance.
Note that the computation time t comp for π cpow-d and π rpow-d with
C = 0 .
03 is smaller than that of π rand with C = 0 .
In Appendix G.2, we show that the results for α = 2 are consistent with the α = 0 .
3 case shown in Table 1.
In Appendix G.5, we also show that for C = 0 .
1 , the results are consistent with the C = 0 .
03 case.
Effect of Mini-batch Size and Local Epochs .
We evaluate the effect of mini-batch size b and local epochs τ on the FMNIST experiments with different sets of hyper-parameters: ( b, τ ) ∈ { (128 , 30) , (64 , 100) } . Note that ( b, τ ) = (64 , 30) is the default hyper-parameter setting for the previous results. The ﬁgures are presented in Appendix G.6.
For b = 128 , we observe that the performance improvement of π pow-d over π rand and π aﬂ is consistent with b = 64 (see Figure 12). In Figure 14, for τ = 100 , with smaller data heterogeneity, the performance gap between π rand and π pow-d is consistent with that of τ = 30 . For larger data heterogeneity, however, increasing the local epochs results in π rand and π pow-d performing similarly.
This shows that with larger data heterogeneity, larger τ results in increasing the selection skew towards speciﬁc clients, and weakens generalization.
C ONCLUDING R EMARKS
In this work, we present the convergence guarantees for federated learning with partial device participation with any biased client selection strategy.
We discover that biasing client selection can speed up the convergence at the rate
O ( 1 Tρ ) where ρ is the selection skew towards clients with higher local losses.
Motivated by this insight, we propose the adaptive client selection strategy
P OWER - OF - CHOICE .
Extensive experiments validate that P OWER - OF - CHOICE yields 3 × faster convergence and 10% higher test accuracy than the baseline federated averaging with random selection.
Even with using fewer clients than random selection, P OWER - OF - CHOICE converges 2 × faster with high test performance.
An interesting future direction is to improve the fairness (Li et al., 2019; Yu et al., 2020; Lyu et al., 2020; Mohri et al., 2019) and robustness (Pillutla et al., 2019) of the P OWER - OF - CHOICE strategy by modifying step 3 of the P OWER - OF - CHOICE algorithm to use a different metric such as the clipped loss or the q -fair loss proposed Li et al.
(2019) instead of F k ( w ) .
A A DDITIONAL T HEOREM
Theorem A.1 (Convergence with Fixed Learning Rate) .
Under Assumptions 3.1 to 3.4, a ﬁxed learning rate η ≤ min { 1 2 µB , 14 L } where B = 1 + 3 ρ 8 , and any client selection strategy π as deﬁned above, the error after T iterations of federated averaging with partial device participation satisﬁes
[Equation]
As T → ∞ the ﬁrst term in (8) goes to 0 and the second term becomes the bias term for the ﬁxed learning rate case.
For a small η , we have that the bias term for the ﬁxed learning rate case in
Theorem A.1 is upper bounded by 8 L Γ 3 µ (cid:16) (cid:101) ρρ − 1 (cid:17) which is identical to the decaying-learning rate case.
The proof is presented in Appendix D.
B P RELIMINARIES FOR P ROOF OF T HEOREM 3.1 AND T HEOREM A.
We present the preliminary lemmas used for proof of Theorem 3.1 and Theorem A.1.
We will denote the expectation over the sampling random source S ( t ) as E S ( t ) and the expectation over all the random sources as E .
Lemma B.1.
Suppose F k is L − smooth with global minimum at w ∗ k , then for any w k in the domain of F k , we have that
[Equation]
Proof.
[Equation]
Lemma B.2 (Expected average discrepancy between w ( t ) and w ( t ) k for k ∈ S ( t ) ) .
[Equation]
Proof. m
[Equation]
Observe from the update rule that k, k (cid:48) are in the same set S ( t ) and hence the terms where k = k (cid:48) in the summation in (14) will be zero resulting in (15).
Moreover for any arbitrary t there is a t 0 such that 0 ≤ t − t 0 < τ that w ( t 0 ) k (cid:48)
=
w ( t 0 ) k since the selected clients are updated with the global model at every τ .
Hence even for an arbitrary t we have that the difference between (cid:107) w ( t ) k (cid:48) − w ( t ) k (cid:107) 2 is upper bounded by τ updates.
With non-increasing η t over t and η t 0 ≤ 2 η t , (15) can be further bounded as,
[Equation]
By taking expectation over (18),
[Equation]
where (22) is because there can be at most m ( m − 1) pairs such that k (cid:54)
=
k (cid:48) in S ( t ) .
Lemma B.3 (Upper bound for expectation over (cid:107) w ( t ) − w ∗ (cid:107) 2 for any selection strategy π ) .
With E [ · ] , the total expectation over all random sources including the random source from selection strategy we have the upper bound:
[Equation]
Proof.
[Equation]
C P ROOF OF T HEOREM 3.
With g ( t ) = 1 m (cid:80) k ∈S ( t ) g k ( w ( t ) k
[Equation]
ξ ( t ) k ) as deﬁned in Section
[Equation]
we have that
[Equation]
First let’s bound A 1 .
[Equation]
where (33) is due to the AM-GM inequality and Cauchy–Schwarz inequality, (35) is due to
Lemma B.1, (36) is due to the µ -convexity of F k , and (37) is due to Lemma B.2.
Next, in expectation, E
[ A 2 ] =
due to the unbiased gradient.
Next again with Lemma B.1 we bound A 3 as follows:
[Equation]
Lastly we can bound A 4 using the bound of variance of stochastic gradients as,
[Equation]
Using the bounds of A 1 , A 2 , A 3 , A 4 above we have that the expectation of the LHS of (27) is bounded as
E
[Equation] [Equation]
where (44) is due to Lemma B
[Equation]
Now we aim to bound A 5 in (44).
First we can represent A 5 in a different form as:
[Equation]
Now with η t < 1 / (4 L ) and ν t = 2 η t (1 − 2 Lη t ) , we have that A 6 can be rewritten and bounded as
[Equation]
ν t
[Equation]
where (48) is due to µ − convexity
[Equation]
(49) is due to Lemma B.1 and the AM-GM inequality and
Cauchy–Schwarz inequality
[Equation]
and (51) is due to the fact that ν t (1 − η t µ ) 2 η t ≤ 1 .
Hence using this bound of A 6 we can upper bound A 5 as
[Equation]
) ) , w ( t ) )
( F ( w ( t ) ) −
K (cid:88) k
= p k F ∗ k )]
+ 2 η t E [ ρ ( S ( π, w ( τ (cid:98) t/τ (cid:99) ) ) , w ∗ )( F ∗ −
K (cid:88) k
= p k F ∗ k )] (54) ≤ 16 η 2 t τ 2 G 2 − ν t (1 − η t L ) ρ
( E [ F ( w ( t ) )] −
K (cid:88) k
= p k F ∗ k ) (cid:124) (cid:123)(cid:122)
(cid:125) A+
η t (cid:101) ρ Γ (55) where (54) is due to the deﬁnition of ρ ( S ( π, w ) , w (cid:48) ) in Deﬁnition 3.2 and (55) is due to the deﬁnition of Γ in Deﬁnition 3.1 and the deﬁnitions of ρ, (cid:101) ρ in Deﬁnition 3.2.
We can expand A 7 in (55) as
[Equation]
where (60) is due to the µ − convexity, (61) is due to − 2 η t (1 − 2 Lη t )(1 − η t L ) ≤ − 34 η t , and (62) is due to − (1 − 2 Lη t )(1 − η t L ) ≤ − (1 − 3 Lη t ) .
Hence we can ﬁnally bound A 5 as
[Equation]
Lη 2 t
[Equation]
Now we can bound E [ (cid:107) w ( t +1) − w ∗ (cid:107) 2 ] as
[Equation]
By deﬁning ∆ t +1 =
E [ (cid:107) w ( t +1) − w ∗ (cid:107) 2 ] , B = 1+ 3 ρ
[Equation]
, C
[Equation]
32 τ 2 G
[Equation]
+ σ
[Equation]
m +
[Equation]
ρL Γ , D = 2Γ( (cid:101) ρ − ρ ) , we have that
[Equation]
By setting ∆ t ≤ ψt + γ , η t = βt + γ and β > 1 µB , γ > 0 by induction we have that
[Equation]
Then by the L-smoothness of F ( · ) , we have that
[Equation]
D P ROOF OF T HEOREM A.
With ﬁxed learning rate η t = η , we can rewrite (65) as
[Equation]
(68) and with η ≤ min { 1 2 µB , 14 L } using recursion of (68) we have that
[Equation]
Using ∆ t ≤ 2 µ ( F ( w ( t ) ) − F ∗ ) and L -smoothness, we have that
[Equation]
E E XTENSION : G ENERALIZATION TO DIFFERENT AVERAGING SCHEMES
While we considered a simple averaging scheme where w ( t +1) = 1 m (cid:80) k
[Equation]
, we can extend the averaging scheme to any scheme q such that the averaging weights q k are invariant in time and satisﬁes (cid:80) k ∈S ( t ) q k = 1 for any t .
Note that q includes the random sampling without replacement scheme introduced by Li et al.
(2020) where the clients are sampled uniformly at random without replacement with the averaging coefﬁcients q k = p k K/m .
With such averaging scheme q , we denote the global model for the averaging scheme q k as (cid:98) w ( t ) , where (cid:98) w ( t +1) (cid:44) (cid:80) k ∈S ( t ) q k (cid:16) w ( t ) k − η t g k ( w ( t ) k ) (cid:17) , and the update rule changes to
[Equation]
[Equation]
where (cid:98) g ( t ) =
(cid:80) k ∈S ( t ) q k g k ( w ( t ) k , ξ ( t ) k ) .
We show that the convergence analysis for the averaging scheme q is consistent with Theorem 3.1.
In the case of the averaging scheme q
[Equation]
we have that
Lemma B
[Equation]
2 and Lemma B
[Equation]
3 shown in Appendix B, each becomes
[Equation]
Then
[Equation]
using the same method we used for the proof of Theorem 3.
[Equation]
we have that
[Equation]
By deﬁning the selection skew for averaging scheme q similar to Deﬁnition 5 as
[Equation]
and
[Equation]
With η t < 1 / (2 L (1 + m ))
[Equation]
using the same methodology for proof of Theorem 3.1 we have that M becomes upper bounded as
[Equation]
Finally we have that
[Equation]
By
[Equation]
, we have that
[Equation]
Again, by setting
[Equation]
by induction we have that
[Equation]
max
[Equation]
Then by the L-smoothness of F ( · )
[Equation]
we have that
[Equation]
With β = mµ , γ = 4 m (1+ m ) L µ and η t = βt + γ , we have that
[Equation]
which is consistent with Theorem 3.1.
F E XPERIMENT D ETAILS
Quadratic Model Optimization.
For the quadratic model optimization, we set each local objective function as strongly convex as follows:
[Equation]
H k ∈ R v × v is a diagonal matrix H k
= h
k
I with h k ∼ U (1 , 20) and e k ∈ R v is an arbitrary vector.
We set the global objective function as F ( w ) =
(cid:80)
Kk =1 p k F k ( w ) , where the data size p k follows the power law distribution P ( x ; a ) = ax a − 1 , 0 ≤ x ≤ 1 , a = 3 .
We can easily show that the optimum for F k ( w ) and F ( w ) is w ∗ k
= H − 1 k e k and w ∗ =
( (cid:80) Kk =1 p k H k ) − 1 ( (cid:80)
Kk =1 p k e k ) respectively.
The gradient descent update rule for the local model of client k in the quadratic model optimization is
[Equation]
[Equation] where the global model is deﬁned as w ( t +1)
= 1 m (cid:80) k ∈S ( t ) w ( t +
[Equation]
k .
We sample m = KC clients for every round where for each round the clients perform τ gradient descent local iterations with ﬁxed learning rate η
and then these local models are averaged to update the global model.
For the implementation of π adapow-d , d was decreased half from d = K for every 5000 rounds.
For all simulations we set τ = 2 , v = 5 , η = 2 × 10 − 5 .
For the estimation of ρ and (cid:101) ρ for the quadratic model, we get the estimates of the theoretical ρ, (cid:101) ρ values by doing a grid search over a large range of possible w , w (cid:48) for ρ ( S ( π, w ) , w (cid:48) ) and ρ ( S ( π, w ) , w ∗ ) respectively.
The distribution of S ( π, w ) is estimated by simulating 10000 iterations of client sampling for each π and w .
Logistic Regression on Synthetic Dataset.
We conduct simulations on synthetic data which allows precise manipulation of heterogeneity.
Using the methodology constructed in (Sahu et al., 2019), we use the dataset with large data heterogeneity, Synthetic (1,1).
We assume in total 30 devices where the local dataset sizes for each device follows the power law.
For the implementation of π adapow-d , d was decreased to d = m from d = K at half the entire communication rounds.
We set the mini batch-size to 50 with τ = 30 , and η = 0 .
05 , where η is decayed to η/ 2 every 300 and 600 rounds.
DNN on FMNIST Dataset.
We train a deep multi-layer perceptron network with two hidden layers on the FMNIST dataset (Xiao et al., 2017).
We construct the heterogeneous data partition amongst clients using the Dirichlet distribution Dir K ( α ) (Hsu et al., 2019), where α determines the degree of the data heterogeneity across clients (the data size imbalance and degree of label skew across clients).
Smaller α indicates larger data heterogeneity.
For all experiments we use mini-batch size of b = 64 , with τ = 30 and η = 0 .
005 , where η is decayed by half for every 150, 300 rounds.
We experiment with three different seeds for the randomness in the dataset partition across clients and present the averaged results.
All experiments are conducted with clusters equipped with one NVIDIA TitanX
GPU.
The number of clusters we use vary by C , the fraction of clients we select.
The machines communicate amongst each other through Ethernet to transfer the model parameters and information necessary for client selection.
Each machine is regarded as one client in the federated learning setting.
The algorithms are implemented by PyTorch.
Pseudo-code of the variants of pow-d: cpow-d and rpow-d. We here present the pseudo-code for π cpow-d and π rpow-d .
Note that the pseudo-code for π cpow-d in Algorithm 1 can be generalized to the algorithm for π pow-d , by changing
[Equation]
1: Input : m, d, p k for k ∈ [ K ] , mini-batch size b = | (cid:98) ξ k | for computing| (cid:98) ξ k | (cid:80) ξ ∈ (cid:98)
ξ k f ( w , ξ ) 2: Output : S ( t ) 3: Initialize : empty sets S ( t ) and
A 4: Global server do 5:
Get A
= { d indices sampled without replacement from [ K ] by p k } 6:
Send the global model w ( t ) to the d clients in A 7:
Receive| (cid:98) ξ k | (cid:80) ξ ∈ (cid:98) ξ k f ( w , ξ ) from all clients in A 8:
Get S ( t ) =
{ m clients with largest| (cid:98) ξ k | (cid:80) ξ ∈ (cid:98) ξ k f ( w , ξ ) (break ties randomly)} 9: Clients in A in parallel do 10:
Create mini-batch (cid:98) ξ k from sampling b samples uniformly at random from B k and compute| (cid:98) ξ k | (cid:80) ξ ∈ (cid:98)
ξ k f ( w , ξ ) and send it to the server 11: return S ( t )
G A DDITIONAL E XPERIMENT R ESULTS G. S ELECTED C LIENT P ROFILE
We further visualize the difference between our proposed sampling strategy π pow-d and the baseline scheme π rand by showing the selected frequency ratio of the clients for K = 30 , C = 0 .
1 for the quadratic simulations in Figure 7.
Note that the selected ratio for π rand reﬂects each client’s dataset size.
We show that the selected frequencies of clients for π pow-d are not proportional to the data size of the clients, and we are selecting clients frequently even when they have relatively low data size like client 6 or 22.
We are also not necessarily frequently selecting the clients that have the highest data size such as client 26.
This aligns well with our main motivation of P OWER - OF - CHOICE that weighting the clients’ importance based on their data size does not achieve the best performance, and rather considering their local loss values along with the data size better represents their importance.
Note that the selected frequency for π rand is less biased than π pow-d .
[Figure]
G. C OMMUNICATION AND C OMPUTATION E FFICIENCY WITH LARGER DATA HETEROGENEITY
In Table 2, we show the communication and computation efﬁciency of P OWER - OF - CHOICE for α = 2 , as we showed for α = 0 .
3 in Table 1 in Section 5.
With C = 0 .
03 fraction of clients, π pow-d , π cpow-d , and π rpow-d have better test accuracy of at least approximately 10% higher test accuracy performance than ( π rand , C = 0 . 1) .
R 60 for π pow-d , π cpow-d , π rpow-d is 0.61, 0.66, 0.
times that of ( π rand , C = 0 . 1) respectively.
This indicates that we can reduce the number of communication rounds by at least 0.6 using 1/3 of clients compared to ( π rand , C = 0 . 1) and still get higher test accuracy performance.
The computation time t comp for π cpow-d and π rpow-d with C = 0 .is smaller than that of ( π rand , C = 0 . 1) .
[Table]
[Figure]
G. I NTERMITTENT C LIENT A VAILABILITY
In real world scenarios, certain clients may not be available due to varying availability of resources such as battery power or wireless connectivity.
Hence we experiment with a virtual scenario, where amongst K clients, for each communication round, we select clients alternately from one group out of two ﬁxed groups, where each group has 0 .
5 K clients.
This altering selection reﬂects a more realistic client selection scenario where, for example, we have different time zones across clients.
For each communication round, we select 0.1 portion of clients from the corresponding group uniformly at random and exclude them from the client selection process.
This random exclusion of certain clients represents the randomness in the client availability within that group for cases such as low battery power or wireless connectivity.
In Figure 8 we show that π pow-d and π rpow-d achieves 10% and 5% test accuracy improvement respectively compared to π rand for α = 2 .
For α = 3 , both π pow-d and π rpow-d shows 10% improvement.
Therefore, we demonstrate that P OWER - OF - CHOICE also performs well in a realistic scenario where clients are available intermittently.
G. R ESULTS FOR DNN ON N ON - IID P ARTITIONED EMNIST D ATASET
To provide further validation of the consistency in our results of π pow-d and its variants on the FMNIST dataset, we present additional experiment results on the EMNIST dataset sorted by digits with K = 500 , C = 0 . We train a deep multi-layer perceptron network with two hidden layers on the dataset partitioned heterogeneously across the clients in the same way as for the FMNIST dataset.
For all experiments, we use b = 64 , τ = 30 , and η = 0 .
005 where η is decayed by half at round 300.
In Figure 9, we show that π pow-d performs with signiﬁcantly higher test accuracy than π rand for varying d for both α = 2 and 0 .
For α = 2 , π aﬂ is able to follow the performance of π pow-d in the later communication rounds, but is slower in achieving the same test accuracy than π pow-d .
Moreover, in
Figure 10, we show that π cpow-d works as good as π pow-d for both large and small data heterogeneity.
The performance of π rpow-d falls behind π pow-d and π cpow-d for smaller data heterogeneity, whereas for larger data heterogeneity, π rpow-d is able to perform similarly with π pow-d and π cpow-d .
[Figure]
[Figure]
G. E FFECT OF THE FRACTION OF SELECTED CLIENTS
[Figure] In Figure for larger C = 0 . 1 with α = 2 , the test accuracy improvement for π pow-d is even higher than the case of C = 0 .
03 with approximately 15% improvement.
π cpow-d performs slightly lower in test accuracy than π pow-d but still performs better than π rand and π aﬂ .
π rpow-d performs as well as π aﬂ .
For α = 0 .
3 , π pow-d , π cpow-d and π rpow-d have approximately equal test accuracy performance, higher than π rand by 5%.
The P OWER - OF - CHOICE strategies all perform slightly better than π aﬂ .
Therefore we show that P OWER - OF - CHOICE performs well for selecting a larger fraction of clients, i e., when we have larger C = 0 . 1 > 0 .
G. E FFECT OF THE LOCAL EPOCHS AND MINI - BATCH SIZE
We present the experiment results elaborated in Section 5 for the different hyper-parameter settings ( b, τ ) ∈ { (128 , 30) , (64 , 100) } in Figure 12, 13, 14, and 15 below.
[Figure] [Figure] [Figure] [Figure]