Personalized Federated Learning towards Communication Efficiency, Robustness and Fairness
Author(s)
Abstract
Personalized Federated Learning faces many challenges such as expensive communication costs, training-time adversarial attacks, and performance unfairnessacross devices.
Recent developments witness a trade-off between a reference modeland local models to achieve personalization.
We follow the avenue and proposea personalized FL method towards the three goals.
When it is time to communicate, our method projects local models into a shared-and-fixed low-dimensionalrandom subspace and uses infimal convolution to control the deviation between thereference model and projected local models.
We theoretically show our method converges for strongly convex objectives with square regularizers and the convergencedependence on the projection dimension is mild.
We also illustrate the benefitsof robustness and fairness on a class of linear problems.
Finally, we conduct alarge number of experiments to show the empirical superiority of our method overseveral state-of-the-art methods on the three aspects.
Introduction
Federated Learning (FL) emerges as a new distributed computing paradigm that would performprivately distributed optimization in large-scale networks of remote clients [51].
For the sake ofprivacy protection, data are generated locally and are kept in the original location during training,which incurs a discrepancy among local data distributions.
Furthermore, the nature that FL works asa decentralized system poses greater challenges to its communication efficiency, robustness againstadversarial attacks, and fairness on resource allocation [32].
To detour the data heterogeneity issue, one considers to personalize local models [35].
A key featurethat any personalization method has is to differ local models from the global model.
The simplestway for personalization is training purely with local data on each device.
Chen et al.
[11] showedthat when the degree of data heterogeneity is above some threshold value, pure local training isminimax optimal; otherwise, the global model is minimax optimal.
In practice, we prefer a methodthat intervenes between the two extremes.
It brings out another popular approach that interpolatesbetween a reference model and local models [20, 21, 13, 14, 26, 64].
Recently, Li et al.
[44] suggestedthat personalization can be leveraged not only to improve accuracy, but also to allow for competingconstraints such as robustness and fairness.
Inspired by their work, we would explore the followingquestion:
Can we balance different constraints of interest (i.e., communication efficiency, robustness andfairness) simultaneously?
In this paper, we give an affirmative answer to the question by proposing a personalized FL methodnamed as lp-proj , whose core is L p -regularization and low-dimensional random projection.
Weemploy the idea of controlling the dissimilarity between the global model and local models via asmoothing kernel of infimal convolution.
Towards the three goals, the smoothing kernel is designed to regularize the projection of local models in a shared low-dimensional random subspace rather thanthe original space.
In this way, each client only communicates the projected models each time andthe server maintains a low-dimensional reference model for regularization.
The random subspace isgenerated once and will not change during training.
It makes local models share a similar part in therandom subspace and adapt to their local data using components beyond that.
Theoretically, we give convergence analysis for strongly convex objective with square regularizer,and show that the convergence dependence on the projection dimension is mild.
By analyzing testlosses and the corresponding variances across the network on federated linear regression, we showthat our proposed method is at least as good as two SOTA methods [14, 44] in terms of Byzantinerobustness [37] and performance fairness (see Definition 1).
Finally, we conduct a large number of experiments to show the empirical superiority of our method.
It not only shows expectant personalization but also promotes fairness by achieving higher testaccuracy more uniformly over clients.
It significantly improves communication efficiency, becausethe subspace dimension is often no more than one-hundredth of the original dimension.
Furthermore,it is more resistant to training-time adversarial attacks.
In summary, we propose a personalized FL algorithm and explore its performance in aspects ofcommunication efficiency, robustness and fairness.
Our results show that low-dimensional projectionbrings multiple benefits and is helpful for algorithm design.
Related Work
There are many works studying personalization and a survey can be found in [35].
It has beenstudied via multi-task learning [57, 28], meta-learning [10, 31, 17], knowledge distillation [41, 70]and transfer learning [63, 49].
Hanzely et al.
[22] provided convergence analysis for a generalpersonalized framework that requires jointly strongly convex and smooth objectives.
Communication Efficiency To reduce the cost of communication in FL with large-scale networks,existing works could be categorized as gradient compression, model compression and reducing thecommunication frequency.
On gradient compression, sparsification [29, 47], quantization [1] andlow-rank approximation [3] are three main directions investigated in the current research.
On modelcompression, Liang et al.
[46] suggested learning local representations and a global model onlyoperates on the local representations, Li et al.
[38] extended the lottery ticket hypothesis and usednetwork pruning in the FL setting.
On communication frequency, Wang et al.
[62] used momentumto delay the global aggregation; McMahan et al.
[51], Karimireddy et al.
[33] performed multiplelocal updates to lessen the communication rounds.
Shahid et al.
[56] presented a survey on currentprogress on communication efficiency in FL.
Our proposed method is a model compression approach.
Differing from previous works that compress the model each time with different basis, our work focuson a shared-and-fixed low-dimensional subspace which is determined at the beginning of trainingand will not change later on.
Fairness in FL Zhou et al.
[71] suggested that there are three types of fairness in FL: performancefairness, collaboration fairness and model fairness.
On performance fairness, an FL system usuallypromotes uniform accuracy distribution across participants, which is closely related to resourceallocation by viewing FL as a joint optimization system over a heterogeneous network.
Li et al.
[44]gave a formal definition (Definition 1) and some efficient methods have been proposed towards thisgoal [44, 27].
Definition 1 (Performance Fairness [44]) .
A model w 1 is more fair than w 2 if the test performance distribution of w 1 across the network with N clients is more uniform than that of w 2 , i.e.var { F k ( w 1 ) } k ∈ [ N ] < var { F k ( w 2 ) } k ∈ [ N ] , where F k ( · ) denotes the test loss of client k ∈
[ N ] andvar denotes the variance 1 .
On collaboration fairness, one expects that each participant receives a reward that can fairly reflectits contribution to the FL system, in which way one can build a sound incentive mechanism.
Lyuet al.
[48] formalized this idea; Yu et al.
[69], Xu and Lyu [67] explored on this aspect.
On modelfairness, one usually concerns ethical issues and seeks to protect some sensitive attributes [16, 23, 36] the objective functions under fairness constraint our work focus on performance fairness, illustrating the benefits of our method through theoretical analysisand numerical experiments.
An extension to collaboration fairness is shown in Appendix D.7.2.
Robust FL Typical adversarial attacks include data poisoning and model update poisoning (Byzantine attacks).
The former injects abnormal sample points into the training dataset [6, 30, 39, 55, 59,65, 18], while the latter manipulates communication messages by sending arbitrary updates to theserver, both attacks hindering the training processes.
In this paper, we mainly aim to defend modelupdate poisoning attacks and achieve Byzantine robustness [37].
An extension to a data poisoningattack is also considered in numerical experiments.
Prior works favor Byzantine-robust SGD variantswhere the server alleviates the attack of Byzantine clients via robust aggregation rules [12, 68, 66, 8].
Beyond that, Li et al.
[42] developed a robust distributed method by incorporating the L p -normregularizer to robustify the objective function.
Li et al.
[44] used robust aggregation rules to achieverobustness and fairness.
Our work leverages the ideas from [42] and [44], but differs from themby embedding the update process in a low-dimensional fixed random subspace.
Theoretical analysis shows that with commonly used regularization parameters, our method is no worse than two
SOTA methods [14, 44] (see Figure 1).
Extensive experiments manifest our method of achievingstate-of-the-art performance under various types and intensities of adversarial attacks.
Methodology
In this section, we present our method which is based on infimal convolution and subspace regularization .
Conventional FL that trains a single global model to fit the “average client” suffers fromstatistical heterogeneity among massive devices.
To enhance accuracy performance, we hope not onlyto leverage the global model, but also stylize it to fit the local data for each client.
To this end, weuse infimal convolution , which is originally proposed to smooth some extended real-valued convexfunction f with a sufficiently smooth kernel function g [52].
We apply this technique in FL to bridgelocal models and global model.
Here, f is the usual objective function in the vanilla case, and g isdesigned to characterize the relationship between local and global models.
Given a general functiong as the smoothing kernel, the personalized FL using infimal convolution is then formulated as abi-level problem:
[Equation]
where G ( · ) is the aggregation function at the server side 2 .
For k ∈ { 1 , · · · , N } ,
[Equation]
Here, ⊗ denotes infimal convolution operator, ξ k is an independent sample drawn from the distribution
D k , and ˜ f k ( x k ; ξ k ) is the loss function corresponding to this sample.
w and x k represent theglobal and local model parameters, respectively.
λ is a hyperparameter controlling the degree ofpersonalization.
Problem (1) is pure local training if λ = 0 , and is synchronized training whenλ → ∞ .
The smoothing kernel function g is task-specific.
Many previous personalized methods can be castinto our infimal convolution framework by setting a proper g .
For example, Dinh et al.
[14], Liet al.
[44] used Moreau Envelopes as the regularizer, which is equivalent to g ( · ) = 1 2 ∥ · ∥ 2 2 .
Li et al.[42] proposed the L p -norm regularization g ( · ) =
∥ · ∥ p instead.
Motivated by the fact that highdimensional data usually has low-dimensional representation that retains meaningful properties [60],and random projection would preserve similarity of data vectors [7], we propose to regularize theprojection of local models in a shared low-dimensional space, which is equivalent to the followingsmoothing kernel
[Equation]
where p ≥ 1 and P is a d sub × d random matrix that is generated initially and will not vary anymore.d sub is the dimension of the shared-and-fixed random subspace.
The choice for P is flexible as theonly requirement in our theory is that all the singular values of P are bounded from both sides.
In thispaper, we consider that P is generated with i.i.d.
Gaussian entries and then normalized to have unit L 2norm for each row as suggested by [40].
We comment that with this g , F k ( w ) is actually a functionof ˜ w =
P w since P ( w − x k ) =
˜ w − P x k .
It implies we can only focus on the low-dimensionalparameter ˜ w ∈ R d sub at the global level for algorithm description and theoretical analysis.
1: Input : Communication rounds T , local update rounds R , client sampling size S , regularization coefficient λ , lower-level problem accuracy ν , step size η , initial global model ˜ w 0 , projection matrix P , speedup control parameter β .
2: for t = 0 to T − 1 do 3:
Server sends ˜ w t to all clients.
4: for all k = 1 to N clients do 5: ˜ w tk, 0 = ˜ w t .
6: for r = 0 to R − 1 do 7:
Independently sample a fresh mini-batch ˜ D k and minimize the loss function (4) up to accuracy level ν to get x tk,r .
Update the local model ˜ w tk,r +1 by (5).
9: end for 10: end for 11:
Server uniformly samples a subset of clients S t of size S .
Each client sends ˜ w tk,R to the server.
Server updates the global model via
[Equation]
13: end for
3.1 The Algorithm
In this section, we introduce the algorithm lp-proj (see Algorithm 1) for the bi-level optimizationproblem (1) with smoothing kernel g given by Eqn.
The algorithm lp-proj is essentially an alternative minimization method on bi-level optimization.
Each client k maintains two parameters: their local parameter x tk,r and a copy of the global parameter˜ w tk,r with additional subscript r denoting inner iterations and superscript t the communication round.
At round t , the server broadcasts the latest global model ˜ w t to all clients.
Then each client initializestheir version of global model ˜ w tk, 0 as ˜ w t (line 5) and starts to solve the problem via alternativeminimization (lines 6-9).
• (line 7)
Given a local version of global model ˜ w tk,r , we use gradient descent ( GD ) to obtain an
approximate solution x tk,r that minimizes ˜ h
k up to accuracy level ν , where
[Equation]
Here ˜ D k is a mini-batch sampled uniformly and ξ k,i refers to a sample from ˜ D k .
The GD iterationis terminated when (cid:13)(cid:13)(cid:13) ∇ ˜
h k ( x tk,r ; ˜ D k , ˜ w tk,r ) (cid:13)(cid:13)(cid:13) 2 2 ≤ ν is satisfied.• (line 8) Given a local parameter x tk,r , the local version of global model ˜ w tk,r is updated by one-stepgradient descent:
[Equation] [Equation]
After R steps of the alternative updates, each client has its own version of the global model ˜
w tk,R .
Then the server accesses a random set of S clients and produces the next global model by a linearcombination of the latest ˜ w t and the average of { ˜ w tk,R } k ∈S t .
Here, a hyperparameter β , which couldbe viewed as a global step-size, is introduced to control the global update process.
Our theorem showsa proper β can speed up convergence, but in practice, we find that the test performance only variesmoderately for different choices of β .
For simplicity, we only consider β = 1 in our experiments.
Communication Efficiency In Algorithm 1, we restrict the global model ˜ w t to lie in a fixed lowdimensional subspace, only ˜ w tk,R of dimension d sub , instead of the full model x tk,r of dimension d ,is communicated to the server during each round, which leads to much fewer bits for communicationcompared to vanilla FL.
Besides, we remark on the difference between our method and otherprojection/sketching-based methods.
On one hand, distributed sketching [4], which directly projectsthe data in a low-dimensional space at the start of training, is “one-shot” rather than iterative, whileour method projects local models every communication round and the local training proceeds withthe full model.
On the other hand, sketched-SGD [29] compresses the transmitted messages with different basis every time, while our random subspace is specified at the beginning and would notchange after that.
Robustness and Fairness For one thing, by applying projection into a low-dimensional subspace, our method only requires (near) consensus of model parameters of different clients in thelow-dimensional subspace, leaving flexibility for the system towards personalization and bettergeneralization to the local data distribution, which could improve performance fairness and robustness when facing adversarial attacks.
For the other, introducing L p -norm regularized term to theobjective function is equivalent to launching an uncertainty set to the model parameter by rewritingthe objective as a constrained optimization problem (e.g., L 1 -norm is the diamond-shaped uncertaintyand L 2 -norm is the spherical uncertainty), in which way we can enhance accuracy by searching fora model adaptive to the local data distribution in the uncertainty set.
Formal analysis on a class oflinear problems is provided in Section 4.2.
Theoretical Analysis
4.1 Convergence
Assumption 1 (Convexity and smoothness) .
˜
f
k is (a) µ -strongly convex or (b) L -smooth, thatis, for any x k , x ′ k ∈ R d and ξ k , we have ˜ f k ( x ′ k ; ξ k ) ≥ ˜ f k ( x k ; ξ k ) + (cid:68) ∇
˜ f k ( x k ; ξ k ) , x ′ k − x k (cid:69) +µ 2 ∥ x ′ k − x k ∥ 22 or (cid:13)(cid:13)(cid:13) ∇ ˜ f k ( x ′ k ; ξ k ) − ∇ ˜ f k
x k ; ξ k ) (cid:13)(cid:13)(cid:13) 2 ≤ L ∥ x ′ k
− x k ∥ 2 .
Assumption 2 (Bounded variance) .
The variance of stochastic gradients in each client is bounded,i.e., E ξ k (cid:13)(cid:13)(cid:13) ∇ ˜ f k ( x k ; ξ k ) −∇ f k ( x k ) (cid:13)(cid:13)(cid:13) 2 2 ≤ γ 2 f .
Assumptions 1 and 2 are standard for convergence analysis.
Assumption 1 also guarantees strongconvexity or smoothness of f k .
Recall that we can view F k as a function of ˜ w instead of w , withsome abuse of notation, we write F k as F k ( ˜ w ) =
min x k ∈ R d (cid:110)
f k ( x k ) + λ 2 ∥
˜ w − P x k ∥ 22 (cid:111) .
Then wehave the following result.
Lemma 1.
Suppose Assumptions 1 (a) and 2 hold.
With probability at least 1 − 2 exp( − cd sub ) ,we have 1 λ
[Equation]
, where
C, c > 0 are constants.
Lemma 1 quantifies the error between the exact gradient ∇ F k ( ˜ w tk,r ) and the approximate gradientλ ( ˜ w tk,r − P x tk,r ) due to mini-batch sampling and optimization error of the inner loop.
From theexpression of δ 2 , we can see that this error has a mild dependence on d sub .
(Numerical verification ofthe mild dependence is shown in Appendix D.7.4.)
Based on Lemma 1, we can establish the convergence of the global model and personalized parametersof Algorithm 1.
Before presenting the main result, we first introduce some necessary notations.
Byconvexity and smoothness of f k , we can show that F k is µ F -strongly convex and L F -smooth withµ F = λµ (1+ C √ d sub /d ) 2 λ + µ and L F = λ .
Then the condition number of F k is κ F = L F /µ F =(1 + C (cid:112) d sub /d ) 2 λ/µ + 1 , and there exists the unique solution ˜ w ∗ =
argmin ˜ w ∈ R d sub F ( ˜ w ) .
Defineσ 2 F = 1 N (cid:80) Nk
=1 ∥∇ F k
( ˜ w ∗ ) ∥ 22 , which measures the bounded diversity of F k .
Then the convergenceof Algorithm 1 is guaranteed by Theorem 1.
Theorem 1.
Suppose that Assumptions 1 (a) and 2 hold.
Let ˆ η 1 = 1 18 L F (1+10 κ F /β ) with β ≥ 1 .
If
T ≥ 2 ˆ η 1 µ F , then with probability at least 1 − 2 exp( − cd sub ) , there exists η ≤ ˆ η 1 βR such that
[Equation]
clientdriftwithmultiplelocalupdatesandapproximationerror
[Equation]
where ∆ 0 = ∥
˜ w 0 − ˜ w ∗ ∥ 22 , c > 0 is a constant, ¯ w T = (cid:80) T − 1 t =
α t ˜ w t /A T with α t =
(1 −ηβRµ F / 2) − ( t +1) and A T =
(cid:80) T − 1 t =
α t , the expectation is w.r.t.
all the randomness except for
See Proposition 10 in
P and ˜ O hides constants and polylogarithmic factors.
Moreover, suppose
that
x
Tk is a solutionsatisfies (cid:13)(cid:13)(cid:13) ∇ ˜
h k ( x Tk ; ˜ D k , ˜ w T ) (cid:13)(cid:13)(cid:13) 2 2 ≤ ν
and O 1 denotes the right-hand side of (6).
Then with probabilityat least 1 − 2 exp( − cd sub ) , we have
[Equation]
From (6), when there is no client sampling ( S = N ), choosing β = Θ( NR ) leads to a quadraticspeedup ˜ O (cid:0) 1 / ( TRN ) 2 (cid:1) w.r.t.
communication rounds.
Eq.
(7) shows the average of personalized parameters (after a linear transformation) converges to a ball with center ˜ w ∗ and radius˜ O (cid:16) λ
δ 2 µ 2 F + σ 2 F λ 2 + δ 2 (cid:17) .
Here λ can be chosen to trade off different terms.
Our Theorem 1 shares the same error bounds as Theorem 1 in [14], except that their approximationerror δ 2 is slightly smaller than ours up to constant factors, since in our case the approximationerror is enlarged by projection.
The constant term O (cid:16) λ
δ 2 µ F (cid:17) appears in both theorems and is causedby biased gradients, i.e., we only get a biased estimate of ∇ F k due to inexact inner optimization(non-zero ν ) and batch data (small | ˜ D k | ).
Hence, our result is comparable to previous work [14]up to constants factors, even if we force optimization in a random subspace, which facilitatescommunication efficiency.
Proposition 1.
Under the conditions of Theorem 1, if | ˜ D k | scales linearly with the index t and ν andη decay at the rate 1 /t , then with high probability, we have E ∥ ˜ w T − ˜ w ⋆ ∥
= O (1 /T ) .
Based onthis, the convergence of personalized parameters is also of the order O (1 /T ) .
Proposition 1 shows that we can carefully tune the parameters to cancel out the enlargement ofapproximation error and achieve any given accuracy ε .
Due to space limitations, the formal results aredeferred to Appendix B. Alternatively, since ν = O ( e − ˜ cR ) for some ˜ c > 0 under the strong convexityassumption, setting R = Ω (log(1 /ε )) and | ˜ D k | = Ω(1 /ε ) also leads to the target accuracy.
4.2 Robustness and fairness
In this subsection, we explore the robustness and fairness benefits of lp-proj on a class of linearproblems and compare lp-proj with Ditto [44] and pFedMe [14].
For ease of analysis, we assumethe rows of P are orthogonal.
In practice, we can directly use the random matrix generated asin Section 3 without explicit orthogonalization, since high-dimensional random vectors are nearlyorthogonal.
Numerical comparison shows that model accuracy would be similar with or withoutorthogonalization (see Appendix D.7.3).
Our Setting We focus on a simplified setting where the number of local update steps is infinite, thereis only one round of communication and all clients participate in the communication.
Then it is naturalto set β = 1 .
Suppose that the true parameter on client k is w k , there are n samples on each client andthe covariate on client k is { ξ k,i } ni =1 and fixed.
The observations are generated by y k,i = ξ ⊤ k,i w k + z k,iwhere the noises z k,i i.i.d.
∼ N (0 , σ 2 ) .
For simplicity, we assume (cid:80) n i =1 ξ k,i ξ ⊤ k,i = bn I d .
Then the testloss on client k is f te k ( x k )
= 12 n (cid:80) ni
=1 (
ξ
⊤ k,i w k + z ′ k,i − ξ ⊤ k,i x k ) 2 , where z ′ k,i ∼ N (0 , σ 2 ) and areindependent of z k,i .
Three Attacks We examine three types of Byzantine attacks.
Denote the message delivered bymalicious client k as ˜ w ( ma ) k , then the attacks are listed as follows
• Same-value attacks : The message sent by a Byzantine client k is set as ˜ w ( ma ) k
= c 1 d sub , where1 d sub ∈ R d sub is the vector of ones and c ∼ N (0 , τ 2 ) .
• Sign-flipping attacks : The transmitted messages are sign-flipped and then scaled, i.e., a Byzantineclient k computes the true value ˜ w k but sends ˜ w ( ma ) k
= −| c |· ˜ w k to the server where c ∼ N (0 , τ 2 ) .
• Gaussian attacks : The message sent by a Byzantine client k is set as ˜ w ( ma )
k ∼
[Equation] [Equation]
[Equation]
The analyses for different attacks are similar, thus we only focus on the same-value attacks herefor illustration.
Results for other attacks are deferred to Appendix C.3.
Suppose that there are
N a malicious clients, and the heterogeneity is uniform in all dimensions in the sense of Eqn.(25)
[Figure] [Figure]
in Appendix C.
[Figure]
where we define Σ 1 to measure data heterogeneity in a single dimension.
Letλ ∗ 1 =
[Equation]
n Σ 1 + NaN 2 ( τ 2 − σ 2 / ( bn )) .
The numerator of λ ∗ 1 is the variance of noises over the number of samples.
The denominator is the sum of data heterogeneity and variance of attacks.
When λ = λ ∗ 1 , pFedMe ,
Ditto and lp-proj all achieve their corresponding minimal losses.
However, we do not know factorsaffecting λ ∗ 1 in advance, implying getting the particular value of λ ∗ 1 is possibly hard.
Therefore, weneed to compare the performance of these methods under different values of λ .
Proposition 2 (Formal statement in Theorem 2, Appendix C.3) .
Denote the averaged losses on benignclients of pFedMe , Ditto and lp-proj by L Me, att1 ( λ ) , L Di,att1 ( λ ) and L l2,att1 ( λ ) respectively.
Theexplicit forms of losses are in Appendix C.3.
Under the same-value attacks, we have (i) L Me,att1 ( λ ) =
L Di,att1 ( λ ) , ∀ λ > 0 and (ii) if λ ∗ 1 < b , L l2, att1 ( λ ) ≤ L Me,att1 ( λ ) if and only if λ ≥
[Equation]
Porposition 2 implies lp-proj outperforms both pFedMe and Ditto once λ is larger than a thresholdvalue [Equation] 1 1 − λ [Equation]
1 /b , which is slightly larger than 2 λ ∗ 1 .
The pattern is captured by Figure 1, where we setn = 200 , N = 100 , N a = 20 , d = 100 , d sub = 10 , b = 1 , σ = 1 , Σ 1 = 0 .
1 and τ = 100 .
Thenλ ∗ 1 ≈ 4 .
9 e − 4 , a pretty small value.
Even for λ < 2 λ [Equation] 1 − λ ∗ 1 /b , the gap between lp-proj and pFedMe/ Ditto is negligible.
Thus, lp-proj has comparable or beter performance for any λ > 0 .
Now we turn to the performance fairness defined in Definition 1.
For simplicity, we further assumethat the true parameters w k are i.i.d.
and distributed as N ( µ w , Σ w ) .
Proposition 3 (Formal statement in Theorem 3, Appendix C.4) .
Denote the variance of test losseson different clients of pFedMe , Ditto and lp-proj by V Me ( λ ) , V Di ( λ ) and V l2 ( λ ) respectively.
Wehave for ∀ λ > 0 , E V l2 ( λ ) ≤ E V Me ( λ ) =
E V Di ( λ ) .
More specifically, E V Me ( λ )
=
O ( d 2 ) and 271 E
V l2 ( λ ) =
O ( d 2sub ) , where the expectation is taken w.r.t.
the randomness of w k .
Proposition 3 shows lp-proj always brings more uniform test losses, no matter what value λ is.
In particular, E V l2 ( λ ) =
O ( d 2 sub ) while E V Me ( λ )
=
O ( d 2 ) .
Since it is likely that d sub ≪ d , theadvantage of lp-proj could be much larger.
It implies lp-proj is more fair than pFedMe and
Ditto .
For the formal theorem, see Appendix C.4.
Numerical Experiments
Experimental Setup We test lp-proj as well as other comparable algorithms on six datasets fromcommon ML and FL benchmarks [50, 9].
We consider both convex and non-convex models.
For thelatter, we consider neural networks including both MLP and CNN.
Details about datasets, modelsand basic information about the FL system are provided in Table 2 in Appendix D. To better modelthe statistical heterogeneity, we distribute the dataset among clients in a non-iid fashion such thateach client only contains partial classes of the data in multi-classification problems.
Furthermore, wealso consider an extension of non-i.i.d.
on data volume skewness in Appendix D.7.1.
For each client, the training and testing data are pre-specified as in the ML community, and 20% oftraining data is randomly extracted to construct a validation set, keeping the remaining 80% as thetraining set.
The training set is used for model fitting and parameter estimation.
For each competingmethod, we use the accuracy performance on the validation set as the tuning criterion and conducta grid search to choose the best hyper-parameter combination among a prescribed candidate set.
[Figure] [Figure]
All reported results are evaluated on the test dataset.
More details about hyperparameter tuning areprovided in Appendix D.2.
Furthermore, to incorporate partial participation [51, 45], we randomlyselect 10% of the clients for aggregation at each communication round.
The projection dimensionof the random subspace for each dataset is chosen based on the full model size and communicationbudget (see Appendix D.2).
For reproduction of our numerical results, we provide the implementedsource code in the supplementary material.
Personalization Accuracy Performance
In order to highlight the empirical performance of ourproposed method, we compare lp-proj with several state-of-the-art personalization methods inthe literature, together with a global method and a pure local method.
Specifically, we considerthe case when p = 1 ( lp-proj-1 ) and p = 2 ( lp-proj-2 ).
For SOTA methods, we consider
Ditto [44], pFedMe [14], Per-fedavg [17], LG-FedAvg [46] and RSA [42].
A brief description ofthese approaches is provided in Appendix D.1.
Due to space limitations, we only show the comparisons on training loss and test accuracy on twostandard datasets ( EMNIST and CIFAR ) in Figure 3.
Results on the other datasets are left in Table 3and Figure 5 in Appendix D.3.
From these figures, we see that lp-proj-1 and lp-proj-2 havelower training loss and higher test accuracy than other methods.
Furthermore, the training process ismore stable as the loss and accuracy curves have less fluctuation.
Communication Efficiency We compare lp-proj with the global baseline FedAvg [51] and fivestandard approaches using gradient and model compression, namely Sketch [29], LBGM [3], QSGD [1],
DGC [47] and LG-FedAvg [46].
We quantify the communication cost via total bytes written and readby active clients each round and capture the relation between test accuracy and communicated bytes.
Due to space limitations, we only show the results on two datasets in Table 1 ( Synthetic(0, 0)and EMNIST ) with the full results left in Table 4 in Appendix D.4.
It is clear that our method is more communication efficient since we only communicate lowdimensional messages.
Given a communication budget of bytes, lp-proj obtains ∼ 26 .
3% and∼ 83 .
5% test accuracy improvement on Synthetic(0, 0) and EMNIST datasets respectively.
Onthe other hand, given a target test accuracy, our proposed method needs much fewer bits than the restand saves the communication cost by 79x and 1320x on the two datasets compared with the bestcompeting method.
Besides, our method owns flexibility on the choice of the projection dimensiond sub , because the convergence dependence of our method on d sub is mild as predicted by Lemma 1with further supported numerical results in Table 16 in Appendix D.7.4.
The compression rate ofour proposed methods can be 1000x or even higher, while that of sketching or gradient compressionmethods typically is no larger than tens.
Robustness In addition to the three Byzantine attacks introduced in Section 4.2, we consider astronger data poisoning attack in the following experiments.
[Table]
• Data poisoning attacks : The training samples on malicious clients are poisoned with uniformlyrandomly chosen noisy labels.
Furthermore, when communicating, these clients would scale theirtransmitted messages to dominate the aggregate update.
For the former three Byzantine attacks, the noise variance τ is set as 100, 10 and 100 respectively.
The corruption levels, i.e., the fractions of malicious clients, are set as { 0 . 1 , 0 . 2 , 0 . 5 , 0 . 8 } .
For
datapoisoning attack, the scaling factor is randomly sampled from N (0 , 20 2 ) and the corruption levelsare from { 0 . 02 , 0 . 05 , 0 . 1 , 0 . 2 } .
Under different types of attacks and different levels of corruption, wecompare the average test accuracy performance on benign clients of lp-proj-1 and lp-proj-2with various defense baselines, including global training augmented with different robust aggregationtechniques, such as median and Krum [8], Ditto and RSA .
Due to space limitations, we show only a representative figure for each attack in Figure 4.
For fullresults, please refer to Appendix D.5.
From Figure 4, we find that under relatively weak attacks, e.g.,same-value and Gaussian attacks, the test accuracy of lp-proj-1 and lp-proj-2 rarely decays asthe fraction of malicious clients increases, while we observe significant drops on the test accuracyfor other algorithms once malicious clients exist.
On the other hand, under strong attacks, e.g.sign-flipping and data poisoning, an increasing fraction of malicious clients deteriorates the accuracyperformance continuously and even collapses the local model if the attack intensity is too large.
For example, under the sign-flipping attack, when the fraction of malicious clients exceeds 20%,only lp-proj-1 , RSA and Global+Krum work, while all other methods fail to produce a solution.
When the attack intensity further increases to 80%, the only robust methods that achieve the desiredaccuracy are lp-proj-1 and RSA .
The numerical results show that our method is resistant to standard malicious attacks, which roots inthe combination of projection and L 1 -norm subspace regularization that attribute to the robustness.
Consider an extreme example: if the subspace dimension is chosen as 0, then the joint optimizationis reduced to pure local training.
No matter how serious the adversarial attack is, the local testperformance would not be affected.
Therefore, random projection helps alleviate the attacks appliedin the original space, while the L 1 -norm helps eliminate outliers further [34].
Fairness To illustrate the accuracy and performance fairness trade-off, we plot the variances ofaccuracies across the network against the corresponding test accuracies for different methods in
Figure 2.
Due to space limitations, we show the results on two datasets here with the full resultsleft in Figure 6 in Appendix D.6.
Furthermore, we also show an extension to collaboration fairness(with its definition) in Appendix D.7.2.
All the results show that lp-proj-1 and lp-proj-2 provideaccurate and fair solutions that are comparable to other SOTA methods.
Concluding Remarks
In this paper, we have proposed a simple but powerful personalized FL approach based on infimalconvolution and subspace projection that we call lp-proj .
We have presented the convergenceresults for strongly convex objectives with square regularizers.
Inherent benefits of robustness andfairness of our method are also illustrated on a class of linear problems.
Empirical results show thatour approach could significantly save communication costs, improve robustness under various kindsof adversarial attacks, and promote performance fairness.
In future work, we would be interestedin establishing convergence results for general L p regularizers, exploring numerical applications onother ML tasks, and considering additional constraints, e.g., differential privacy.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] The Introduction part (Section 1) summarizes our
contribution and the Related Work part (Section 2) discusses the relationship betweenour work and previous ones.
(b) Did you describe the limitations of your work?
[Yes] See the conclusion part.
We
will consider convergence analysis for more general L p regularizers and additional
constraints in our future work.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] Societal
impacts are not applicable to our work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 4.
for the full assumptions used in convergence analysis and Section 4.2 for the setting
considered in robustness and fairness analysis.
(b) Did you include complete proofs of all theoretical results?
[Yes] They are deferred in
Appendix A, B and C.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] The source code
is provided in the supplementary material and the experimental details are described in
Appendix D.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were
chosen)?
[Yes] The data split strategy is described in Section 5 and the hyperparameter
settings for all the competing methods are described in Appendix D.2.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] The complete results are reported in Appendix D.
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] We use GPU for computing and
the type is described in Appendix D.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]