SoteriaFL : A Uniﬁed Framework for Private Federated Learning with Communication Compression
Author(s)
AfﬁliationAddress
Abstract
To enable large-scale machine learning in bandwidth-hungry environments suchas wireless networks, signiﬁcant progress has been made recently in designingcommunication-efﬁcient federated learning algorithms with the aid of communication compression.
On the other end, privacy-preserving, especially at the clientlevel, is another important desideratum that has not been addressed simultaneouslyin the presence of advanced communication compression techniques yet.
In thispaper, we propose a uniﬁed framework that enhances the communication efﬁciencyof private federated learning with communication compression.
Exploiting bothgeneral compression operators and local differential privacy, we ﬁrst examine a simple algorithm that applies compression directly to differentially-private stochasticgradient descent, and identify its limitations.
We then propose a uniﬁed framework
SoteriaFL for private federated learning, which accommodates a general familyof local gradient estimators including popular stochastic variance-reduced gradient methods and the state-of-the-art shifted compression scheme.
We provide acomprehensive characterization of its performance trade-offs in terms of privacy,utility, and communication complexity, where SoteriaFL is shown to achieve bettercommunication complexity without sacriﬁcing privacy nor utility than other privatefederated learning algorithms without communication compression.
Introduction
With the proliferation of mobile and edge devices, federated learning (FL) [35, 41] has recentlyemerged as a disruptive paradigm for training large-scale machine learning models over a vast amountof geographically distributed and heterogeneous devices.
For instance, Google uses FL in the Gboardmobile keyboard for next word predictions [23].
FL is often modeled as a distributed optimizationproblem [34, 35, 41, 29, 52], aiming to solvemin x ∈ R d (cid:40) f ( x ; D ) :
= 1 n n (cid:88) i =
f ( x ; D i ) (cid:41) , where f ( x
[Equation]
Here, D denotes the entire dataset distributed across all n clients, where each client i has a localdataset D i = { d i,j } mj =1 of equal size m , 1 x ∈ R d denotes the model parameters, f ( x ; D ) , f ( x ; D i ) ,and f ( x ; d i,j ) denote the nonconvex loss function of the current model x on the entire dataset D , thelocal dataset D i , and a single data sample d i,j , respectively.
For simplicity, we use f ( x ) , f i ( x ) andf i,j ( x ) to denote f ( x ; D ) , f ( x ; D i ) and f ( x ; d i,j ) , respectively.
1.1 Motivation: privacy-utility-communication trade-offs
To unleash the full potential of FL, it is extremely important that the algorithm designed to solve (1)needs to meet several competing desiderata.
Communication efﬁciency Communication between the server and clients is well recognized asthe main bottleneck for optimizing the latency of FL systems, especially when the clients—such asmobile devices—have limited bandwidth, the number of clients is large, and/or the machine learningmodel has a lot of parameters—
for example, the language model GPT-3 [6] has billions of parametersand therefore cumbersome to share directly.
Therefore, it is very important to design FL algorithms to reduce the overall communication cost,which takes into account both the number of communication rounds and the communication costper communication round for reaching a desired accuracy.
With these two quantities in mind, thereare two principal approaches for communication-efﬁcient FL: 1) local methods , where in eachcommunication round, clients run multiple local update steps before communicating with the server,in the hope of reducing the number of communication rounds, e.g., FedAvg [41], Local-SVRG [21],
SCAFFOLD [32], FedPAGE [57], and ProxSkip [43]; 2) compression methods , where clients sendcompressed communication message to the server, in the hope of reducing the communication costper communication round, e.g., [3, 33, 51, 25, 31, 42, 44, 22, 38, 45, 37].
While both categorieshave garnered signiﬁcant attention in recent years, we will focus on the second approach based oncommunication compression to enhance communication efﬁciency.
Privacy preserving While FL holds great promise of harnessing the inferential power of private datastored on a large number of distributed clients, these local data at clients often contain sensitive orproprietary information without consent to share.
Although FL may appear to protect the data privacyvia storing data locally and only sharing the model updates (e.g., gradient information), the trainingprocess can nonetheless reveal sensitive information as demonstrated by, e.g., Zhu et al.
It isthus desirable for FL to preserve privacy in a guaranteed manner [19, 29, 46, 52].
To ensure the training process does not accidentally leak private information, advanced privacypreserving tools such as differential privacy (DP) [16] have been widely integrated into trainingalgorithms [14, 8, 15, 1, 50, 26, 11, 18].
A notable example is DP-SGD [1], which developed adifferentially-private stochastic gradient descent (SGD) algorithm in the centralized (single-node)setting.
More recently, several differentially-private algorithms [27, 53, 47, 40] are proposed forthe more general distributed ( n -node) setting suitable for FL.
In this paper, we also follow the DPapproach to preserve privacy.
In particular, we adopt local differential privacy (LDP) to respect theprivacy of each client, which is critical in FL.
Encouraged by recent advances in communication compression techniques, and the widespreadsuccess of differentially-private methods, a natural question is
Can we develop a uniﬁed framework for private federated learning with communication compression,and understand the trade-offs between privacy, utility, and communication?
Note that there have been a handful of works that simultaneously address compression and privacyin FL.
Unfortunately, they only provide partial answers to the above question.
Most of the existingworks only consider speciﬁc, elementary or tailored compression schemes that are applied directly tothe gradient messages in DP-SGD [2, 54, 20, 60, 56, 13].
A number of works [48, 9, 10, 30, 17, 49]extended and considered different compression schemes, but did not provide concrete trade-offsin terms of privacy, utility and communication.
Furthermore, existing theoretical analyses can belimited only to convex problems [20], lacking in some aspects such as utility [60], or deliveringpessimistic guarantees on utility and / or communication due to strong assumptions [56, 13].
Finally,existing work only studied the DP framework for direct compression, while it is known that therecently developed shifted compression scheme [42, 24] achieves much better convergence.
Due tonoise injection for privacy-preserving, it is a priori unclear if the shifted compression scheme is alsocompatible with privacy.
1.2 Our contributions
In this paper, we answer the above question by providing a general approach that enhances thecommunication efﬁciency of private federated learning in the nonconvex setting, through a uniﬁedframework called SoteriaFL (see Algorithm 2).
Speciﬁcally, we have the following contributions.
[Table]
shifted comp.
(1) Wang et al.
[53] considered the “global” ( (cid:15), δ ) -DP (which only protects the privacy for entire dataset D , i.e., the local dataset D i on node i may leak to other nodes j (cid:54) = i ) without communication compression.
However, we consider the “local” ( (cid:15), δ ) -LDP which can protect the local datasets D
i ’s at the client level.
(2) Ding et al.
[13] adopted a slightly different compression assumption E
[ (cid:107)C ( x ) − x (cid:107) 2 ] ≤ ˜ σ 2 , with ˜ σ 2 playing a similar role as (1 + ω ) in ours.
It only works for small (cid:15) = 1 / √ T , which leads to an accuracy no better than [Equation] [Equation]
[Equation]
, a factor of T 1 / 6 worse than the utility of the other algorithms including ours.
(3) Zhang et al.
[56] only considered random- k sparsiﬁcation, which is a special case of our general compression operator.
Moreover, it requires 1 + ω (cid:28) log T , i.e., at least k (cid:29) d log T out of d coordinates need to be communicated, and its utility hides logarithmic factors larger than 1 + ω .
The communication complexity n 7 / 2 is due to their convergence condition T > n 5 .
(4) Here, τ := (1+ ω ) 3 / 2 n 1 / 2 .
If n ≥ (1 + ω ) 3 (which is typical in FL), then τ < 1 , and we can drop the terms involving τ from SoteriaFL .
We ﬁrst present a simple CDP-SGD (Algorithm 1) that directly combines communicationcompression and DP-SGD .
We provide theoretical analysis for CDP-SGD in Theorem 1and show its limitations in communication efﬁciency.2.
We then propose a general framework SoteriaFL for private FL, which accommodates ageneral family of local gradient estimators including popular stochastic variance-reducedgradient methods and the state-of-the-art shifted compression scheme.
We provide a uniﬁedcharacterization of its performance trade-offs in terms of privacy, utility (convergenceaccuracy), and communication complexity.3.
We apply our uniﬁed analysis for SoteriaFL and obtain theoretical guarantees for severalnew private FL algorithms, including SoteriaFL-GD , SoteriaFL-SGD , SoteriaFL-SVRG ,and SoteriaFL-SAGA .
All of these algorithms are shown to perform better than the plain
CDP-SGD (Algorithm 1), and have lower communication complexity compared with otherprivate FL algorithms without compression.
We provide detailed comparisons between the proposed approach and prior arts in Table 1.
To thebest of our knowledge, SoteriaFL is the ﬁrst uniﬁed framework that simultaneously enables localdifferential privacy and shifted compression, and allows ﬂexible local computation protocols at theclient level.
Preliminaries
[Figure]
Let [ n ] denote the set { 1 , 2 , · · · , n } and (cid:107)·(cid:107) denote the Euclidean norm of a vector and the spectralnorm of a matrix.
Let (cid:104) u , v (cid:105) denote the standard Euclidean inner product of two vectors u and v .
Let f ∗ :=
min x f ( x ) denote the optimal value of problem (1).
In addition, we use the standardorder notation O ( · ) to hide absolute constants.
We now introduce the deﬁnitions of the compressionoperator and local differential privacy, and some assumptions for the objective functions.
Compression operator We introduce the notion of a randomized compression operator , whichis used to compress the gradients to save communication.
The following deﬁnition of unbiasedcompressors is standard and has been used in many distributed/federated learning algorithms [3, 33,42, 24, 39, 22, 38].
Deﬁnition 1 (Compression operator)
A randomized map C : R d (cid:55)→ R d is an ω -compressionoperator if for all x ∈ R d , it satisﬁes
[Equation]
In particular, no compression ( C ( x ) ≡ x ) implies ω = 0 .
Note that the conditions (2) are satisﬁed by many practically useful compression operators, e.g.,random sparsiﬁcation and random quantization [3, 39, 38].
A useful rule of thumb is that thecommunication cost is often reduced by a factor of 1 1+ ω due to compression [3].
Next, we brieﬂydiscuss an example called random sparsiﬁcation to provide more intuition.
Example 1 (Random sparsiﬁcation).
Given x ∈ R d , the random- k sparsiﬁcation operator is deﬁnedby C ( x ) := dk · ( ξ k (cid:12) x ) , where (cid:12) denotes the Hadamard (element-wise) product and ξ k ∈ { 0 , 1 } d isa uniformly random binary vector with k nonzero entries ( (cid:107) ξ k {+(cid:107) 
+} = k ).
This random- k sparsiﬁcationoperator C satisﬁes (2) with ω = dk − 1 , and the communication cost is reduced by a factor of 1 1+ ωsince
we transmit k = d 1+ ω (due to its ω = dk − 1 ) coordinates rather than d coordinates.
Local differential privacy We not only want to train the machine learning model using fewercommunication bits, but also want to maintain each client’s local privacy, which is a key componentfor FL applications.
We follow the framework of (local) differential privacy [4, 7, 58].
We say thattwo datasets D and D (cid:48) are neighbors if they differ by only one entry.
We have the following deﬁnitionfor local differential privacy (LDP).
Deﬁnition 2 (Local differential privacy (LDP))
A randomized mechanism M : D → R withdomain D and range R is ( (cid:15), δ ) -locally differentially private for client i if for all neighboringdatasets
D i , D [-
-](cid:48)
i ∈ D on client i and for all events S ∈ R in the output space of M , we have
[Equation]
The deﬁnition of LDP (Deﬁnition 2) is very similar to the original deﬁnition of ( (cid:15), δ ) -DP
[16, 15],except that now in the FL setting, each client protects its own privacy by encoding and processing itssensitive data locally, and then transmitting the encoded information to the server without coordinationand information sharing between the clients.
Assumptions about the functions Recalling (1), we consider the nonconvex FL setting, where thefunctions { f i,j } are arbitrary functions satisfying the following standard smoothness assumption(Assumption 1) and bounded gradient assumption (Assumption 2).
Assumption 1 (Smoothness) There exists some L ≥ 0 , such that for all i ∈ [ n ] , j ∈ [ m ] , thefunction f i,j is L -smooth,
[Equation]
Assumption 2 (Bounded gradient)
There exists some G ≥ 0 , such that for all i ∈ [ n ] , j ∈ [ m ] andx ∈ R d , we have (cid:107)∇ f i,j ( x ) (cid:107) ≤ G .
The smoothness assumption is very standard for the convergence analysis, and the bounded gradientassumption is also standard for the differential privacy analysis [5, 50, 26, 18].
Warm-up: Plain Compressed Differentially-Private SGD
There are two methods to combine privacy and compression: (1) ﬁrst perturb and then compress,and (2) ﬁrst compress and then perturb.
The advantage of the ﬁrst method is that it is very simpleand general, since compression will preserve the differential privacy and work seamlessly withany previous existing privacy mechanisms.
However, the second method needs to design carefulperturbation mechanisms (otherwise the perturbation might diminish the communication saving ofcompression), e.g., binomial perturbation [2] or discrete Gaussian perturbation [30].
In addition,it is observed that the ﬁrst method achieves better utility compared with the second one in somesettings [13].
Thus, we also apply the ﬁrst method in this paper: ﬁrst perturb then compress.
As a warm-up, we ﬁrst introduce a simple algorithm CDP-SGD (described in Algorithm 1), whichsubsumes some existing algorithms as special cases (e.g., [56, 13, 60]) for private FL with bettertheoretical guarantees.
The procedure for CDP-SGD is very simple: at each round t , each client ﬁrstcomputes a local stochastic gradient ˜ g
ti using its local dataset D
i
(Line 3 in Algorithm 1).
Then, ituses Gaussian mechanism [1] to achieve LDP (Line 4 in Algorithm 1) and transfers the compressedperturbed private gradient information to the server (Line 5 in Algorithm 1).
Finally, the serveraggregates the compressed information and update the model parameters (Line 7–8 in Algorithm 1).
// all nodes
SGD method v t
1 n ni =1 v ti 8: x t +1 = x
t − η t v t 9:
Now we present the theoretical guarantees for CDP-SGD in the following theorem.
Theorem 1 (Utility and communication for CDP-SGD ) Suppose that Assumptions 1 and 2 hold.
By choosing the algorithm parameters properly and letting the total number of communication rounds
T =
O (cid:16)
[Equation]
, CDP-SGD (Algorithm 1) satisﬁes ( (cid:15), δ ) -LDP and theutility
[Equation]
The proposed CDP-SGD (Algorithm 1) is simple but effective.
When the compression parameter ωis a constant (i.e., constant compression ratio), CDP-SGD achieves the same utility O (cid:16)
[Equation]
as DP-SGD in the single-node case n = 1 .
Our utility is better than [13] by a factor of T 1 / 6 , and ourcommunication complexity is much better than [56] (see Table 1).
However, the communication complexity of CDP-SGD still has room for improvements due to directcompression (Line 5 in Algorithm 1).
In particular, if the local dataset size m stored on clients isdominating, then CDP-SGD (even if we compute local full gradients as CDP-GD ) requires O ( m 2 )communication rounds (see Theorem 1), while previous distributed differentially-private algorithmswithout communication compression ( Distributed DP-SRM [53], LDP SVRG and LDP SPIDER [40])only need O ( m ) communication rounds (see Table 1).
SoteriaFL : Uniﬁed Private FL Framework with Shifted Compression
Due to the limitations of plain CDP-SGD , we now present an advanced and uniﬁed private FLframework called SoteriaFL in this section, which allows a large family of local gradient estimators(Line 3 in Algorithm 2 and Line 3–11 in Algorithm 3).
Via adopting the advanced shifted compression(Line 5 in Algorithm 2), SoteriaFL reduces the total number of communication rounds O ( m 2 ) of
CDP-SGD to O ( m ) , which matches previous uncompressed DP algorithms (see Table 1), and furtherreduces the total communication complexity due to less communication cost per round.
We ﬁrst introduce our uniﬁed SoteriaFL framework in Section 4.1 and then provide a genericassumption and a uniﬁed analysis that covers privacy, utility, and communication for SoteriaFL in
Section 4.2.
4.1 Uniﬁed SoteriaFL framework
Our SoteriaFL framework is described in Algorithm 2.
At each round t , each client will compute alocal (stochastic) gradient estimator ˜ g
ti using its local dataset D
i (Line 3 in Algorithm 2).
One canchoose several optimization methods for computing this local gradient estimator such as standardgradient descent ( GD ), stochastic GD ( SGD ), stochastic variance-reduced gradient ( SVRG ) [28, 36],and SAGA [12] (see e.g., Line 3–11 in Algorithm 3).
Then, each client adds a Gaussian perturbationξ ti on its gradient estimate ˜ g ti to provide LDP (Line 4 in Algorithm 2).
However, different from
CDP-SGD (Algorithm 1) where we directly compress the perturbed stochastic gradients, now eachclient maintains a reference s ti and compress the shifted message ˜
g ti − s ti (Line 5 in Algorithm 2).
This extra shift operation achieves much better convergence behavior (fewer communication
rounds)than
CDP-SGD , and thus can achieve lower communication complexity.
Algorithm 2 SoteriaFL ( framework for compressed private FL)
Input: initial point x 0 , stepsize η t , shift stepsize γ t , variance σ 2 p 1: for t = 0 , 1 , 2 , . .
, T do 2: for each node i ∈ [ n ] do in parallel 3:
Compute local gradient estimator ˜ g ti // it allows many methods, e.g., SGD, SVRG, and SAGA 4: Privacy : g ti = ˜ g ti + ξ ti , where ξ ti ∼ N ( 0 , σ 2 p I ) 5: Compression : let v ti =
C ti ( g ti − s ti ) and send to the server // shifted compression 6:
Update shift s t
+1 i
[Equation]
C ti ( g ti
[Equation]
s ti ) 7: end each node 8:
Server aggregates compressed information
[Equation]
=
[Equation]
ti 9:
[Equation]
[Equation]
11: end for
4.2 Generic assumption and uniﬁed theory
We provide a generic Assumption 3 which is very ﬂexible to capture the behavior of several existing(and potentially new) gradient estimators, while simultaneously maintaining the tractability to enablea uniﬁed and sharp theoretical analysis.
Assumption 3 (Generic assumption of local gradient estimator for SoteriaFL )
The gradient estimator ˜ g ti (Line 3 of Algorithm 2) is unbiased E t [˜ g ti ] = ∇ f i ( x t ) for i ∈ [ n ] , where E t takes theexpectation over all randomness before round t .
Moreover, it can be decomposed as two terms˜ g ti :=
A ti + B ti and there exist constants G A , G B , C 1 , C 2 , C 3 , C 4 , θ and a random sequence { ∆ t }such that
[Equation] 3a
[Equation]
where ϕ ti,j and ψ ti,j are bounded by G A and G B respectively, I b usually denotes a random minibatchwith size b , and E t takes the expectation conditioned on all history before round t .
Here, ϕ ti,j andψ ti,j should be viewed as functions related to the j -th sample d i,j stored on client i .
A few comments are in order.
Concretely, the decomposition (3a) is used for our uniﬁed privacyanalysis (i.e., Theorem 2).
We can let one of them be 0 if the gradient estimator only contains oneterm or is not decomposable.
The parameters C 1 and C 2 in (3b) capture the variance of the gradientestimators, e.g., C 1 = C 2 =
if the client computes local full gradient ˜ g ti = ∇ f i ( x t ) , and C 1 (cid:54) = 0(note that ∆ t will shrink in (3c)) and C 2 = 0 if the client uses variance-reduced gradient estimatorssuch as SVRG / SAGA .
Finally, the parameters θ, C 3 and C 4 in (3c) capture the shrinking behaviorof the variance (incurred by the gradient estimators), where different variance-reduced gradientmethods usually have different shrinking behaviors.
More concrete examples to follow in Lemma 1in Section 5.
Uniﬁed theory for privacy-utility-communication trade-offs Given our generic Assumption 3,we can obtain a uniﬁed analysis for SoteriaFL framework.
The following Theorem 2 uniﬁes theprivacy analysis and Theorem 3 uniﬁes the utility and communication complexity analysis.
Theorem 2 (Privacy for SoteriaFL ) Suppose that Assumption 3 holds.
There exist constants c andc (cid:48) , for any (cid:15) < c (cid:48) b 2 T/m 2 and δ ∈ (0 , 1) , SoteriaFL (Algorithm 2) is ( (cid:15), δ ) -LDP if we choose [Equation]
[Equation]
Theorem 3 (Utility and communication for SoteriaFL ) Suppose that Assumptions 1 and 3 hold,and the compression operator C ti (used in Line 5 of Algorithm 2) satisﬁes (2).
Set the stepsize asη t ≡ η ≤ min
[Equation]
where α = 3 βC 1 2(1+ ω ) θL 2 , ∀ β > 0 , the shift stepsize as γ t ≡ 1 1+ ω , and the privacy variance σ 2 p accordingto Theorem 2.
Then, SoteriaFL (Algorithm 2) satisﬁes ( (cid:15), δ ) -LDP and the following
[Equation]
where Φ 0 :
= f ( x 0 ) − f ∗ + {+αL ∆ 
+}
+ βLn
[Equation]
ni =
[Equation]
f i
[Equation]
x 0 ) − s
i (cid:107) 2 .
By further choosing the totalnumber of communication rounds T as
[Equation]
SoteriaFL has the following utility (accuracy) guarantee:
[Equation]
Theorem 3 is a uniﬁed theorem for our SoteriaFL framework, which covers a large family of localstochastic gradient methods under the generic Assumption 3.
In the next Section 5, we will show thatmany popular local gradient estimators ( GD , SGD , SVRG , and SAGA ) satisfy Assumption 3, and thuscan be captured by our uniﬁed analysis.
Some Algorithms within SoteriaFL Framework
In this section, we propose several new algorithms ( SoteriaFL-GD , SoteriaFL-SGD , SoteriaFLSVRG and SoteriaFL-SAGA ) captured by our SoteriaFL framework.
We give a detailed Algorithm 3 which describes all these four SoteriaFL -type algorithms in a nutshell.
To begin, we show that these local gradient estimators ( GD , SGD , SVRG , and SAGA ) satisfy Assumption 3 in the following main lemma, by detailing the corresponding parameter values (i.e.,
G A , G B , C 1 , C 2 , C 3 , C 4 , and θ ).
Input: initial point x 0 , stepsize η t , shift stepsize γ t , variance σ 2 p , minibatch size b , probability p 1: for t = 0 , 1 , 2 , . .
, T do 2: for each node i ∈ [ n ] do in parallel 3:
Option I: SGD 4:
Compute local SGD estimator ˜ g ti = 1 b (cid:80) j ∈I b ∇ f i,j ( x t ) //
GD if choose b = m 5:
Option II: SVRG 6:
Compute local SVRG estimator ˜ g ti = 1 b (cid:80) j ∈I b ( ∇ f i,j ( x t ) − ∇ f i,j ( w t ))
+ ∇ f i ( w t ) 7:
Update SVRG snapshot point w t +1 = (cid:26) x t , with probability p w t , with probability 1 − p 8:
Option III: SAGA 9:
Compute local SAGA estimator: ˜ g ti = 1 b (cid:80) j ∈I b ( ∇ f i,j ( x t ) − ∇ f i,j ( w ti,j ))
+ 1 m (cid:80) mj
=1 ∇ f i,j ( w ti,j ) 10:
Update SAGA variables w t
+1 i,j = (cid:26) x t , for j ∈ I b w ti,j , for j / ∈ I b 11:
End Options 12:
Privacy : g ti = ˜
g ti + ξ ti , where ξ ti ∼ N ( 0 , σ 2 p I ) 13:
Compression : let v ti =
C ti ( g ti − s ti ) and send to the server 14:
Update shift s t +1 i =
s ti + γ t C ti
( g ti − s ti ) 15: end each node 16:
Server aggregates compressed information v t =
s t + 1 n (cid:80) ni
=1 v ti 17: x t +1 = x
t − η t v t 18: s t +
=
s t + γ t 1 n (cid:80) ni
=1 v ti 19: end for
Lemma 1 (SGD/SVRG/SAGA estimators satisfy Assumption 3)
Suppose that Assumptions 1and 2 hold.
The local SGD estimator ˜ g ti (Option I in Algorithm 3) satisﬁes Assumption 3 with G A = G, G [Equation]
[Equation]
The local SVRG estimator ˜ g ti (Option II in Algorithm 3) satisﬁes Assumption 3 with
[Equation]
The local SAGA estimator ˜ g ti (Option III in Algorithm 3) satisﬁes Assumption 3 with
[Equation]
With Lemma 1 in hand, we can plug their corresponding parameters into the uniﬁed Theorem 3 to obtain detailed utility and communication bounds for the resulting methods ( SoteriaFLSGD / SoteriaFL-GD , SoteriaFL-SVRG , and SoteriaFL-SAGA ).
Formally, we have the followingcorollaries.
Corollary 1 ( SoteriaFL-SGD / SoteriaFL-GD ) Suppose that Assumption 1 and 2 hold and we combine Theorem 3 and Lemma 1, i.e., choosing stepsize η t ≡ η ≤ 1 (1+ √ (1+ ω ) 3 /n )
L , where we set β =(1+ ω )n (1+ √ (1+ ω ) 3 /n ) , shift stepsize γ t ≡ 1 1+ ω , and privacy variance σ 2 p =
c G 2 T log(1 /δ ) m 2 (cid:15).
If we further setthe minibatch size b = min (cid:110) m(cid:15)G √ β √ (1+ ω )
Ld log(1 /δ ) , m (cid:111) and the total number of communication rounds T = O (cid:16) √ nLm(cid:15)
G √ (1+ ω ) d log(1 /δ ) (1 + √ τ ) (cid:17) , where τ := (1+ ω ) 3 / 2 n 1 /, then SoteriaFL-SGD satisﬁes ( (cid:15), δ ) - LDP and the following utility grarantee 1 T (cid:80)
[Equation]
E [Equation] √
[Equation]
√ [Equation] (1+ √ τ ) (cid:17) .
[Table] [Table]
[Table] [Table]
Corollary 2 ( SoteriaFL-SVRG )
Suppose that Assumption 1 and 2 hold and we combine Theorem 3and Lemma 1, i.e., choosing stepsize η t ≡ η ≤ p 2 / 3 b 1 / 3 min { 1 , √ n/ (1+ ω ) 3 }
L , where we set β =p 4 / 3 b 2 / 3 (1+ ω ) 2 min { 1 ,n/ (1+ ω ) 3 } n and p 2 / 3 b 1 / 3 ≤ 1 , shift stepsize γ t ≡ 1 1+ ω , and privacy varianceσ 2 p =
c 5 G 2 T log(1 /δ ) m 2 (cid:15).
If we further let the minibatch size b = m 2 / 3 , the probability p =
b/m ,and the total number of communication rounds T
= O (cid:16)
[Table] [Equation]
√ [Equation] ) max (cid:8) 1 , τ (cid:9)(cid:17) , whereτ : = (1+ ω )
[Equation]
2 n 1 /, then SoteriaFL-SVRG satisﬁes ( (cid:15), δ ) -LDP and the following utility guarantee
[Equation]
The utility and communication complexity for SoteriaFL-SAGA are the same as SoteriaFL-SVRG ,and we defer its detailed corollary to the Appendix.
Interestingly, SoteriaFL -style algorithms are more communication-efﬁcient than CDP-SGD whenthe local dataset size m is large, with a communication complexity of O ( m ) , in contrast to O ( m 2 )for CDP-SGD .
In terms of utility, SoteriaFL-SVRG and SoteriaFL-SAGA can achieve the sameutility as CDP-SGD , while SoteriaFL-GD and SoteriaFL-SGD achieve a slightly worse guaranteethan that of CDP-SGD by a factor of 1 + √ τ , where τ :=
(1+ ω ) 3 / 2 n 1 /is small when the number ofclients n is large.
Gradient complexity of SoteriaFL -style algorithms
Although the utility and the communicationcomplexity are the most important considerations in private FL, another worth-noting criterion isthe gradient complexity , which is deﬁned as the total number of stochastic gradients computed byeach client.
Although SoteriaFL-GD , SoteriaFL-SGD , SoteriaFL-SVRG and SoteriaFL-SAGAhave similar communication complexity (see Table 1), they actually have very different gradientcomplexities—summarized in Table 2—since the minibatch sizes and gradient update rules forthese algorithms vary a lot.
The gradient complexity of SoteriaFL-SVRG / SoteriaFL-SAGA isusually smaller than SoteriaFL-SGD , and all of them are smaller than SoteriaFL-GD .
In sum, werecommend SoteriaFL-SVRG / SoteriaFL-SAGA due to its superior utility and gradient complexitywhile maintaining almost the same communication complexity as SoteriaFL-SGD / SoteriaFL-GD .
Conclusion
We propose SoteriaFL , a uniﬁed framework for private FL, which accommodates a general family oflocal gradient estimators including popular stochastic variance-reduced gradient methods and thestate-of-the-art shifted compression scheme.
A uniﬁed characterization of its performance trade-offsin terms of privacy, utility (convergence accuracy), and communication complexity is presented,which is then instantiated to arrive at several new private FL algorithms.
All of these algorithms areshown to perform better than the plain CDP-SGD algorithm especially when the local dataset size islarge, and have lower communication complexity compared with other private FL algorithms withoutcompression.