O N
C ONVERGENCE
OF F EDERATED A VERAGING
L
ANGEVIN D YNAMICS
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose a federated averaging Langevin algorithm (FA-LD) for uncertainty quantiﬁcation and mean predictions with distributed clients.
In particular, we generalize beyond normal posterior distributions and consider a general class of models.
We develop theoretical guarantees for FA-LD for strongly log-concave distributions with non-i.i.d data and study how the injected noise and the stochasticgradient noise, the heterogeneity of data and the varying learning rates affect the convergence.
Such an analysis sheds light on the optimal choice of local updates to minimize communication cost.
Important to our approach is that the communication efﬁciency does not deteriorate with the injected noise in the Langevin algorithms.
In addition, we examine in our FA-LD algorithm both independent and correlated noise used over different clients.
We observe there is a trade-off between the pairs among communication, accuracy and data privacy.
As local devices may become inactive in federated networks, we also show convergence results based on different averaging schemes where only partial device updates are available.
In such a case, we discover an additional bias that does not decay to zero.
I NTRODUCTION
Federated learning (FL) allows multiple parties to jointly train a consensus model without sharing user data.
Compared to the classical centralized learning regime, federated learning keeps training data on local clients, such as mobile devices or hospitals, where data privacy, security, and access rights are a matter of vital interest.
This aggregation of various data resources heeding privacy concerns yields promising potential in areas of internet of things (Chen et al., 2020), healthcare (Li et al., 2020d; 2019b), text data (Huang et al., 2020), and fraud detection (Zheng et al., 2020).
A standard formulation of federated learning is a distributed optimization framework that tackles communication costs, client robustness, and data heterogeneity across different clients (Li et al., 2020a).
Central to the formulation is the efﬁciency of the communication, which directly motivates the communication-efﬁcient federated averaging (FedAvg) (McMahan et al., 2017).
FedAvg introduces a global model to synchronously aggregate multi-step local updates on the available clients and yields distinctive properties in communication.
However, FedAvg often stagnates at inferior local modes empirically due to the data heterogeneity across the different clients (Charles & Koneˇcn`y, 2020;
Woodworth et al., 2020).
To tackle this issue, Karimireddy et al.
(2020); Pathaky & Wainwright (2020) proposed stateful clients to avoid the unstable convergence, which are, however, not scalable with respect to the number of clients in applications with mobile devices (Al-Shedivat et al., 2021).
In addition, the optimization framework often fails to quantify the uncertainty accurately for the parameters of interest, which are crucial for building estimators, hypothesis tests, and credible intervals.
Such a problem leads to unreliable statistical inference and casts doubts on the credibility of the prediction tasks or diagnoses in medical applications.
To unify optimization and uncertainty quantiﬁcation in federated learning, we resort to a Bayesian treatment by sampling from a global posterior distribution , where the latter is aggregated by infrequent communications from local posterior distributions.
We adopt a popular approach for inferring posterior distributions for large datasets, the stochastic gradient Markov chain Monte Carlo (SGMCMC) method (Welling & Teh, 2011; Vollmer et al., 2016;
Teh et al., 2016; Chen et al., 2014;
Ma et al., 2015), which enjoys theoretical guarantees beyond convex scenarios (Raginsky et al., 2017; Zhang et al., 2017; Mangoubi & Vishnoi, 2018; Ma et al., 2019).
In particular, we examine in the federated learning setting the efﬁcacy of the stochastic gradient Langevin dynamics (SGLD) algorithm, which differs from stochastic gradient descent (SGD) in an additionally injected noise for exploring the posterior.
The close resemblance naturally inspires us to adapt the optimization-based
FedAvg to a distributed sampling framework.
Similar ideas have been proposed in federated posterior averaging (Al-Shedivat et al., 2021), where empirical study and analyses on Gaussian posteriors have shown promising potential of this approach.
Compared to the appealing theoretical guarantees of optimization-based algorithms in federated learning (Pathaky & Wainwright, 2020; Al-Shedivat et al., 2021), the convergence properties of approximate sampling algorithms in federated learning is far less understood.
To ﬁll this gap, we proceed by asking the following question:
Can we build a uniﬁed algorithm with convergence guarantees for sampling in FL?
In this paper, we make a ﬁrst step in answering this question in the afﬁrmative.
We propose the federated averaging Langevin dynamics for posterior inference beyond the Gaussian distribution.
We list our contributions as follows:
• We present a novel non-asymptotic convergence analysis for FA-LD from simulating strongly log-concave distributions on non-i.i.d data when the learning rate is ﬁxed.
The frequently used bounded gradient assumption of (cid:96) 2 norm in FedAvg optimization is not required.
• The convergence analysis indicates that injected noise, data heterogeneity, and stochasticgradient noise are all driving factors that affect the convergence.
Such an analysis provides a concrete guidance on the optimal number of local updates to minimize communications.
• We can activate partial device updates to avoid straggler’s effects in practical applications and tune the correlation of injected noises to protect privacy.
• We also provide differential privacy guarantees, which shed light on the trade-off between data privacy and accuracy given a limited budget.
For related works of other federated learning approaches, we refer interested readers to section H.
P RELIMINARIES 2. A N OPTIMIZATION PERSPECTIVE ON FEDERATED AVERAGING
Federated averaging (FedAvg) is a standard algorithm in federated learning and is typically formulated into a distributed optimization framework as follows
[Equation]
where θ ∈ R d , l ( θ ; x c,j ) is a certain loss function based on θ and the data point x c,j .
FedAvg algorithm requires the following three iterate:
• Broadcast :
The center server broadcasts the latest model, θ k , to all local clients.
• Local updates : For any c ∈ [ N ] , the c -th client ﬁrst sets the auxiliary variable β ck
= θ k
and then conducts K ≥ 1 local steps:
β ck +
=
β ck − ηn c ∇ (cid:101) (cid:96) c
( β ck ) , where η is the learning rate and ∇ (cid:101) (cid:96) c is the unbiased estimate of the exact gradient ∇ (cid:96) c .
• Synchronization : The local models are sent to the center server and then aggregated into a unique model θ k + K :
= (cid:80)
Nc =1 p c β ck + K , where p c as the weight of the c -th client such that p c = n c (cid:80) Ni =1 n i ∈ (0 , 1) and n c > 0 is the number of data points in the c -th client.
From the optimization perspective, Li et al.
(2020c) proved the convergence of the FedAvg algorithm on non-i.i.d data such that a larger number of local steps K and a higher order of data heterogeneity slows down the convergence.
Notably, Eq.(1) can be interpreted as maximizing the likelihood function, which is a special case of maximum a posteriori estimation (MAP) given a uniform prior.
2. S TOCHASTIC GRADIENT L ANGEVIN DYNAMICS
Posterior inference offers the exact uncertainty quantiﬁcation ability of the predictions.
A popular method for posterior inference with large dataset is the stochastic gradient Langevin dynamics
(SGLD)
(Welling & Teh, 2011), which injects additional noise into the stochastic gradient such that θ k +
=
θ k − η ∇ (cid:101) f
( θ k ) + √ 2 τηξ k , where τ is the temperature and ξ k is a standard Gaussian vector.
f ( θ ) :
= (cid:80) Nc =1 (cid:96) c ( θ ) is an energy function.
∇ (cid:101) f
( θ ) is an unbiased estimate of ∇ f ( θ ) .
In the longtime limit, θ k converges weakly to the distribution π ( θ ) ∝ exp( − f ( θ ) /τ )
(Teh et al., 2016) as η → 0 .
P OSTERIOR INFERENCE VIA FEDERATED AVERAGING L ANGEVIN DYNAMICS
The increasing concern for uncertainty estimation in federated learning motivates us to consider the simulation of the distribution π ( θ ) ∝ exp( − f ( θ ) /τ ) with distributed clients.
Problem formulation We propose the federated averaging Langevin dynamics (FA-LD) based on the FedAvg framework in section 2.1.
We follow the same broadcast step and synchronization step but propose to inject random noises for local updates .
In particular, we consider the following scheme: for any c ∈ [ N ] , the c -th client ﬁrst sets θ ck = θ
k
and then conducts K ≥ 1 local steps:
Local updates
[Equation]
where ∇ f c ( θ ) = 1 p c ∇ (cid:96) c ( θ ) ; ∇ (cid:101)
f c ( θ ) is the unbiased estimate of ∇ f c ( θ ) ; Ξ ck is some Gaussian vector in Eq.(6).
Summing Eq.(2) from clients c = 1 to N and setting θ k
= ( θ 1 k , · · · , θ Nk ) , we have
[Equation]
where β k =
N (cid:88) c
= p c β ck ,
[Equation]
By the nature of synchronization , we always have β k
= θ k for any k ≥
and the process follows
[Equation]
which resembles SGLD except that we have a different “gradient” operator ∇ and θ k is not accessible when k mod K (cid:54) = 0 .
Since our target is to simulate from π ( θ ) ∝ exp( − f ( θ ) /τ ) , we expect ξ k is a standard Gaussian vector.
By the concentration of independent Gaussian variables, it is natural to set
[Equation] [Equation]
where
ξ k = (cid:80) N
c
=1 p c Ξ c k = (cid:80) N c =1 √ p c ξ c k and ξ c k is a also standard Gaussian vector.
Now we present the algorithm based on independent inject noise ( ρ = 0 ) and the full-device update (3) in Algorithm 1, where ρ is the the correlation coefﬁcient and will be further studied in section 4.3.3.
We observe Eq.(7) maintains a temperature τ/p c >
τ to converge to the stationary distribution π .
Such a mechanism may limit the disclosure of individual data and shows a potential to protect the data privacy.
Algorithm 1 Federated averaging Langevin dynamics Algorithm (FA-LD), informal version of
Algorithm 4.
Denote by θ ck the model parameter in the c -th client at the k -th step.
Denote the one-step intermediate result by β ck .
ξ ck is an independent standard d -dimensional Gaussian vector at iteration k for each client c ∈ [ N ] ; ˙ ξ k is a d -dimensional Gaussian vector shared by all the clients; ρ denotes the correlation coefﬁcient.
S k is sampled according to a device-sampling rule based on scheme I or II.
[Equation]
where Π device ( β ck +1 ) = (cid:80)
Nc =1 p c β ck +1 for full device and = (cid:80) c
∈S k +1 1 S β ck
+1 for partial device.
C ONVERGENCE ANALYSIS
In this section, we show that FA-LD converges to the stationary distribution π ( θ ) in the 2-Wasserstein ( W 2 ) distance at a rate of O (1 / √ T (cid:15) ) for strongly log-concave and smooth density.
The W 2 distance is deﬁned between a pair of Borel probability measures µ and ν on R d as follows
[Equation]
where (cid:107) · (cid:107) 2 denotes the (cid:96) 2 norm on R d and the pair of random variables ( β µ , β ν
) ∈ R d × R d is a coupling with the marginals following L ( β µ ) = µ and L ( β ν )
=
ν .
Note that L ( · ) denotes a distribution of a random variable.
4. A SSUMPTIONS
We make standard assumptions on the smoothness and convexity of the functions f 1 , f 2 , · · · , f N , which naturally yields appealing tail properties of the stationary measure π .
Thus, we no longer require a restrictive assumption on the bounded gradient in (cid:96) 2 norm as in Koloskova et al.
; Yu et al.
; Li et al.
(2020c).
In addition, to control the distance between ∇ f c and ∇ (cid:101)
f c , we also assume a bounded variance of the stochastic gradient in assumption 4.3.
Assumption 4.1 (Smoothness) .
For each c ∈ [ N ] , f c is L -smooth if for some L >
[Equation]
Assumption 4.2 (Strongly convexity) .
For each c ∈ [ N ] , f c is m -strongly convex if for some m >
[Equation]
Assumption 4.3 (Bounded variance) .
For each c ∈ [ N ] , the variance of noise in the stochastic gradient ∇ (cid:101) f c ( x ) in each client is upper bounded such that
[Equation]
Quality of non-i.i.d data Denote by θ ∗ the global minimum of f .
Next, we quantify the degree of the non-i.i.d data by γ :=
max c ∈ [ N ] (cid:107)∇ f c ( θ ∗ ) (cid:107) 2 , which is non-negative and yields a larger scale if the data is less identically distributed.
4. P ROOF SKETCH
The proof hinges on showing the one-step result in the W 2 distance.
To facilitate the analysis, we ﬁrst deﬁne an auxiliary continuous-time process (¯ θ t ) t ≥ 0 that synchronizes for almost any time t ≥
[Equation]
[Equation] where ¯ θ t = (cid:80)
Nc =1 p c ¯ θ c t , ∇ f
( ¯ θ t ) = (cid:80) N c =1 p c ∇ f c (¯ θ c t ) , ¯ θ ct is the continuous-time variable at client c , and W is a d -dimensional Brownian motion.
The continuous-time algorithm is known to converge to the stationary distribution π ( θ )
∝ e − f ( θ ) τ , where f ( θ ) =
(cid:80)
Nc =1 p c f c ( θ ) .
Assume that ¯ θ 0 simulates from the stationary distribution π , then it follows that ¯ θ t ∼ π for any t ≥ 0 .
4.2. D OMINATED CONTRACTION IN FEDERATED LEARNING
The ﬁrst target is to show a certain contraction property of (cid:107) β − θ − η ( ∇ f ( β ) − ∇ f ( θ ))
(cid:107) 22 based on distributed clients with infrequent communications.
Consider a standard decomposition
[Equation]
Using Eq.(4), we decompose I and apply Jensen’s inequality to obtain a lower bound of I .
In what follows, we have the following lemma.
Lemma 4.4 (Dominated contraction property, informal version of Lemma B.1) .
Assume assumptions 4.1 and 4.2 hold.
For any learning rate η ∈ (0 , 1 L + m ] , any { θ c } Nc =1 , { β c } Nc =1 ∈ R d , we have
[Equation]
where β = (cid:80)
Nc =1 p c β c , θ = (cid:80)
Nc =1 p c θ c , ∇ f ( β ) =
(cid:80)
Nc =1 p c ∇ f c ( β c ) , and ∇ f ( θ )
= (cid:80)
Nc =1 p c ∇ f c ( θ c ) .
It implies that as long as the local parameters θ c , β c and global θ, β don’t differ each other too much, we can guarantee the desired convergence.
In a special case when the communication is conducted at every iteration, the divergence term disappears and recovers the standard contraction (Dalalyan & Karagulyan, 2019).
4.2. B OUNDING DIVERGENCE
The following result shows that given a ﬁnite number of local steps K , the divergence between θ c in local client and θ in the center is bounded in (cid:96) 2 norm.
Notably, since the Brownian motion leads to a lower order term O ( η ) instead of O ( η 2 ) , a naïve proof framework such as Li et al.
(2020c) may lead to a crude upper bound for the ﬁnal convergence.
Lemma 4.5 (Bounded divergence, informal version of Lemma B.3) .
Assume assumptions 4.1, 4.2, and 4.3 hold.
For any η ∈ (0 , 2 /m ) and (cid:107) θ c 0 − θ ∗ (cid:107) 22 ≤ d D 2 for any c ∈ [ N ] and some constant D , we have the (cid:96) 2 upper bound of the divergence between local clients and the center
[Equation]
The result relies on a uniform upper bound in (cid:96) 2 norm, which avoids bounded gradient assumptions.
4.2. C OUPLING TO THE STATIONARY PROCESS
Note that ¯ θ t is initialized from the stationary distribution π .
The solution to the continuous-time process Eq.(8) follows:
[Equation]
Set t → ( k + 1) η
and ¯ θ
→ ¯ θ kη for Eq.(9) and consider a synchronous coupling such that W ( k +1)
η −
W kη := √ ηξ k is used to cancel the noise terms, we have
[Equation]
Subtracting Eq.(5) from Eq.(10) and taking square and expectation on both sides yield that
E
[Equation]
+
[Equation]
22 + divergence term + time error .
Eventually, we arrive at the one-step error bound for establishing the convergence results.
Lemma 4.6 (One step update, informal version of Lemma B.5) .
Assume assumptions 4.1, 4.2, and 4.3 hold.
Consider Algorithm 1 with any η ∈ (0 , 1 2 L ) and (cid:107) θ c 0 − θ ∗ (cid:107) 2 2 ≤ d D 2 , ρ = 0 , and full device participation for any c ∈ [ N ] , where θ ∗ is the global minimum for the function f .
Then
[Equation]
where µ k denotes the probability measure of θ k and κ =
L/m is the condition number.
Given small enough η , the above Lemma indicates that the algorithm will eventually converge
4. F ULL DEVICE PARTICIPATION 4.3. C ONVERGENCE BASED ON INDEPENDENT NOISE
When the synchronization step is conducted at every iteration k , the FA-LD algorithm is essentially the standard SGLD algorithm
(Welling & Teh, 2011).
Theoretical analysis based on the 2-Wasserstein distance has been established in Durmus & Moulines (2019); Dalalyan (2017); Dalalyan & Karagulyan (2019).
However, in scenarios of K > 1 with distributed clients, a divergence between the global variable θ k and local variable θ ck appears and unavoidably affects the performance.
The upper bound on the sampling error is presented as follows.
Theorem 4.7 (Main result, informal version of Theorem B.6) .
Assume assumptions 4.1, 4.2, and 4.
hold.
Given Algorithm 1 with η ∈ (0 , 12 L ] , ρ = 0 , full device, and well initialized { θ c 0 } Nc =1 , we have
[Equation]
where µ k denotes the density of θ k at iteration k , K is the local updates, κ :=
L/m , γ := max c ∈ [ N ] (cid:107)∇ f c ( θ ∗ ) (cid:107) 2 , and H 0 :
= D 2 + max c ∈ [ N ] τmp c + γ 2 m 2 d + σ 2 m 2 .
We observe that the initialization, injected noise, data heterogeneity, and stochastic gradient noise all affect the convergence.
Similar to Li et al.
(2020c)
, FA-LD with K -local steps resembles the one-step
SGLD with a large learning rate and the result is consistent with the optimal rate (Durmus & Moulines, 2019), despite multiple inaccessible local updates.
Nevertheless, given more smoothness assumptions, we may obtain a better dimension dependence (Durmus & Moulines, 2019; Li et al., 2022).
Bias reduction (Karimireddy et al., 2021) can be further adopted to alleviate the data heterogeneity.
Optimal choice of K .
To achieve the precision (cid:15) based on the learning rate η , we can set
[Equation]
This readily leads to ηm ≤ O
[Equation]
Denote by T (cid:15) the number of iterations to achieve the target accuracy (cid:15) .
Plugging into the upper bound of ηm , it sufﬁces to set T (cid:15) =
Ω( (cid:15) − 2 dκ
( K 2 + κ ) H 0 · log( d/(cid:15) )) .
Note that H
= Ω( D 2 ) , thus the number of communication rounds is of the order T (cid:15) K = Ω (cid:18)
K + κK (cid:19) , where the value of T (cid:15) K ﬁrst decreases and then increases w.r.t.
K , which indicates setting K either too large or too small leads to high communication costs.
Ideally, K should be selected in the scale of Ω( √ κ ) .
Combining the deﬁnition of T (cid:15) , this shows that the optimal K for FA-LD is in the order of O ( √ T (cid:15) ) , which matches the optimization-based results (Stich, 2019; Li et al., 2020c).
4.3. C ONVERGENCE GUARANTEES VIA VARYING LEARNING RATES
Theorem 4.8 (Informal version of Theorem B.7) .
Assume assumptions 4.1, 4.2, and 4.3 hold.
Consider Algorithm 1 with ρ = 0 , full device, an initialization satisfying (cid:107) θ c 0 − θ ∗ (cid:107) 22 ≤ d D 2 for any c ∈ [ N ] , and the varying learning rate following η k = 1 2 L +(1 / 12) mk .
Then for any k ≥ 0 , we have
W 2 ( µ k , π ) ≤ 45 κ (cid:112) (( K − 1) 2 + κ ) H 0 · (cid:0)
η k md (cid:1) 1 / 2 , ∀ k ≥ 0 .
To achieve the precision (cid:15) , we need W 2 ( µ k , π ) ≤ (cid:15) , i.e. W 2 ( µ k , π ) ≤ 45 κ (cid:112) ( K 2 + κ ) H 0 · (cid:18) md
[Table]
mk (cid:19) 1 /.
We therefore require Ω( (cid:15) − 2 d ) iterations to achieve the precision (cid:15) , which improves the Ω( (cid:15) − 2 d log( d/(cid:15) ))
rate for FA-LD with a ﬁxed learning rate by a O (log( d/(cid:15) )) factor.
4.3. P RIVACY - ACCURACY TRADE - OFF VIA CORRELATED NOISES
The local updates in Eq.(2) with Ξ ck = ξ ck / √ p c requires all the local clients to generate the independent noise ξ ck .
Such a mechanism enjoys the implementation convenience and yields a potential to protect the data privacy and alleviates the security issue.
However, the large scale noise inevitably slows down the convergence.
To handle this issue, the independent noise can be generalized to correlated noise based on a correlation coefﬁcient ρ between different clients.
Replacing Eq.(7) with
[Equation]
where ˙ ξ k is a d -dimensional standard Gaussian vector shared by all the clients at iteration k and ˙ ξ k is independent with ξ ck for any c ∈ [ N ] .
Following the synchronization step based on Eq.(3), we have
[Equation]
where ξ k = ρ ˙ ξ k + (cid:112) 1 − ρ 2 (cid:80) Nc =1 √ p c ξ ck .
Since the variance of i.i.d variables is additive, it is clear that ξ k follows the standard d -dimensional Gaussian distribution.
The correlated noise implicitly reduces the temperature and naturally yields a trade-off between federation and accuracy.
Since the inclusion of correlated noise doesn’t affect the iterate of Eq.(12), the algorithm property maintains the same except the scale of the temperature τ and efﬁcacy of federation are changed.
Based on a target correlation coefﬁcient ρ ≥ 0 , Eq.(11) is equivalent to applying a temperature
T c,ρ = τ ( ρ 2 +(1 − ρ 2 ) /p c ) .
In particular, setting ρ = 0 leads to T c,
= τ/p c , which exactly recovers
Algorithm 1; however, setting ρ = 1 leads to T c, 1 = τ , where the injected noise in local clients is reduced by 1 /p c times.
Now we adjust the analysis as follows
Theorem 4.9 (Informal version of Theorem B.8) .
Assume assumptions 4.1, 4.2, and 4.3 hold.
Consider Algorithm 1 with ρ ∈ [0 , 1] , η ∈ (0 , 12 L ] and (cid:107) θ c 0 − θ ∗ (cid:107) 22 ≤ d D 2 for any c ∈ [ N ] , we have
[Equation]
where µ k denotes the probability measure of θ k , H ρ := D 2 + 1 m max c ∈ [ N ] T c,ρ + γ 2 m 2 d
+ σ 2 m 2 .
Such a mechanism leads to a trade-off between data privacy and accuracy and may motivate us to exploit the optimal ρ under differential privacy theories (Wang et al., 2015).
4. P ARTIAL DEVICE PARTICIPATION
Full device participation enjoys appealing convergence properties.
However, it suffers from the straggler’s effect in real-world applications, where the communication is limited by the slowest device.
Partial device participation handles this issue by only allowing a small portion of devices in each communication and greatly increased the communication efﬁciency in a federated network.
The ﬁrst device-sampling scheme I (Li et al., 2020b) selects a total of S devices, where the c -th device is selected with a probability p c .
The ﬁrst theoretical justiﬁcation for convex optimization has been proposed by Li et al.
(2020c).
The second device-sampling scheme II is to uniformly select S devices without replacement.
We follow Li et al.
(2020c)
and assume S indices are selected uniformly without replacement.
In addition, the convergence also requires an additional assumption on balanced data (Li et al., 2020c).
Both schemes are formally deﬁned in section C.1.
Theorem 4.10 (Informal version of Theorem C.3) .
Under mild assumptions, we run Algorithmwith ρ ∈ [0 , 1] , a ﬁxed η ∈ (0 , 1 2 L ] and (cid:107) θ c 0 − θ ∗ (cid:107) 22 ≤ d D 2 for any c ∈ [ N ] , we have
[Equation]
where C S = 1 for Scheme I and C S =
N − S N − 1 for Scheme II.
Partial device participation leads to an extra bias regardless of the scale of η .
To reduce it, we suggest to consider highly correlated injected noise, such as ρ = 1 , to reduce the impact of the injected noise.
Further setting O ( (cid:112) d/S ) ≤ (cid:15)/ 3 and following a similar η as in section 4.3.1, we can achieve the precision (cid:15) within Ω( (cid:15) − 2 d log( d/(cid:15) ))
iterations given enough devices satisfying S = Ω( (cid:15) − 2 d ) .
The device-sampling scheme I provides a viable solution to handle the straggler’s effect, which is rather robust to the data heterogeneity and doesn’t require the data to be balanced.
In more practical cases where a system can only operate based on the ﬁrst S messages for the local updates, Scheme II can achieve a reasonable approximation given more balanced data with uniformly sampled device.
If
S = 1 , our Scheme II matches the result in the Scheme I; If S = N , then our Scheme II recovers the result in the full device setting; If S = N − o ( N ) , our Scheme II bound is better than scheme I.
D IFFERENTIAL P RIVACY G UARANTEES
We consider the ( (cid:15), δ ) -differential
privacy with respect to the substitute-one relation (cid:39) s (Balle et al., 2018).
Two datasets S (cid:39) s S (cid:48) if they have the same size and differ by exactly one data point.
For (cid:15) ≥ 0 and δ ∈ [0 , 1] , a mechanism M is ( (cid:15), δ ) -differentially private w.r.t.
(cid:39) s if for any pair of input datasets S (cid:39) s S (cid:48) , and every measurable subset E ⊂ Range ( M ) , we have
[Equation]
Since partial device participation is more general, we focus on analyzing the differential privacy guarantee based on updates with partial devices.
Here, we present the result under scheme II.
For the result under scheme I, please refer to Theorem G.3 in the appendix.
Theorem 5.1 (Partial version of Theorem G.3) .
Assume assumptions G.1 and G.2 holds.
For any
[Equation] [Equation]
, then Algorithm 1 under scheme II is ( (cid:15) (3) K,T , δ (3) K,T ) - differentially private w.r.t.
(cid:39) s after T ( T = EK with E ∈ N , E ≥ 1 ) iterations where (cid:15)
[Equation]
min
[Equation]
with (cid:101)
[Equation]
According to
Theorem 5.
and section
G,
Algorithm is at least
[Equation] [Equation]
, SN γTδ
[Equation]
) -differentially private.
Moreover, if
[Equation]
, then (cid:15)
[Equation] [Equation]
= O
[Equation]
There is a trade-off between privacy and utility.
By Theorem 5.1, (cid:15) (3) K,T is an increasing function of η τ (1 − ρ 2 ) , SN , and T .
δ (3) K,T is an increasing function of SN , γ , and T .
However, by Theorem 4.10, the upper-bound of W 2 ( µ T , µ ) is a decreasing function of ρ , T , S and is an increasing function of τ and
N .
There is an optimal η to minimize W 2 ( µ T , µ ) for ﬁxed T while we can make (cid:15) (3) K,T arbitrarily small by decreasing η for any ﬁxed T .
In practice, users can tune hyper-parameters based on DP and accuracy budget.
For example, under some DP budget ( (cid:15) ∗ , δ ∗ ) , we can select the largest ρ ∈ [0 , 1] and S ∈ [ N ] such that (cid:15) (3) K,T ≤ (cid:15) ∗ and δ (3) K,T ≤ δ ∗ to achieve the target error W 2 ( µ T , µ ) .
E XPERIMENTS
Simulations For each c ∈ [ N ] , where N = 50 , we sample θ c from a 2d Gaussian distribution
N (0 , αI 2 ) and sample n c points from N ( θ c , Σ) , where Σ = (cid:20)− 2 − (cid:21) .
Thus, l ( θ ; x c,i ) =
12 ( θ − x c,i ) (cid:62) Σ − 1 ( θ − x c,i ) + log(
π | Σ | 12 ) , (cid:96) c
( θ ) = (cid:80) n c i =1 l ( θ ; x c,i ) .
The temperature is τ = 1 .
The target density is N ( u, 1 n Σ) with u = 1 n (cid:80)
Nc =1 (cid:80) n c
i =1 x c,i .
We choose a Gaussian posterior to facilitate the calculation of the W2 distance to verify theoretical properties.
Other details are detailed in section I.
[Figure]
(a
Study of K
[Figure]
Optimal local steps: We study the choices of local step K for Algorithm 1 based on ρ = 0 , full device, and different α ’s, which corresponds to different levels of data heterogeneity modelled by γ .
We choose α = 0 , 1 , 10 , 100 , 1000 and the corresponding γ is around 1 × 10 8 , 4 × 10 11 , 4 × 10 12 , 4 × 10 13 , and 4 × 10 14 , respectively.
We ﬁx η = 10 − 7 .
We evaluate the (log) number of communication rounds to achieve the accuracy (cid:15) =
10 − 3 and denote it by T (cid:15) .
As shown in Figure 1(a), a small K leads to an excessive amount of communication costs; by contrast, a large K results in large biases, which in turn requires high communications.
The optimal K that minimizes communication is aroundand the communication savings can be as large as 30 times .
Data heterogeneity and correlated noise: We study the impact of γ on the convergence based on ρ = 0 , full device, different γ from { 1 × 10 8 , 4 × 10 11 , 4 × 10 12 , 4 × 10 13 , and 4 × 10 14 } .
We set
K = 10 .
As shown in Figure 1(b), the W 2 distances under different γ all converge to some levels around 10 − 3 after sufﬁcient computations.
Nevertheless, a larger γ does slow down the convergence, which suggests adopting more balanced data to facilitate the computations.
In Figure 1(c), we study the impact of ρ on the convergence of the algorithm.
We choose K = 100 and γ = 10 8 and observe that a larger correlation slightly accelerates the computation, although it risks in privacy concerns.
Approximate samples:
In Figure 1(e), we plot the empirical density according to the samples from
Algorithm 1 with ρ = 0 , full device, K = 10 and γ = 10 8 , η = 10 − 7 .
For comparison, we show the true density plot of the target distribution in Figure 1(d).
The empirical density approximates the true density very well, which indicates that the potential of FA-LD in federated learning.
For the convergence based on partial device participation , we refer interested readers to section I.
(Fashion) MNIST To evaluate the performance of FA-LD under different local steps K on realworld datsets, we apply FA-LD to train a logistic regression model with the cross entropy loss on the
MNIST and Fashion-MNIST dataset.
The training dataset is split uniformly at random into N =subsets of equal size for 10 clients.
In each setting, we collect one parameter sample after everycommunication rounds and average the predicted probabilities made by all the previous collected parameter samples to calculate three test statistics, accuracy, Brier Score (BS) (Brier et al., 1950), and Expected Calibration Error (ECE) (Guo et al., 2017) on the test dataset.
We tune the step sizes η for the best performance and plot the curves of those test statistics against communication rounds under different local steps K = 1 , 10 , 20 , 50 , 100 in Figure 2.
Other details are provided in section J.
[Figure]
According to Figure 2, under the same communication budget , FA-LD with K =
(i.e. the standard
SGLD algorithm) performs the worst in terms of all three test statistics, which indicates the necessity of multiple local updates in federated learning.
Moreover, for different test statistics, the optimal local step K could be different; e.g., for the MNIST dataset, the optimal K in terms of accuracy is between 50 and 100 (Figure 2(a)), while the optimal K in terms of BS is around 20 (Figure 2(b)).
To better visualize the convergence, we also plot the curves of accuracy, BS, and ECE when a warmup period consisting of the ﬁrst 500 communication rounds is removed.
Such results are detailed in section J.
C ONCLUSION
We propose a novel convergence analysis for federated averaging Langevin dynamics (FA-LD) with distributed clients.
Our results no longer require the bounded gradient assumption in (cid:96) 2 norm as in the optimization-driven literature in federated learning.
The theoretical guarantees yield a concrete guidance on the selection of the optimal number of local updates to minimize communication costs.
In addition, the convergence highly depends on the data heterogeneity and the injected noises, where the latter also inspires us to consider correlated injected noise and partial device updates to balance between differential privacy and prediction accuracy with theoretical guarantees.
APPENDIX
Roadmap.
In Section A, we layout the formulation of the algorithm, basic notations, and deﬁnitions.
In Section B, we present the main convergence analysis for full device participation.
We discuss the optimal number of local updates based on a ﬁxed learning rate, the acceleration achieved by varying learning rates, and the privacy-accuracy trade-off through correlated noises.
In Section C, we analyze the convergence of partial device participation through two device-sampling schemes.
In Section D, we provide lemmas to upper bound the contraction, discretization and divergence for proving the main convergence results.
In Section E, we include supporting lemmas to prove results in the previous section.
In Section F, we establish the initial condition.
In Section G, we prove differential privacy guarantees.
In Section H, we discuss the related work and literature.
In Section I, we include additional experiments.
A P RELIMINARIES A. B ASIC NOTATIONS AND BACKGROUNDS
Let N denote the number of clients.
Let T (cid:15) denote the number of global steps to achieve the precision (cid:15) .
Let K denote the number of local steps.
For each c ∈ [ N ] := { 1 , 2 , · · · , N } , we use f c and ∇ f c denote the loss function and gradient of the function f c in client c .
Notably, ∇ f is not a standard gradient operator acting on f when multiple local steps are adopted ( K > 1 ).
For the stochastic gradient oracle, we denote by ∇ (cid:101)
f c ( · ) the unbiased estimate of the exact gradient ∇ f c of client c .
In addition, we denote p c as the weight of the c -th client such that p c ≥ 0 and (cid:80)
Nc =1 p c = 1 .
ξ ck is an independent standard d -dimensional Gaussian vector at iteration k for each client c ∈ [ N ] and ˙ ξ k is a unique Gaussian vector shared by all the clients.
Algorithm
Federated averaging Langevin dynamics algorithm
(FA-LD).
Denote by θ ck the model parameter in the c -th client at the k -th step.
Denote the immediate result of one step SGLD update from θ ck by β ck .
ξ ck is an independent standard d -dimensional Gaussian vector at iteration k for each client c ∈ [ N ] .
A global synchronization is conducted every K steps.
This is a transformed version of Algorithm 1 with ρ = 0 and full device participation for ease of analysis.
[Equation]
Inspired by Li et al.
(2020c)
, we deﬁne two virtual sequences
[Equation]
which are both inaccessible when k mod K (cid:54) = 0 .
For the gradients and injected noise, we also deﬁne ∇ f ( θ k ) =
N (cid:88)
[Equation]
[Equation]
In what follows, it is clear that E ∇ (cid:101)
f
( θ ) = (cid:80)
Nc =1 p c E ∇ (cid:101) f
c ( θ c ) = ∇ f ( θ ) for any θ c ∈ R d and any c ∈ [ N ] .
Summing Eq.(14) from clients c
= 1 to N and combining Eq.(16) and Eq.(17), we have
[Equation]
Moreover, we always have β k
= θ k whether k + 1 mod K = 0 or not by Eq.(15) and Eq.(16).
In what follows, we can write
[Equation]
which resembles the SGLD algorithm
(Welling & Teh, 2011) except that the construction of stochastic gradients is different and θ k is not accessible when k mod K (cid:54) = 0 .
A.1. F EDERATED AVERAGING L ANGEVIN DIFFUSION
To facilitate the analysis, we deﬁne the federated averaging Langevin diffusion for any c ∈ [ N ]
[Equation]
where ¯ θ ct is the continuous-time variable at time t {+for client c and ¯ β ct is the immediate result; W c is a d -dimensional Brownian motion and is mutually independent for different c ; S and D are two sets that follow R ≥ 
+} = S ∪ D such that [Equation]
where ∇ t ∈ R + denotes the synchronization frequency. Starting from some t > 0 and δ > 0 such that t ∈ S and t + δ ∈ (cid:18) ∇ t (cid:22) t ∇ t (cid:23) , ∇ t (cid:18)(cid:22) t ∇ t (cid:23) + 1 (cid:19)(cid:19) , we have for any c ∈ [ N ] [Equation] We take the weighted average and obtain
[Equation] where ¯ β t = (cid:80)
Nc =1 p c ¯ β ct , ¯ θ t = (cid:80) Nc =1 p c ¯ θ ct , W is a d -dimensional Brownian motion. Sending ∇ t → 0 , the continuity implies that
[Equation] Since the synchronization is conducted via ∇ t → 0 in continuous time, we recover an auxiliary
Langevin diffusion (¯ θ t ) for any t ∈ S
[Equation]
Moreover, we always have ¯ θ t = ¯ β t = (cid:80)
Nc =1 p c ¯ β ct on both the set D of measure 0 and set S . It is equivalent to studying the standard Langevin diffusion (¯ θ t ) t ≥ [Equation] which converges to the stationary distribution π (¯ θ ) ∝ exp( − f (¯ θ ) /τ ) and f (¯ θ )
= (cid:80)
Nc =1 p c f c (¯ θ ) .
A. A SSUMPTIONS AND DEFINITIONS
Assumption A.1 (Smoothness, restatement of Assumption 4.1) .
For each c ∈ [ N ] , we say f c is
L -smooth if for some L >
[Equation]
Note that the above assumption is equivalent to saying that
[Equation]
Assumption A.2 (Strong convexity, restatement of Assumption 4.2) .
For each c ∈ [ N ] , f c is m -strongly convex if for some m >
[Equation]
An alternative formulation for strong convexity is that
[Equation]
Assumption A.3 (Bounded variance, restatement of Assumption 4.3) .
For each c ∈ [ N ] , the variance of noise in the stochastic gradient ∇ (cid:101) f c ( x ) in each client is upper bounded such that
[Equation]
The bounded variance in the stochastic gradient is a rather standard assumption and has been widely used in Cheng et al.
(2018); Dalalyan & Karagulyan (2019);
Li et al.
(2020c).
Extension of bounded variance to unbounded cases such as E [ (cid:107)∇ (cid:101)
f c ( x ) − ∇ f c ( x ) (cid:107) 22 ] ≤ δ ( L 2 x 2 + B 2 ) for some M and δ ∈ [0 , 1) is quite straightforward and has been adopted in assumption A.4 stated in Raginsky et al.
The proof framework remains the same.
Quality of non-i.i.d data Denote by θ ∗ the global minimum of f .
Next, we quantify the degree of the non-i.i.d data by γ :=
max c ∈ [ N ] (cid:107)∇ f c ( θ ∗ ) (cid:107) 2 , which is a non-negative constant and yields a smaller scale if the data is more evenly distributed.
Deﬁnition A.4.
We deﬁne parameter T c,ρ H ρ , κ and γ
[Equation]
data heterogeneity
[Equation]
stochastic noise
[Equation]
B F ULL DEVICE PARTICIPATION B. O NE - STEP UPDATE
Wasserstein distance We deﬁne the 2-Wasserstein distance between a pair of Borel probability measures µ and ν on R d as follows
[Equation]
where (cid:107) · (cid:107) 2 denotes the (cid:96) 2 norm on R d and the pair of random variables ( β µ , β ν
) ∈ R d × R d is a coupling with the marginals following L ( β µ ) = µ and L ( β ν )
=
ν .
L ( · ) denotes a distribution of a random variable.
The following result provides a crucial contraction property based on distributed clients with infrequent synchronizations.
Lemma B.1 (Dominated contraction property, restatement of Lemma 4.4) .
Assume assumptions A. and A.2 hold.
For any learning rate η ∈ (0 ,
[Equation] L [Equation] , any { θ c } Nc =1 , { β c } Nc =1 ∈ R d , we have
[Equation]
N (cid:88) c
= p c · (
[Equation]
{+β c − β (cid:107) 
 + (cid:107) θ c − θ+} (cid:107) 22 ) .
where β = (cid:80)
Nc =1 p c β c , θ = (cid:80)
Nc =1 p c θ c , ∇ f ( θ )
=
(cid:80)
Nc =1 p c ∇ f c ( θ c ) , and ∇ f ( β ) =
(cid:80)
Nc =1 p c ∇ f c ( β c ) .
We postpone the proof into Section D.1.
The above result implies that as long as the local parameters θ c , β c and global θ, β don’t differ each other too much, we can guarantee the desired convergence.
The following result ensures a bounded gap between ¯ θ cs and ¯ θ cη (cid:98) sη (cid:99) in (cid:96) 2 norm for any s ≥ 0 and c ∈ [ N ] .
We postpone the proof of Lemma B.2 into Section D.2.
Lemma B.2 (Discretization error) .
Assume assumptions A.1, A.2, and A.3 hold.
For any s ≥ 0 , any learning rate η ∈ (0 , 2 /m ) and (cid:107) θ c 0 − θ ∗ (cid:107) 22 ≤ d D 2 for any c ∈ [ N ] , the iterates of (¯ θ s ) based on the continuous dynamics of Eq.(20) satisfy the following estimate
[Equation]
The following result shows that given a ﬁnite number of local steps K , the divergence between θ c in local client and θ in the center is bounded in (cid:96) 2 norm.
Notably, since the non-differentiable Brownian motion leads to a lower order term O ( η ) instead of O ( η 2 ) in (cid:96) 2 norm, a naïve proof may lead to a crude upper bound.
We delay the proof of Lemma B.3 into Section D.3.
Lemma B.3 (Bounded divergence, restatement of Lemma 4.5) .
Assume assumptions A.1, A.2, and
A.3 hold.
For any learning rate η ∈ (0 , 2 /m ) and (cid:107) θ c 0 − θ ∗ (cid:107) 2 2 ≤ d D 2 for any c ∈ [ N ] , we have the (cid:96) 2 upper bound of the divergence between local clients and the center as follows
[Equation]
where H ρ , κ and γ 2 are deﬁned as Deﬁnition A.4.
The following presents a standard result for bounding the gap between ∇ f ( θ ) and ∇ (cid:101) f
( θ ) .
We delay the proof of Lemma B.4 into Setion D.
Lemma B.4 (Bounded variance) .
Given assumption A.3, we have
[Equation]
Having all the preliminary results ready, now we present a crucial lemma for proving the convergence of all the algorithms.
Lemma B.5 (One step update, restatement of Lemma 4.6) .
Assume assumptions A.1, A.2, and A. hold.
Consider Algorithm 2 with independently injected noise ρ = 0 , any learning rate η ∈ (0 , 12 L ) and (cid:107) θ c 0 − θ ∗ (cid:107) 22 ≤ d D 2 for any c ∈ [ N ] , where θ ∗ is the global minimum for the function f .
Then
[Equation]
where µ k denotes the probability measure of θ k , H 0 , κ and γ 2 are deﬁned as Deﬁnition A.4.
Proof.
The solution of the continuous-time process Eq.(20) follows that
[Equation]
Set t → ( k + 1) η
and ¯ θ
→ ¯ θ kη for Eq.(21) and consider a synchronous coupling such that W ( k +1) η
[Equation]
W kη := √ ηξ k
[Equation]
We ﬁrst denote ζ k := ∇ (cid:101)
f ( θ k ) − ∇ f ( θ k ) .
Subtracting Eq.(19) from Eq.(22) yields that
[Equation]
Taking square and expectation on both sides
[Equation]
we have
[Equation]
independence
[Equation]
where the ﬁrst inequality follows by the AM-GM inequality for any q > 0 , the second inequality follows by Lemma B.1 and Assumption A.3.
The third inequality follows by Lemma B.3 with ρ = 0 ; moreover, the continuous-time process conducts synchronization at any time step, hence ¯ θ ckη = ¯ θ kη .
Since the learning rate follows 12 L ≤ 1 m + L ≤ 2 m , the requirement of the learning rate for Lemma B. and Lemma B.3 is clearly satisﬁed.
Recall that φ = 1 − ηm , we get 1+ φ 2 = 1 − 12 ηm .
Choose q = 1+ φ 2 φ − 1 so that (1 + q ) φ = (1+ φ )= 1 − 12 ηm .
In addition, we have 1 + 1 q = 1+ qq = 1+ φ 1 − φ ≤ 2 ηm .
It follows that
[Equation]
where the second inequality holds because η ∈ (0 , 12 L ] ≤ 12 m .
For the term E (cid:107) Y k (cid:107) 22 in Eq.(24), we have the following estimate [Equation]
[Equation]
where the ﬁrst inequality follows by Hölder’s inequality, the second inequality follows by Jensen’s inequality, the third inequality follows by Assumption A.1, and the last inequality follows by Lemma
B.2.
The last equality holds since (cid:8) (cid:8)(cid:8) κ d γ 2 + Lτ ≤ LmH 0 and κ = L/m .
Plugging Eq.(25) and Eq.(26) into Eq.(24), we have
[Equation]
Choose the speciﬁc Langevin diffusion ¯ θ in stationary regime, we have W 22 ( µ k , π ) =
E (cid:107) ¯ θ kη − θ k (cid:107)and W 22 ( µ k +1 , π ) ≤ E (cid:107) ¯ θ ( k +1)
η − θ k +1 (cid:107) 22 .
Arranging the terms, we have
[Equation]
+ [Equation] where η ≤ 12 L , κ ≥ 1 , mτ ≤ Lτ ≤ LτN ≤ L max c ∈ [ N ] T c, 0 ≤ LmH 0 , and σ 2 ≤ L 2 H 0 are applied to the result.
B. C ONVERGENCE VIA INDEPENDENT NOISES
Theorem B.6 (Restatement of Theorem 4.7) .
Assume assumptions A.1, A.2, and A.3 hold.
Consider
Algorithm 2 with a ﬁxed learning rate η ∈ (0 , 12 L ] and (cid:107) θ c 0 − θ ∗ (cid:107) 22 ≤ d D 2 for any c ∈ [ N ] , we have
[Equation]
where µ k denotes the probability measure of θ k , H 0 , κ and γ 2 are deﬁned as Deﬁnition A.4.
Proof.
Iteratively applying Theorem B.5 and arranging terms, we have that
[Equation]
[Equation]
where κ = Lm .
By Lemma F.1 and the initialization condition (cid:107) θ c 0 − θ ∗ (cid:107) 22 ≤ d D 2 for any c ∈ [ N ] , we have that
[Equation]
Applying the inequality (1 − ηm 2 ) ≤ (1 − ηm 4 ) 2 completes the proof.
Discussions
Optimal choice of K .
To ensure the algorithm to achieve the (cid:15) precision based on the total number of steps T (cid:15) and the learning rate η , we can set
[Equation]
This directly leads to ηm ≤ min
[Equation]
m
[Equation]
[Equation]
(cid:19) . Plugging into the upper bound of η , it implies that to reach the precision level (cid:15) , it sufﬁces to set
[Equation]
Since H 0 = Ω( D 2 + τm ) , we observe that the number of communication rounds is around the order
[Equation]
where the value of T (cid:15) K ﬁrst decreases and then increases with respect to K , indicating that setting K either too large or too small may lead to high communication costs and hurt the performance
Ideally,
K should be selected in the scale of Ω( √ κ ) .
Combining the deﬁnition of T (cid:15) in Eq.(28), this suggests an interesting result that the optimal K should be in the order of O ( √ T (cid:15) ) .
Similar results have been achieved by Stich (2019);
Li et al.
(2020c).
B. C ONVERGENCE VIA VARYING LEARNING RATES
Theorem B.7 (Restatement of Theorem 4.8) .
Assume assumptions A.1, A.2, and A.3 hold.
Consider
Algorithm 2 with an initialization satisfying (cid:107) θ c 0 − θ ∗ (cid:107) 22 ≤ d D 2 for any c ∈ [ N ] and varying learning rate following
[Equation]
Then for any k ≥ 0 , we have
[Equation]
Proof.
We ﬁrst denote
[Equation]
Next, we proceed to show the following inequality by the induction method
[Equation]
where the decreasing learning rate follows that
[Equation]
(i)
For the case of k = 0 , since
[Equation]
where the last inequality follows by Lemma F.1 and (cid:107) θ c 0 − θ ∗ (cid:107) 22 ≤ d D 2 for any c ∈ [ N ] .
It is clear that W 2 ( µ 0 , π ) ≤ 14 C [Equation]
[Equation] md by Eq.(30).
(ii)
If now that Eq.(29) holds for some k ≥ 0 , it follows by Lemma B.
that
[Equation]
Since (cid:0) 1 − η k m 6 (cid:1) ≤ (cid:0) 1 − η k m 12 (cid:1) 2 , we have
[Equation]
To prove W 2 ( µ k +1 , π ) ≤ 1 .
5 C κ (cid:0) η k +1 md (cid:1) 1 / 2 , it sufﬁces to show (cid:0) 1 − η k m 12 (cid:1)
η 1 / 2 k ≤ η k +1 , which is detailed as follows
[Equation]
where the last inequality follows since
[Equation]
The above result implies that to achieve the precision (cid:15) , we require
[Equation]
The means that we only require k = Ω( d(cid:15) 2 ) to achieve the precision (cid:15) .
By contrast, the ﬁxed learning rate requires T (cid:15) =
Ω (cid:18) d(cid:15) 2 · log (cid:0) d/(cid:15) (cid:1)(cid:19) , which is much slower than the algorithm with varying learning rate by O (cid:0) log (cid:0) d/(cid:15) (cid:1)(cid:1) times.
B. P RIVACY - ACCURACY TRADE - OFF VIA CORRELATED NOISES
Note that Algorithm 2 requires all the local clients to generate the independent noise ξ ck .
Such a mechanism enjoys the convenience of the implementation and yields a potential to protect the privacy of data and alleviates the security issue.
However, the scale of noises is maximized and inevitable slows down the convergence.
For extensions, it can be naturally generalized to correlated noise based on a hyperparameter, namely the correlation coefﬁcient ρ between different clients.
Replacing
Eq.(14) with
[Equation]
[Equation]
where ˙ ξ k is a d -dimensional standard Gaussian vector shared by all the clients at iteration k , ξ ck is a unique d -dimensional Gaussian vector generated by client c ∈ [ N ] only.
Moreover, ˙ ξ k is dependent with ξ ck for any c ∈ [ N ] .
Following the same synchronization step based Eq.(15), we have
[Equation]
(32) where ξ k = ρξ
k + (cid:112) 1 − ρ 2 (cid:80)
Nc =1 √ p c ξ ck .
Since the variance of i.i.d variables is additive, it is clear that ξ k follows the standard d -dimensional Gaussian distribution.
The inclusion of the correlated noise implicitly reduces the temperature and naturally yields a trade-off between federation and accuracy.
We refer to the algorithm with correlated noise as the hybrid Federated Averaging Langevin dynamics (gFA-LD) and present it in Algorithm 3.
Since the inclusion of correlated noise doesn’t affect the formulation of Eq.(32), the algorithm property maintains the same except the scale of the temperature τ and federation are changed.
Based on a target correlation coefﬁcient ρ ≥ 0 , Eq.(31) is equivalent to applying a temperature
T c,ρ = τ ( ρ 2 + (1 − ρ 2 ) /p c ) .
In particular, setting ρ = 0 leads to T c,
=
(1 − ρ 2 ) /p c , which exactly recovers Algorithm 2; however, setting ρ = 1 leads to T c, 1 = τ , where the injected noise in local clients is reduced by 1 /p c times.
Now we adjust the analysis as follows
Theorem B.8 (Restatement of Theorem 4.9) .
Assume assumptions A.1, A.2, and A.3 hold.
Consider Algorithm 3 with a correlation coefﬁcient ρ ∈ [0 , 1] , a ﬁxed learning rate η ∈ (0 , 12 L ] and (cid:107) θ c 0 − θ ∗ (cid:107) 22 ≤ d D 2 for any c ∈ [ N ] , we have
[Equation]
where µ k denotes the probability measure of θ k , H ρ , κ and γ 2 are deﬁned as Deﬁnition A.4.
Proof.
The proof follows the same techniques as in Theorem B.6 except that H 0 is generalized to
H ρ to accommodate to the changes of the injected noise .
The details are omitted.
Algorithm 3 Hybrid federated averaging Langevin dynamics algorithm.
Denote by θ ck the model parameter in the c -th client at the k -th step.
Denote the immediate result of one step SGLD update from θ ck by β ck .
ξ ck is an independent standard d -dimensional Gaussian vector at iteration k for each client c ∈ [ N ] and ˙ ξ k is a d -dimensional standard Gaussian vector shared by all the clients.
ρ denotes the correlation coefﬁcient of the injected noises.
A global synchronization is conducted every K steps.
This is a clean version of Algorithm 1 based on full device updates for ease of analysis.
[Equation]
p c β ck + if k + 1 mod K = 0 .
C P ARTIAL DEVICE PARTICIPATION
Full device participation enjoys appealing convergence properties.
However, it suffers from the straggler’s effect in real-world applications, where the communication is limited by the slowest device.
Partial device participation handles this issue by only allowing a small portion of devices in each communication and greatly increased the communication efﬁciency in a federated network.
C. U NBIASED SAMPLING SCHEMES
The ﬁrst device-sampling scheme I (Li et al., 2020b) selects a total of S devices, where the c -th device is selected with a probability p c .
The ﬁrst theoretical justiﬁcation for convex optimization has been proposed by Li et al.
(2020c).
(Scheme I: with replacement).
Assume S k
= { n 1 , n 2 , · · · , n S } , where n j ∈ [ N ] is a random number that takes a value of c with a probability p c for any j ∈ { 1 , 2 , · · · , S } .
The synchronization step follows that θ k = 1 S (cid:80) c ∈S k θ ck .
Another strategy is to uniformly select S devices without replacement.
We follow Li et al.
(2020c)
and assume S indices are selected uniformly without replacement and the synchronization step is the same as before.
In addition, the convergence also requires an additional assumption on balanced data (Li et al., 2020c).
(Scheme II: without replacement).
Assume S k
= { n 1 , n 2 , · · · , n S } , where n j ∈ [ N ] is a random number that takes a value of c with a probability 1 S for any j ∈ { 1 , 2 , · · · , S } .
Assume the data is balanced such that p 1 = · · · = p N = 1 N .
The synchronization step follows that θ k = NS (cid:80) c ∈S k p c θ ck = 1 S (cid:80) c ∈S k θ ck . Lemma C.1 (Unbiased sampling scheme) . For any k mod K = 0 based on scheme I or II, we have E
[Equation]
k [Equation] ξ ck is the independent Gaussian vector proposed by each client c ∈ [ N ] and ˙ ξ k is a unique
Gaussian vector shared by all the clients.
ρ denotes the correlation coefﬁcient.
A global synchronization is conducted every K steps.
S k is a subset that contains S indices according to a device-sampling rule based on scheme I or II.
This is a clean version of Algorithm 1 for ease of analysis.
[Equation]
Proof.
According to the deﬁnition of scheme I or II, we have θ k = 1 S (cid:80) c
∈S k θ ck .
In what follows,
E θ k
= 1 S E (cid:80) c ∈S k θ ck = 1 S (cid:80) c 0 ∈S k (cid:80)
Nc =1 p c β ck =
(cid:80)
Nc =1 p c β ck , where p 1 = p 2 = · · · = p N for scheme II in particular. [Equation]
B OUNDED DIVERGENCE BASED ON PARTIAL DEVICE
Lemma C.2 (Bounded divergence based on partial device) .
Assume assumptions A.1, A.2, and A. hold.
Consider Algorithm 4 with a correlation coefﬁcient ρ ∈ [0 , 1] , any learning rate η ∈ (0 , 2 /m ) and (cid:107) θ c 0 − θ ∗ (cid:107) 22 ≤ d D 2 for any c ∈ [ N ] , we have the following results
For Scheme I, the divergence between θ k and β k is upper bounded by
[Equation]
For Scheme II, assuming the data is balanced such that p 1 = · · · = p N = 1 N , the divergence between θ k and β k is upper bounded by
[Equation]
where H ρ , κ and γ 2 are deﬁned as Deﬁnition A.4.
Proof.
We prove the bounded divergence for the two schemes, respectively.
For scheme I with replacement, ¯ θ k = (cid:80)
c
∈S k 1 S β c k for a subset of indices S k .
Taking expectation with respect to S k , we have
[Equation]
where the ﬁrst equality follows by the independence and unbiasedness of θ n i k for any i ∈
[ S ] .
To further upper bound Eq.(33), we follow the same technique as in Lemma B.3.
Since k mod K = 0 , k 0 = k − K is also the communication time, which yields the same θ ck 0 for any c ∈ [ N ] .
in what follows,
N (cid:88)
[Equation]
where the last inequality follows by β k = (cid:80)
Nc =1 p c β ck and E (cid:107) x
− E x (cid:107) 22 ≤ E (cid:107) x
(cid:107) 22 .
Combining
Eq.(33) and Eq.(34), we have
[Equation]
where the last inequality follows a similar argument as in Lemma B.3.
For scheme II , given p 1 = p 2 = · · · = p N = 1 N , we have θ k = 1 S (cid:80) c
∈S k β ck , which leads to
E (cid:107)
[Equation]
where I A is an indicator function that equals to 1 if the event A happens.
Plugging the facts that P ( c ∈ S k )
=
SN and P ( c 1 , c 2 ∈ S k )
=
S ( S − 1) N ( N − 1) for any c 1 (cid:54) =
c 2 ∈ [ N ] into the above equation [Equation] we have
[Equation]
22 , where the last equality holds since (cid:80) {+c ∈ [ N ] (cid:107) β ck − β k (cid:107) 
 + (cid:80) c 1 (cid:54) = c 2 (cid:104)+}
{+β c 1 k − β k , β c 2 k − β k (cid:105) = (cid:107) β k − β k (cid:107) 
 =+}
Eventually, we have
[Equation]
where the ﬁrst inequality follows a similar argument as in Eq.(34) and the last inequality follows by
Lemma B.3.
C. C ONVERGENCE VIA PARTIAL DEVICE PARTICIPATION
Theorem C.3 (Restatement of Theorem 4.10) .
Assume assumptions A.1, A.2, and A.3 hold.
Consider Algorithm 4 with a correlation coefﬁcient ρ ∈ [0 , 1] , a ﬁxed learning rate η ∈ (0 , 12 L ] and (cid:107) θ c 0 − θ ∗ (cid:107) 22 ≤ d D 2 for any c ∈ [ N ] , we have
[Equation]
where C K = ηmK 1 − e − ηmK 2 , C S = 1 for Scheme I and C S = N − S N − 1 for Scheme II .
Proof.
Note that
[Equation]
where the last equality follows by the unbiasedness of the device-sampling scheme in Lemma C.1.
If k + 1 mod K (cid:54) = 0 , we always have β k
+
=
θ k +1 and E (cid:107) β
k +1 − θ k +1 (cid:107)
= 0 .
Following the same argument as in Lemma B.5, both schemes lead to the one-step iterate as follows
[Equation]
If k + 1 mod K = 0 , combining Lemma C.2 and Lemma B.5, we have
[Equation]
where C S = 1 for Scheme I and C S =
N − S N − 1 for Scheme II .
Repeatedly applying Eq.(35) and Eq.(36) and arranging terms, we have that
[Equation]
where the second inequality follows by (1 − r )
K ≤ e − rK for any r ≥ 0 .
D B OUNDING CONTRACTION , DISCRETIZATION , AND DIVERGENCE D. D OMINATED CONTRACTION PROPERTY
Proof of Lemma B.1 .
Given a client index c ∈ [ N ] , applying Theorem
[Equation]
1.12 (Nesterov, 2004) leads to
[Equation]
In what follows, we have
[Equation]
For the second item I in the right hand side, we have I = [Equation] =
[Equation]
≥ −
[Equation]
where the ﬁrst inequality follows by the AM-GM inequality and Eq.(37) [Equation] respectively; the last inequality follows by Jensen’s inequality such that
[Equation]
[Table]
[Equation]
Plugging Eq.(39) into Eq.(38), we have
[Equation]
where the last inequality follows by 2 L m + L ≥ 1 , m ≤ L , 1 − 2 a ≤ (1 − a ) 2 for any a , and η ∈ (0 , 1 m + L ] .
D. D ISCRETIZATION ERROR
Proof of Lemma B.2.
In the continuous-time diffusion (20), the synchronization is conducted in inﬁnitesimal time ∇ t → 0 .
It follows that ∇ f (¯ θ ) = (cid:80) Nc =1 p c f c (¯ θ ) for any ¯ θ ∈ R d and it is straightforward to verify that f satisﬁes both Assumption A.1 and A.2 with the same smoothness factor L and convexity constant m when ∇ t → 0 (it does not hold for federated settings with non-trivial synchronization frequency, i.e. ∇ t > 0 ).
For any s ∈ [0 , ∞ ) , there exists a certain k ∈ N + such that s ∈
[ kη, ( k + 1) η ) .
By the continuous dynamics of Eq.(20) , we have
[Equation]
which suggests that
[Equation]
We ﬁrst square the terms on both sides and take Young’s inequality and expectation
E sup s ∈ [ kη,
[Equation]
k +
[Equation]
η )
[Equation]
Then, by Cauchy Schwarz inequality and the fact that | s − kη | ≤ η , we have
[Equation]
where the last inequality follows by Burkholder-Davis-Gundy inequality (50) and Itô isometry.
By Young’s inequality, the smoothness assumption A.1 and ∇ f ( θ ∗ )
= 0 when ∇ t → 0 , we have
[Equation]
where the second inequality follows by Theorem 17 (Cheng et al., 2018) since ¯ θ 0 is simulated from the stationary distribution π and ¯ θ s is stationary.
Combining Eq.(40) and Eq.(41), we have
[Equation]
D. B OUNDED DIVERGENCE
Proof of Lemma B.3.
For any k ≥ 0 , consider k
=
K (cid:98) kK (cid:99) such that k ≤ k 0 and θ ck
= θ k 0 for any k ≥ 0 .
It is clear that k − k 0 ≤ K − 1 for all k ≥ 0 .
Consider the non-increasing learning rate such that η k 0 ≤ 2 η k for all k − k 0 ≤ K − 1 .
By the iterate Eq.(19), we have
[Equation]
≤
[Equation]
c [Equation] where the ﬁrst inequality holds by E (cid:107) θ − E θ (cid:107) 22 ≤ E (cid:107) θ (cid:107) 22 for a stochastic variable θ ; the second inequality follows by ( (cid:80) K − 1 i =1 a i ) 2 ≤ ( K − 1) (cid:80) K − 1 i =1 a 2 i ; the last inequality follows by Lemma E. and η 2 k 0 ≤ 4 η 2 k .
H ρ is deﬁned in Deﬁnition A.4.
D. B OUNDED VARIANCE
Proof of Lemma B.4.
By assumption A.3, we have
[Equation]
E U NIFORM UPPER BOUND E. D ISCRETE DYNAMICS
Lemma E.1 (Discrete dynamics) .
Assume assumptions A.1, A.2, and A.3 hold.
We consider the generalized formulation in Algorithm 3 with the temperature
[Equation]
given a correlation coefﬁcient ρ .
For any learning rate η ∈ (0 , 2 /m ) and (cid:107) θ c 0 − θ ∗ (cid:107) 22 ≤ d D 2 for any c ∈ [ N ] , we have the (cid:96) 2 norm upper bound as follows
[Equation]
where γ := max c ∈ [ N ] (cid:107)∇ f c ( θ ∗ ) (cid:107)
and θ ∗ denotes the global minimum for the function f .
Proof.
First, we consider the k -th iteration, where k ∈ { 1 , 2 , · · · , K − 2 , ( K − 1) − } and ( K − 1) − denotes the ( K − 1) -step before synchronization.
Following the iterate of Eq.(14) in a local client of c ∈ [ N ] , we have [Equation]
[Equation]
where the last equality follows from E ξ k = 0 and the conditional independence of θ ck − θ ∗ − (cid:101)
f c ( θ ck ) and ξ k .
Note that
[Equation]
where the ﬁrst step follows from simple algebra
[Equation]
the second step follows from the unbiasedness of the stochastic gradient [Equation] and the last step follows from Assumption A.3.
For any q > [Equation] we can upper bound the ﬁrst term of Eq.(43) as follows
[Equation]
where the ﬁrst inequality follows by the AM-GM inequality; the second inequality is a special case of Lemma B.1 based on Assumption A.2, where no local steps is involved before the synchronization step.
Similar results have been achieved in Theorem 3 (Dalalyan, 2017).
In addition, γ := max c ∈ [ N ] (cid:107)∇ f c ( θ ∗ ) (cid:107) 2 .
Choose q =
( 1+ ψ 2 ψ ) 2 − 1 so that (1 + q ) ψ 2 =
(1+ ψ ) 2.
Moreover, since ψ = 1 − ηm 2 , we get 1+ ψ 2 = 1 − 14 ηm .
In addition, we have 1 + 1 q = 1+ qq =
[Equation]
1+ ψ
[Equation]
2 (1 − ψ )(1+3 ψ ) ≤ 2 ηm .
It follows that
[Equation]
Combining Eq.(42), Eq.(43), Eq.(44), and Eq.(45), we have the following iterate
[Equation]
Note that
[Equation]
given η ∈ (0 , 2 m ) .
Recursively applying the above equation k times, where k ∈ { 1 , 2 , · · · , K − 1 , K − } and K − denotes the K -step without synchronization, it follows that
[Equation]
where the second inequality holds by g ( η ) ≤ 1 , the third inequality holds because (cid:107) θ [Equation] − θ ∗ (cid:107) 22 ≤ d Dfor any c ∈ [ N ] and η < 2 m .
In particular, the K -th step before synchronization yields that
[Equation]
Having all the results ready, for the K -local step after synchronization, applying Jensen’s inequality
[Equation]
Now starting from iteration K
[Equation]
we adapt the recursion of Eq.(46) for the k -th step, where k ∈ { K + 1 , · · · , 2 K − 1 , (2 K ) − } and (2 K ) − denotes the 2 K -step without synchronization, we have [Equation]
[Equation]
where the second inequality follows by Eq.(48), the fact that 1 − g ( η ) ≥ ηm/ 3 and η ≤ 2 m , and the deﬁnition of U .
The third one holds since g ( η ) ≤ 1 .
By repeating Eq.(48) and (49), we have that for all k ≥ 0 , we can obtain the desired uniform upper bound.
Discussions: Since the above result is independent of the learning rate η , it can be naturally applied to the setting with decreasing learning rates.
The details are omitted.
E. B OUNDED GRADIENT
Lemma E.2 (Bounded gradient in (cid:96) 2 norm) .
[-where γ := max c ∈ [ N ] (cid:107)∇ f c ( θ ∗ ) (cid:107) 
-]
Given assumptions A.1, A.2, and A.3 hold, for any client c and any learning rate η ∈ (0 , 2 /m ) and (cid:107) θ c 0 − θ ∗ (cid:107) 22 ≤ d D 2 for any c ∈ [ N ]
[Equation]
we have the (cid:96)norm upper bound as follows
[Equation]
where
[Equation]
Proof.
Decompose the (cid:96) 2 of the gradient as follows
[Equation]
where the ﬁrst inequality follows by Assumption A.3; the second inequality follows by Young’s inequality; the third inequality follows by Assumption A.1 and the deﬁnition that γ := max [Equation] f c ( θ [Equation]
[Equation]
2 ; the fourth inequality follows by Lemma E.1; the last inequality follows by κ :=
Lm ≥ 1 .
F I NITIAL CONDITION
Lemma F.1 (Initial condition) .
Let µ 0 denote the Dirac delta distribution at θ 0 .
Then, we have
[Equation]
Proof.
By Cheng et al.
(2018), there exists an optimal coupling between µ 0 and π such that
[Equation]
where the second step follows from triangle inequality, the last step follows from Lemma 12 (Durmus & Moulines, 2019) and the temperature τ is included to adapt to the time scaling.
Burkholder-Davis-Gundy inequality Let φ : [0 , ∞ )
→ R r × d for some positive integers r and d .
In addition, we assume E (cid:82) ∞ 0 | φ ( s ) |
d s < ∞ and let Z ( t ) = (cid:82) t 0 φ ( s )
d W s , where W s is a d -dimensional
Brownian motion.
Then for all t ≥ 0 , we have
[Equation]
G M ORE ON D IFFERENTIAL P RIVACY G UARANTEES
We make the following assumptions for the analysis of DP.
Assumption G.1 (Bounded (cid:96) 2 -sensitivity) .
The gradient of loss function l : R d × X → R , ( θ, x ) (cid:55)→ l ( θ ; x ) w.r.t.
θ has a uniformly bounded (cid:96) 2 -sensitivity for ∀ θ ∈ R d :
[Equation]
For example, when l ( θ ; · ) is M -Lipschitz for any θ ∈ R d , ∆ l ≤ 2 M .
Following the tradition of DP guarantees, we assume that the unbiased gradient is computed as follows.
Assumption G.2 (Unbiased gradient estimates) .
The unbiased estimate of ∇ f c ( θ ) is calculated using a subset (denoted by S c ) of { x c,i } n c i =1 sampled uniformly at random from all the subsets of size γn c of { x c,i } n c i =1 :
[Equation]
Theorem G.3.
Assume assumptions G.1 and G.2 holds.
For any δ 0 ∈ (0 , 1) , if η ∈ (cid:16) 0 ,
[Equation]
l log(1 . 25 /δ 0 ) (cid:105) , then Algorithm 1 is ( (cid:15) (3) K,T , δ (3) K,T ) -differentially private w.r.t.
(cid:39) s after executed for T ( T = EK with E ∈ N , E ≥ 1 ) iterations where
[Equation]
K min (cid:40)(cid:115) 2 T
K log (cid:18)δ(cid:19) + T ( e (cid:101) (cid:15) K − 1)
K , T
K (cid:41) , δ (3) K,T = T K (cid:101)
δ K
{++ δ 2 , with (cid:101) (cid:15) K = (cid:40) log (cid:16) 
 + (cid:16) 1 − (cid:0) 1 − 1 N (cid:1) S (cid:17) ( e (cid:15) K − 1) (cid:17) , under scheme I , log (cid:0) 1 + SN ( e (cid:15) K − 1) (cid:1) , under scheme II , (cid:101) δ K = (cid:40) (cid:80) Ss =1 (cid:0)+}
Ss (cid:1) (cid:0) 1 N (cid:1) s (cid:0) 1 − 1 N (cid:1) S − s δ K,s , under scheme I ,
{+SN ( Kγδ 0 + δ 1 ) , under scheme II , δ K,s = ( e (cid:15) K − 1) δ K,s,e (cid:15) K /s −, δ K,s, 0 = 1 . 
 Kγ (cid:18) δ1 .(cid:19) 1 /s+ δ 1 , (cid:15) K+}
{+= (cid:15) 1 min (cid:110)(cid:112) 2 K log(1 /δ 1 ) + K ( e (cid:15) 1 − 1) , K (cid:111) , (cid:15) 
+}
= 2∆ l (cid:115)
η log(1 .
25 /δ 0 )
τ (1 − ρ 2 ) min c ∈ [ N ] p c , and δ 1 , δ 2 ∈ [0 , 1) .
Proof.
Let D c := { x c,i } i ∈ [ n c ] denote the dataset of the c -th client for c ∈ [ N ] .
Let D :=
∪ c ∈
[ N ] D c denote the whole dataset.
As FedAvg algorithms can be divided into the processes of local updates, synchronization, and broadcasting with risks of information leakage in synchronization (local model uploading and aggregation) and broadcasting, we consider the differential privacy guarantees in synchronization and broadcasting similar to Wei et al.
Since there is no involvement of data in model aggregation and broadcasting, they are post-processing processes.
Thus, it sufﬁces to analyze the differential privacy guarantees in local model uploading.
For any two datasets D (cid:39) s D (cid:48) , there exists c ∈ [ N ] such that D c (cid:39) s D (cid:48)
c and D c (cid:48)
=
D (cid:48) c (cid:48) for any c (cid:48) ∈
[ N ] , c (cid:48) (cid:54)
=
c .
Consider the function m c ( S ; θ )
=
θ − ηγ ∇ f c ( θ ; S )
= θ − ηγp c (cid:80) x ∈S ∇ l c ( θ ; x ) with | S | = γn c ( | A | denotes the cardinality of set A ).
By Assumption G.1, for any θ ∈ R d , the sensitivity of m c ( D c ; θ ) is
[Equation]
For the mechanism M c ( S ; θ ) :
= m c ( S ; θ ) + (cid:112) 2 ητρ 2 ˙ ξ + (cid:112) 2 η (1 − ρ 2 ) τ/
p c ξ with ˙ ξ and ξ being two independent standard d -dimensional Gaussian vector, since ˙ ξ is broadcasted to all the clients, it can be treated as some known constant which does not contribute to the differential privacy.
Thus, the standard deviation of the added Gaussian noise is (cid:112) 2 ητ (1 − ρ 2 ) /p
c at each dimension.
Then, according to the Gaussian mechanism Dwork et al.
(2014), M c ( D c ; θ ) is ( (cid:15) 0 ,c , δ 0 ) -differentially private for any θ ∈ R d with
[Equation]
For S c sampled uniformly at random from all the subsets of size γn c of D c , deﬁne (cid:102)
M c ( D c ; θ ) :=
M c ( S c ; θ ) .
Then, according to Theorem 9 in Balle et al.
(2018), (cid:102)
M c is (log (1 + γ ( e (cid:15) 0 ,c − 1)) , γδ 0 ) - differentially private.
Notice that for any (cid:15) 0 ,c ∈ [0 , 1] (i.e., 0 ≤ η ≤ 2 p c τ (1 − ρ 2 ) γ 2 ∆ 2 l c ( δ 0 )), we have 0 ≤ e (cid:15) 0 ,c − 1 ≤ 2 (cid:15) 0 ,c and log (1 + γ
[Equation]
≤
[Equation]
Deﬁne
[Equation]
Then, we have max c ∈ [ N ] (cid:15) 1 ,c
= (cid:15) 1 and max c ∈
[ N ] (cid:15) 0 ,c ∈ [0 , 1] if
[Equation]
log(
[Equation]
[Equation] Thus, for 0 ≤ η ≤ τ (1 − ρ 2 ) γ 2 min c (cid:48)∈ [ N ] p c (cid:48)
∆l log(1 .
25 /δ 0 ) , (cid:102)
M c ( D c ; θ ) is ( (cid:15) 1 , γδ 0 ) -differentially private for any θ ∈ R d .
From now on, we assume that 57 holds.
Deﬁne M Kc ( D c ; θ ) to be the K -fold composition of (cid:102) M c
( D c ; θ ) .
According to the composition rules of ( (cid:15), δ ) -differential privacy (Theorem 3.1 and 3.3 in Dwork et al.
, M Kc ( D c ; θ ) is ( (cid:15) K , δ K ) -differentially private with
[Equation] [Equation]
(59) for any δ 1 ∈ [0 , 1) .
By 56, if
[Equation]
(60) we have (cid:15) 1 ∈ (cid:20) 0 , log (cid:18)
[Equation]
+ (cid:113) 2log
[Equation]
[Equation]
(cid:19)(cid:21) which implies that K(cid:15) 1 ( e (cid:15) 1 − 1) ≤ (cid:112) 2 K log(1 /δ 1 ) (cid:15)and (cid:15) K ≤ 2 (cid:112) 2 K log(1 /δ 1 ) (cid:15) 1 .
In the synchronization process, S clients selected via device-sampling scheme I or II send their local models to the center.
Thus, for scheme I (with replacement) and scheme II (without replacement), according to Theorem 10 and Theorem 9 in Balle et al.
(2018) respectively, each synchronization process is ( (cid:101) (cid:15) K , (cid:101) δ K ) -differentially private with
[Equation]
under scheme I ,
[Equation]
under scheme II ,
[Equation]
[Equation]
under scheme I ,
[Equation]
under scheme II . [Equation]
) where
[Equation]
The aggregation and broadcasting process is post-processing and preserves the guarantees of differential privacy (Proposition 2.1 in Dwork et al.
When executed T iterations, Algorithm 1 is the T/K -fold composition of local updates, synchronization, and broadcasting.
According to the composition rules of ( (cid:15), δ ) -differential privacy (Theorem 3.1 and 3.3 in Dwork et al.
Algorithm 1 is ( (cid:15) (3) K,T , δ (3) K,T ) -differentially private after T iterations with
[Equation]
65) for δ 1 , δ 2 ∈ [0 , 1) and δ 0 ∈ (0 , 1) .
Notice that under scheme II, e (cid:101) (cid:15) K − 1 = SN ( e (cid:15) K − 1) , thus, (cid:15)
[Equation]
[-T = (cid:101) (cid:15) K min (cid:110)(cid:113) 2 TK log(1 /δ 2 ) + TSKN ( e (cid:15) K − 1) , TK (cid:111) and δ (3) K,T = SN γTδ 0 + TSKN δ 
 + δ 2 .-][Equation]
Discussion on the differential privacy guarantees of Algorithm 1 under scheme II By 58, 59, 62, 63, 64, and 65, by letting δ 1 , δ 2 = 0 , we have that Algorithm 1 is at least ( TK log (cid:0) 1 + SN ( e K(cid:15) 1 − 1) (cid:1) , SN γTδ 0 )
-differentially private.
If
[Equation]
(66) we have TK (cid:101) (cid:15) K ( e (cid:101) (cid:15) K − 1) ≤ (cid:113) 2 TK log(1 /δ 2 ) (cid:101) (cid:15) K and therefore
[Equation]
Now assume η satisﬁes 60, then by 61 and 67,
[Equation]
Notice that if
[Equation]
by 56, 61, 62, and 63, we have
[Equation]
[Equation]
Thus
[Equation]
we have 68 holds.
Plugging 56 into 68, we have
[Equation]
For any K ≥ 1 , if ηK (cid:28) 1 and T (cid:29) 1 , using log(1 + x ) ≈ x and e x − 1 ≈ x when | x | (cid:28) 1 , we can write 69 as
[Equation]
and 73 as
[Equation]
H R ELATED WORK
Federated Learning Current federated learning follows two paradigms.
The ﬁrst paradigm asks every client to learn the model using private data and communicate in model parameters.
The second one uses encryption techniques to guarantee secure communication between clients.
In this paper, we focus on the ﬁrst paradigms Dean et al.
; Shokri & Shmatikov (2015); McMahan et al.
(2016; 2017); Huang et al.
There is a long list of works showing provable convergence algorithm for
FedAvg types of algorithms in the ﬁeld of optimization Li et al.
(2020c; 2021); Huang et al.
Khaled et al.
; Yu et al.
; Wang et al.
; Karimireddy et al.
One line of research Li et al.
(2020c); Khaled et al.
; Yu et al.
; Wang et al.
; Karimireddy et al.
(2020) focuses on standard assumptions in optimization (such as, convex, smooth, stronglyconvex, bounded gradient).
The other line of work Li et al.
; Huang et al.
(2021) proves the convergence in the regime where the model of interest is an over-parameterized neural network (also called NTK regime Jacot et al.
Extensions to general partial device participation, and arbitrary communication schemes have been well addressed in Avdyukhin & Kasiviswanathan (2021);
Haddadpour & Mahdavi (2019).
Scalable Monte Carlo methods SGLD Welling & Teh (2011) is the ﬁrst stochastic gradient Monte
Carlo method that tackles the scalability issue in big data problems.
Ever since, variants of stochastic gradient Monte Carlo methods were proposed to accelerate the simulations by utilizing more general
Markov dynamics Ma et al.
; Chen et al.
(2014), Hessian approximation Ahn et al.
(2012), parallel tempering Deng et al.
(2020), as well as higher-order numerical schemes Chen et al.
; Li et al. (2019c); Cheng et al.
; Ma et al.
; Mou et al.
; Shen & Lee (2019).
Distributed Monte Carlo methods Sub-posterior aggregation was initially proposed in
Neiswanger et al.
; Wang & Dunson; Minsker et al.
(2014) to accelerate MCMC methods to cope with large datasets.
Other parallel MCMC algorithms Nishihara et al.
Ahn et al.
; Chen et al.
; Chowdhury & Jermaine (2018);
Li et al.
(2019a) propose to improve the efﬁciency of Monte Carlo computation in distributed or asynchronous systems.
Gürbüzbalaban et al.
(2021) proposed stochastic gradient Monte Carlo methods in decentralized systems.
Al-Shedivat et al.
; Mekkaoui et al.
; Chen & Chao (2021) introduced empirical studies of posterior averaging in federated learning.
A Concurrent Work Parallel to our work, QLSD Vono et al.
(2021) also studied the convergence of
SGLD in federated settings and a compression operator was proposed to alleviate the communication overhead.
By contrast, we follow the tradition in the FL community and achieve this target by solely conducting multiple local steps to balance accuracy and communication.
Other interesting Bayesian federated learning algorithms can be found in Kotelevskii et al.
; Zhang et al.
Our averaging scheme is deterministic and may be limited when activating all the devices is costly; we also refer interested readers to the study of federated averaging Langevin dynamics based on a probabilistic averaging scheme in Plassier et al.
I S IMULATIONS
Experimental details We repeat each experiment R = 300 times.
At the k -th communication round, we obtain a set of R simulated parameters { θ k,j } Rj =1 , where θ k,j denotes the parameter at the k -th round in the j -th independent run.
The underlying distribution µ k at round k is approximated by a Gaussian variable with the empirical mean u k = 1 R (cid:80)
Rj =1 θ k,j and covariance matrix Σ k = [Figure] −
[Figure] i
[Figure]
Partial device participation We study the convergence of two popular device-sampling schemes I and II.
We ﬁx the number of local steps K = 100 and the total devices N = 50 .
We try to sample S devices based on different ﬁxed learning rates η .
The full device updates are also presented for a fair evaluation.
As shown in Figure 3(a), larger learning rates converge faster but lead to larger biases; small learning rates, by contrast, yield diminishing biases consistently, where is in accordance with
Theorem 4.7.
However, in partial device scenarios, the bias becomes much less dependent on the learning rate in the long run.
We observe in Figure 3(b), Figure 3(c), Figure 3(d), and Figure 3(e) that the bias caused by partial devices becomes dominant as we decrease the number of partial devices S for both schemes.
Unfortunately, such a phenomenon still exists even when the algorithms converge, which suggests that the proposed partial device updates may be only appropriate for the early period of the training or simulation tasks with low accuracy demand.
J E XPERIMENTS ON THE MNIST AND F ASHION -MNIST DATASET
Experimental details For both the MNIST and Fashion-MNIST dataset, the temperature τ is set to be 0.05.
We split the training dataset of size 60000 into 10 non-overlapping subsets uniformly at random for 10 clients.
During the training, the stochastic gradient of the energy function at each step is calculated with a batch size of 200 for each client.
Warmup period To better observe the convergence under different local steps K with the existence of a warmup period, we use a warmup period of 500 communication rounds for different local steps
K = 1 , 10 , 20 , 50 , 100 .
Speciﬁcally, in each setting, after the ﬁrst 500 communication rounds, we collect one parameter sample every 10 communication rounds and average the predicted probabilities made by all the previous collected parameter samples to calculate three test statistics, accuracy, Brier
Score (BS), and Expected Calibration Error (ECE), on the test dataset.
We tune the step sizes η for the best performance and plot the curves of those test statistics against communication rounds in
Figure 4.
[Figure]
According to Figure 4, for the MNIST dataset
[Figure]
FA-LD with K = 1 performs the worst in terms of all three test statistics.
For the Fashion-MNIST dataset, FA-LD with K = 1 performs the worst in terms of accuracy and BS and does not perform the best in terms of ECE.
Thus, it is beneﬁcial to have multiple local updates in federated learning under a ﬁxed communication budget.
Among K = 1 , 10 , 20 , 50 , 100 , according to Figure 4(a), 4(b), and 4(c), for the MNIST dataset, the optimal local step K is 20 in terms of accuracy, BS, and ECE.
According to Figure 4(d), 4(e), and 4(f), for the Fashion-MNIST dataset, the optimal K in terms of accuracy is 20, the optimal K in terms of BS is 10, while the optimal K in terms of ECE is 50.
It is worth mentioning that the optimal K with a warmup period could be different from the optimal
K without a warmup period (see Figure 2).
For example, the optimal K in terms of BS for the
Fashion-MNIST dataset is 100 without a warmup period (Figure 2(e)) but is 10 with a warmup period of 500 communication rounds (Figure 4(e)), which indicates that the budget of communication also plays a rule in determining the optimal local step K because the optimal K changes when the samples in the ﬁrst 500 communication rounds are not collected.