F ED REP: A B YZANTINE -R OBUST , C OMMUNICATIONE FFICIENT AND P RIVACY -P RESERVING F RAMEWORK FOR F EDERATED L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Federated learning (FL) has recently become a hot research topic, in which Byzantine robustness, communication efﬁciency and privacy preservation are three important aspects.
However, the tension among these three aspects makes it hard to simultaneously take all of them into account.
In view of this challenge, we theoretically analyze the conditions that a communication compression method should satisfy to be compatible with existing Byzantine-robust methods and privacy-preserving methods.
Motivated by the analysis results, we propose a novel communication compression method called consensus sparsiﬁcation (ConSpar).
To the best of our knowledge, ConSpar is the ﬁrst communication compression method that is designed to be compatible with both Byzantine-robust methods and privacypreserving methods.
Based on ConSpar, we further propose a novel FL framework called FedREP , which is Byzantine-robust, communication-efﬁcient and privacypreserving.
We theoretically prove the Byzantine robustness and the convergence of FedREP.
Empirical results show that FedREP can signiﬁcantly outperform communication-efﬁcient privacy-preserving baselines.
Furthermore, compared with Byzantine-robust communication-efﬁcient baselines, FedREP can achieve comparable accuracy with an extra advantage of privacy preservation.
I NTRODUCTION
Federated learning (FL), in which participants (also called clients) collaborate to train a learning model while keeping data privately-owned, has recently become a hot research topic (Kone v cn ` y et al., 2016; McMahan & Ramage, 2017).
Compared to traditional data-center based distributed learning (Haddadpour et al., 2019;
Jaggi et al., 2014; Lee et al., 2017; Lian et al., 2017; Shamir et al., 2014; Sun et al., 2018;
Yu et al., 2019a; Zhang & Kwok, 2014;
Zhao et al., 2017; 2018;
Zhou et al., 2018; Zinkevich et al., 2010), service providers have less control over clients and the network is usually less stable with smaller bandwidth in FL applications.
Furthermore, participants will also take the risk of privacy leakage in FL if privacy-preserving methods are not used.
Consequently, Byzantine robustness, communication efﬁciency and privacy preservation have become three important aspects of FL methods (Kairouz et al., 2021) and have attracted much attention in recent years.
Byzantine robustness.
In FL applications, failure in clients or network transmission may not get discovered and resolved in time (Kairouz et al., 2021).
Moreover, some clients may get attacked by an adversarial party, sending incorrect or even harmful information purposely.
The clients in failure or under attack are also called Byzantine clients.
To obtain robustness against Byzantine clients, there are mainly three different ways, which are known as redundant computation, server validation and robust aggregation, respectively.
Redundant computation methods (Chen et al., 2018; Konstantinidis & Ramamoorthy, 2021; Rajput et al., 2019) require different clients to compute gradients for the same training instances.
These methods are mostly for traditional data-center based distributed learning, but unavailable in FL due to the privacy principle.
In server validation methods (Xie et al., 2019b; 2020b), server validates clients’ updates based on a public dataset.
However, the performance of server validation methods depends on the quantity and quality of training instances.
In many scenarios, it is hard to obtain a large-scale high-quality public dataset.
The third way is to replace the mean aggregation on server with robust aggregation (Alistarh et al., 2018;
Bernstein et al., 2019;
Blanchard et al., 2017; Chen et al., 2017; Ghosh et al., 2020; Karimireddy et al., 2021; Li et al.,
[Table]
[Table]
usually has a wider scope of application.
Many Byzantine-robust FL methods (Wang et al., 2020; Xie et al., 2019a) take this way
Communication efﬁciency.
In many FL applications, server and clients are connected by wide area network (WAN), which is usually less stable and has smaller bandwidth than the network in traditional data-center based distributed machine learning.
Therefore, communication cost should also be taken into consideration.
Local updating technique (Kone v cn`y et al., 2016; McMahan et al., 2017; Yu et al., 2019b; Zhao et al., 2017; 2018), where clients locally update models for several iterations before global aggregation, is widely used in FL methods.
Communication cost can also be reduced by communication compression techniques, which mainly include quantization (Alistarh et al., 2017; Faghri et al., 2020; Gandikota et al., 2021; Safaryan & Richt ´ arik, 2021; Seide et al., 2014; Wen et al., 2017), sparsiﬁcation (Aji & Heaﬁeld, 2017; Chen et al., 2020;
Stich et al., 2018;
Wangni et al., 2018)
and sketching 1 (Rothchild et al., 2020).
Error compensation (also known as error feedback) technique (Gorbunov et al., 2020; Wu et al., 2018;
Xie et al., 2020c) is proposed to alleviate the accuracy decrease for communication compression methods.
Moreover, different techniques can be combined to further reduce communication cost (Basu et al., 2020; Lin et al., 2018).
Privacy preservation.
Most of the existing FL methods send gradients or model parameters during training process while keeping data decentralized due to the privacy principle.
However, sending gradients or model parameters may also cause privacy leakage problems (Kairouz et al., 2021; Zhu et al., 2019).
Random noise is used to hide the true input values in some privacy-preserving techniques such as differential privacy (DP) (Abadi et al., 2016; Jayaraman et al., 2018; McMahan et al., 2018) and sketching (Liu et al., 2019; Zhang & Wang, 2021).
Secure aggregation (SecAgg) (Bonawitz et al., 2017; Choi et al., 2020) is proposed to ensure the privacy of computation.
Based on secure multiparty computation (MPC) and Shamir’s t -out-of- n secret sharing (Shamir, 1979), SecAgg allows server to obtain only the average value for global model updating without knowing each client’s local model parameters (or gradients).
Since noises can be simply added to stochastic gradients in most of the exsiting FL methods to provide input privacy, we mainly focus on how to combine SecAgg with Byzantine-robust and communication-efﬁcient methods in this work.
There are also some methods that consider two of the three aspects (Byzantine robustness, communication efﬁciency and privacy preservation), including RCGD (Ghosh et al., 2021), F 2 ed-Learning (Wang et al., 2020), SHARE (Velicheti et al., 2021) and SparseSecAgg (Ergun et al., 2021), which we summarize in Table 1.
However, the tension among these three aspects makes it hard to simultaneously take all of the three aspects into account.
In view of this challenge, we theoretically analyze the tension among Byzantine robustness, communication efﬁciency and privacy preservation, and propose a novel FL framework called FedREP.
The main contributions are listed as follows:
• We theoretically analyze the conditions that a communication compression method should satisfy to be compatible with Byzantine-robust methods and privacy-preserving methods.
Motivated by the analysis results, we propose a novel communication compression method called consensus sparsiﬁcation (ConSpar).
To the best of our knowledge, ConSpar is the ﬁrst communication compression method that is designed to be compatible with both Byzantine-robust methods and privacy-preserving methods.
• Based on ConSpar, we further propose a novel FL framework called FedREP , which is Byzantine-robust, communication-efﬁcient and privacy-preserving.
• We theoretically prove the Byzantine robustness and the convergence of FedREP.
• We empirically show that FedREP can signiﬁcantly outperform existing communicationefﬁcient privacy-preserving baselines.
Furthermore, compared with Byzantine-robust communication-efﬁcient baselines, FedREP can achieve comparable accuracy with an extra advantage of privacy preservation.
P RELIMINARY
In this work, we mainly focus on the conventional federated learning setup with m clients and a single server (Kairouz et al., 2021), which collaboratively to solve the ﬁnite-sum optimization problem: min
[Equation]
where w is the model parameter and d is the dimension of parameter.
f i ( w ) is the empirical loss of parameter w on the i -th training instance.
D k denotes the index set of instances stored on the k -th client and F k ( w ) is the local loss function of the k -th client.
We assume that D k ∩ D k (cid:48)
= ∅
when k (cid:54) =
k (cid:48) , and consider the instances on different clients with the same value as several distinct instances.
p k is the weight of the k -th client satisfying that p k > 0 and (cid:80) mk =
1 p k = 1 .
A common setting of p k is that p k = |D
k | / ( (cid:80) mk =1 |D k | ) .
For simplicity, we assume |D k | =
|D k (cid:48) | for all k, k (cid:48) ∈ [ m ] and thus p k = 1 /m .
The analysis in this work can be extended to general cases in a similar way.
Most federated learning methods (Karimireddy et al., 2020; McMahan et al., 2017; 2018) to solve problem (1) are based on distributed stochastic gradient descent and its variants, where clients locally update model parameters according to its own training instances and then communicate with server for model aggregation in each iteration.
However, the size of many widely-used models (Devlin et al., 2018;
He et al., 2016) is very large, leading to heavy communication cost.
Thus, techniques to reduce communication cost are required in FL.
Moreover, FL methods should also be robust to potential
Byzantine attack and privacy attack in real-world applications.
Byzantine attack.
Let [ m ] = { 1 , 2 , . . .
, m } denote the set of clients.
G ⊆
[ m ] denotes the set of good (non-Byzantine) clients, which will execute the algorithm faithfully.
The rest clients [ m ] \ G are
Byzantine, which may act maliciously and send arbitrary values.
The server, which is usually under service provider’s control, will faithfully execute the algorithm as well.
This Byzantine attack model is consistent to that in many previous works (Karimireddy et al., 2021).
Although there are some works (Burkhalter et al., 2021) focusing on another types of attacks called backdoor attacks (Kairouz et al., 2021), in this paper we mainly focus on Byzantine attacks.
The purpose of Byzantine attacks is to degrade the model performance.
One typical technique to defend against Byzantine attacks is robust aggregation (Kairouz et al., 2021), which guarantees bounded aggregation error even if
Byzantine clients send incorrect values.
Privacy attack.
In a typical FL method, server is responsible for using the average of clients’ local updating values for global model updating.
However, local updating information may be used to recover client’s training instances (Zhu et al., 2019), which will increase the risk of privacy leakage.
Thus, server is prohibited to directly receive individual client’s updating information by the requirement of privacy preservation (Kairouz et al., 2021).
Secure aggregation (Bonawitz et al., 2017) is a typical privacy-preserving method, which only allows server to have access to the average value for global model updating.
There are mainly two different types of FL settings, which are also called cross-silo FL and crossdevice FL (Kairouz et al., 2021).
We mainly focus on the cross-silo FL setting in this paper, where the number of clients m is usually not too large and all clients can participate in each training iteration.
Meanwhile, in this paper we mainly focus on synchronous FL methods.
M ETHODOLOGY
In this section, we analyze the conditions that a communication compression method should satisfy to be compatible with Byzantine-robust methods and privacy-preserving methods.
Based on the analysis, we propose a novel communication compression method called consensus sparsiﬁcation and a novel federated learning framework called FedREP that is Byzantine-robust, communication-efﬁcient and privacy-preserving.
In FedREP, we adopt robust aggregation technique to obtain Byzantine robustness due to its wider scope of application than redundant computation and server validation.
For privacy preservation, we mainly focus on secure aggregation, which is a widely used technique in FL to make server only have access to the average of clients’ local updating values.
3. M OTIVATION
We ﬁrst analyze the compatibility of existing communication compression methods with secure aggregation (SecAgg) (Bonawitz et al., 2017).
SecAgg is usually adopted together with quantization since it requires to operate on a ﬁnite ﬁeld to guarantee the privacy preservation.
Traditional quantization methods that represent each coordinate in lower bits can compress gradients in ﬂoating point number ( 32 bits) only up to 1 / 32 of the original size.
Even with quantization, SecAgg still suffers from heavy communication cost.
Thus, sparsiﬁcation is required to further reduce the communication cost (Ergun et al., 2021).
However if we simply combine traditional sparsiﬁcation methods (e.g., random- K and top- K sparsiﬁcation) with SecAgg, the random mask in SecAgg will damage the sparsity.
Thus, non-Byzantine clients should agree on the non-zero coordinates in order to keep the sparsity in SecAgg.
Then we analyze the compatibility of sparsiﬁcation with robust aggregation.
As previous works (Karimireddy et al., 2021) have shown, to obtain Byzantine robustness, it requires the distances between compressed updates from different clients (a.k.a. dissimilarity between clients) to be small.
Speciﬁcally, we present the deﬁnition of ( δ, c ) -robust aggregator in Deﬁnition 1.
Deﬁnition 1 ( ( δ, c ) -robust aggregator (Karimireddy et al., 2021; 2022)) .
Assume constant δ ∈ [0 , 12 ) and index set G ⊆
[ m ] satisﬁes |G| ≥ (1 − δ ) m .
Suppose that we are given m random vectors v 1 , . .
v m ∈ R d such that E (cid:107) v k − v k (cid:48) (cid:107) 2 ≤ ρ 2 for any ﬁxed k, k (cid:48) ∈ G .
v k can be arbitrary value if k ∈ [ m ] \ G .
Aggregator Agg ( · ) is said to be ( δ, c ) -robust
if the aggregation error e = Agg [Equation] | [Equation]
∈G
[Equation]
k satisﬁes that
[Equation]
As shown in previous works (Karimireddy et al., 2022), many widely-used aggregators, such as
Krum (Blanchard et al., 2017), geoMed
(Chen et al., 2017) and coordinate-wise median (Yin et al., 2018), combined with averaging in buffers (please refer to Section 3.3), satisfy Deﬁnition 1.
Moreover, O ( δρ 2 ) is the tightest order (Karimireddy et al., 2021).
Thus, a compression method which is compatible with robust aggregation should satisfy the condition that the expectation of dissimilarity between clients’ updates is kept small after compression.
Therefore, we theoretically analyze the expectation of dissimilarity after sparsiﬁcation.
For space saving, we only present the results here.
Proof details can be found in Appendix B.
Theorem 1.
Let { v k } m k =1 denote random vectors that satisfy E (cid:107) v k − v k (cid:48) (cid:107)
=
( ρ k,k (cid:48) ) 2 and E (cid:107) v k (cid:107) 2 = ( µ k ) 2 for any ﬁxed k, k (cid:48) ∈ G .
More speciﬁcally, E [( v k ) j − ( v k (cid:48) )
j ] 2 = ξ k,k (cid:48) ,j ( ρ k,k (cid:48) )and E [( v k ) 2 j ] = ζ k,j ( µ k ) 2 , where ξ k,k (cid:48) ,j > 0 , µ k,j > 0 , (cid:80) j ∈ [ d ] ξ k,k (cid:48) ,j = 1 and (cid:80) j ∈ [ d ] ζ k,j = 1 for any ﬁxed k, k (cid:48) ∈ G .
Let C ( · ) denote any sparsiﬁcation operator and N k denote the set of non-zero coordinates in C ( v k ) .
For any ﬁxed k, k (cid:48) ∈ G , we have:
[Equation]
[Equation]
Please note that when dissimilarity between the k -th and the k (cid:48) -th
clients is not too large, ( µ k ) 2 and ( µ k (cid:48) ) 2 are usually much larger than ( ρ k,k (cid:48) ) 2 .
In Equation (3), terms ( µ k ) 2 and ( µ k (cid:48) ) 2 vanish if and only if N k \ N k (cid:48)
=
N k (cid:48) \ N k = ∅ with probability 1 , which is equivalent to that N k = N k (cid:48) with probability 1 .
Furthermore, in order to lower the dissimilarity bewteen any pair of non-Byzantine clients, all non-Byzantine clients should agree on the non-zero coordinates of sparsiﬁed vectors.
Motivated by the analysis results in these two aspects, we propose the consensus sparsiﬁcation .
3. C ONSENSUS S PARSIFICATION
We introduce the consensus sparsiﬁcation (ConSpar) method in this section.
For simplicity, we assume the hyper-parameter K is a multiple of client number m .
We use u tk to denote the local memory for error compensation (Stich et al., 2018) on client k at the t -th iteration.
Initially, u 0 k = 0 .
Let g tk denote the updates vector to be sent from client k at the t -th iteration.
Client k ﬁrst generates a coordinate set T tk by top- Km sparsiﬁcation criterion.
More speciﬁcally, T tk contains the coordinates according to the largest Km absolute values in g tk .
Then, set ˜ T tk is generated by randomly selecting ( Km − r tk ) elements from T tk , where random variable r tk follows the binomial distribution B ( Km , α ) (0 ≤ α ≤ 1) .
Thus, | ˜ T tk | =
Km − r tk .
Then, set R tk is generated by randomly selecting r tk elements from [ d ] \ ˜ T tk .
Finally, client k computes I
tk = ˜ T
tk ∪ R tk and sends I tk to server.
The main purpose of the operations above is to obfuscate the top- Km dimension for privacy preservation.
Larger α could provide stronger privacy preservation on the coordinates, but may degrade the model accuracy, as we will empirically show in Section 5.
The sent coordinates are totally random when α = 1 .
When server has received {I tk } mk =1 from all clients, it decides the coordinate set of sparsiﬁed gradients
I t = ∪ mk =
I tk , and broadcasts I t to all clients.
Finally, each client receives I t , and computes the sparsiﬁed ˜ g
tk according to it.
2 More speciﬁcally, (˜ g tk )
j = ( g tk ) j if j ∈ I t
and (˜ g tk )
j
=otherwise, where (˜ g tk ) j denotes the value in the j -th coordinate of ˜ g tk .
Since |I tk |
=
Km , we have |I t | = | ∪ mk =1 I tk | ≤ (cid:80) mk
=1 |I tk | = K .
Please note that clients only need to send (˜ g tk ) I t to the server since clients and the server have reached an agreement on the non-zero coordinates I t of sparsiﬁed gradients.
More speciﬁcally, (˜ g tk ) I t
= ((˜ g tk )
j 1 , . .
, (˜ g tk ) j |I t | ) , where j s ∈
I t ( s = 1 , . . .
, |I t | ) are in ascending order.
When server has received { (˜ g tk ) I t } mk =1 , it uses robust aggregation to obtain ( ˜ G t ) I t
=
Agg ( { (˜ g tk ) I t } mk =1 ) .
Please note that ( ˜ G t ) I t is still a sparsiﬁed vector.
Moreover, server is only required to broadcast ( ˜ G t )
I t since I t has already been sent to clients before.
Thus, ConSpar is naturally a two-way sparsiﬁcation method without the need to adopt DoubleSqueeze technique (Tang et al., 2019).
Then we analyze the dissimilarity between clients after consensus sparsiﬁcation.
Please note that we do not assume the behaviour of Byzantine clients, which may send arbitrary I tk .
Proposition 1.
Let { ˜ g tk } mk =1 denote the consensus sparsiﬁcation results of vectors { g tk } mk =
and then we have E (cid:107) ˜
g tk − ˜ g tk (cid:48)
(cid:107) 2 ≤ E (cid:107) g
tk − g tk (cid:48) (cid:107) 2 for any ﬁxed k, k (cid:48) ∈ G .
Proposition 1 indicates that ConSpar will not enlarge the dissimilarity between clients, which is consistent with Theorem 1.
Meanwhile, SecAgg can be used in the second communication round and random masks are needed to add on the consensus non-zero coordinates only.
Then we analyze the privacy preservation of the mechanism that we use to generate I tk in ConSpar.
To begin with, we present the deﬁnition of (cid:15) -differential privacy ( (cid:15) -DP) in Deﬁnition 2.
Deﬁnition 2.
Let (cid:15) > 0 be a real number.
A random mechanism M is said to provide (cid:15) -differential privacy if for any two adjacent input datasets T 1 and T 2 and for any subset of possible outputs S : Pr [Equation] M
[Equation]
T
[Equation]
∈
[Equation] ] ≤ exp( (cid:15) ) · Pr [ M ( T 2 ) ∈ S ] . In the mechanism that we use to generate I tk in ConSpar, T 1 and T 2 are the top- Km coordinate sets.
Deﬁnition 2 leaves the deﬁnition of adjacent datasets open.
In this work, coordinate sets T 1 and Tthat satisfy T 1 , T 2 ⊆
[ d ] and |T 1 | = |T 2 | =
Km are deﬁned to be adjacent if T 1 and T 2 differ only on one element.
Liu et al.
(2020) provides DP guarantee for sparsiﬁcation methods with only one selected coordinate.
Our deﬁnition is more general and includes the one coordinate special case where |T 1 | = |T 2 | = 1 .
Now we show that the coordinate generation mechanism provides (cid:15) -DP.
Theorem 2.
For any α ∈ (0 , 1] , the mechanism in consensus sparsiﬁcation that takes the set of top coordinates T tk as an input and outputs
I tk provides (cid:16) ln (cid:16) (1+ α ) ·
Km ( d − Km +1)
2 α (cid:17)(cid:17) -differential privacy.
Finally, we analyze the communication complexity of ConSpar.
Clients need to send candidate coordinate set I tk , receive I t , send local gradient in the form of (˜ g tk ) I t , and then receive ( ˜ G t ) I t in each iteration.
Thus, each client needs to communicate no more than ( Km + K ) integers and 2 K ﬂoating point numbers in each iteration.
When each integer or ﬂoating point number is represented by 32 bits ( 4 bytes), the total communication load of each client is no more than (96 + 32 m ) K bits in each iteration.
The communication load is not larger than that of vanilla top- K sparsiﬁcation, in which 4 × 32 K
= 128 K bits are transmitted in each iteration.
Meanwhile, although ConSpar requires two communication rounds, the extra communication round is acceptable.
For one reason, there is little computation between the two rounds, which will not signiﬁcantly increase the risk of client disconnection during the aggregation process.
For another reason, the cost of the extra communication round is negligible when combined with SecAgg since SecAgg already requires multiple communication rounds and can deal with ofﬂine clients.
3. F ED REP
As we have shown, ConSpar is compatible with each of robust aggregation and SecAgg.
However, robust aggregation and SecAgg can not be simply applied together since SecAgg is originally designed for linear aggregation (such as summation and averaging) while robust aggregation is usually nonlinear.
This is also known as the tension between robustness and privacy (Kairouz et al., 2021).
Buffers on server are widely studied in Byzantine-robust machine learning (Karimireddy et al., 2022;
Velicheti et al.,
; Wang et al., 2020; Yang & Li, 2021), which can be used to make such a trade-off between robustness and privacy.
We also introduce buffers in FedREP.
The details of FedREP are illustrated in Algorithm 1 and Algorithm 2 in Appendix A.
Let integer s denote the buffer size.
For simplicity, we assume client number m is a multiple of buffer size s and hence there are ms buffers on server.
At the beginning of the t -th global iteration, each client k locally trains model using optimization algorithm A and training instances D k based on w t and obtain model parameter w t
+1 k
= A ( w t ; D k ) .
The update to be sent is g tk = u tk + ( w t − w t +1 k ) , where u tk is the local memory for error compensation with u 0 k = 0 .
Then client k generates coordinate set I tk by consensus sparsiﬁcation (please see Section 3.2), and sends I tk to the server.
When server receives all clients’ suggested coordinate sets, it broadcasts I t = ∪ mk =
I tk , which is the set of coordinates to be transmitted in the current iteration, to all clients.
In addition, server will randomly assign a buffer for each client.
More speciﬁcally, server randomly picks a permutation π of [ m ] and assign buffer b l to clients { π ( ls + k ) } sk =1 ( l = 0 , 1 , . .
. , ms − 1) .
Then for each buffer, b l = 1 s (cid:80) sk =1 (˜ g tπ ( ls + k ) )
I t is obtained by secure aggregation 3 and global update ( ˜ G t )
I t =
Agg ( { b l } m/sl =1 ) is obtained by robust aggregation among buffers.
During this time, clients could update the local memory for error compensation by computing u t
+1 k
= g tk − ˜ g tk .
Finally, ( ˜ G t ) I t is broadcast to all clients for global updating by w t
+1 =
w t − ˜ G t .
We have noticed that the consensus sparsiﬁcation is similar to the cyclic local top- K sparsiﬁcation (CLT- K ) (Chen et al., 2020), where all clients’ non-zero coordinates are decided by one client in each communication round.
However, there are signiﬁcant differences between these two sparsiﬁcation methods.
CLT- K is designed to be compatible with all-reduce while consensus sparsiﬁcation is designed to be compatible with robust aggregation and SecAgg in FL.
In addition, when there are
Byzantine clients, CLT- K does not satisfy the d (cid:48)
-contraction property since Byzantine clients may purposely send wrong non-zero coordinates.
Meanwhile, some works (Karimireddy et al., 2022) show that averaging in groups before robust aggregation (as adopted in FedREP) can help to enhance the robustness of learning methods on heterogeneous datasets.
We will further explore this aspect in future work since it is beyond the scope of this paper.
Finally, we would like to discuss more about privacy.
In the ideal case, server would learning nothing more than the aggregated result.
However, obtaining the ideal privacy-preserving property itself would be challenging, and even more so when we attempt to simultaneously guarantee Byzantine robustness and communication efﬁciency (Kairouz et al., 2021).
In FedREP, server has access to the partially aggregated mean b l and the coordinate set I tk .
However, as far as we know, the risk of privacy leakage increased by the two kinds of information is limited.
Server does not know the momentum sent from each single client, and only has access to the coordinate set I t k without knowing the corresponding values or even the signs.
Although it requires further work to study how much information can be obtained from the coordinates, to the best of our knowledge, there are almost no exsiting methods that can recover the training data only based on the coordinates.
C ONVERGENCE
In this section, we theoretically prove the convergence of FedREP.
Due to limited space, proof details are in Appendix B.
Firstly, we present the deﬁnition of d (cid:48) -contraction operator (Stich et al., 2018).
Deﬁnition 3 ( d (cid:48) -contraction) .
C : R d → R d is called a d (cid:48) -contraction operator (0 < d (cid:48)
≤ d ) if
[Equation]
The d (cid:48) -contraction property of consensus sparsiﬁcation is shown in Proposition 2.
Proposition 2.
If the fraction of Byzantine clients is not larger than δ (0 ≤ δ < 12 ) , consensus sparsiﬁcation is a d (cid:48) cons -contraction operator, where d (cid:48) cons =
d (
[Equation]
Therefore, the existing convergence results of d (cid:48) -contraction operator (Stich et al., 2018) can be directly applied to consensus sparsiﬁcation when there is no Byzantine attack.
Then we theoretically analyze the convergence of FedREP.
For simplicity in the analysis, we consider the secure aggregation and the robust aggregation on server as a unit secure robust aggregator, which is denoted by
SRAgg ( · ) .
Therefore, ˜ G t = SRAgg ( { ˜ g tk } mk =1 ) .
The assumptions are listed below.
Assumption 1 (Byzantine setting) .
The fraction of Byzantine clients is not larger than δ (0 ≤ δ < 12 ) and the secure robust aggregator SRAgg ( · ) is ( δ, c ) -robust with constant c ≥ 0 .
Assumption 2 (Lower bound) .
F ( w ) is bounded below: ∃ F ∗ ∈ R , F ( w ) ≥ F ∗ , ∀ w ∈ R d .
Assumption 3 ( L -smoothness) .
Global loss function F ( w ) is differentiable and L -smooth:
[Equation]
Assumption 4 (Bounded bias) .
∀ k ∈ G , we have E [ ∇ f i tk ( w )]
= ∇ F k ( w ) and there exists B ≥such that (cid:107)∇ F k [Equation] w
[Equation]
− ∇ F ( w ) (cid:107) ≤ B, ∀ w ∈ R d .
Assumption 5 (Bounded gradient) .
∀ k ∈ G , stochastic gradient ∇ f i tk ( w ) has bounded expectation: ∃ D ∈ R + , such that (cid:107)∇ F k ( w ) (cid:107) ≤ D, ∀ w ∈ R d .
Assumption 1 is common in Byzantine-robust distributed machine learning, which is consistent with previous works (Karimireddy et al., 2022).
The rest assumptions are common in distributed stochastic optimization.
Assumption 5 is widely used in the analysis of gradient compression methods with error compensation.
We ﬁrst analyze the convergence for a special case of FedREP where the training algorithm
A is local SGD with learning rate η and interval I .
Speciﬁcally, w t +1 k is computed by the following process: (i) w t +1 , 0 k [Equation] iii) [Equation]
=
[Equation]
, where i t,jk is uniformly sampled from D k .
Assumption 6 is made for this case.
Assumption 6 (Bounded variance) .
Stochastic gradient ∇ f i tk ( w ) is unbiased with bounded variance:
E [ ∇ f i tk ( w )]
= F k ( w ) and ∃ σ ∈ R + , such that E (cid:107)∇ f i tk ( w ) − ∇ F k ( w ) (cid:107) 2 ≤ σ 2 , ∀ w ∈ R d , ∀ k ∈ G .
According to Assumption 5 and 6, the second order moment of stochastic gradient ∇ f i tk ( w ) is bounded by ( D 2 + σ 2 ) .
Let u t
=
1 |G| (cid:80) k
∈G u tk and let e t = SRAgg ( { ˜ g tk } mk =1 ) − 1 |G| (cid:80)
k ∈G ˜
g tk denote the aggregation error .
We ﬁrst show that E (cid:107) u tk (cid:107) 2 and E (cid:107) e t (cid:107) 2 are both bounded above.
Lemma 1.
Under Assumption 1, 2, 3, 4, 5 and 6, let constant H =
d/d (cid:48) cons and take learning rate η t
= η > 0 , we have E (cid:107) u tk (cid:107) 2 ≤ 4 H 2 I 2 ( D 2 + σ 2 ) · η 2 , ∀ k ∈ G .
Lemma 2.
Under the same conditions in Lemma 1, we have E (cid:107) e t (cid:107) 2 ≤ 8 cδI 2 (4 H 2 +1)
( D 2 + σ 2 ) · η 2 .
Based on Lemma 1 and Lemma 2, we have the following theorem.
Theorem 3.
For FedREP, under the same conditions in Lemma 1 and Lemma 2, we have:
[Equation]
ηIT [Equation] where γ [-1 = 2 IL · [2(1 − I − 1 ) LD + 2 HD √ D 2 + σ 
-][Equation]
γ
[Equation] and [Equation] 4 H 2 + 1)( D 2 + σ 2 ) D.
When taking η = O (1 / √ T ) , Theorem 3 guarantees that FedREP has a convergence rate of O (1 / √ T ) with an extra error ∆ , which consists of two terms.
The ﬁrst term 2 BD comes from the bias of stochastic gradients, which reﬂects the degree of heterogeneity between clients.
The term vanishes in i.i.d.
cases where B = 0 .
The second term 4 (cid:112) 2 cδ (4 H 2 + 1)( D 2 + σ 2 ) D comes from the aggregation error.
The term vanishes when there is no Byzantine client ( δ = 0 ).
Namely, the extra error ∆ vanishes in i.i.d.
cases without Byzantine clients.
Then we analyze the convergence of FedREP with general local training algorithms that satisfy Assumption 7, which illustrates two important properties of a training algorithm.
Assumption 7.
Let w (cid:48)
=
A ( w ; D k ) .
There exist constants η A > 0 , A 1 ≥ 0 and A 2 > 0 such that local training algorithm
A satisﬁes (cid:107) E [ G A ( w ; D k )]
− ∇ F k ( w ) (cid:107) ≤ A 1 and E (cid:107) G A ( w ; D k ) (cid:107) 2 ≤ ( A 2 ) 2 , where G A ( w ; D k )
= ( w − w (cid:48) )
/η
A , ∀ k ∈ [ m ] .
In Assumption 7, G A ( w ; D k ) can be deemed as an estimation of gradient ∇ F k ( w ) by algorithm A with bounded bias A 1 and bounded second order moment ( A 2 ) 2 .
The expectation appears due to the randomness in algorithm A .
Many widely used algorithms satisfy Assumption 7.
For vanilla SGD, let η A be the learning rate and G A ( w ; D k ) is exactly the stochastic gradient.
Thus, we have A 1 =and ( A 2 )
= D 2 + σ 2 under Assumption 5 and 6.
Moreover, previous works (Allen-Zhu et al., 2020;
El-Mhamdi et al., 2020; Karimireddy et al., 2021) have shown that using history information such as momentum is necessary in Byzantine-robust machine learning.
We show that local momentum SGD also satisﬁes Assumption 7 in Proposition 3 in Appendix B.
Theorem 4.
Let constant H =
d/d (cid:48) cons .
For FedREP, under Assumption 1, 2, 3, 4, 5 and 7, we have:
[Equation]
where
[Equation] and [Equation]
Compared to the error ∆ in Theorem 3, there is an extra term 2 A 1 D in ∆ A , which is caused by the bias of gradient estimation in algorithm A .
Meanwhile, we would like to point out that Theoremprovides convergence guarantee for general algorithms.
For some speciﬁc algorithm, tighter upper bounds may be obtained by adopting particular analysis technique.
We will leave it for future work since we mainly focus on a general framework in this paper.
E XPERIMENT
In this section, we evaluate the performance of FedREP and baselines on image classiﬁcation task.
Each method is evaluated on CIFAR-10 dataset (Krizhevsky et al., 2009) with a widely used deep learning model ResNet-
(He et al., 2016).
Training instances are equally and uniformly distributed to each client.
All experiments in this work are conducted by PyTorch on a distributed platform with dockers.
More speciﬁcally, we set 32 dockers as clients, among which 7 clients are Byzantine.
One extra docker is set to be the server.
Each docker is bound to an NVIDIA Tesla K80 GPU.
Unless otherwise stated, we set local training algorithm
A to be local momentum SGD with momentum hyper-parameter β = 0 .
9 (see Equation (182) in Appendix B) for FedREP.
We run each method in the same environment for 120 epochs.
Initial learning rate is chosen from { 0 . 1 , 0 . 2 , 0 . 5 , 1 . 0 , 2 . 0 , 5 . 0 } .
At the 80 -th epoch, learning rate will be multiplied by 0 .
1 as suggested in (He et al., 2016).
The best top- 1 accuracy w.r.t.
epoch is used as ﬁnal metrics.
We test each method under bit-ﬂipping attack, ‘A Little is Enough’ (ALIE) attack (Baruch et al., 2019) and ‘Fall of Empires’ (FoE) attack (Xie et al., 2020a).
The updates sent by Byzantine clients with bit-ﬂipping attack are in the opposite direction.
ALIE and FoE are two omniscient attacks, where attackers are assumed to know the updates on all clients and use them for attack.
We set attack magnitude hyper-parameter to be 0 .
5 for FoE attack.
For FedREP, we test the performance when the robust aggregator is geometric median (geoMed)
(Chen et al., 2017), coordinate-wise trimmed-mean (TMean) (Yin et al., 2018) and centered-clipping (CClip) (Karimireddy et al., 2021), respectively.
More speciﬁcally, we adopt Weiszfeld’s algorithm (Pillutla et al., 2019) with iteration number set to be 5 for computing geoMed.
The trimming fraction in TMean is set to 7 / 16 .
For CClip, we set clipping radius to be 0 .
5 and iteration number to be 5 .
Batch size is set to be 25 .
[Figure] [Figure]
We ﬁrst empirically evaluate the effect of α on the performance of FedREP.
We set I = 1 , s = 2 and
K = 0 .
05 d .
We compare the performance of FedREP when α = 0 , 0 .
95 , 0 . 99 and 1 .
As illustrated in Figure
[Figure]
the performance of FedREP with CClip changes little when α ranges from 0 to 0 .
Final accuracy will decrease rapidly when α continues to increase.
A possible reason is that some coordinates could grow very large in local error compensation memory when α is near 1 .
More results of FedREP with geoMed and TMean are presented in Appendix C.1.
Since the effect of α is small when 0 ≤ α ≤ 0 .
95 , we set α = 0 in the following experiments.
In addition, as the empirical results in Appendix C.2 show, Byzantine attacks on coordinates have little effect on the performance of FedREP.
Thus, we assume no attacks on the coordinates in the following experiments.
Then we compare FedREP with a Byzantine-robust communication-efﬁcient baseline called Robust Compressed Gradient Descent with Error Feedback (RCGD-EF) (Ghosh et al., 2021).
For fairness, we set the compression operator Q ( · ) in RCGD-EF to be top- K sparsiﬁcation, and set Γ = 0 . 05 , which is the ratio of transmitted dimension number to total dimension number, for both FedREP and RCGD-EF.
Local updating interval
I is set to 5 for each method.
As illustrated in Figure 2, FedREP has comparable performance to RCGD-EF under bit-ﬂipping and FoE attack, but outperforms RCGDEF under ALIE attack.
Meanwhile, FedREP is naturally a two-way sparsiﬁcation method while RCGD-EF is not.
Moreover, FedREP provides extra privacy preservation compared to RCGD-EF.
Due to limited space, more empirical results are presented in the appendices.
Empirical results in
Appendix C.3 show the effect of momentum hyper-parameter β .
Empirical results in Appendix C. show that FedREP can signiﬁcantly outperform the communication-efﬁcient privacy-preserving baseline SparseSecAgg (Ergun et al., 2021).
Empirical results in Appendix C.5 show that compared with the Byzantine-robust privacy-preserving baseline SHARE (Velicheti et al., 2021), FedREP has comparable convergence rate and accuracy with much smaller communication cost.
C ONCLUSION
In this paper, we theoretically analyze the tension among Byzantine robustness, communication efﬁciency and privacy preservation in FL.
Motivated by the analysis results, we propose a novel
Byzantine-robust, communication-efﬁcient and privacy-preserving FL framework called FedREP .
Theoretical guarantees for the Byzantine robustness and the convergence of FedREP are provided.
Empirical results show that FedREP can signiﬁcantly outperform communication-efﬁcient privacypreserving baselines.
Furthermore, compared with Byzantine-robust communication-efﬁcient baselines, FedREP can achieve comparable accuracy with an extra advantage of privacy preservation.
Reproducibility Statement.
In this work, we empirically test the performance of FedREP and the baselines on the public dataset CIFAR-10 (Krizhevsky et al., 2009) with the widely used deep learning model ResNet-20 (He et al., 2016).
Common settings for all the experiments in this work are presented at the beginning of Section 5.
More settings for each single experiment are presented along with the empirical results in Section 5 and
Appendix C.
In addition, we provide the core part of our code in the supplementary material.
All the proof details for the theoretical results in this work can be found in Appendix B.
A D ETAILS OF F ED REP
The detailed algorithms of FedREP on server and clients are illustrated in Algorithm 1 and Algorithm 2, respectively.
Input: client number m size s robust aggregator
Agg
; for t = 0 to T − 1 do
Receive {I tk } mk =1 from all clients and compute I t
= ∪ mk =
I tk ;
Broadcast I t to all clients;
Pick a random permutation π of [ m ] ;
Assign buffer b l to clients { π ls k ) } sk =1 for l = 0 , 1 , . .
. , ms − 1 ; for l = 0 to ms − 1 do
Obtain b l = 1 s (cid:80) sk =
(˜ g tπ ls
) I t via SecAgg protocol; end for
Compute: ( ˜ G
=
Agg
Broadcast
G
; end for
Input local optimization
Initialization: model parameter w local memory u 0 k = 0 ; for t
0 to T − 1 do /* Locally learning model update error compensation
Locally train learning model obtain parameter w t
+1 k
= A ( w t ; D k )
Compute g tk = u tk + ( w t − w t +1 k ) ; /
* Two-stage aggregation with sparsiﬁcation */
Generate coordinate set I tk by consensus sparsiﬁcation (see Section 3.2 in the main text);
Send I tk to the server;
Receive coordinate set I t and assigned buffer number l from the server;
Compute (˜ g tk ) I t and send it to the assigned buffer b l via SecAgg protocol;
Update memory
Receive ( from the server and according
B P ROOF D ETAILS
In this section
[Equation]
we present the proof details of the theoretical results in the paper.
B. P ROOF OF T HEOREM
Proof.
For any ﬁxed k
[Equation]
k (cid:48) ∈ [ m ] ,
[Equation]
B. P ROOF OF P ROPOSITION
Proof.
Let I t denote the set of non-zero coordinates after consensus sparsiﬁcation.
In general cases, for any ﬁxed k, k (cid:48) ∈ [ m ] , we have:
[Equation]
B. P ROOF OF T HEOREM
Proof.
Let M be the the mechanism in consensus sparsiﬁcation that takes the set of top coordinates T as an input and outputs a random coordinate set. T 1 , T 2 ⊆
[ d ] are two arbitrary adjacent input coordinate sets that satisfy |T 1 | = |T 2 | =
Km and S is any subset of possible outputs of M .
When S is empty, Pr [ M ( T 1 ) ∈ S ] =
Pr [ M ( T 2 ) ∈ S ]
= 0 .
Thus, for any (cid:15) > 0 , we have:
[Equation]
Without loss of generality, we suppose that S is non-empty.
For any I ∈ S , let |T 1 ∩ I| =
Uand |T 2 ∩ I| = U 2 .
T 1 and T 2 only differ on one element since they are adjacent.
Thus, we have
U
=
U 1 − 1 , U 1 or U 1 + 1 .
Set ˜ T 1 is generated by randomly selecting ( Km − r 1 ) elements from T 1 , where r 1 follows the binomial distribution
Pr ( Km , α ) .
Thus, ∀ i = 0 , 1 , . .
, K/m ,
[Equation]
To obtain I as the ﬁnal output, only the elements in T 1 ∩ I can be selected.
Thus, r 1 should equal or be larger than |T 1 \ I| =
Km − U 1 .
Furthermore, for r 1 ≥ Km − U 1 , the probability that all elements are selected from T 1 ∩ I is (cid:0)
[Equation]
Finally, the r 1 elements in I \ ˜ T 1 should be selected from [ d ] \ ˜ T 1 , of which the probability is 1 / (cid:0) d
− K/m
[Equation]
1 r(cid:1) since | [ d ] \ ˜ T 1 | = d − ( K/m − r 1 ) =
d − K/m + r 1 .
Thus, we have:
[Equation]
Thus
[Equation]
Pr [ M ( T 1 ) =
I ] is monotonically increasing with respect to U 1 .
Similarly,
[Equation]
which is monotonically increasing with respect to U 2 .
Thus
[Equation]
Pr [ M ( T 1 )=
I ]
Pr [ M ( T 2 )
=
I ] takes the maximum value when U
= U 2 + 1 .
Therefore,
Pr
[Equation]
M
[Equation]
I
[Equation] [Equation]
Let
[Equation]
i
[Equation]
(26) and
[Equation]
i (cid:19)(cid:35) .
We have
[Equation] [Equation]
Since U 1 = U 2 + 1 ≤ K/m , we have U 2 ≤ K/m − 1 .
Thus,
S 1 ( α ) =
U 2 +1 (cid:88) i
= (cid:0)
[Equation]
− αα (cid:1)
i
[Equation]
(cid:0) d
[Equation]
/m
[Equation]
U 2 +1 i (cid:1) (cid:0)
[Equation]
d −
[Equation]
/m (cid:1) (32) ≤
U 2 (cid:88) i = (cid:0) 1 −
[Equation]
(cid:1)
i − 1 (cid:0) d d
[Equation]
/m (cid:1)(cid:0) U 2 +1 i (cid:1) (cid:0)
[Equation]
d −
[Equation]
/m (cid:1)
+
[Equation]
(cid:88) i =
[Equation]
(cid:0) 1 −
[Equation]
(cid:0)
[Equation]
/m (cid:1)(cid:0) U 2 +1 i (cid:1) (cid:0)
d − i d − K/m (cid:1) (33) =
U 2 (cid:88) i = (cid:0) 1 − αα (cid:1)
i − 1 (cid:0) d d − K/m (cid:1)(cid:0)
[Equation]
(36) =
[Equation]
i
[Equation]
[Equation] [Equation]
[Equation] [Equation]
2 α
[Equation]
· S
[Equation] [Equation]
) (40) ≤ (1 +
[Equation]
) · Km ( d − Km + 1)
[Equation]
· S 0 (
[Equation]
Therefore,
Pr [ M ( T 1 ) = I ]
Pr [ M ( T 2 ) =
I ] ≤
S 1 ( α )
S 0 ( α ) ≤ (1 + α ) ·
Km ( d − Km + 1) 2 α .
Consequently,
[Equation] [Equation]
(43) which shows that M provides ln (cid:16)
[Table]
(cid:17)
[Table]
Then we provide a stronger result for the case where α is close to 1 .
Speciﬁcally, when 12 ≤ α ≤ 1 ,
[Equation]
Since U 1 = U 2 + 1 ≤ K/m
[Equation]
we have U 2 ≤ K/m − 1 .
Thus,
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
(cid:19) U+ (cid:0) 1 − αα
[Equation] [Equation]
≤
[Equation] [Equation]
Consequently,
[Equation]
It shows that when ≤ α ≤ 1 ,
M provides (cid:15) M -DP, where (cid:15)
M = ln (cid:0) 1 + 1 − αα · (2 K/m
+ 1)
· d K/m (cid:1) .
Specially, when α = 1 , (cid:15) M = ln(1 + 0) = 0 .
It is consistent to that the coordinate set is totally random when α = 1 .
B. P ROOF OF P ROPOSITION
Proof.
∀ k ∈ G , ∀ 0 ≤ t < T , we have:
[Equation]
Therefore,
[Equation] [Equation]
[Equation] [Equation]
For any j (cid:54)∈ I tk ,
Pr (cid:2) j ∈
I t |I tk (cid:3) ≥
Pr  
j ∈   (cid:91) k (cid:48)
∈G\{ k }
I tk
[Equation]

 (cid:12)(cid:12)(cid:12)(cid:12) I tk   (62) =
Pr  
j ∈   (cid:91) k (cid:48)
∈G\{ k } (cid:16) ˜ T
[Equation]
(cid:48)
[Equation]
(cid:48) (cid:17) (cid:12)(cid:12)(cid:12)(cid:12)
[Equation]

[Equation]
(63) =
Pr  
[Equation]
 
[Equation]
k
[Equation] [Equation]
[Equation]
 
[Equation]
 
[Equation]
k (cid:48)
[Equation]
{ k }
[Equation]
(cid:48)
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
  (64)
[Equation]
Pr
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
k (cid:48)
[Equation]
\{ k }
[Equation]
tk (cid:48) 
 \  
[Equation]
k (cid:48)
∈G\{ k } ˜ T tk (cid:48) 
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)
I tk   .
For simplicity, let
[Equation]
=
Pr
[Equation]
66) and we have:
[Equation]
tk (cid:48) 
 (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) I tk , j (cid:54)∈   (cid:91) k (cid:48)
∈G\{ k } ˜ T tk (cid:48) 
 
 (67) =
ν
+ (1 − ν ) ·
  1 − Pr  
j (cid:54)∈   (cid:91) k (cid:48)
∈G\{ k }
R tk (cid:48) 
 (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) I tk , j (cid:54)∈   (cid:91) k (cid:48)
∈G\{ k } ˜ T tk (cid:48) 
 
   (68) =
ν
+ (1 − ν ) ·   1 − (cid:89) k (cid:48)
∈G\{ k }
Pr  
j (cid:54)∈ R tk (cid:48) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) I tk , j (cid:54)∈   (cid:91) k (cid:48)
∈G\{ k } ˜ T tk (cid:48) 
 
   (69) (i) =
ν
+ (1 − ν ) ·   1 − (cid:89) k (cid:48)
∈G\{ k }  
K/m (cid:88) i =
Pr [ r tk (cid:48)
=
i ] · (cid:18) 1 − i d − K/m + i (cid:19) 
   (70) (ii) ≥ ν
+ (1 − ν ) ·   1 −
[Equation] [Equation]
where (i) holds because when r tk = i , the probability that element j is among the i randomly selected elements from [ d ] \ ˜ T tk is i d − K/m
+ i since | [ d ] \ ˜ T tk | =
d − K/m
+ i .
Inequality (ii) holds because i ≤ K/m .
Equation (iii) holds because r tk (cid:48) follows the binomial distribution B ( Km , α ) .
Inequality (iv) holds because 1 − (cid:16) 1 − E [ r tk ] d (cid:17) (1 −
[Equation]
) m
[Equation]
1 ≤ 1 .
Since 0 ≤ α ≤ 1 and 0 < Km < d , we have 0 ≤ αKmd < 1 .
Thus,
[Equation]
Therefore,
Pr
[Equation] [Equation]
Substituting it into (61), it is obtained that
[Equation]
Take total expectation and we have:
[Equation]
Also,
E
[Equation] [Equation]
 (cid:88)
j ∈T t k ( g tk ) 2 j (cid:12)(cid:12)(cid:12) r tk   (89)
[Equation]
+
[Equation] [Equation]
(90) =
[Equation]
d
[Equation]
(cid:107)
[Equation]
(cid:107)
[Equation]
(91) =
[Equation] [Equation]
[Equation] [Equation]
) · (cid:107)
[Equation]
(cid:107)(92) =(
K/m )
[Equation] [Equation]
/m
[Equation]
r tk ) d (
[Equation]
) · (cid:107) g tk (cid:107)(93) =
K md (cid:107) g
tk (cid:107) 2 .
Thus,
E   (cid:88)
j ∈I tk ( g tk ) 2 j   =
E  
E   (cid:88)
j ∈I tk ( g tk ) 2 j (cid:12)(cid:12)(cid:12) r tk   
 ≥ E (cid:20) K
md (cid:107) g tk (cid:107)(cid:21) =
K md (cid:107) g
tk (cid:107) 2 .
[Equation]
Substituting (95) into (84), we have:
[Equation]
Since ˜ g tk is the consensus sparsiﬁcation result of g tk
[Equation]
we have:
[Equation]
By deﬁnition, consensus sparsiﬁcation is a d (cid:48) cons -contraction operator, where
[Equation]
B. P ROOF OF L EMMA
Proof.
When training algorithm
A is I -iteration local SGD with learning rate η , we have w t +1 , 0 k = w t , w t
+1 ,j +1 k
= w t
[Equation]
1 ,j k
[Equation]
t,
[Equation] [Equation]
) ( j = 0 , 1 , . . .
, I
[Equation]
1) and w t
[Equation]
1 k = w t
[Equation]
, where i t,jk is uniformly sampled from D k .
Therefore, we have the following inequality for all k ∈ G :
[Equation]
where (i) is derived based on Proposition 2.
(ii) is derived based on that (cid:107) x + y (cid:107) 2 ≤ (1 + θ ) (cid:107) x (cid:107) 2 + (1+ θ − 1 ) (cid:107) y (cid:107) 2 for any constant θ > 0 .
(iii) is derived based on that (1 − d (cid:48)
cons d )(1+ d (cid:48) cons 2 d ) < 1 − d (cid:48) cons 2 d and (1 − d (cid:48)
cons d
[Equation]
+
[Equation]
cons )
[Equation]
d d (cid:48) cons .
(iv) is derived based on Assumption 5 and Assumption 6.
When η t = b √ t + λ where constant b > 0 and λ = 4 d d (cid:48) cons , the second term on the RHS
2 I 2 d
[Equation]
Combining (110) and (117), we have
[Equation]
Therefore,
[Equation]
Recursively using (119), we have
[Equation]
Thus
[Equation]
Finally,
[Equation]
When η t = η , by (110)
[Equation]
we have
[Equation] [Equation]
Recursively using (124), we have
[Equation]
Thus,
[Equation] [Equation]
Finally,
[Equation] [Equation]
B. P ROOF OF L EMMA
Proof.
Based on Assumption 6 and Assumption 5, we have that ∀ k ∈ G ,
[Equation]
By Lemma 1, if η t = b √ t + λ where constant b > 0 and λ = 4 d d (cid:48) cons , we have
[Equation]
[Equation] [Equation]
Therefore
[Equation]
by Deﬁnition 1 and (132),
[Equation]
Similarly
[Equation]
if η t = η >
[Equation]
we have
[Equation]
and
[Equation]
B
[Equation]
P ROOF OF T HEOREM
Proof.
Let u t
=
1 |G| (cid:80) k
∈G u tk be the averaging memory of non-Byzantine clients.
ˆ w is deﬁned as ˆ w t =
w t − u t .
The iteration rule for ˆ w is derived as follows:
[Equation]
− e
[Equation]
142) where e t =
SRAgg
( { ˜ g tk } mk =1 ) − 1 |G| (cid:80) k ∈G ˜
g tk is the estimation error of 1 |G| (cid:80) k ∈G ˜
g tk .
Let ¯ G t =
( w t − 1 |G| (cid:80) k
∈G w t +1 k ) / ( ηI ) .
Then we have ¯ G t =
I |G| (cid:88) k
[Equation]
I − 1 (cid:88)
j =
[Equation]
f i [Equation] ( w [Equation]
+
[Equation] k [Equation] , t = 0 , 1 , . .
, T − 1 , (143) and ˆ
[Equation] [Equation]
The equation can be interpreted as that ˆ w t +1 is obtained by performing an SGD step on ˆ w t with learning rate ηI , gradient approximation ¯ G t and error e t .
Based on Assumption 3 and the inequality that (cid:107) x
+ y (cid:107) 2 ≤
[Equation]
(cid:107) x (cid:107)
[Equation]
+
[Equation]
(cid:107) y (cid:107)
[Equation] [Equation]
[Equation] [Equation] [Equation]
[Equation]
Taking expectation on both sides
[Equation]
we have
[Equation]
Based on Assumption 3 and that −(cid:107)
[Equation]
we have:
[Equation]
In addition
[Equation]
using Assumption 3, Assumption
[Equation]
Assumption 5 and Equation (143)
[Equation]
we have:
[Equation]
w t , u t (cid:105)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (157) ≤ D
I |G| (cid:88) k
∈G
I − 1 (cid:88)
j =
E (cid:104)(cid:13)(cid:13)(cid:13) ∇ F ( w t +1 , 0 k − u t ) − ∇ F ( w t +1 ,j k ) (cid:13)(cid:13)(cid:13)(cid:12)(cid:12)(cid:12) w t , u t (cid:105) + D
I |G| (cid:88) k
∈G
I − 1 (cid:88)
j =
E (cid:104)(cid:13)(cid:13)(cid:13) ∇ F ( w t +1 ,j k ) − ∇ F k ( w t +1 ,j k ) (cid:13)(cid:13)(cid:13)(cid:12)(cid:12)(cid:12) w t , u t (cid:105) (158) ≤ D
I |G| (cid:88) k
∈G
I − 1 (cid:88)
j = (cid:16) L · E (cid:104)(cid:13)(cid:13)(cid:13) w t +1 , 0 k − u t − w t
+1 ,j k (cid:13)(cid:13)(cid:13)(cid:12)(cid:12)(cid:12) w t , u t (cid:105) + B (cid:17) (159) ≤ DL
I |G| (cid:88) k
∈G
I − 1 (cid:88)
j = (cid:16) (cid:107) u t (cid:107) + E (cid:104)(cid:13)(cid:13)(cid:13)
w t +1 , 0 k − w t
+1 ,j k (cid:13)(cid:13)(cid:13)(cid:12)(cid:12)(cid:12) w t , u t (cid:105)(cid:17) + BD (160) ≤ DL
I |G| (cid:88) k
∈G
I − 1 (cid:88)
j
= (cid:16) j − 1 (cid:88) j (cid:48) =
E (cid:104)(cid:13)(cid:13)(cid:13)
w t +1 ,j (cid:48)
k − w t
+1 ,j (cid:48)
+1 k (cid:13)(cid:13)(cid:13)(cid:12)(cid:12)(cid:12) w t , u t (cid:105)(cid:17) + DL · (cid:107) u t (cid:107) + BD.
With Assumption 5, we have:
E (cid:104)(cid:13)(cid:13)(cid:13)
w t +1 ,j (cid:48)
k − w t
+1 ,j (cid:48)
+1 k (cid:13)(cid:13)(cid:13)(cid:12)(cid:12)(cid:12) w t , u t (cid:105) =
E (cid:104)(cid:13)(cid:13)(cid:13) η · ∇ f i t,j (cid:48) k
( w t +1 ,j (cid:48) k ) (cid:13)(cid:13)(cid:13)(cid:12)(cid:12)(cid:12) w t , u t (cid:105) ≤ ηD.
Therefore, − E [Equation] w
[Equation] [ ¯ G [Equation]
− [Equation]
F
[Equation] ˆ w [Equation]
)] (cid:12)(cid:12)(cid:12) [Equation] t , u t [Equation]
[Equation]
Note that E [ XY ] ≤ (cid:112) E
[ X 2 ] · E [ Y 2 ] .
Using Assumption 5 and Lemma 2, we have:
[Equation]
According to Assumption 5 and 6,
[Equation]
  (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 1 I |G| (cid:88) k ∈G
I − 1 (cid:88) j = ∇ f i t,jk ( w t +1 ,j k ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) w t , u t   (171) ≤
I |G| (cid:88) k ∈G I − 1 (cid:88) j =
E (cid:20)(cid:13)(cid:13)(cid:13) ∇ f i t,jk ( w t +1 ,j k ) (cid:13)(cid:13)(cid:13) 2 (cid:12)(cid:12)(cid:12)(cid:12) w t , u t (cid:21) (172) ≤ I |G| (cid:88) k
∈G I − 1 (cid:88) j = ( D 2 + σ 2 ) (173) = D 2 + σ 2 .
Substituting (137), (154), (166), (170) and (174) into (150), we have: E [ F [Equation] ˆ w
[Equation]
+
[Equation]
[Equation]
u
[Equation]
(cid:107)∇ [Equation] [Equation] (cid:107)
[Equation]
[Equation]
[Equation] ] (cid:12)(cid:12)(cid:12) [Equation] t (cid:105)
[Equation]
Note that E (cid:107) u t (cid:107) =
(cid:112)
[ E (cid:107) u t (cid:107) ] 2 ≤ (cid:112)
[ E (cid:107) u t (cid:107) 2 ] .
Taking total expectation on both sides and using that
E (cid:107) u t (cid:107)
[Equation]
[Equation] [Equation]
2 + σ 2 ) · η 2 , we have:
[Equation]
Namely,
[Equation]
By taking summation from t = 0 to T −
[Equation]
we have:
[Equation]
Note that ˆ w 0 = w 0 and F ( ˆ w T ) ≥ F ∗ .
Thus,
[Equation]
DL + 2 HD (cid:112) D 2 + σ {+
+}
+ ( D 2 + σ 2 ) + 8 cδ (4 H 2 + 1)( D 2 + σ 2 ) (cid:105) + η 2 · (cid:104) 8 H 2 I 2 L 2 ( D 2 + σ 2 ) (cid:105) + 2 (cid:104) BD + (cid:112) 8 cδ (4 H 2 + 1)( D 2 + σ 2 ) D (cid:105) .
In summary,
[Equation]
where γ 1 = 2 IL · [2(1 − I − 1 ) LD + 2 HD √ D 2 + σ {+
+} + ( D 2 + σ 2 ) + 8 cδ (4 H 2 + 1)( D 2 + σ 2 )] , γ 2 = 8 H 2 I 2 L 2 ( D 2 + σ 2 ) and ∆ = 2 BD + 4 (cid:112) 2 cδ (4 H 2 + 1)( D 2 + σ 2 ) D .
B. A NALYSIS FOR L OCAL M OMENTUM SGD
We present the following proposition, which illustrates that Assumption 7 holds when A is set to be local momentum SGD.
Proposition 3.
Under Assumption 3, 5 and 6, local momentum SGD satisﬁes Assumption 7.
Moreover, for local momentum SGD with learning rate η > 0 , update interval I ∈ N + and momentum hyperparameter β ∈ [0 , 1) , we have η A = ηI , A 1 = β (1 − β I ) I (1 − β ) D + √ D
+ σ 2 + ( I − 12 + β 2 (1 − β I − 1 ) I (1 − β )− β ( I − 1) I (
1 − β ) )
· L √ D 2 + σ 2 and ( A 2 )
= D 2 + σ 2 .
Proof.
When A is set to be local momentum SGD with learning rate η , update interval I and momentum hyper-parameter β , let m 0 ,jk = 0 be the initial momentum and w t
+1 k
( t = 0 , 1 , . . .
, T − [Equation] ) is computed by the following process:
[Equation]
Let η A = ηI , we have
[Equation]
[Equation]
In addition,
[Equation]
Now we prove that E (cid:107) m t,j k (cid:107) 2 ≤ D 2 + σ 2 ( j = 0 , 1 , . .
, I ) by deduction on t .
Step 1.
When t = 0 , we have E (cid:107) m 0 ,jk (cid:107)
2 = 0 ≤ D 2 + σ 2 .
Step 2 (deduction).
Suppose E (cid:107) m t,jk (cid:107) 2 ≤ D 2 + σ 2 , we have E (cid:107) m t +1 , 0 k (cid:107)
= E (cid:107) m t,Ik (cid:107) 2 ≤ D 2 + σand E (cid:107)∇ f i t,j (cid:48) k
( w t +1 ,j (cid:48) k ) (cid:107) 2 ≤ D 2 + σ 2 .
Since β j +1 + (1 − β ) j (cid:88) j (cid:48) =
[Equation]
j − j (cid:48)
[Equation]
, (188) m t +1 ,j +1 k can be deemed as a weighted averaging of m t +1 , 0 k and {∇ f i t,j (cid:48) k
( w t +1 ,j (cid:48) k ) } jj (cid:48)
=0 .
Thus,
E (cid:107) m t +1 ,j +1 k (cid:107) 2 ≤ D 2 + σ 2 , j = 1 , 2 , . . . , I.
By mathematical deduction, ∀ t = 0 , 1 , . . .
, T , we have
[Equation]
Therefore,
[Equation]
Substituting (187) into (183), we have:
[Equation]
I I − 1 (cid:88) j = (cid:104) β
j +1 m t +1 , 0 k
+ (1 − β ) j (cid:88)
j (cid:48) =
β j − j (cid:48) ∇ f i t,j (cid:48) k
( w t +1 ,j (cid:48) k ) (cid:105) (192) =
β (1 − β I ) I (1 − β ) m t +1 , 0 k + 1 − β
I
I − 1 (cid:88)
j =  
j (cid:88) j (cid:48) =
β j − j (cid:48) ∇ f i t,j (cid:48) k
( w t +1 ,j (cid:48) k ) 
 (193) =
β (1 − β I )
I (1 − β ) m t +1 , 0 k + 1 − β
I
I − 1 (cid:88) j (cid:48)
=  
I − 1 (cid:88)
j = j (cid:48)
β j − j (cid:48) ∇ f i t,j (cid:48) k
( w t +1 ,j (cid:48) k ) 
 (194) =
β (1 − β I )
I (1 − β ) m t +1 , 0 k + 1 − β
I
I − 1 (cid:88) j (cid:48)
= (cid:34) 1 − β
I − j (cid:48) 1 − β ∇ f i t,j (cid:48) k
( w t +1 ,j (cid:48) k ) (cid:35) (195) =
I   β (1 − β I ) 1 − β m t +1 , 0 k +
I − 1 (cid:88) j (cid:48)
= (1 − β
I − j (cid:48) )
∇ f i t,j (cid:48)
k
( w t +1 , [Equation]
Therefore,
[Equation]
Since E (cid:107) m t
[Equation]
[Equation]
+ D
and that
[Equation]
we have (cid:107) E [ G A ( w t [Equation] D k )] − ∇ F k ( w t ) (cid:107) ≤ I (cid:34)
[Equation]
I
[Equation]
[Equation]
j (cid:48)
[Equation] [Equation]
I
[Equation]
j (cid:48)
[Equation]
[ (cid:112)
[Equation]
(cid:48)
[Equation]
(cid:112)
[Equation]
2 ] (cid:35) (202)
[Equation]
I
[Equation]
I − 1 (cid:88) j (cid:48) =
( j (cid:48) − j (cid:48) β
I − j (cid:48) )
(cid:35) (203)
[Equation]
I
[Equation]
[Equation]
[Equation]
[Equation]
B. P ROOF OF T HEOREM
Proof.
Similar to Lemma 1 and Lemma 2, we have the following inequalities to bound the local memory and the aggregation error, respectively, for general training algorithm
A that satisﬁes
Assumption 7:
[Equation]
where (i) is derived based on Proposition 2.
(ii) is derived based on that (cid:107) x + y (cid:107) 2 ≤ (1 + θ ) (cid:107) x (cid:107) 2 + (1+ θ − 1 ) (cid:107) y (cid:107) 2 for any constant θ > 0 .
(iii) is derived based on that (1 − d (cid:48)
cons d )(1+ d (cid:48) cons 2 d ) < 1 − d (cid:48) cons 2 d and (1 − d (cid:48)
cons d )(1 + 2 d d (cid:48) cons ) < 2 d d (cid:48) cons .
(iv) is derived based on Assumption 7.
Therefore,
[Equation]
[Equation]
Recursively using (212), we have
[Equation]
Thus,
[Equation]
Let H =
d (cid:48) cons /d .
Finally
[Equation]
it is obtained that
[Equation]
Based on Assumption
[Equation]
we have that ∀ k ∈ G ,
[Equation]
Thus
[Equation]
Therefore, by Deﬁnition 1 and (220),
[Equation]
Let ¯ w t +1 = 1 |G| (cid:80) k
∈G w t +1 k .
Combining with Equation (142) [Equation] we have:
[Equation]
The equation can be interpreted as that ˆ w t +1 is obtained by adding a small term − ( w t − ¯ w t +1 ) on ˆ w t with error e t .
Therefore,
[Equation]
[Equation]
Taking expectation on both sides, we have
[Equation]
By using Assumption 3, Assumption 4, Assumption 7, we have:
[Equation]
Note that E [ XY ] ≤ (cid:112) E
[ X 2 ] E [ Y 2 ] .
Based on Assumption 5, Assumption 7 and (223), we have:
[Equation]
According to Assumption 7,
[Equation]
Substituting (154), (223), (236), (240) and (241) into (231), we have:
[Equation]
[Equation]
Note that E (cid:107) u t (cid:107) =
(cid:112)
[ E (cid:107) u t (cid:107) ] 2 ≤ (cid:112)
[ E (cid:107) u t (cid:107) 2 ] and that E (cid:107) u t (cid:107) 2 = 4 H 2 ( A 2 ) 2 ( η A ) 2 .
Taking total expectation on both sides, we have:
[Equation] [Equation]
Namely,
[Equation]
By taking summation from t = 0 to T −
[Equation]
we have:
[Equation]
Note that ˆ w 0 = w 0 and F ( ˆ w T ) ≥ F ∗ .
Thus,
[Equation]
A
[Equation]
In summary,
[Equation]
where γ A , 1 = 2( A 2 ) 2 L + 4 HA 2 DL + 16 cδ (4 H 2 + 1)( A 2 ) 2 L , γ A , 2 = 8 H 2 ( A 2 ) 2 L 2 and ∆ A = 2 A 1 D + 2 BD + 4 (cid:112) 2 cδ (4 H 2 + 1)
A 2 D .
C M ORE E XPERIMENTAL R ESULTS
In this section, we present more empirical results, which are consistent to the ones in the main text of this paper and further support our conclusions.
C. M ORE E XPERIMENTS ABOUT THE E FFECT OF A LPHA
We present more empirical results about FedREP with aggregators geoMed and TMean in this section.
The experimental settings are the same as those in the main text.
As illustrated in Figure 3, the empirical results are consistent with that in the main text.
In addition, we have also noticed that the performance of FedREP with TMean is not stable enough under ALIE attack.
A possible reason is that the aggregator TMean is not robust enough against ALIE attack since FedREP with each of the other two aggregators (geoMed and CClip) has a relatively stable empirical results.
We will further study this phenomenon in future works
[Figure]
C. E XPERIMENTS ABOUT B YZANTINE A TTACKS ON C OORDINATES
In each iteration of FedREP, clients will send the coordinate set I tk to server.
However, Byzantine clients may send arbitrary coordinates.
Although the theoretical analysis in the main text has included this case, we also provide empirical results about Byzantine behaviour on sending coordinates.
We set α = 0 for non-Byzantine clients and consider four different Byzantine settings, where
Byzantine clients send the correct coordinates (noAtk), send the coordinates of K m smallest absolute values (minAtk), send random coordinates (randAtk) and send coordinates that is the same as a non-Byzantine client (sameAtk), respectively.
We set K = 0 .
d while the other settings are the same as those in Section 5 in the main text.
As illustrated in Figure 4, although the Byzantine attack on coordinates slightly changes the communication cost, it has little effect on the convergence rate and ﬁnal top- 1 accuracy.
The main reason is that the top- Km coordinates of each non-Byzantine client can always be sent to server in FedREP, no matter what is sent from Byzantine clients.
[Figure]
[Figure]
C. E XPERIMENTS ABOUT L OCAL M OMENTUM
Previous works (Karimireddy et al., 2021) have shown that using momentum can help to reduce the variance of stochastic gradients and obtain stronger Byzantine robustness.
We also provide empirical results about the momentum in this section.
The experimental settings keep the same as in Sectionin the main text.
As illustrated in Figure 5, using local momentum can make FedREP more robust to
Byzantine attack ALIE, which is consistent with previous works (Karimireddy et al., 2021).
C. C OMPARISON WITH S PARSE S EC A GG
We ﬁrst empirically compare the performance of FedREP with the communication-efﬁcient privacypreserving FL baseline
SparseSecAgg
(Ergun et al., 2021) when there is no attack.
We test the performance of FedREP with buffer size s = 4 , 8 and 16 , respectively.
We set Γ = 0 . 05 and 0 .for
SparseSecAgg in the two experiments, respectively.
Correspondingly, we set K = 0 .
05 d and K = 0 .
1 d in the two experiments for FedREP since the transmitted dimension number in FedREP is uncertain but not larger than K .
Thus, we have Γ ≤ K/d for FedREP.
The top-1 accuracy w.r.t.
epochs is illustrated in Figure 6.
The results show that FedREP can signiﬁcantly outperform the existing communication-efﬁcient privacy-preserving baseline SparseSecAgg when there is no Byzantine attack.
In addition, we have tried different learning rates for SparseSecAgg and it has the best performance when learning rate equals 5 .
As illustrated in Figure 7, FedREP signiﬁcantly outperforms
SparseSecAgg on top- 1 accuracy when communication cost is similar.
The communication cost of
SparseSecAgg is much more than FedREP when the performance on top- 1 accuracy is comparable.
For one reason, FedREP is based on top- K sparsiﬁcation while SparseSecAgg is based on random- K sparsiﬁcation.
For another reason, FedREP adopts error-compensation technique while SparseSecAgg does not.
[Figure] [Figure]
[Figure]
C. C OMPARISON WITH SHARE
We empirically compare the performance of FedREP and SHARE (Velicheti et al., 2021) in this section.
Both FedREP and SHARE are FL frameworks that can work with various local training algorithms on clients.
Compared to SHARE, the main advantage of FedREP is the consensus sparsiﬁcation.
Moreover, FedREP degenerates to SHARE when consensus sparsiﬁcation hyperparameter K = md .
Therefore, we compare SHARE and FedREP with different Γ when keeping other conditions the same.
Speciﬁcally, we set buffer size (a.k.a.
cluster size in SHARE) to beand local training algorithms to be vanilla SGD for each method.
The other settings are the same as those in Section 5 of the main text of this paper.
Since the actually transmitted dimension number is uncertain in each communication round for FedREP, we count the average transmitted dimension number of all communication rounds and use it as a measurement of communication cost.
The experimental results of FedREP and SHARE when there are no Byzantine clients are illustrated in
Figure 8.
The experimental results of FedREP and SHARE when there are 3 Byzantine clients under bit-ﬂipping attack, ALIE attack and FoE attack are illustrated in Figure 9, Figure 10 and Figure 11, respectively.
As we can see from the empirical results, compared to SHARE, there is almost no loss on the convergence rate and ﬁnal accuracy when Γ is about 0 .
079 for FedREP.
In addition, there is only a little loss on ﬁnal accuracy when Γ is as low as about 0 . 017 .
Interestingly, when under ALIE attack, empirical results show that FedREP has an even higher ﬁnal accuracy compared to SHARE.
We conduct an extra experiment to compare the performance of
FedREP and SHARE when there are 7 Byzantine clients with ALIE attack.
We set local training algorithm to be momentum SGD with β = 0 .
9 and buffer size s = 2 in the extra experiment.
The other settings are the same.
As illustrated in Figure 12, FedREP can still outperform SHARE in this setting.
A possible reason is that the consensus sparsiﬁcation in FedREP can lower the dissimilarity between the updates of different clients and thus lower the aggregation error (please see Deﬁnitionin the main text for more details).
However, it requires more effort to further explore this aspect and we leave it for future work.
In summary, FedREP has a comparable performance to SHARE on convergence rate and ﬁnal accuracy, but has much less communication cost than SHARE.
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]