FSL: F EDERATED S UPERMASK L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Federated learning (FL) allows multiple clients with (private) data to collaboratively train a common machine learning model without sharing their private training data.
In-the-wild deployment of FL faces two major hurdles: robustness to poisoning attacks and communication efﬁciency.
To address these concurrently, we propose F ederated S upermask L earning (FSL).
FSL server trains a global subnetwork within a randomly initialized neural network by aggregating local subnetworks of all collaborating clients.
FSL clients share local subnetworks in the form of rankings of network edges; more useful edges have higher ranks.
By sharing integer rankings, instead of ﬂoat weights, FSL restricts the space available to craft effective poisoning updates, and by sharing subnetworks, FSL reduces the communication cost of training.
We show theoretically and empirically that FSL is robust by design and also signiﬁcantly communication efﬁcient; all this without compromising clients’ privacy.
Our experiments demonstrate the superiority of FSL in real-world FL settings; in particular, (1) FSL achieves similar performances as state-of-the-art FedAvg with signiﬁcantly lower communication costs: for CIFAR10, FSL achieves same performance as Federated Averaging while reducing communication cost by ∼ 35% .
(2) FSL is substantially more robust to poisoning attacks than state-of-the-art robust aggregation algorithms.
I NTRODUCTION
Federated Learning (FL) is an emerging AI technology, where mutually untrusted clients (e.g., Android devices) collaborate to train a shared model, called the global model , without explicitly sharing their local training data.
FL training involves a server (e.g., Google server) which collects model updates from selected FL clients in each round of training, and uses them to update the global model.
FL, although highly promising, faces multiple challenges (Kairouz et al., 2019; Li et al., 2020b) to its practical deployment, in particular, communication efﬁciency and robustness, which are the focus of our work.
Privacy preservation is another major challenge to FL, but is orthogonal to our work.
We present F ederated S upermask L earning (FSL), a novel approach to perform FL while concurrently achieving the two goals of robustness and communication efﬁciency.
FSL is built on a novel learning paradigm called supermasks (Zhou et al., 2019; Ramanujan et al., 2020), which allows it to reduce communication costs while achieving signiﬁcantly higher robustness.
Speciﬁcally, in FSL, clients collaborate to ﬁnd a subnetwork within a randomly initialized neural network which we call the supernetwork (this is in contrast to conventional FL where clients collaborate to train a neural network).
The goal of training in FSL is to collaboratively identify a supermask, which is a binary mask of 1’s and 0’s, that is superimposed on the random neural network (the supernetwork) to obtain the ﬁnal subnetwork.
The subnetwork is then used for downstream tasks, e.g., image classiﬁcation, hence it is equivalent to the global model in conventional FL.
Note that in entire FSL training, weights of the supernetwork do not change.
More speciﬁcally, each FSL client computes the importance of the edges of the supernetwork based on their local data.
The importance of the edges is represented as a ranking vector.
Each FSL client will use the edge popup algorithm
(Ramanujan et al., 2020) and their data to compute their local rankings (the edge popup algorithm aims at learning which edges in a supernetwork are more important over the other edges by minimizing the loss of the subnetwork on their local data).
Each client then will send their local edge ranking to the server.
Finally, the FSL server uses a novel voting mechanism to aggregate client rankings into a supermask, which represents which edges of the random neural network (the supernetwork) will form the global subnetwork.
Intuitions on FSL’s robustness: In traditional FL algorithms, clients send large-dimension model updates ∈ R d (real numbers) to the server, providing malicious clients a signiﬁcant ﬂexibility in fabricating malicious updates.
By contrast, FSL clients merely share the rankings of the edges of the supernetwork, i.e., integers ∈ [1 , d ] , where d is the size of the supernetwork.
Therefore, as we will show both theoretically and empirically, FSL provides robustness by design and reduces the impact of untargeted poisoning attacks.
Furthermore, unlike most existing robust FL frameworks,
FSL does not require any knowledge about the percentages of malicious clients.
Intuitions on FSL’s communication efﬁciency: In FSL, the clients and the server communicate just the rankings of the edges in the supernetwork, i.e., a permutation of indices in [1 , d ] .
Ranking vectors are generally signiﬁcantly smaller than the global model.
This, as we will show, signiﬁcantly reduces the upload and download communication in FSL compared to Federated Averaging (FedAvg) (McMahan et al., 2017), where clients communicate model parameters, each of 32/64 bits.
Empirical results: We experiment with three datasets in real-world heterogeneous FL settings and show that: (1) FSL achieves similar performance (e.g., model accuracy) as state-of-the-art FedAvg but with signiﬁcantly reduced communication costs: for CIFAR10, the accuracy and communication cost per client are 85.4% and 40.2MB for FedAvg, while 85.3% and 26.2MB for FSL.
(2) FSL is highly robust to poisoning attacks as compared to state-of-the-art robust aggregation algorithms: from 85.4% in the benign setting, 10% malicious clients reduce the accuracy of FL to 56.3% and 58.8% with Trimmed-Mean (Xie et al., 2018;
Yin et al., 2018) and Multi-Krum (Blanchard et al., 2017), respectively, while FSL’s performance only decreases to 79.0%.
We also compare FSL with two communication reduction methods, SignSGD
(Bernstein et al., 2019) and TopK (Alistarh et al., 2018a) and show that FSL produces comparable communication costs and model accuracies.
For instance, on CIFAR10, FSL, SignSGD, and TopK achieve 85.3%, 79.1%, and 82.1% test accuracy, respectively, when the corresponding communication costs (download and upload) are 26.2MB, 20.73MB, and 30.79MB.
On the other hand, FSL offers a signiﬁcantly superior robustness.
For instance, on CIFAR10, 10% (20%) malicious clients reduce the accuracy of SignSGD to 39.7% (10.0%), but FSL’s accuracy decreases to only 79.0% (69.5%).
TopK is incompatible with existing robust aggregation algorithms, hence uses Average aggregation and is as vulnerable as FedAvg, especially in the real-world heterogeneous settings.
R ELATED W ORKS
Supermask Learning:
Modern neural networks have a very large number of parameters.
These networks are generally overparameterized (Dauphin & Bengio, 2013; Denil et al., 2013), i.e., they have more parameters than they need to perform a particular task, e.g., classiﬁcation.
The lottery ticket hypothesis (Frankle & Carbin, 2019) states that a fully-trained neural network, i.e., supernetwork , contains sparse subnetworks , i.e., subsets of all neurons in supernetwork, which can be trained from scratch (i.e., by training same initialized weights of the subnetwork) and achieve performances close to the fully trained supernetwork.
The lottery ticket hypothesis allows for massive reductions in the sizes of neural networks.
(Ramanujan et al., 2020) offer a complementary conjecture that an overparameterized neural network with randomly initialized weights contains subnetworks which perform as good as the fully trained network.
Poisoning Attacks and Defenses for Federated Learning (FL): FL involves mutually untrusting clients.
Hence, a poisoning adversary may own or compromise some of the FL clients, called malicious clients , with the goal of mounting a targeted or untargeted poisoning attack.
In a targeted attack, the goal is to reduce the utility of the model on speciﬁc test inputs, while in the untargeted attack, the goal is to reduce the utility for all (or most) test inputs.
It is shown (Blanchard et al., 2017) that even a single malicious client can mount an effective untargeted attack on FedAvg.
In order to make FL robust to the presence of such malicious clients, the literature has designed various robust aggregation rules (AGR)
(Blanchard et al., 2017; Mhamdi et al., 2018;
Yin et al., 2018; Chang et al., 2019), which aim to remove or attenuate the updates that are more likely to be malicious according to some criterion.
For instance, Multi-krum (Blanchard et al., 2017) repeatedly removes updates that are far from the geometric median of all the updates, and Trimmed-mean (Xie et al., 2018;
Yin et al., 2018) removes the largest and smallest values of each update dimension and calculates the mean of the remaining values.
Unfortunately, these robust AGRs are not very effective in non-convex FL settings and multiple works have demonstrated strong targeted (Wang et al., 2020; on them.
Communication Cost of FL:
In many real-world applications of FL, it is essential to minimize the communication between FL server and clients.
Especially in cross-device FL, the clients (e.g., mobile phones and wearable devices) have limited resources and communication can be a major bottleneck.
There are two major types of communication reduction methods: (1) Qunatization methods reduce the resolution of (i.e., number of bits used to represent) each dimension of a client update.
For instance, SignSGD
(Bernstein et al., 2019) uses the sign (1 bit) of each dimension of model updates.
(2) Sparsiﬁcation methods propose to use only a subset of all the update dimensions.
For instance, in TopK (Aji & Heaﬁeld, 2017; Alistarh et al., 2018a), only the largest K% update dimensions are sent to the server in each FL round.
We note that, communication reduction methods primarily focus on and succeed at reducing upload communication (client → server), but they use the entire model in download communication (server → client).
P RELIMINARIES 3. F EDERATED L EARNING
In FL (McMahan et al., 2017; Kairouz et al., 2019; Kone ˇ cn ` y et al., 2016), N clients collaborate to train a global model without directly sharing their data.
In round t , the service provider (server) selects n out of N total clients and sends them the most recent global model θ t .
Each client trains a local model for E local epochs on their data starting from the θ t using stochastic gradient descent (SGD).
Then the client send back the calculated gradients ( (cid:79) k for k th client) to the server.
The server then aggregates the collected gradients and updates the global model for the next round.
FL can be either cross-device or cross-silo (Kairouz et al., 2019).
In cross-device FL, N is large (from few thousands to billions) and only a small fraction of clients is chosen in each FL training round, i.e., n (cid:28) N .
By contrast, in cross-silo FL, N is moderate (up to 100) and all clients are chosen in each round, i.e., n = N .
In this work, we evaluate the performance of FSL and other FL baselines for cross-device FL under realistic production FL settings.
3. E DGE - POPUP A LGORITHM
a subnetwork of size k % of the entire network θ 1: Input: number of local epochs E , training data D , initial weights θ w and scores θ s , subnetwork size k %, learning rate η 2: for e ∈
[ E ] do 3:
B ← Split D in B batches 4: for batch b ∈
[ B ] do 5:
EP F ORWARD ( θ w , θ s , k, b ) 6: θ s = θ s − η ∇ (cid:96) ( θ s ; b ) 7: end for 8: end for 9: return θ s 10: function EP FORWARD ( θ w , θ s , k, b ) 11: m ← sort ( θ s ) 12: t ← int ((1 − k ) ∗ len ( m )) 13: m [: t ] =14: m [ t :] =15: θ p
= θ w (cid:12) m 16: return θ p ( b ) 17: end function
The edge-popup (EP) algorithm (Ramanujan et al., 2020) is a novel optimization method to ﬁnd supermasks within a large, randomly initialized neural network, i.e., a supernetwork, with performances close to the fully trained supernetwork.
EP algorithm does not train the weights of the network, instead only decides the set of edges to keep and removes the rest of the edges (i.e., pop).
Speciﬁcally, EP algorithm assigns a positive score to each of the edges in the supernetwork.
On forward pass, it selects top k% edges with highest scores, where k is the percentage of the total number of edges in the supernetwork that will remain in the ﬁnal subnetwork.
On the backward pass, it updates the scores with the straight-through gradient estimator (Bengio et al., 2013).
Algorithmpresents EP algorithm; we defer further details to Appendix D.
F EDERATED S UPERMASK L EARNING : D ESIGN
In this section, we provide the design of our federated supermask learning (FSL) algorithm.
FSL clients collaborate (without sharing their local data) to ﬁnd a subnetwork within a randomly initialized, untrained neural network called the supernetwork .
Algorithm 2 describes FSL’s training.
Training a global model in FSL means to ﬁrst ﬁnd a unanimous ranking of supernetwork edges and then use the subnetwork of the top ranked edges as the ﬁnal output.
We detail a round of FSL training and depict it in Figure 1, where we use a supernetwork with six edges e i ∈ [0 , 5] to demonstrate a single FSL round and consider three clients C j ∈ [1 , 3] who aim to ﬁnd a subnetwork of size k =50% of the original supernetwork.
←
←
13: end
Server: Majority Vote 15:
←
16: end for 17: function V OTE
V ← A
S
A ← UM
20: return
21: end function
4. S ERVER : I NITIALIZATION P HASE (O NLY FOR ROUND t = 1 )
In the ﬁrst round, the FSL server chooses a random seed S EED to generate initial random weights θ w and scores θ s for the global supernetwork θ ; note that, θ w , θ s , and S EED remain constant during the entire FSL training.
Next, the FSL server shares S EED with FSL clients, who can then locally reconstruct the initial weights θ w and scores θ s using S EED .
Figure 1- 1 depicts this step.
Recall that, the goal of FSL training is to ﬁnd the most important edges in θ w without changing the weights.
Unless speciﬁed otherwise, both server and clients use the Singed Kaiming Constant algorithm (Ramanujan et al., 2020) to generate random weights and the Kaiming Uniform algorithm (He et al., 2015) to generate random scores.
However, in Appendix C.1, we also explore the impacts of different initialization algorithms on the performance of FSL.
We use the same seed to initialize weights and scores.
At the beginning, the FSL server ﬁnds the global rankings of the initial random scores (Algorithmline 4), i.e., R 1 g =
A RG S ORT ( θ s ) .
We deﬁne rankings of a vector as the indices of elements of vector when the vector is sorted from low to high, which is computed using A RG S ORT function (argsort).
4. C LIENTS : C ALCULATING THE RANKS (F OR EACH ROUND t )
In the t th round, FSL server randomly selects n clients among total N clients, and shares the global rankings R tg with them.
Each of the selected n clients locally reconstructs the weights θ w ’s and scores θ s ’s using SEED (Algorithm 2 line 9).
Then, each FSL client reorders the random scores based on the global rankings, R tg (Algorithm 2 line 10); we depict this in Figure 1- 2a .
Next, each of the n clients uses reordered θ s and ﬁnds a subnetwork within θ w using Algorithm 1; to ﬁnd a subnetwork, they use their local data and E local epochs (Algorithm 2 line 11).
Note that, each iteration of Algorithm 1 updates the scores θ s .
Then client u computes their local rankings R tu using the ﬁnal updated scores ( S ) and A RGSORT (.), and sends R tu to the server.
Figure
1- 2a shows how each of the selected n clients reorders the random scores using global rankings.
For instance, the initial global rankings for this round are R tg =
[2 , 3 , 0 , 5 , 1 , 4] , meaning that edge e 4 should get the highest score ( s 4 = 1 . 2 ), and edge e 2 should get the lowest score ( s 2 = 0 . 2 ).
Figure 1- 2b shows, for each client, the scores and rankings they obtained after ﬁnding their local subnetwork.
For example, rankings of client C 1 are R t 1 = [4 , 0 , 2 , 3 , 5 , 1] , i.e., e 4 is the least important and e 1 is the most important edge for C 1 .
Considering desired subnetwork size to be 50%, Cuses edges { 3,5,1 } in their ﬁnal subnetwork.
4. S ERVER : M AJORITY V OTE (F OR EACH ROUND t )
The server receives all the local rankings of the selected n clients, i.e., R t { u ∈ U } .
Then, it performs a majority vote over all the local rankings using V OTE (.) function.
Note that, for client u , the index i represents the importance of the edge R tu [ i ] for C u .
For instance, in Figure 1- 2b , rankings of C 1 are
R t 1 = [4 , 0 , 2 , 3 , 5 , 1] , hence the edge e 4 at index=0 is the least important edge for C 1 , while the edge e 1 at index=5 is the most important edge.
Consequently, V OTE (.) function assigns reputation=0 to edge e 4 , reputation=1 to e 0 , reputation=2 to e 2 , and so on.
In other words, for rankings R tu of C u and edge e i , V OTE (.) computes the reputation of e i as its index in R tu .
Finally, V OTE (.) computes the total reputation of e i as the sum of reputations from each of the local rankings.
In Figure 1- 2b , reputations of e 0 are 1 in R t 1 , 1 in R t 2 , and 0 in R t 3 , hence, the total reputation of e 0 is 2.
We depict this in Figure 1- 3 ; here, the ﬁnal total reputations for edges e { i ∈ [0 , 5] } are A = [2 , 12 , 3 , 11 , 8 , 9] .
Finally, the server computes global rankings R t
+1 g to use for round t + 1 by sorting the ﬁnal total reputations of all edges, i.e., R t +1 g
[Equation]
Note that, all FSL operations that involve sorting, reordering, and voting are performed in a layerwise manner .
For instance, the server computes global rankings R tg in round t for each layer separately, and consequently, the clients selected in round t reorder their local randomly generated scores θ s for each layer separately.
F EDERATED S UPERMASK L EARNING : S ALIENT F EATURES
In this section, we discuss the two salient features of FSL that are instrumental for any distributed learning algorithm to be practical: robustness to poisoning attacks and communication efﬁciency .
5. R OBUSTNESS OF FSL TO P OISONING A TTACKS
FSL is a distributed learning algorithm with mutually untrusting clients.
Hence, a poisoning adversary may own or compromise some of FSL clients, called malicious clients , and mount a targeted or untargeted poisoning attack.
In our work, we consider the untargeted attacks as they are more severe than targeted attacks and can cause denial-of-service for all collaborating clients (Shejwalkar et al., 2021) and show that FSL is secure against such poisoning attacks by design.
Intuition on FSL’s robustness: Existing FL algorithms, including robust FL algorithms, are shown to be vulnerable to targeted and untargeted poisoning attacks (Shejwalkar et al., 2021) where malicious clients corrupt the global model by sharing malicious model updates.
One of the key reasons behind the susceptibility of existing algorithms is that their model updates can have arbitrary values.
For instance, to manipulate vanilla FedAvg, malicious clients send very large updates (Blanchard et al., 2017), and to manipulate Multi-krum and Trimmed-mean, (Fang et al., 2020; Shejwalkar &
Houmansadr, 2021) propose to perturb a benign update in a speciﬁc malicious direction.
On the other hand, in FSL, clients must send a permutation of indices ∈ [1 , n (cid:96) ] for each layer.
Hence, FSL signiﬁcantly reduces the space of the possible malicious updates that an adversary can craft.
Majority voting in FSL further reduces the chances of successful attack.
Intuitively, this makes FSL design robust to poisoning attacks.
Below, we make this intuition more concrete.
The worst-case untargeted poisoning attack on FSL: Here, the poisoning adversary aims to reduce the accuracy of the ﬁnal global FSL subnetwork on most test inputs.
To achieve this, the adversary should replace the high ranked edges with low ranked edges in the ﬁnal subnetwork.
For the worst-case analysis of FSL, we assume a very strong adversary (i.e., threat model): 1) each of the malicious clients has some data from benign distribution; 2) malicious clients know the entire FSL algorithm and its parameters; 3) malicious clients can collude.
Under this threat model we design a worst case attack on FSL (Algorithm 3 in Appendix A.1), which executes as follows:
First, all malicious clients compute rankings on their benign data and use V OTE (.) algorithm to compute an aggregate rankings.
Finally, each of the malicious clients uses the reverse of the aggregate rankings to share with the FSL server in given round.
The adversary should invert the rankings layer-wise as the FSL server will aggregate the local rankings per layer too, and it is not possible to mount a model-wise attack.
Now we justify why the attack in Algorithm 3 is the worst case attack on FSL for the strong threat model we consider.
Note that, FSL aggregation, i.e., V OTE (.), computes the reputations using clients’ rankings and sums the reputations of each network edge.
Therefore, the strongest poisoning attack would want to reduce the reputation of good edges.
This can be achieved following the aforementioned procedure of Algorithm 3 to reverse the rankings computed using benign data.
[Figure]
Theoretical analysis of robustness of FSL algorithm:
In this section, we prove an upper bound on the failure probability of robustness of FSL, i.e., the probability that a good edge will be removed from the ﬁnal subnetwork when malicious clients mount the worst case attack.
Following the work of Bernstein et al.
(2019), we make two assumptions in order to facilitate a concrete robustness analysis of FSL: a) each malicious client has access only to its own data, and b) we consider a simpler V OTE (.) function, where the FSL server puts an edge e i in the ﬁnal subnetwork if more than half of the clients have e i (a good edge) in their local subnetworks.
In other words, the rankings that each client sends to the server is just a bit mask showing that each edge should or should not be in the ﬁnal subnetwork.
The server makes a majority vote on the bit masks, and if an edge has more than half votes, it will be in the global subnetwork.
Our V OTE (.) mechanism has more strict robustness criterion, as it uses more nuanced reputations of edges instead of bit masks.
Hence, the upper bound on failure probability in this section also applies to the FSL VOTE (.) function.
The probability that our voting system fails is the probability that more than half of the votes do not include e i in their subnetworks.
The upper bound on the probability of failure would be 1 / 2 (cid:113) np
(1 − p ) ( n ( p + α (1 − 2 p ) − 1 / 2)) 2 , where n is the number of clients being processed
, p shows the probability that a benign client puts e i in their top ranks, and α is the fraction of malicious clients.
Due to space limitations, we defer the detailed proof to Appendix A.2.
Figure 2 shows the upper bound on the failure of V OTE (.) for different values of α and p .
We note that, the higher the probability p , the higher the robustness of FSL.
5. FSL’ S C OMMUNICATION C OSTS
In FL, and especially in the cross-device setting, clients have limited communication bandwidth.
Hence, FL algorithms must be communication efﬁcient.
We discuss here the communication cost of FSL algorithm.
In the ﬁrst round, the FSL server only sends one seed of 32 bits to all the FSL clients, so they can construct the random weights ( θ w ) and scores ( θ s ).
In a round t , each of selected
FSL clients receives the global rankings R tg and sends back their local rankings R tu .
The rankings are a permutation of the indices of the edges in each layer, i.e., of [0 , n (cid:96) − 1] ∀ (cid:96) ∈
[ L ] where L is the number of layers and n (cid:96) is the number of parameters in (cid:96) th layer.
We use the naive approach to communicate layer-wise rankings, where each FSL client exchanges a total of (cid:80) (cid:96)
∈
[ L ] n (cid:96) × log( n (cid:96) ) bits.
Because, for the layer (cid:96) , the client receives and sends n (cid:96) ranks where each one is encoded with log( n (cid:96) ) bits.
On the other hand, a client exchanges (cid:80) (cid:96) ∈
[ L ] n (cid:96) ×bits in FedAvg, when 32 bits are used to represent each of n (cid:96) weights in layer (cid:96) .
In Appendix E, we compare theoretical communication costs of various FL algorithms.
Sparse-FSL: Here, we propose Sparse-FSL, a simple extension of FSL to further reduce the communication cost.
In Sparse-FLS, a client sends only the most important ranks of their local rankings to the server for aggregation.
For instance, in Figure 1, client C 1 sends R t 1 =
[4 , 0 , 2 , 3 , 5 , 1] in case of FSL.
But in sparse-FSL, with sparsity set to 50%, client C 1 sends just the top 3 rankings, i.e., sends R (cid:48) t
= [3 , 5 , 1] .
For each client, the sparse-FSL server assumes 0 reputation for all of the edges not included in the client’s rankings, i.e., in Figure 1, sparse-FSL server will assign reputation=0 for edges e 4 , e 0 , and e 2 .
Then the server uses V OTE (.) to compute total reputations of all edges and then sort them to obtain the ﬁnal aggregate global rankings, i.e., R t +1 g , to use for subsequent rounds.
We observe in out experiments, that sparse-FSL performs very close to FSL, even with sparsity as low as 10%, while also signiﬁcantly reducing the communication cost.
Due to space limitation, we defer the communication cost comparison of FSL with FedAvg, SingSGD, and LotteryFL (Li et al., 2020a) to Appendix E.
E XPERIMENTS
In this section, we investigate the utility, robustness, and communication cost of our FSL framework.
We use MNIST, CIFAR10, and FEMNIST data and distribute them in non-iid fashion (using Dirichlet distribution) among 1000, 1000, and 3400 clients respectively.
At the end of the training, we calculate the test accuracy of all the clients on the ﬁnal global model, and we report the mean and standard deviation of all clients’ test accuracies in our experiments.
We provide further details of experimental setup in Appendix B.
In addition to FSL, we also evaluate Sparse-FSL in different settings.
We use SFSL top x% to denote a Sparse-FSL clients who sends top x% of ranks in each round.
6. C OMMUNICATION C OST A NALYSIS
In FSL, both clients and server communicate just the edge ranks instead of weight parameters.
Thus, FSL reduces both upload and download communication cost.
Table 1 illustrates the utility, i.e., the accuracy on test data, and communication cost of FSL and state-of-the-art quantization, i.e., SignSGD (Bernstein et al., 2019), and sparsiﬁcation, i.e., TopK (Alistarh et al., 2018b; Aji &
Heaﬁeld, 2017) communication-reduction methods.
FSL versus SignSGD: We note that, FSL is signiﬁcantly more accurate than SignSGD.
For instance, on CIFAR10, distributed non-iid among 1000 clients, FSL achieves 85.3% while SignSGD achieves 79.1% , or on FEMNIST, FSL achieves 84.2% while SignSGD achieves 79.3%.
This is because, FSL clients send more nuanced information via rankings of their subnetworks compared to
SignSGD, where clients just send the signs of their model updates.
SignSGD in FL reduces only the upload communication, but for efﬁciency reasons, the server sends all of the weight parameters (each of 32 bits) to the newly selected clients.
Hence, SignSGD has very efﬁcient upload communication, but very inefﬁcient download communication.
For instance, on CIFAR10, for both upload and download, FSL achieves 13.1MB each while SignSGD achieves 0.63MB and 20.1MB, respectively.
FSL versus TopK:
We compare FSL with TopK with K ∈ { 10 , 50 } % .
FSL is more accurate than Topk for MNIST and CIFAR10: on CIFAR10, FSL accuracy is 85.3%, while TopK accuracies are 82.1% and 77.8% with K =50% and K =10%, respectively.
Similar to SignSGD, Topk is more efﬁciently reduces upload cost, but has very high download communication cost.
Therefore, the [Table] combined upload and download communication cost per client per round is 26.2MB for FSL and 30.79MB for TopK with K =50%, and TopK still has worse performance.
Communication cost reduction due to Sparse-FSL (SFSL): We now evaluate SFSL explained in Section 5.2.
In SFSL with top 50% ranks, clients send the top 50% of their ranks to the server, which reduces the upload bandwidth consumption by half.
Please note that SFSL download cost is the same as FSL download cost since the FSL server should send all the global rankings to selected clients for the next FSL round.
We note from Table 1 that, by sending fewer ranks, SFSL reduces upload communication at a small cost of performance.
For instance, on CIFAR10, SFLS with top 50% reduces the upload communication by 50% at the cost reducing accuracy from 85.4% to 77.6%.
6. S ECURITY A NALYSIS
We compare FSL with state-of-the-art robust aggregation rules (AGRs):
Mkrum (Blanchard et al., 2017), and Trimmed-mean (Xie et al., 2018;
Yin et al., 2018).
Table 2 gives the performances of robust AGRs, SignSGD, and FSL with different percentages of malicious clients.
Here, we make a rather impractical assumption in favor of the robust AGRs: we assume that the server knows the exact % of malicious clients in each FL round.
FSL does not require this knowledge.
FSL achieves higher robustness than state-of-the-art robust AGRs: We note from Table
that,
FSL is more robust to the presence of malicious clients who try to poison the global model compared to Multi-Krum, Trimmed-mean, and SignSGD for both 10% and 20% malicious clients rates.
For instance, on CIFAR10, 10% malicious clients can decrease the accuracy of FL models to 56.3%, 58.8%, and 39.8% for Trimmed-mean, Multi-Krum, and SignSGD respectively; 20% malicious clients can decrease the accuracy of the FL models to 20.5%, 25.6%, 10.0% for Trimmed-mean,
Multi-Krum, and SignSGD respectively.
On the other hand, FSL performance decreases to 79.0% and 69.5% for 10% and 20% attacking ratio respectively.
We make similar observations for MNIST and FEMNIST datasets: for FEMNIST, 10% (20%) malicious clients reduce accuracy of the global model from 85.8% to 72.7% (56.2%) for Trimmed-Mean, to 80.9% (23.7%) for Multi-krum, and 76.7% (55.1%) for SignSGD, while FSL accuracy decreases to 83.0% (65.8%).
We omit evaluating TopK, because even a single malicious client (Blanchard et al., 2017) can jeopardize its accuracy. [Table]
6. M ISCELLANEOUS D ISCUSSIONS
Due to space limitations, we defer a detailed discussion of ablation studies of FSL to Appendix C and below give their important takeaways.
Initialization matters in FSL:
In FSL, the weight parameters are randomly initialized at the start and remain ﬁxed throughout the training.
An appropriate initialization is instrumental to the success of FSL, since the clients are trying to ﬁnd the most important weight parameters.
We study efﬁcacy of three initializing strategies that use three different distributions: Glorot Normal, Kaiming Normal, and Singed Kaiming Constant.
Table 5 shows the results.
We observe from Table 5 that, Singed
Kaiming Constant initialization achieves the best results that are closest to FedAvg.
Varying the sparsity of edge-popup algorithm in FSL: Figure 4 illustrates how the performance of FSL varies with the sizes of local subnetworks that the clients share with the server.
In other words, when we vary the sparsity k % of edge popup algorithm during local subnetwork search k ∈ [10 , 20 , 30 , 40 , 50 , 60 , 70 , 80 , 90]% .
Interestingly we note that, FSL performs the worst when clients use all ( k =100%) or none ( k =0%) of the edges.
This is because, it is difﬁcult to ﬁnd a subnetwork with small number of edges.
While using all of the edge essentially results in using a random neural network.
As we can see FSL with k ∈ [40 , 70]% , gives the best performances for all the three datasets.
Hence, we set k =50% by default in our experiments.
C ONCLUSIONS
We designed a novel collaborative learning algorithm, called F ederated S upermask L earning (FSL), to address the issues of robustness to poisoning and communication efﬁciency in existing FL algorithms.
We argue that a core reason for the susceptibility of existing FL algorithms to poisoning is the use of arbitrary values in their model updates.
Hence, in FSL, we use ranks of edges of a randomly initialized neural network contributed by collaborating clients to ﬁnd a global ranking and then use a subnetwork based only on the top edges.
Use of rankings in a ﬁxed range restricts the space available to poisoning adversaries to craft malicious updates, and also allows FSL to use sophisticated communication reduction methods.
We show, both theoretically and empirically, that ranking based collaborative learning can effectively mitigate the robustness issue as well as reduce the communication costs involved.
R EPRODUCIBILITY S TATEMENT
We have attached our implementation code and experiment notebooks as supplementary materials.
There are separate experiment notebooks showing the process of FSL training for MNIST, CIFAR and FEMNIST (results that we used in Table 1 and Table 2).
We have explained all the hyperparameters including optimizer, learning rate, momentum, weight decay, and batch size in Appendix B.1 for both FSL and FedAvg (including using robust AGRs) settings.
We also explained the model architectures we used in our experiments in Table 4 explaining different layers with number of parameters inside each.
We also explained how we distributed MNIST and CIFAR10 samples among 1,000 clients using Dirichlet distribution to make them non-iid in Appendix B.3.
We propose F ederated S upermask L earning (FSL) where clients ﬁnd a subnetwork within a randomly initialized neural network, without training the weights of the network.
Our extensive evaluation demonstrated FSL can provide better communication cost and more robustness to untargeted poisoning attacks compared to existing FL compressors and Byzantine-robust aggregation rules.
A M ISSING D ETAILS OF R OBUSTNESS OF FSL A. FSL W ORST C ASE P OISONING A TTACK A LGORITHM
Algorithm 3 shows the rankings poisoning attack explained in Section 5.
Algorithm
1: Input: number of malicious clients M , number of malicious local epochs E (cid:48) , seed SEED , global ranking R tg , learning rate η , subnetwork size k % 2: function C RAFTMALICIOUSUPDATE ( M, SEED , R tg , E (cid:48) , η, k ): 3: for mu ∈
[ M ] do (cid:46)
For all the malicious clients 4:
Malicious Client Executes: 5: θ s , θ w ← Initialize scores and weights using SEED 6: θ s [ R tg ] ← SORT ( θ s ) 7:
S ← Edge-PopUp( E (cid:48) , D tru , θ w , θ s , k, η ) 8:
R tmu ← A RG S ORT ( S ) (cid:46)
Ranking of the malicious client 9: end for 10:
Aggregation: 11:
R tm ← V OTE
( R tmu ∈
[ M ] ) (cid:46) Majority vote aggregation 12: return R EVERSE ( R tm ) (cid:46) reverse the ranking 13: end function
A. T HEORETICAL ANALYSIS OF ROBUSTNESS OF FSL
In this section, we detail the proof of robustness of FSL.
In other words, we prove an upper bound on the failure probability of robustness of FSL, i.e., the probability that a good edge will be removed from the ﬁnal subnetwork when malicious clients mount the worst case attack.
Inspired from
SignSGD (Bernstein et al., 2019), for this proof, We assume a simpler V OTE (.) function where if more than half of the clients add an edge e i to their subnetworks, then the FSL server adds it to the ﬁnal global subnetwork.
We also assume that the malicious clients cannot collude in our proof.
Assume that edge e i is a good edge, i.e., having e i in the ﬁnal subnetwork improves the performance of the ﬁnal subnetwork. Let Z be the random variable that represents the number of clients who vote for the edge e i to be in the ﬁnal subnetwork, i.e., the number of clients whose local subnetwork of size k % of the entire supernetwork (Algorithm 2 line 11) contains e i .
Therefore, Z ∈ [0 , n ] where n is the number of clients being processed in a given FSL round.
Let G and B be the random variable that represent the number of benign and malicious clients that vote for edge e i , respectively; the malicious clients inadvertently exclude the good edge e i in their local subnetwork based on their benign training data.
There are total of αn malicious clients, where α is the fraction of malicious clients that B of them decides that e i is a bad edge and should not be removed.
Each of the malicious clients computes the subnetwork on its own benign training data, so B of them do not conclude that e i is a good edge.
Hence, Z = G + B .
We can say that G and B have binomial distribution , i.e., G ∼ binomial ([(1 − α ) n, p ] and B ∼ binomial ([ αn, 1 − p ] where p is the probability that a benign client has this edge in their local subnetwork and α is the fraction of malicious clients.
Note that the probability that our voting in simpliﬁed FSL fails is P [ failure ] =
P [ Z < = n 2 ] , i.e., when more than half of the clients vote against e i , i.e., they do not include e i in their local subnetworks.
We can ﬁnd the mean and variance of Z as follows:
[Equation]
Cantelli (1929) provides an inequality where for a random variable X with mean µ and variance σwe have P [ µ − X > = λ ] < = 1 1+ λ 2 σ [-
-]2 . Using this inequality, we can write: [Equation] [Figure]
[Figure]
malicious ratio=10% [Figure] ratio= [Figure]
% malicious ratio=30% malicious ratio=40%
[Equation]
because 1 + x 2 > = 2 x , we have:
[Equation]
What this means is that the probability that the simpliﬁed V OTE (.) fails is upper bounded as in (4). We show the effect of the different values of α and p in Figure 2. We can see from Figure 2, if the benign clients can train better supermasks (better chance that a good edge ended in their subnetwork), the probability that the attackers succeed is lower (more robustness). V OTE (.)
in FSL (Section 4.3) is more sophisticated and puts more constraints on the malicious clients, hence the about upper bound also applies to FSL.
We show the theoretical relationship between upper bound on the failure of V OTE (.) for different values of malicious rate ( α ) in Figure 2.
To validate our theoretical bounds, we measure the least number of malicious clients that the adversary needs to control to remove a good edge from the global subnetwork in Figure 3.
In this Figure, we report the votes of one particular edge in the ﬁrst layer of LeNet (trained on MNIST) where there are 288 edges in the ﬁrst layer (Table B.1 shows the number of edges in each layer).
We consider this edge as a good edge as it would be in the ﬁnal global subnetwork if there were no malicious clients.
We are selecting 25 clients in each round, so we consider 2 × α × 25 for thresholds.
For instance, when we assume there are 20% malicious clients (on average there are 5 malicious among 25 selected clients) that means that 5 votes of benign votes decreases and 5 votes is added to malicious votes, so the threshold would be 10.
[Table]
B M ISSING D ETAILS OF E XPERIMENTAL S ETUP B. D ATASETS AND MODEL ARCHITECTURES
MNIST is a 10-class class-balanced classiﬁcation task with 70,000 gray-scale images, each of size 28 × 28.
We experiment with LeNet architecture given in Table 4. For local training in each FSL/FL round, each client uses 2 epochs.
For training weights (experiments with FedAvg, SignSGD, TopK), we use SGD optimizer with learning rate of 0.01, momentum of 0.9, weight decay of 1e-4, and batch size 8.
For training supermasks (experiments with FSL), we use SGD with learning rate of 0.4, momentum 0.9, weight decay 1e-4, and batch size 8.
CIFAR10 (Krizhevsky & Hinton, 2009) is a 10-class classiﬁcation task with 60,000 RGB images (50,000 for training and 10,000 for testing), each of size 32 × 32.
We experiment with a VGG-like architecture given in Table 4, which is identical to what Ramanujan et al.
(2020) used. For local training in each FSL/FL round, each client uses 5 epochs.
For training weights (experiments with FedAvg, SignSGD, TopK), we use SGD with learning rate of 0.1, momentum of 0.9, weight decay of 1e-4, and batch size of 8. For training supermasks (experiments with FSL), we optimize SGD with learning rate of 0.4, momentum of 0.9, weight decay of 1e-4, and batch size of 8.
FEMNIST (Caldas et al., 2018;
Cohen et al., 2017) is a character recognition classiﬁcation task with 3,400 clients, 62 classes (52 for upper and lower case letters and 10 for digits), and 671, gray-scale images.
Each client has data of their own handwritten digits or letters.
We use LeNet architecture given in Table 4. For local training in each FSL/FL round, each client uses 2 epochs.
For training weights (experiments with FedAvg, SignSGD, TopK), we use SGD with learning rate of 0.15, momentum of 0.9, weight decay of 1e-4, and batch size of 10. For training supermasks (experiments with FSL), we optimize SGD with learning rate of 0.2, momentum of 0.9, weight decay of 1e-4, and batch size of 10. B. HYPERPARAMETERS T UNING
We optimize the hyperparameters based on FSL and other baselines independently. The hyperparameters that we used in our experiments are tuned in scenario with no malicious clients. Table B. shows the performance of FSL and other baselines on CIFAR10 (distributed over 1000 users using Dirichlet distribution) for different values of hyperparameters when there are 10% malicious clients among the clients.
This table shows the robustness of FSL still persists even if we change the hyperparameters.
We reported mean of accuracies and standard deviation of accuracies for all the clients at the ﬁnal FSL round.
B. N ON - IID D ATA D ISTRIBUTION
Using Dirichlet Distribution: Considering the heterogeneous data in the real-word cross-device
FL, similar to previous works (Reddi et al., 2020; Hsu et al., 2019), we distribute MNIST and CIFAR10 among 1,000 clients in a non-iid fashion using Dirichlet distribution with parameter β = 1 . Note that increasing β results in more iid datasets.
Next, we split datasets of each client into training ( 80% ) and test ( 20% ).
At the end of the FL rounds, we calculate the test accuracy of each client for its test data, and we report the average of test accuracies of all the clients.
We run all the experiments for 2000 global rounds of FSL and FL and select 25 clients in each round. Assigning only two classes to each client: McMahan et al. (2017) used a more extreme heterogeneous data assignment.
For assignment of MNIST and CIFAR10 among 1000 clients using this
Non-iid-ness method, we sort all the training and validation data inside MNIST and CIFAR10, then partition them into 2000 shards.
This means that each shards of training MNIST has 30 images and each CIFAR10 shard has 25 images. Then we assign two random shards to each client resulting in each client have at most data of two classes, and in CIFAR10 experiments, each client hastraining images, and 10 test images, and in MNIST experiments, each client has 60 training images and 10 test images. We only use this assignment in Section C.3. B. B ASELINE FL A LGORITHMS
Federated averaging In non-adversarial FL settings, i.e., without any malicious clients, the dimension-wise Average (FedAvg) (Koneˇcn`y et al., 2016; McMahan et al., 2017) is an effective
AGR. In fact, due to its efﬁciency, Average is the only AGR implemented by FL applications in practice (Ludwig et al., 2020; Paulik et al., 2021).
SignSGD is a quantization method used in distributed learning to compress each dimension of updates into 1 bit instead of 32 or 64 bits.
To achieve this, in SignSGD (Bernstein et al., 2019)
the clients only send the sign of the gradient updates to the server, and the server runs a majority vote on them.
SignSGD is designed for distributed learning where all the clients participate in each round, so all the clients are aware of the most updated weight parameters of the global model.
However, using SignSGD in FL just provides beneﬁt in upload bandwidth, but to achieve good overall performance of the global model, the server should send all the weight parameters (each of 32 bits) to the newly selected clients in each round.
This makes SignSGD very efﬁcient in upload cost, but it is as inefﬁcient as FedAvg in download.
TopK is a sparsiﬁcation method used in distributed learning that transmits only a few elements in each model update to the server.
In TopK (Aji & Heaﬁeld, 2017; Alistarh et al., 2018a), the clients ﬁrst sort the absolute values of their local gradient updates, and send the Top K% largest gradients update dimensions to the server for aggregation.
TopK suffers from the same problem as SignSGD: for performance reasons, the server should send the entire updated model weights to the new selected clients. B. M ODEL P OISONING A TTACK FOR R OBUSTNESS E VALUATIONS
To evaluate robustness of various FL algorithms, we use state-of-the-art model poisoning attack proposed by Shejwalkar & Houmansadr (2021) in our robustness experiments.
The attack proposes a general FL poisoning framework and then tailors it to speciﬁc FL settings. It ﬁrst computes an average ∇ b of the available benign updates and perturbs it in a dynamic, data-dependent malicious direction ω to compute the ﬁnal poisoned update ∇ (cid:48)
= ∇ b + γω . DYN-OPT ﬁnds the largest γ that [Table]
successfully circumvents the target AGR. DYN-OPT is much stronger, because unlike STAT-OPT, it ﬁnds the largest γ and uses a dataset tailored ω .
C M ISSING E XPERIMENTS C. FSL: I NITIALIZATION M ATTERS
[Table]
In FSL
[Table]
the weight parameters are ﬁxed throughout the FSL protocol and they are initialized randomly at the beginning of the protocol.
It is very important to appropriately initialize the weights since the clients will ﬁnd the subnetworks within these weights.
We use three different distribution for initializing the weight parameters as follows:
Glorot Normal (Glorot & Bengio, 2010) where we denote by X N .
Previous work Zhou et al.
(2019) used this initialization to demonstrate that subnetworks of randomly weighted neural networks can achieve impressive performance.
Kaiming Normal (He et al., 2015) where we denote by N k deﬁned as N K = N (cid:16) 0 , (cid:112) 2 /n
(cid:96) − 1 (cid:17) where N shows normal distribution.
n (cid:96) shows the number of parameters in the (cid:96) th layer.
Singed Kaiming Constant (Ramanujan et al., 2020) where all the wights are a constant σ but they are assigned { + , −} randomly.
This constant, σ , is the standard deviation of Kaiming Normal.
We show this initialization with U K as we are sampling from {− σ, + σ } where σ = (cid:16)(cid:112) 2 /n (cid:96) − 1 (cid:17) .
Table 5 shows the results of running FSL for three datasets under the three aforementioned initialization algorithms.
We compare FSL with FedAvg and report the mean, standard deviation, minimum, and maximum of the accuracies for the clients’ local subnetwork (for FSL) and local models (for FedAvg) at the end of FSL/FedAvg training.
As we can see under three different random initialization, using Signed Kaiming Normal ( U K ) results in better performance.
We note from Tablethat FSL with Signed Kaiming Normal ( U K ) initialization achieves performance very close to the performance of FedAVg. Note that, since the FSL clients update scores in each round, unlike initialization of weights, initialization of scores does not have signiﬁcant impact on the ﬁnal global subnetwork search. Therefore, we do not explore different randomized initialization algorithms for scores and simply use Kaiming
Uniform initialization for scores.
[Figure]
[Figure]
[Figure]
T e s t
A cc u r a c y ( % )
FedAvg
FSL (a)
[Figure]
[Figure]
T e s t
A cc u r a c y ( % )
FedAvg FSL (b [Figure] CIFAR    80 k % 72.
T e s t
A cc u r a c y ( % )
FedAvg FSL [Figure] Ramanujan et al.
(2020) also considered these three initialization to ﬁnd the best subnetwork in centralized machine learning setting.
They also showed that using Singed Kaiming Normal gives the best supermasks.
Our results align with their conclusions, hence we use Singed Kaiming Normal to initialize the weights and Kaiming Uniform to initialize the scores of global supernetwork
. C. P ERFORMANCES OF FSL WITH V ARYING S IZES OF S UBNETWORKS
In FSL, each client uses Edge-Pop Algorithm (Ramanujan et al., 2020) and their local data to ﬁnd a local subnetwork within a randomly initialized global network, which we call supernetwork .
EdgePop algorithm use parameter k which represents the % of all the edges in a supernetwork which will remain in the ﬁnal subnetwork.
For instance, k = 50% denotes that each client ﬁnds a subnetwork within a supernetwork that has half the number of edges as in the supernetwork.
Figure 4 illustrates how the performance of the global subnetwork in FSL varies with the size of subnetwork; note that, all of the clients collaborate to ﬁnd the global subnetwork.
We train nine FSL models with k ∈ { 10 , 20 , 30 , 40 , 50 , 60 , 70 , 80 , 90 } % and a FedAvg model (shown using a horizontal line); FedAvg model updates all the weights, hence it is a supermask with k = 100% . C. P ERFORMANCES OF FSL WITH DIFFERENT H ETEROGENEOUS D ATA D ISTRIBUTION M ETHODS
Table 6 shows the performances of FSL and FedAvg using different methods of non-iid assignment.
We distribute the data between 1000 clients with two methods: (I) Dirichlet distribution with β =similar to (Reddi et al., 2020; Hsu et al., 2019) and (II) each client has data of 2 random classes similar to (McMahan et al., 2017). In this table, we can see that FSL can achieve the same performance of FedAvg in different heterogeneous data distributions.
D M ISSING D ETAILS OF E DGE -P OPUP AND FSL A LGORITHM
Suppose in a fully connected neural network, there are L layers and layer (cid:96) ∈ [1 , L ] has n (cid:96) neurons, denoted by V (cid:96) =
{ V (cid:96) 1 , ..., V (cid:96)n (cid:96) } .
If I v and Z v denote the input and output for neuron v respectively, then the input of the node v is the weighted sum of all nodes in previous layer, i.e.,
I v = (cid:80) u ∈ V (cid:96) − 1 W uv Z u .
Here, W uv is the weight of the edge connecting u to v .
Edge-popup algorithm tries to ﬁnd subnetwork E , so the input for neuron v would be: I v = (cid:80) ( u,v ) ∈
E W uv Z u .
Updating scores.
Consider an edge E uv that connects two neurons u and v , W uv be the weight of
E uv , and s uv be the score assigned to the edge E uv by Edge-popup algorithm.
Then the edge-popup algorithm removes edge E uv from the supermask if its score s uv is not high enough.
Each iteration of supermask training updates the scores of all edges such that, if having an edge E uv in subnetwork reduces loss (e.g., cross-entropy loss) over training data, the score s uv increases.
[Table] The algorithm selects top k% edges (i.e., ﬁnds a subnetwork with sparsity of k%) with highest scores, so I v reduces to I v =
(cid:80) u ∈ V (cid:96) − 1 W
uv Z u h ( s uv ) where h ( . ) returns 1 if the edge exists in topk% highest score edges and 0 otherwise.
Because of existence of h ( . ) , which is not differentiable, it is impossible to compute the gradient of loss with respect to s uv .
Recall that, the edge-popup algorithm use straight-through gradient estimator (Bengio et al., 2013) to compute gradients.
In this approach, h ( . ) will be treated as the identity in the backward pass meaning that the upstream gradient (i.e., ∂L∂I v ) goes straight-through h () .
Now using chain rule, we can derive ∂L∂I v ∂I v ∂s uv =
[Equation]
u where L is the loss to minimize.
Then we can SGD with step size η to update scores as
[Equation]
Ramanujan et al.
(2020) proved that when edge ( a, b ) replaces ( c, b ) in layer (cid:96) and the rest of the subnetwork remains ﬁxed then the loss of the supermask learning decreases (provided the loss is sufﬁciently smooth).
Motivated by their proof, we can show when these two edges are swapped in
FSL, the loss decreases for FSL optimization too.
Theorem 1: when edge ( a, b ) replaces ( c, b ) in layer (cid:96) and the rest of the subnetwork remains ﬁxed then the loss of the FSL optimization will decrease (provided the loss is sufﬁciently smooth).
proof. First, we know that the optimization problem of FSL is as follow: [Equation] where λ i shows the importance of the i th client in empirical risk minimization which λ i = 1 N gives same importance to all the participating clients.
m is the ﬁnal mask that contains the edges of top ranks, and L i is the loss function for the i th client.
θ w (cid:12) m shows the subnetwork inside the random θ w that all clients unanimously vote for.
In this optimization, the FSL clients try to minimize F by ﬁnding the best global ranking R g .
We now wish to show F ( θ w , R t +1 g ) <
F ( θ w , R tg ) when in FSL round t + 1 , the edge ( a, b ) replaces ( c, b ) in layer (cid:96) and the rest of the subnetwork remains ﬁxed.
Suppose global rank of edge ( a, b ) was
R tg [( a, b )] and global rank of edge ( c, b ) was R tg
[( c, b )] in round t , so we have:
[Equation]
where the order of all remaining global ranks remain ﬁxed, and only these two edges are swapped in global ranking.
Now let s t,iab shows the score of weight w ab in round t and client i th and s t
+1 ,i ab shows the updated score of it after local training.
As in our majority vote, we are calculating the sum of the reputation of edges we will have:
[Equation]
We also know that Edge-popup algorithm updates the scores in the i th client as follow:
[Equation]
Based on 9, 10 and 11, we can say:
[Equation]
[Equation]
We also know that:
[Equation]
Based on 12, 13 and 14, we can say:
[Equation]
So based on 15, and what Ramanujan et al.
(2020) proved for each supermask training we can show 16.
We assume that loss is smooth and the input to the nodes that their edges are swapped are close before and after the swap.
[Equation]
that means:
[Equation]
E M ISSING D ETAILS A BOUT C OMMUNICATION C OST C OMPARISON
[Figure] C o mm u n i c
i c a t i o n C o s t ( K B ) [Figure]
One of the features of the FSL training is its communication efﬁciency.
In Section 5.2, we show that if the FSL clients send and receive rankings, the communication cost will be (cid:80) (cid:96) ∈
[ L ] n (cid:96) × log( n (cid:96) ) bits per client. In this section, we are providing a lower bound on the FSL communication cost, and then compare it with FedAvg and SignSGD. Lowerbound of communication cost of FSL: Since the FSL clients send and receive layer-wise rankings of indices, i.e., integers ∈ [0 , n (cid:96) − 1] , for layer (cid:96) , there are n (cid:96) ! possible permutations for layer (cid:96) ∈
[ L ] .
If we use the best possible compression method in FSL, an FSL client needs to send and receive (cid:80) (cid:96) ∈
[ L ] log( n (cid:96) !)
bits.
Therefore, the download and upload bandwidth for each FSL client would be (cid:80) (cid:96)in [ L ] log ( n (cid:96) ∗ ( n (cid:96) − 1) ∗ ... ∗ 2 ∗ 1) =
(cid:80) (cid:96) ∈ [ L ] (cid:80) n (cid:96) i =1 log ( i ) bits.
Please note that in our experiment, FSL clients send and receive the rankings without any further compression, and (cid:80) (cid:96) ∈
[ L ] (cid:80) n (cid:96) i =1 log ( i ) just shows a lower-bound of communication cost of FSL.
In Section 6.1, we measure the performance and communication cost of FSL with other existing FL compressors signSGD
(Bernstein et al., 2019) and TopK (Aji & Heaﬁeld, 2017;
Alistarh et al., 2018a).
In Figure 5, we compare the communication cost of one client per FL round for FedAvg, SignSGD, and different variant of FSL for different number of parameters.
Similar work in this domain is LotteryFL (Li et al., 2020a), a personalization framework that each
FL client learns a lottery ticket network (LTN) by pruning the base model using Lottery Ticket hypothesis (Frankle & Carbin, 2019).
In LotteryFL, each client sends and receives the update for its subnetwork, and at the end, they have an extra step for personalization.
FSL is different from
LotteryFL as the FSL clients ﬁnd subnetworks within a random and ﬁxed network and send the ranks of their subnetwork edges instead of what LotteryFL clients do that train their weights and ﬁnd a subnetwork by freezing some weights and send their actual model update.
LotteryFL is based on FedAvg that the clients can send any update to the server, which is vulnerable to the same attacks that existed for FedAVG.
In terms of communication cost, FSL is very close to LotteryFL as they report 1.81x improvement over CIFAR10 which is close to FSL and SFSL(50%) which provide 1.53x, 3.09x improvement respectively over CIFAR10. [Table] F A DDITIONAL C OMPARISONS
Figure 6 is showing the learning curve of FSL for different numbers of local epochs for CIFAR experiment.
On the x-ais we have accumulated communication cost: e × 13 .
1 × 2 × 25 MB where e is the FSL round, 13.1MB is the cost of FSL per client, 2 is for download+upload cost
[Figure]
andclients are selected in each round
[Figure]
Table 7 is showing the effect of other settings on performance of