DReS-FL: Dropout-Resilient Secure Federated Learning for Non-IID Clients via Secret Data Sharing
Anonymous Author(s)
AffiliationAddress email
Abstract
Federated learning (FL) strives to enable collaborative training of machine learningmodels without centrally collecting clients’ private data.
Different from centralizedtraining, the local datasets across clients in FL are non-independent and identicallydistributed (non-IID).
In addition, the data-owning clients may drop out of thetraining process arbitrarily.
These characteristics will significantly degrade thetraining performance.
This paper proposes a Dropout-Resilient Secure Federated
Learning (DReS-FL) framework based on Lagrange coded computing (LCC) totackle both the non-IID and dropout problems.
The key idea is to utilize Lagrangecoding to secretly share the private datasets among clients so that the effects ofnon-IID distribution and client dropouts can be compensated during local gradientcomputations.
To provide a strict privacy guarantee for local datasets and correctlydecode the gradient at the server, the gradient has to be a polynomial function ina finite field, and thus we construct polynomial integer neural networks (PINNs)to enable our framework.
Theoretical analysis shows that DReS-FL is resilient toclient dropouts and provides privacy protection for the local datasets.
Furthermore,we experimentally demonstrate that DReS-FL consistently leads to significantperformance gains over baseline methods.
Introduction
Federated learning (FL) [1] is a machine learning framework in which a central server coordinates alarge number of clients to collaboratively train a shared model.
The key idea of FL is to train themodel locally by individual clients and aggregate updates globally by the server.
The main target is toprovide privacy protection for clients’ local samples and solve the “data islands” problem.
However,it has been shown recently that local models may reveal substantial information about the localdatasets and the private training data can be reconstructed through model inversion attacks [2, 3, 4].
Besides, as local data are typically non-independent and identically distributed (non-IID), the modeldivergence during the local update may lead to unstable and slow convergence [5, 6, 7].
With manyclients involved in the training, some of the clients could drop out of the training process unexpectedly(due to poor connectivity, battery level, etc), and it will cause detrimental model performance [8].
Thus, effective mechanisms are needed to tackle the non-IID data distribution and client dropouts,while preserving the privacy of local datasets, which motivates this work.
To prevent information leakage from the local models, secure aggregation protocols [9, 10, 11, 12,13, 14, 15] have been developed to allow for global aggregation without revealing the parameters ofclients’ models.
Even if some clients may drop out, these protocols can still recover the aggregated results of the surviving clients.
However, their training performance may degrade significantly inthe non-IID setting due to the aggregation bias.
To alleviate the non-IID problem, existing methodstypically follow algorithm-based approaches [5, 16, 17, 18, 19] and add regularization terms tomitigate the model divergence.
However, these methods are not dropout-resilient evidenced by theempirical results in [20].
This can be explained by the greatly varying data distributions amongdifferent rounds.
Another fold of strategy for dealing with the non-IID problem is data-centricapproach [21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31], which generates extra training samples toconstruct a more balanced data distribution for each client.
The common practices are to share thesynthesized samples [23, 24, 25, 26] or GAN-based augmented data [27, 28, 29, 30, 31].
However,these methods may leak private information about local datasets and violate the privacy criterion in FL.
Contributions.
In this work, we develop a D ropout- Re silient S ecure F ederated L earning (DReS-FL)framework to address the above problems via Lagrange coded computing (LCC) [32].
The keyidea of LCC is to encode the datasets using Lagrange polynomials, in order to create computationalredundancy across the workers in a privacy-preserving way.
At the beginning of the training, theclients secretly share their private datasets with each other via Lagrange coding.
This allows clients toaccess an encoded version of the global dataset 1 for local gradient computations while guaranteeingprivacy in an information-theoretic sense.
After collecting the computation results from a certainnumber of surviving clients, the server performs polynomial interpolation to decode the gradient, andthus it is resilient to client dropouts.
As the local gradients are computed on the mini-batches uniformlysampled from the global dataset, the server obtains global gradient 2 after decoding.
Therefore, thetraining process in DReS-FL is made equivalent to the centralized training and eliminates the non-IIDproblem.
To provide a privacy guarantee for the local datasets and correctly decode the gradient at theserver, the gradient has to be a polynomial function in a finite field, which is a main design challengeof DreS-FL.
Our main contributions are summarized as follows:
• The proposed DReS-FL framework provides a unified approach to tackle two critical
problems of FL, namely, non-IID data distribution and client dropouts .
Meanwhile, itmaintains privacy and security guarantees such that no information about local datasets canbe leaked beyond the global model
parameters.• We construct polynomial integer neural networks (PINNs) to ensure that the gradient isa polynomial, so that cryptographic primitives can be applied for secure computation.
A
PINN consists of affine transformation layers with parameters constrained in an integer set,and it adopts the quadratic function as the activation function.
The convergence analysis of DReS-FL with PINNs is also provided.
• We conduct extensive experiments on FL benchmark datasets to demonstrate the effective-
ness of DReS-FL.
It is shown that DReS-FL outperforms baseline methods under the setting
where local datasets are heterogeneous and clients may drop out of the training processarbitrarily.
Related Works
Secure aggregation.
Secure model aggregation [9, 10, 11, 13, 12, 14, 15] is a key component of FL that protects the privacy of each client’s model while allowing their global aggregation amidstpossible user dropouts.
Existing protocols essentially rely on two main principles, including apairwise random-seed agreement for mask cancellation and secret sharing of the random seeds toconstruct the dropped masks [9, 10, 11, 12, 14, 13, 15].
However, these approaches may suffer from
[Figure]
ode
[Figure]
ha r i ng
[Figure]
severe performance degradation in non-IID FL, since the surviving clients in each round vary greatly,and thus the local gradients are biased towards different data distributions.
Non-IID data and client dropouts.
Training with heterogeneous data is a unique challenge for FL[1], which significantly affects the convergence performance [8].
The client dropouts exacerbatethe non-IID problem as the data distributions among different rounds could vary greatly.
Manyalgorithm-based methods [5, 16, 17, 18, 19] attempt to mitigate the clients’ model divergence, butthese methods cannot solve the essence of the non-IID problem due to the intrinsic difference betweenminimizing the local empirical loss and minimizing the global empirical loss.
Another line ofwork adopts data-centric methods [27, 28, 29, 30, 31] to modify the local distributions.
Ideally, aperfect data sharing mechanism should achieve that the local datasets have the same distributionas the global dataset while maintaining the privacy guarantee.
Common practices include sharingraw datasets [21, 22], synthesized samples, [23, 24, 25, 26] or augmented data [27, 28, 29, 30, 31].
However, these works cannot fully preserve local data privacy in an information-theoretic sense[33].
A special data-centric method is the secret coding scheme, which has been widely utilizedin homomorphic encryption (HE) [34, 35, 36, 37, 38, 39, 40] and multiparty computation (MPC)techniques [41, 42, 43].
This coding scheme allows computations to be performed on encrypteddata and has been used for privacy-preserving machine learning [36, 40, 43].
However, the HEmethods often suffer from time-consuming cryptographic tools, and MPC techniques are difficult togeneralize such primitives to a large number of clients.
Recently, distributed secure machine learningframeworks [44, 45] have been proposed for logistic regression problems.
They apply Lagrangecoding for secret data sharing and approximate the Sigmoid function by a polynomial function.
Thispaper proposes DReS-FL to further extend these works to train deep neural networks in the FLsetting.
System Model
We consider a federated learning framework as shown in Fig.
1 that consists of one central serverand N data-owning clients.
Each client i ∈
[ N ] holds a local dataset ( X ( i ) , Y ( i ) ) of size
m i , where
X ( i ) ∈
R m i ×
d x represents the set of input features of dimension d x and Y ( i ) ∈
R m i × d y correspondsto the output vector of dimension d y .
Accordingly, the size of the global dataset ( X , Y ) whichconcatenates all local datasets ( X ( i ) , Y ( i ) ) , ∀ i ∈
[ N ] is denoted as m ≜ (cid:80) [-Ni =
-]N i =
1 m i .
The clients aimto jointly train a neural network based on their local datasets without sharing private data samples.
Particularly, the gradients are computed locally and aggregated globally.
However, the local data maybe highly heterogeneous, and the clients may drop out at any time unexpectedly, which makes thetraining process unstable.
Our goal is to improve the convergence performance by secret data sharingwhile preserving the privacy of local datasets.
Threat model and privacy requirements.
We consider a threat model where the clients are honestbut-curious .
In particular, clients follow the protocol honestly but may collude amongst themselves tolearn additional information.
Besides, we assume that the server is also honest-but-curious, but doesnot collude with any other clients.
To avoid leaking private information from the shared samples andprotect the local updates against model inversion attacks [2, 3, 4], we impose two privacy requirementsin the training process.
First, the clients learn nothing about the private datasets of others from theshared samples even if up to T clients collude.
Second, the server learns no information about theprivate datasets of clients from a single local computation result beyond the model parameters.
Lagrange coded computing.
The LCC framework considers a scenario involving computations overmassive datasets stored distributedly across multiple clients [32].
The key idea is to encode the datausing Lagrange polynomial for redundant distributed computing, which fits nicely with federatedlearning due to its dropout-resiliency and privacy guarantees.
DReS-FL applies LCC to secretly sharethe private datasets among clients for local gradient computations, and the global gradient can bedecoded by the server for model updating.
To provide a strong privacy guarantee for the datasets andcorrectly decode the gradient at the server, the gradient should be a polynomial function in a finitefield.
However, existing neural networks cannot satisfy this requirement, since the datasets are in thereal field and the gradients are not polynomial functions due to the non-linear operations.
Polynomial Integer Neural Networks.
We define a class of polynomial integer neural networks(PINNs) to ensure that the gradient is a polynomial function in a finite field F p with a prime numberp .
First, we transform the dataset ( X , Y ) from the real domain to the finite domain ( X , Y ) .
Besides,a PINN consists of affine transformation layers (e.g., fully connected layers and convolutional layers)and utilizes the quadratic function as the activation function.
The model parameters of PINNs aredefined in the integer set Z p ≜ {−⌊ p +12 ⌋ , . . .
, ⌊ p − 12 ⌋} .
Given a feed-forward function f
( X ; w ) andselecting the mean squared error (MSE) as the loss function, the gradient of the input samples is amultivariate polynomial with integer coefficients, i.e., g ( X , Y ; w ) ≜ ∇ w ∥ Y − f
( X ; w ) ∥ 22 ∈ Z d w ,where d w represents the number of model parameters.
In particular, to avoid wrap-around whencomputing gradient in the finite field F p , we assume the prime number p is sufficiently large withoutleading to overflow errors in the integer set Z p .
The Proposed DReS-FL Framework
DReS-FL consists of two main phases, as shown in Fig.
In the first phase, the private datasets aretransformed from the real domain to the finite field, and data-owning clients secretly share datasetsby Lagrange coding.
Then, the server and the clients train a PINN iteratively via (1) local gradientcomputations and (2) gradient decoding and model updating.
4.1 Data Transformation and Secret Sharing
To guarantee information-theoretic privacy, each client has to mask the datasets in a finite field F pusing uniformly random matrices.
Firstly, the local datasets ( X ( i ) , Y ( i ) ) are converted from thereal domain to the finite field ( X ( i ) , Y ( i ) ) .
Considering an element-wise function ϕ
( z ) =
z + cthat transforms a real value to a non-negative number by adding a proper scalar c 3 , we define
X ≜ Round (2 l · ϕ ( X )) , where the rounding operation is element-wise that quantizes each entry toits closest integer, and l ∈ Z controls the quantization loss.
We adopt the notation D to represent theglobal dataset , which is the concatenation of all the local datasets ( X ( i ) , Y ( i ) ) for i ∈ [ N ] .
After converting the private datasets to the finite field, clients adopt T -private Lagrange coding [32]to secretly share their local data samples across other clients, where T ≥ 1 is the privacy parameterin our system ensuring that the encoded datasets do not leak any information about the originaldatasets even if T clients collude.
First, each client i ∈
[ N ] partitions its local dataset to K shardsas X ( i ) ≜
[ X ( i ) T, . . .
, X ( i ) T K ] T and Y ( i ) ≜
[ Y ( i ) T, . . .
, Y ( i ) T K ] T .
Assuming that m i is divisible [Table] [Table] [Table]
Y ( I t ) j ; w ( t ) ) to the server.
7: end for 8: if Server receives at least deg( g )( K + T − 1) + 1 uploads then 9:
Decode K stochastic gradients (cid:101) g
( D ( I t ) k ; w ( t ) ) for k ∈ [ K ] by polynomial interpolation. Convert gradients from the finite field to the integral domain g ( D ( I t ) k ; w ( t ) ) by (5). Update the global model by w ( t +1)
=
w ( t ) − Q ( ηbK (cid:80) Kj =1 g ( D ( I t ) k ; w ( t ) )) based on (6). 12: end if 13: end for by K , we have X ( i ) k ∈ F miK × d x p and Y ( i ) k ∈ F miK × d y p for k ∈ [ K ] . A large value of K helps toreduce the communication overhead in secret data sharing and computation costs in local gradientcomputations 4 .
Then, the clients add padding from T uniform random masks to the data samplesfor privacy protection.
Each client i ∈
[ N ] forms the following polynomials
u i : F p →
F miK × d x p andv i : F p → F miK × d y p of degree K + T − 1 to encode the local dataset:u i
[Equation]
U [Equation] (1) v i ( z ) ≜ (cid:88) k ∈ [ K ]
Y (
[Equation]
(cid:89) j ∈
[Equation]
K
[Equation]
β j β k −
[Equation]
(cid:88)
[Equation] (cid:89) j ∈ [Equation] (2) where { U ( i ) k } ’s and { V ( i ) k } ’s are random noise matrices uniformly sampled from F mK × d x p and F mK × d y p ,respectively.
These matrices mask the local datasets and provide a privacy guarantee against up to Tcolluding workers.
The clients and the server agree on K + T distinct elements { β 1 , . .
, β K + T }from the finite field F p in advance.
Particularly, setting z = β k for k ∈ [ K ] , we obtain u
i
β k )
=
X ( i ) kand v i ( β k ) =
Y ( i ) k .
All the clients use the same N distinct elements { α 1 , . .
. , α N } selected from F p to encode the private datasets, where { α j } j ∈ [ N ] ∩ { β k } k ∈ [ K + T ] = ∅ .
Each client i obtains Nencoded datasets ( (cid:101) X ( i ) j , (cid:101) Y ( i ) j ) ≜ ( u i ( α j ) , v i ( α j )) for j ∈ [ N ] , where ( (cid:101) X ( i ) j , (cid:101) Y ( i ) j ) is sent to clientj from client i .
All the received encoded datasets at client j are represented as ( (cid:101) X j , (cid:101) Y j ) , where(cid:101)
X j ≜ [ (cid:101) X (1) T j , . .
, (cid:101) X ( N ) T j ] T ∈ F (cid:101) m × d x p and (cid:101)
Y j ≜ [ (cid:101) Y (1) T j , . .
, (cid:101) Y ( N ) T j ] T ∈ F (cid:101) m × d y p for j ∈ [ N ]
[-Accordingly, the number of samplesin the encoded dataset is (cid:101) m ≜ 1 K (cid:80) ni =
-]
4.2 Federated Training
Local Gradient Computation.
The server randomly initializes a PINN at the beginning of thetraining process, and the model parameters are constrained to an integer set Z p in the trainingprocess.
In each communication round, the server sends the model parameters to the clients, andthey compute the stochastic gradient over the mini-batches with size b .
Particularly, we assumethat all the clients use the same row selection matrix C ( t ) ∈ { 0 , 1 } b × (cid:101) m for data sampling in eachround t 5 , and the mini-batch at each client j ∈ [ N ] is determined by [ (cid:101)
X
( I t )
j , (cid:101) Y ( I t )
j ] = C ( t )
[ (cid:101) X j , (cid:101)
Y j ] .
Here, I t = { l ( t ) 1 , . . .
, l ( t ) b } ⊆
[ (cid:101) m ] is a randomly selected index set in the t -th round with l i ∈
[ (cid:101) m ]for i ∈ [ b ] .
The entries of C ( t ) satisfy C ( t ) i,l i = 1 for i ∈
[ b ] , and other entries are set to zero.
Eachclient j computes the stochastic gradient (cid:101) g
( (cid:101) X
( I t )
j , (cid:101) Y ( I t )
j ; w ( t ) )
≡ g ( (cid:101) X
( I t )
j , (cid:101) Y ( I t )
j ; w ( t ) )
mod pin the finite field, and uploads the result to the server.
Particularly, each (cid:101) g ( (cid:101) X
( I t )
j , (cid:101) Y ( I t )
j ; w ( t ) )amounts to an evaluation of the polynomial (cid:101) g ( u I t ( z ) , v I t ( z ); w ( t ) ) at the point z
= α j , where two( K + T − 1) -degree polynomial functions u
I t : F p → F b × d x p and v I t : F p → F b × d y p are defined asfollows:
[Equation]
with
X ( I t ) k
[Equation]
Every ( X ( I t ) k , Y ( I t ) k ) =
( u I t ( β k ) , v I t ( β k )) for k ∈ [ K ] is a global mini-batch selected from theglobal dataset D .
For notational simplicity, we define D ( I t ) k
[Equation]
( I t ) k , Y ( I t ) k ) .
Gradient Decoding and Model Updating.
After receiving any deg ( g )( K + T − 1) + 1 local computation results 6 , the server decodes the stochastic gradients (cid:101) g ( D ( I t ) k ; w ( t ) ) ≜ (cid:101) g ( X ( I t ) k , Y ( I t ) k ; w ( t ) )for k ∈
[ K ] by interpolating the polynomial (cid:101) g
( u I t ( z ) , v I t ( z ); w ( t ) ) .
As the polynomial function(cid:101) g ( u I t
( z ) , v I t ( z ); w ( t ) ) is a composition of the encoding polynomials and the gradient function, itsdegree is deg ( g )( K + T − 1) .
Therefore, the server needs at least deg ( g )( K + T − 1)+1 evaluationsto interpolate it. After recovering the coefficients of the polynomial (cid:101) g
( u I t ( z ) , v I t ( z ); w ( t ) ) , theserver decodes K stochastic gradients (cid:101) g
( D ( I t ) k ; w ( t ) )
= (cid:101) g ( u I t ( β k ) , v I t ( β k ); w ( t ) ) on the globalmini-batches D ( I t ) k by letting z = β k for k ∈
[ K ] .
Then, the server converts the gradient from the finite field to the integer set Z p by g ( D ( I t ) k ; w ( t ) )
=
ψ ( (cid:101) g
( D ( I t ) k ; w ( t ) )) , where ψ ( z ) is an element-wisefunction defined as follows:
[Equation]
As we assume that the prime number p is sufficiently large, the converted gradients do nothave overflow errors.
Thus, the central sever updates the global model by w ( t +1) =
w ( t ) −
Q
[Equation]
; w ( t ) )) , where η denotes the learning rate and bK represents the global batch size 7 .
Q ( z ) is a stochastic quantization function to ensure the model parameters are in the integer set 203 Z p after updating, which is defined as follows:
[Equation]
Besides, the probability of rounding z to ⌊ z ⌋ is proportional to the proximity of z to ⌊ z ⌋ so that thestochastic rounding is unbiased.
The overall procedure is summarized in Algorithm 1.
Theoretical Analysis
5.1 Privacy and Security Guarantees
Before training starts, client j receives an encoded version of the global dataset ( (cid:101)
X j , (cid:101) Y j ) from otherclients.
Lagrange coding in DReS-FL provides a strong privacy guarantee that the clients cannot inferanything about the private datasets based on the received datasets even if up to T clients collude.
Thefollowing theorem shows that our DReS-FL satisfies the first privacy requirement in Section 3, andthe proof is available in Section IV of [32].
Theorem 1.
( T -private coding scheme [32])
Employing Lagrange coding with the privacy parameter
T in DReS-FL, we have that for every subset of clients T ⊆
[ N ] of size at most T , the mutualinformation I ( D ; { (cid:101) X j , (cid:101) Y j } j ∈T ) = 0 .
Besides, DReS-FL can provide a security guarantee such that the server learns no information fromthe local gradients beyond the global model w .
This property corresponds to the second privacyrequirement in Section 3.
The following theorem provides a rigorous statement and its proof isdeferred to Appendix B.1.
Theorem 2.
For any i, j ∈ [ N ] and index set I t , the conditional mutual information
I ( X ( i ) , Y ( i ) ; (cid:101) g ( (cid:101) X
( I t ) j , (cid:101) Y ( I t ) j ; w ( t ) ) | w ( t ) ) equals to zero.
5.2 Dropout-resiliency and Convergence
In the FL setting, it is common for clients to drop out at any time during protocol execution, whichleads to model divergence especially when the clients’ datasets are highly heterogeneous.
Thefollowing theorem shows that DReS-FL is resilient to a certain number of client dropouts.
Theorem 3.
(Dropout-resiliency) Consider N clients in the federated learning system thatuse a T -private coding scheme to secretly share the local samples [ X ( i ) T, . .
, X ( i ) T K ] T and[ Y ( i ) T, . . .
, Y ( i ) T K ] T for i ∈
[ N ] for local gradient computations.
DReS-FL guarantees that theserver can decode the global gradient when there are no more than D = N − deg( g )
( K + T − 1) − 1client dropouts.
Proof.
Theorem 1 of [32] shows that given a number of N computing nodes and a K -shard dataset,the LCC framework provides a T -private coding scheme for computing any polynomial g , as long asdeg( g )
( K + T − 1)+1 ≤ N .
Thus, it can tolerate at most N − deg( g )(
K + T − 1) − 1 dropouts.
Remark 1.
There is a tradeoff among privacy guarantee ( T ), gradient computation cost ( 1 /K ),and dropout-resilience ( D ).
Parameter T reflects the privacy threshold of Lagrange coding, andparameter K accounts for the computation load reduction.
In particular, the local batch size ofeach client is 1 /K of the global batch size.
DReS-FL can achieve any T , K , and D as long as
D deg( g )( K + T − 1) + 1 ≤ N .
As T and K increase, DReS-FL can tolerate fewer client dropouts.
Remark 2.
Our DReS-FL framework can be extended to more general cases in which clients canrun s ( s ≥ 1 ) local SGD steps each round.
More discussion has been deferred to Appendix D.
[Table]
Next, we characterize the convergence performance of PINNs, which relies on the fact that the globalgradients in the training process are unbiased.
Define the empirical risk as ℓ ( w ) ≜
E ( X , Y ) ∼D ∥ Y −f ( X ; w ) ∥ 22 and the corresponding gradient as g e ( w ) ≜ E ( X , Y ) ∼D [ g ( X , Y ; w )] .
The variables( X , Y ) are drawn from the distribution of the global dataset D .
To prove that DReS-FL guaranteesconvergence to the optimal model parameters, we first present the following amputations to facilitatethe analysis.
Assumption 1.
( L -smoothness)
There exists a constant L > 0 such that for all w 1 , w 2 ∈ Z d w p , wehave [Equation]
Assumption 2.
(Unbiased and variance-bounded stochastic gradient)
There exists a constantσ > 0 such that any stochastic gradient g
( D k ; w ( t ) ) satisfies E (cid:2) 1 b g ( D k ; w ( t ) ) (cid:3) =
g e ( w ( t ) ) and
E (cid:2) ∥
1 b g ( D k ; w ( t ) )
− g e ( w ( t ) )
∥ 2 (cid:3) ≤
[Equation]
Assumption 3.
(Unbiased and variance-bounded rounding operation)
There exists a constant γ > 0such that for any z ∈ R , the stochastic quantization operation Q ( · ) satisfies E [ Q ( z )]
=
z and
E (cid:2)
[Equation] Q [Equation]
With the above preparations, we have the following theorem which ensures the convergence.
Theproof is deferred to Appendix B.2.
Theorem 4.
(Convergence)
Denote w ∗ as the first-order optimal solution.
With Assumption 1-3,selecting the learning rate as η = O (cid:16) 1 / √ τ ′ (cid:17) such that Ψ ≜ 1 − ηL/ 2 − ηγ 2 L/ 2 > 0 , after τ ′times of model updates, we have:
[Equation]
Experiments
6.1 Experimental Setup
Dataset.
We evaluate our proposed algorithm on several benchmark datasets: MNIST [46], FashionMNIST [47], EMNIST (Balanced) [48], CIFAR-10 [49], CIFAR-100 [49], and SVHN [50].
Tosimulate the non-IID data distribution, we assume there are N = 20 clients in the learning systemand adopt the skewed label partition [51] to shuffle the datasets.
Specifically, we sort a dataset by thelabels, divide it into N shards, and assign one shard to each client.
To simulate the client dropouts inthe training process, we consider an extreme scenario, where the dropout rate of each client is set to0.99 with a probability of 0.5 or is uniformly sampled from [0 , 0 . 1] otherwise.
Model structures.
We adopt a multi-layer perception (MLP) with two hidden layers for the image classification tasks on MNIST, Fashion-MNIST, and EMNIST datasets.
Besides, we utilizean ImageNet pretrained VGG19 model [52, 53] for CIFAR-10, CIFAR-100, and SVHN datasets.
Specifically, the parameters in convolutional layers of VGG19 are fixed, and we utilize an MLP withtwo hidden layers as a classifier.
The baseline methods train the neural networks on the real field andselect the rectified linear unit (ReLU) function as the activation function.
In each communicationround, clients perform one SGD step for the local model update.
DReS-FL.
Our method adopts the same size PINNs to replace MLPs in the federated training, andthe degree of gradient is deg( g )
= 8 .
Particularly, the extracted features from the last convolutionallayer of VGG19 are secretly shared with other clients.
We set the parameters K = 1 and T = 1 inthe Lagrange coding, and the minimum number of clients needed to decode the global gradient is 9.
6 Round 1e
A cc u r a c y ( % )
FedAvg FedAvg-IS DReS-FL (Ours)
(a) MNIST     6 Round 1e
A cc u r a c y ( % )
FedAvg FedAvg-IS DReS-FL (Ours)
(b) Fashion-MNIST     6 Round 1e
A cc u r a c y ( % )
FedAvg FedAvg-IS DReS-FL (Ours) (c) EMNIST     6 Round 1e
A cc u r a c y ( % )
FedAvg FedAvg-IS DReS-FL (Ours)
(d) CIFAR6 Round 1e
A cc u r a c y ( % )
FedAvg FedAvg-IS DReS-FL (Ours)
(e) CIFAR6 Round 1e
A cc u r a c y ( % )
FedAvg FedAvg-IS DReS-FL (Ours) (f) SVHN
Baselines.
We select algorithm-based methods as baselines, including FedAvg [1], FedAvg withimportance sampling (FedAvg-IS) [54, 55], and SCAFFOLD [5].
Specifically, we assume that the
FedAvg-IS method knows the dropout distribution, and the local updates are weighted by the dropoutprobabilities to mitigate bias.
Besides, we adopt the centralized training scheme as a performanceupper bound in the comparison. More details of the experimental settings are deferred to Appendix A. 6.2 Performance Evaluation
1.5 Round 1e
A cc u r a c y ( % )
SCAFFOLD
We compare the performance of our DReS-FL method with baselines.
The experimental results are shown in Table 2 and Fig.
The FedAvg method achieves worse performance than the centralized training scheme. This is attributed to the non-IID data andclient dropouts.
The FedAvg-IS method improves the test accuracycompared with FedAvg, but there is still a noticeable performancegap with the centralized training scheme.
It shows that using theknowledge of dropout distribution can partially compensate for thebiases in the aggregated models, but the local data distributions arestill heterogeneous and degrade the performance.
Besides, SCAFFOLD has a low accuracy on all the settings.
As the frequency ofupdating local control variates is low, the estimation of the update direction is highly inaccurate suchthat the model does not converge as shown in Fig.
These results are consistent with the findings in[20].
Our DReS-FL method is superior to all the baseline methods as the server can obtain globalgradients after polynomial interpolation.
In addition, DReS-FL achieves comparable performanceto the centralized training scheme on some datasets, which demonstrates the effectiveness of ourproposed framework in solving the non-IID and dropout problems.
Conclusions
This paper proposed a Dropout-Resilient Secure Federated Learning (DReS-FL) framework via
Lagrange coded computing (LCC) to simultaneously solve the data heterogeneity and dropoutproblems of FL, while providing privacy guarantees for the local datasets.
The polynomial integerneural networks (PINNs) have been constructed to ensure that the server can correctly decode theglobal gradient without privacy leakage.
Extensive experimental results validated the effectiveness ofthe proposed method.
Potential limitations of our method have been deferred to Appendix E.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Appendix E.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] The proofs areprovided in Section 5 and Appendix B. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[No] The code isproprietary, but the public datasets used in the experiments and the instructions needed
for reproduction are presented in Section 6 and
Appendix A.
(b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
[Yes] Please refer to Section 6 and Appendix A.
(c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?
[Yes] See Section 6.
(d) Did you include the total amount of compute and the type of resources used (e g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix A.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] See Appendix A.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
The code is proprietary and the datasets used in our work are public datasets.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A] We choose the commonly-used datasets in the experiments.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]