DReS-FL: Dropout-Resilient Secure Federated Learning for Non-IID Clients via Secret Data Sharing
Abstract
Federated learning (FL) strives to enable privacy-preserving training of machine learning models without centrally collecting clients’ private data.
Despite its advantages, the local datasets across clients in FL are non-independent and identically distributed (non-IID), and the data-owning clients may drop out of the training process arbitrarily.
These characteristics will significantly degrade the training performance.
Therefore, we propose a Dropout-Resilient Secure Federated Learning (DReS-FL) framework based on Lagrange coded computing (LCC) to tackle both the non-IID and dropout problems.
The key idea is to utilize Lagrange coding to secretly share the private datasets among clients so that each client receives an encoded version of the global dataset 1 , and the local gradient computation over this dataset is unbiased.
To correctly decode the gradient at the server, the gradient function has to be a polynomial in a finite field, and thus we construct polynomial integer neural networks (PINNs) to enable our framework.
Theoretical analysis shows that DReS-FL is resilient to client dropouts and provides strong privacy guarantees.
Furthermore, we experimentally demonstrate that DReS-FL consistently leads to significant performance gains over baseline methods.
Introduction
Federated learning (FL) [1] is a machine learning framework in which a central server coordinates a large number of clients to collaboratively train a shared model.
The key idea of FL is to train the model locally by individual clients and aggregate updates globally by the server.
The main target is to provide privacy protection for clients’ local samples and solve the “data islands” problem.
However, as local data are typically non-independent and identically distributed (non-IID), the model divergence during the local update may lead to unstable and slow convergence [2, 3, 4].
With many clients involved in the training, some of the clients could drop out of the training process unexpectedly (due to poor connectivity, battery level, etc), and it will cause detrimental model performance [5].
Thus, effective mechanisms are needed to tackle the non-IID data distribution and client dropouts, while preserving the privacy of local datasets, which motivates this work.
To alleviate the non-IID problem, existing methods typically follow algorithm-based approaches [2, 6, 7, 8, 9] and add regularization terms to mitigate the model divergence.
However, these methods are not dropout-resilient evidenced by the empirical results in [10].
This can be explained by the greatly varying data distributions among different rounds.
Another fold of strategy for dealing with the non-IID problem is data-centric approach [11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21], which generates extra training samples to construct a more balanced data distribution for each client.
The common practices are to share the synthesized samples [13, 14, 15, 16] or GAN-based augmented data [17, 18, 19, 20, 21].
However, these methods may leak private information about local datasets and violate the privacy criterion in FL.
In this work, we develop a D ropout- Re silient S ecure F ederated L earning (DReS-FL) framework to address the above problems via Lagrange coded computing (LCC)
The key idea of LCC is to encode the datasets using Lagrange polynomials that create computational redundancy across the workers in a privacy-preserving way to tolerate client dropouts.
Before the training starts, the clients secretly share their encoded datasets with each other.
This allows clients to access an encoded version of the global dataset that solves the data heterogeneous problem.
In each communication round of federated training, the clients perform local gradient computations on the mini-batches sampled from the encoded datasets. After collecting the uploaded computation results from surviving clients, the server performs polynomial interpolation to decode the global gradient 2 for model training.
Therefore, the training process in DReS-FL is made equivalent to centralized training and eliminates the non-IID and dropout problems.
With respect to privacy protection, the proposed framework has two salient features:
• It guarantees the privacy of local datasets during data sharing,
i
e., no private information can be inferred from the encoded data even if a certain number of clients collude.
• It achieves the same privacy guarantee as the secure aggregation protocols,
i e., the server learns no information about the private dataset from a single client’s computation result. Note that to correctly decode the gradient at the server, the gradient has to be a polynomial function in a finite field, which is a main design challenge of DreS-FL.
To sum up, our main contributions are summarized as follows:
• The proposed DReS-FL framework provides a unified approach to tackle two critical problems of FL, namely, non-IID data distribution and client dropouts .
Meanwhile, it maintains privacy and security guarantees such that no information about local datasets can be leaked beyond the global model parameters.
• We construct polynomial integer neural networks (PINNs) to ensure that the gradient is a polynomial, so that cryptographic primitives can be applied for secure computation.
A PINN consists of affine transformation layers with parameters constrained in an integer set, and it adopts the quadratic function as the activation function.
The convergence analysis of DReS-FL with PINNs is also provided.
• We conduct extensive experiments on FL benchmark datasets to demonstrate the effectiveness of DReS-FL.
It is shown that DReS-FL outperforms baseline methods under the setting where local datasets are heterogeneous and clients may drop out of the training process arbitrarily.
Related Works
Non-IID data and client dropouts.
Training with heterogeneous data is a unique challenge for FL [1], which significantly affects the convergence performance [5].
The client dropouts exacerbate the non-IID problem as the data distributions among different rounds could vary greatly.
Many algorithm-based methods [2, 6, 7, 8, 9] attempt to mitigate the clients’ model divergence, but these methods cannot solve the essence of the non-IID problem due to the intrinsic difference between minimizing the local empirical loss and minimizing the global empirical loss.
Another line of work adopts data-centric methods [17, 18, 19, 20, 21] to modify the local distributions.
Ideally, a perfect data sharing mechanism should achieve that the local datasets have the same distribution as the global dataset while maintaining the privacy guarantee.
Common practices include sharing raw datasets [11, 12], synthesized samples, [13, 14, 15, 16] or augmented data [17, 18, 19, 20, 21].
However, these works cannot fully preserve local data privacy in an information-theoretic sense [23].
A special data-centric method is the secret coding scheme, which has been widely utilized [Figure] M ode l S ha r i ng
[Figure]
in homomorphic encryption (HE) [24, 25, 26, 27, 28, 29, 30] and multiparty computation (MPC) techniques [31, 32, 33].
This coding scheme allows computations to be performed on encrypted data and has been used for privacy-preserving machine learning [26, 30, 33].
However, the HE methods often suffer from time-consuming cryptographic tools, and MPC techniques are difficult to generalize such primitives to a large number of clients.
Recently, distributed secure machine learning frameworks [34, 35] have been proposed for logistic regression problems.
They apply Lagrange coding for secret data sharing and approximate the Sigmoid function by a polynomial function.
This paper proposes DReS-FL to further extend these works to train deep neural networks in the FL setting. Secure aggregation.
It has been shown recently that the clients’ updates in FL may reveal substantial information about the local datasets, and the private training data can be reconstructed through model inversion attacks [36, 37, 38].
To prevent information leakage from the local models, secure aggregation protocols [39, 40, 41, 42] have been developed to allow for global aggregation without revealing the parameters of clients’ models. Even if some clients may drop out, these protocols can still recover the aggregated results of the surviving clients. Existing protocols essentially rely on two main principles, including a pairwise random-seed agreement for mask cancellation and secret sharing of the random seeds to construct the dropped masks [39, 43, 40, 41, 44, 42]. However, these approaches may suffer from severe performance degradation in non-IID settings, since the surviving clients in each round vary greatly, and thus the aggregate gradient is biased towards different data distributions.
Different from previous works, our proposed DReS-FL framework achieves the same privacy guarantee while solving the data heterogeneity problem.
System Model
We consider a federated learning framework as shown in Fig.
1 that consists of one central server and N data-owning clients.
Each client i ∈
[ N ] holds a local dataset ( X i , Y i ) of size
m i , where
X i ∈
R m i ×
d x represents the set of input features of dimension d x and Y i ∈
R m i × d y corresponds to the output vector of dimension d y .
Accordingly, the size of the global dataset ( X , Y ) which concatenates all local datasets ( X i , Y i ) , ∀ i ∈
[ N ] is denoted as m ≜ (cid:80) N i =
1 m i .
The clients aim to jointly train a neural network based on their local datasets without sharing private data samples.
Particularly, the gradients are computed locally and aggregated globally.
However, the local data may be highly heterogeneous, and the clients may drop out at any time unexpectedly, which makes the training process unstable.
Our goal is to improve the convergence performance by secret data sharing while preserving the privacy of local datasets.
3.1 Lagrange Coded Computing for Federated Learning
The Lagrange coded computing (LCC) framework enables private computing in distributed settings to provide resiliency and efficiency [22].
The key idea is using Lagrange coding to encode the data for redundant distributed computing, which fits nicely with federated learning due to its dropout-resiliency and privacy requirement.
Specifically, the clients share their encoded datasets with each other and perform gradient computation over the encoded samples. The server decodes the aggregate gradient after receiving the uploaded computation results from clients.
To provide a strong privacy guarantee for the datasets and correctly decode the gradient at the server, the gradient function should be a polynomial function in a finite field.
However, existing neural networks cannot satisfy this requirement, since the datasets are in the real field and the gradients are non-polynomial.
Polynomial integer neural networks.
We define a class of polynomial integer neural networks (PINNs) to ensure that the gradient is a polynomial function in a finite field F p with a prime number p .
First, we transform the dataset ( X , Y ) from the real domain to the finite domain ( X , Y ) .
Besides, a PINN consists of affine transformation layers (e.g., fully connected layers and convolutional layers) and utilizes the quadratic function as the activation function.
The model parameters of PINNs are defined in the integer set Z p ≜ {−⌊ p +1 2 ⌋ , . . .
, ⌊ p − 1 2 ⌋} .
Given a feed-forward function f
( X ; w ) and selecting the mean squared error (MSE) as the loss function, the gradient of the input samples is a multivariate polynomial with integer coefficients, i.e., g ( X , Y ; w ) ≜ ∇ w ∥ Y − f
( X ; w ) ∥ 22 ∈ Z d w , where d w represents the number of model parameters.
Denoting the number of quadratic activation layers as L , the degree of the gradient function 3 g ( X , Y ; w ) is deg( g ) =
2 L +1 . In particular, to avoid wrap-around when computing gradient in the finite field F p , we assume the prime number p is sufficiently large without leading to overflow errors in the integer set Z p . Lagrange coding.
The proposed DReS-FL framework uses Lagrange polynomials to achieve a D -resilient, T -private, and K -efficient coding scheme. D -Resiliency means that the global gradient can be decoded by the server in the presence of up to D client dropouts. T -privacy denotes that no information about local datasets can be inferred from the encoded data even if up to T clients collude.
K -efficiency corresponds to the complexity of the coding scheme.
Specifically, each private dataset is split into K shards in Lagrange coding, and the size of the encoded dataset is proportional to 1 /K .
Therefore, increasing the value of K reduces the communication overhead of data sharing.
The following theorem characterizes the ( D, T, K ) -achievable coding scheme, and its proof is available in Section IV of [22].
Theorem 1.
Given the client number N and the degree of the gradient function deg( g ) , a D -resilient,
T -private, and K -efficient Lagrange coding scheme is achievable, as long as
[Equation]
Remark 1.
As shown in Theorem 1, there is a tradeoff among resiliency ( D ), privacy ( T ), and efficiency ( K ).
As the sum of T and K increases, the proposed framework tolerates fewer client dropouts.
Specifically, the maximum value of D is N − 1 − deg( g ) by setting T =
K = 1 .
Remark 2.
Setting the privacy parameter T ≥ 1 , the gradient computation over the encoded samples leaks no private information according to the data process inequality.
This implies that the proposed DReS-FL framework achieves the same privacy guarantee as the secure aggregation protocols.
Specifically, the server learns no information about the private dataset from a single client’s computation result.
DReS-FL consists of two main phases, as shown in Fig.
In the first phase, the private datasets are transformed from the real domain to the finite field, and data-owning clients secretly share datasets by Lagrange coding.
Then, the server and the clients train a PINN iteratively via (1) local computations and ( decoding and model updating [Figure]
4.1 Data Transformation and Secret Sharing
To guarantee information-theoretic privacy, each client has to mask the datasets in a finite field F p using uniformly random matrices.
Firstly, the local datasets ( X i , Y i ) are converted from the real domain to the finite field ( X i , Y i ) .
Considering an element-wise function ϕ
( z ) =
z + c that transforms a real value to a non-negative number by adding a proper scalar c 4 , we define
X ≜ Round (2 l · ϕ ( X )) , where the rounding operation is element-wise that quantizes each entry to its closest integer, and l ∈ Z controls the quantization loss.
We adopt the notation D to represent the global dataset , which is the concatenation of all the local datasets ( X i , Y i ) for i ∈ [ N ] .
After converting the private datasets to the finite field, the clients adopt a ( D, T, K ) -achievable Lagrange coding to encode local data for secret sharing.
First, each client i ∈
[ N ] partitions its local dataset to K shards as X i ≜ [ X (1) T i , . . .
, X ( K ) T i ] T and Y i ≜
[ Y (1) T i , . .
, Y ( K ) T i ] T .
Assuming that m i is divisible by K , we have X ( k )
i ∈ F miK × d x p and Y ( k ) i ∈
F miK × d y p for k ∈ [ K ] .
A large value of K helps to reduce the complexity in secret data sharing.
Then, the clients add padding from T uniform random masks to the data samples for privacy protection.
Each client i ∈
[ N ] forms the following polynomials
u i : F p →
F miK × d x p and v i : F p → F miK × d y p of degree K + T − 1 to encode the local dataset:
[Equation]
where { U ( k ) i } ’s and { V ( k ) i } ’s are random noise matrices uniformly sampled from F mK × d x p and
F mK × d y p , respectively.
These matrices mask the local datasets and provide a privacy guarantee against up to T colluding workers.
The clients and the server agree on K + T distinct elements { β 1 , . .
, β K + T } from the finite field F p in advance.
Particularly, setting z = β k for k ∈ [ K ] , we reconstruct the data shard ( u i ( β k ) , v i ( β k )) =
( X ( k ) i , Y ( k ) i ) .
All the clients use the same N distinct elements { α 1 , . .
. , α N } selected from F p to encode the private datasets, where { α j } j ∈ [ N ] ∩ { β k } k ∈ [ K + T ] = ∅ .
Each client i obtains N encoded datasets ( (cid:101) X i → j , (cid:101) Y i
→ j ) ≜ ( u i ( α j ) , v i ( α j )) for j ∈ [ N ] , where each ( (cid:101) X i → j , (cid:101) Y i → j ) is sent to client j from client i .
All the received encoded [Table] , (cid:101) Y ( I t ) j ) . Upload local computation results (cid:101) g
( (cid:101) X ( I t ) j , (cid:101) Y ( I t ) j ; w ( t ) ) to the server.
7: end for 8: if Server receives at least deg( g )( K + T − 1) + 1 uploads then 9: Decode K global gradients (cid:101) g ( X ( k ) I t , Y ( k ) I t ; w ( t ) ) for k ∈ [ K ] by polynomial interpolation. Convert gradients from the finite field to the integral domain g ( X ( k )
I t , Y ( k ) I t ; w ( t ) ) by (6). Update the model by (7) based on the aggregate gradient (cid:80) Kk =1 g ( X ( k ) I t , Y ( k )
I t ; w ( t ) ) .
12: end if 13: end for datasets at client j are represented as ( (cid:101) X j , (cid:101) Y j ) , where (cid:101) X j ≜
[ (cid:101) X T 1 → j , . . . , (cid:101) X TN → j ] T ∈ F (cid:101) m × d x p and (cid:101)
Y j ≜ [ (cid:101) Y T 1 → j , . . .
, (cid:101) Y TN → j ] T ∈ F (cid:101) m × d y p for j ∈ [ N ] {+. Accordingly, the number of samples in the encoded dataset is (cid:101) m ≜ 1 K (cid:80) ni =
 m i . Fig.+}
2 demonstrates the secret data sharing scheme.
4.2 Federated Training
Local Gradient Computation.
The server randomly initializes a PINN at the beginning of the training process, and the model parameters are constrained to an integer set Z p during the training process.
In each communication round, the server sends the model parameters to the clients, and they compute the stochastic gradient over the mini-batches with size b .
Particularly, we assume that all the clients use the same row selection matrix C ( t ) ∈ { 0 , 1 } b × (cid:101) m for data sampling in each round t 5 , and the mini-batch at each client j ∈ [ N ] is determined by [ (cid:101)
X
( I t )
j , (cid:101) Y ( I t )
j ] ≜ C ( t ) [ (cid:101) X j , (cid:101) Y j ] .
Here, I t = { l ( t ) 1 , . . .
, l ( t ) b } ⊆
[ (cid:101) m ] is a randomly selected index set in the t -th round with l i ∈
[ (cid:101) m ] for i ∈ [ b ] .
The entries of C ( t ) satisfy C ( t ) i,l i = 1 for i ∈
[ b ] , and other entries are set to zero.
Each client j computes the stochastic gradient (cid:101) g
( (cid:101) X
( I t )
j , (cid:101) Y ( I t )
j ; w ( t ) ) ≜ g ( (cid:101) X
( I t )
j , (cid:101) Y ( I t )
j ; w ( t ) )
mod p in the finite field, and uploads the result to the server.
Particularly, each (cid:101) g ( (cid:101) X
( I t )
j , (cid:101) Y ( I t )
j ; w ( t ) ) amounts to an evaluation of the polynomial (cid:101) g
( u I t ( z ) , v I t ( z ); w ( t ) ) at the point z
= α j , where two ( K + T − 1) -degree polynomial functions u
I t : F p → F b × d x p and v I t : F p → F b × d y p are defined as follows: [Equation] z − β j
[Equation]
with
[Equation]
Every ( X ( k ) I t , Y ( k ) I t ) =
( u I t ( β k ) , v I t ( β k )) for k ∈ [ K ] is a global mini-batch selected from the global dataset D .
Gradient Decoding and Model Updating.
According to (4) and (5), the server can obtain the global gradients (cid:101) g ( X ( k ) I t , Y ( k ) I t ; w ( t ) ) for k ∈
[ K ] by evaluating the polynomial (cid:101) g
( u I t ( z ) , v I t ( z ); w ( t ) ) at the point z = β k .
But, the server needs to first recover the coefficients of this polynomial, which is a composition of the encoding polynomials ( u I t ( z ) , v I t ( z )) and the gradient function (cid:101) g .
As the degree of the composite polynomial is deg ( g )( K + T − 1) , the server requires at least deg ( g )( K + T − 1)+ local computation results (i.e., evaluation points) to interpolate it 6 .
This implies that the proposed
DReS-FL framework can tolerate at most D = N − deg( g )( K + T − 1) − 1 client dropouts.
After decoding the global gradients, the server converts them from the finite field to the integer set Z p by g ( X ( k ) I t , Y ( k ) I t ; w ( t ) )
= ψ ( (cid:101) g ( X ( k ) I t , Y ( k ) I t ; w ( t ) )) , where ψ ( z ) is an element-wise function defined as follows:
[Equation]
As we assume that the prime number p is sufficiently large, the converted gradients do not have overflow errors.
Thus, the central sever updates the global model by w ( t +1) =
w ( t ) −
Q ( ηbK (cid:80)
Kk =1 g ( X ( k ) I t , Y ( k ) I t ; w ( t ) ) , where η denotes the learning rate and bK represents the global batch size 7 .
Q ( z ) is a stochastic quantization function to ensure the model parameters are in the integer set Z p after updating, which is defined as follows:
[Equation]
Besides, the probability of rounding z to ⌊ z ⌋ is proportional to the proximity of z to ⌊ z ⌋ so that the stochastic rounding is unbiased.
The overall procedure is summarized in Algorithm 1.
Convergence Analysis
In this section we characterize the convergence performance of PINNs, which relies on the fact that the global gradients in the training process are unbiased.
Define the empirical risk as ℓ ( w ) ≜
E ( X , Y ) ∼D
∥ Y − f
( X ; w ) ∥ 22 and the corresponding gradient as g e ( w ) ≜ E ( X , Y )
∼D [ g ( X , Y ; w )] .
The variables ( X , Y ) are drawn from the distribution of the global dataset D .
To prove that DReS-FL guarantees convergence to the optimal model parameters, we first present the following assumptions to facilitate the analysis.
Assumption 1.
( L -smoothness)
There exists a constant L > 0 such that for all w 1 , w 2 ∈ Z d w p , we have ∥ g e ( w 1 ) − g e ( w 2 ) ∥ 2 ≤ L ∥ w 1 − w 2 ∥ 2 .
[Table]
Assumption 2.
(Unbiased and variance-bounded stochastic gradient)
There exists a constant σ >such that any stochastic gradient g ( X ( k ) I t , Y ( k ) I t ; w ( t ) )
satisfies E (cid:104) 1 b g ( X ( k ) I t , Y ( k ) I t ; w ( t ) ) (cid:105) =
g e ( w ( t ) ) and E (cid:104) ∥
1 b g ( X ( k ) I t , Y ( k ) I t ; w ( t ) )
− g e ( w ( t ) )
∥ 2 (cid:105) ≤ σ 2 .
Assumption 3.
(Unbiased and variance-bounded rounding operation)
There exists a constant γ >such that for any z ∈ R , the stochastic quantization operation Q ( · ) satisfies E [ Q ( z )]
=
z and
E (cid:2)
∥ Q (
[Equation]
−
[Equation]
With the above preparations, we have the following theorem which ensures the convergence.
The proof is deferred to Appendix A.
Theorem 2.
(Convergence)
Denote w ∗ as the first-order optimal solution.
With Assumption 1-3, selecting the learning rate as η = O (cid:16) 1 / √ τ ′ (cid:17) such that Ψ ≜ 1 − ηL/ 2 − ηγ 2 L/ 2 > 0 , after τ ′ times of model updates, we have:
[Equation]
Experiments 6.1 Experimental Setup
Dataset.
We evaluate our proposed algorithm on several benchmark datasets: MNIST [45], FashionMNIST [46], EMNIST (Balanced) [47], CIFAR-10 [48], CIFAR-100 [48], and SVHN [49].
Specifically, the extra training samples in the SVHN dataset are not utilized.
To simulate the non-IID data distribution, we assume there are N = 20 clients in the learning system and adopt the skewed label partition [50] to shuffle the datasets.
Specifically, we sort a dataset by the labels, divide it into N shards, and assign one shard to each client.
To simulate the client dropouts in the training process, we consider an extreme scenario, where the dropout rate of each client is set to 0.99 with a probability of 0.5 or is uniformly sampled from [0 , 0 . 1] otherwise. More details of the datasets are deferred to
Appendix B.
Model structures.
We adopt a multi-layer perception (MLP) with two hidden layers for the image classification tasks on MNIST, Fashion-MNIST, and EMNIST datasets. Each hidden layer contains 64 neurons.
For CIFAR-10, CIFAR-100, and SVHN datasets, we resize the input images from 32 × 32 to 224 × 224 and adopt the convolutional layers of a pretrained VGG model to extract 25088-dimensional features. To classify the extracted features, we select a two-layer MLP model with 4096 hidden units each.
The baseline methods train the neural networks on the real field and select the rectified linear unit (ReLU) function as the activation function.
In each communication round, clients perform one SGD step for the local model update.
DReS-FL.
Our method adopts the same size PINNs to replace MLPs in the federated training, and the degree of gradient is deg( g )
= 8 .
Particularly, the extracted features from the last convolutional layer of VGG19 are secretly shared with other clients.
We set the parameters K = 1 and T = 1 in the Lagrange coding, and the minimum number of clients needed to decode the global gradient is 9. Baselines.
In the experiments, data-centric approaches [11, 15, 17, 33, 34, 35] are not compared since some of them [11, 15, 17] lack strong privacy guarantees while others [33, 34, 35] cannot support federated neural network training with multiple clients.
We select algorithm-based methods as baselines, including FedAvg [1], FedAvg with importance sampling (FedAvg-IS) [51, 52], and
6 Round 1e
A cc u r a c y ( % )
FedAvg FedAvg-IS DReS-FL (Ours)
(a) MNIST     6 Round 1e
A cc u r a c y ( % )
FedAvg FedAvg-IS DReS-FL (Ours)
(b) Fashion-MNIST     6 Round 1e
A cc u r a c y ( % )
FedAvg FedAvg-IS DReS-FL (Ours) (c) EMNIST     6 Round 1e
A cc u r a c y ( % )
FedAvg FedAvg-IS DReS-FL (Ours)
(d) CIFAR6 Round 1e
A cc u r a c y ( % )
FedAvg FedAvg-IS DReS-FL (Ours)
(e) CIFAR6 Round 1e
A cc u r a c y ( % )
FedAvg FedAvg-IS DReS-FL (Ours) (f) SVHN
SCAFFOLD [2], since these methods can be easily combined with secure aggregation methods 8 .
Particularly, we assume that the FedAvg-IS method knows the dropout distribution, and the local computation results are weighted by the participation probability (i.e., 1 - dropout probability) to mitigate bias in aggregation.
Besides, we also select the centralized training scheme as a performance upper bound, where the server can access all the clients’ datasets for model training.
6.2 Performance Evaluation
1.5 Round 1e
A cc u r a c y ( % )
SCAFFOLD
The experimental results are shown in Table 2 and Fig. The
FedAvg method achieves worse performance than the centralized training scheme.
This is attributed to the non-IID data and client dropouts.
The FedAvg-IS method improves the test accuracy compared with FedAvg, but there is still a noticeable performance gap with the centralized training scheme.
It shows that using the knowledge of dropout distribution can partially compensate for the biases in the aggregated models, but the local data distributions are still heterogeneous and degrade the performance.
Besides, SCAFFOLD has a low accuracy on all the settings.
As the frequency of updating local control variates is low, the estimation of the update direction is highly inaccurate such that the model does not converge as shown in Fig.
These results are consistent with the findings in [10].
Our DReS-FL method is superior to all the baseline methods as the server can obtain global gradients after polynomial interpolation.
In addition, DReS-FL achieves comparable performance to the centralized training scheme on some datasets, which demonstrates the effectiveness of our proposed framework in solving the non-IID and dropout problems.
Conclusions
This paper proposed a Dropout-Resilient Secure Federated Learning (DReS-FL) framework via
Lagrange coded computing (LCC) to simultaneously solve the data heterogeneity and dropout problems of FL, while providing privacy guarantees for the local datasets.
The polynomial integer neural networks (PINNs) have been constructed to ensure that the server can correctly decode the global gradient without privacy leakage.
Extensive experimental results validated the effectiveness of the proposed method.
Potential limitations of our method include that the degree of the gradient in a PINN increases exponentially with the number of layers, which hinders training a deep model for complex tasks. Besides, performing multiple local SGD steps largely increases the finite field size as the range of results grows exponentially with the number of multiplications, and thus it will lead to substantial communication overhead in model transmission. Despite some limitations, we believe DReS-FL is a promising framework for many practical FL application scenarios given its effectiveness in resolving both the non-IID and client dropout problems, while with strong privacy guarantees.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section(c)
Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] The proofs are provided in Appendix A. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL
[No] The code is proprietary, but the public datasets used in the experiments and the instructions needed for reproduction are presented in Section 6 and
Appendix B.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] Please refer to Section 6 and Appendix B. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Section 6.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix B. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] See Appendix B. (c) Did you include any new assets either in the supplemental material or as a URL?
[No] The code is proprietary and the datasets used in our work are public datasets.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] We choose the commonly-used datasets in the experiments.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
For simplicity, we denote g ( t ) = 1 bK (cid:80) Kk =1 g ( X ( k ) I t , Y ( k ) I t ; w ( t ) ) in this section.
The server updates the global model by w ( t +1) = w ( t ) − Q ( η g ( t ) ) in each round t after receiving deg( g )( K + T − 1)+ uploads from clients. We first provide an important lemma to show that the model update Q ( η g ( t ) ) on the server is an unbiased estimate of η g e ( w ( t ) ) . Lemma 1.
(Unbiased and variance-bounded model update)
In the t -th round, the model update
Q ( η g ( t ) ) has the following properties:
[Equation]
Proof.
According to Assumption 2-3, we directly obtain that E (cid:2) Q ( η g ( t ) ) (cid:3) =
E
[Equation]
η
[Equation]
= g e ( w ) .
Since the batch sampling and rounding operation cause independent errors, the variance is upper bounded as follows:
[Equation]
where (a) follows Assumption 3.
(b) is due to Assumption 2 and the independence of mini-batch sampling noises among clients.
With Lemma
[Equation]
we prove Theorem 2 as follows:
Proof.
The model update in the t -th iteration can be expressed as w ( t +1) =
w ( t ) − Q ( η g ( t ) ) .
According to the Taylor’s expansion, we have:
[Equation]
where (c) follows
Lemma 1, (d) holds according to the fact that E (cid:10) ∇ Q ( η g ( t ) ) − η g e ( w ( t ) ) , g e ( w ( t ) ) (cid:11) = 0 , and (e) is due to Assumption 2-3.
If it holds
[Table]
[Table]
[Table]
that η − η 2 L 2 − η 2 γ 2 L>
[Table]
we summarize the above inequality over t = 1 , 2 , . . .
[Table]
τ ′ to conclude the proof.
B Additional Experimental Details
All experiments are performed by Pytorch on an Intel Xeon Gold 6246R CPU @ 3.40 GHz and
a
Geforce RTX 3090.
Some details of the datasets are summarized in Table 3.
We adopt mini-batch
SGD with a batch size of 64 to optimize the models in federated training.
{+The communication round is set to be 7 × 
+}
[Table]
and the clients perform one local SGD step in each round.
The learning rate is initialized as 0.
[Table]
and it will decay with a factor of 0.65 after every 1500 rounds.
Other parameters in our DRes-FL framework are summarized in Table 4.
C Complexity Analysis and Comparison
In this part, we analyze the communication and computational complexities of the proposed DReS-FL framework with respect to the parameters ( N, T, K, τ, d w , b g ) .
Parameter N is the number of clients, and T denotes the privacy threshold in Lagrange coding [22].
Parameter K denotes the number of shards in the local datasets.
A large value of K reduces the communication and computation overheads in the proposed DReS-FL framework.
In the federated training, the parameter τ corresponds to the number of communication rounds.
Parameters d w and b g denote the model size and the global batch size, respectively.
Before training starts, each client’s computation cost for Lagrange coding and communication complexity for data sharing are O
( N log 2 ( K + T ) log log( K + T )) and O ( N/K ) , respectively.
In each round of federated training, the local computation complexity is O
( d w b g /K ) , and the model uploading cost is O
( d w ) .
Besides, the communication overhead of the server for model distributing is O ( Nd w ) , and the model decoding complexity by polynomial interpolation is
O ( R log 2 R log log Rd w ) , where R denotes the minimum uploads needed for gradient decoding.
Different from our method, secure aggregation approaches [39, 53, 40, 42, 44, 41] generate random masks to protect the local model parameters.
In each round, clients first share coded masks with each other, which allows for aggregating the masked models at the server.
As some clients may drop out of the training process unexpectedly, the surviving clients upload the shared information belonging to the dropped clients to reconstruct the aggregated model.
The main drawback of such approaches is that the clients need to generate new masks in each round, and their computational and communication complexities increase linearly with the number of training rounds.
In comparison, the data sharing phase of our method only introduces extra costs for one time, which is independent of the training rounds.
In the scenario that the number of training rounds is very large, the proposed DReS-FL method achieves lower computational and communication costs than the secure aggregation protocols.
[Table]
FedAvg with
LightSecAgg
[Table]
[Table]
with
LightSecAgg
[Table]
The detailed comparisons among FedAvg, FedAvg with LighSecAgg [41], and our DReS-FL method are summarized in Table 5 and 6.
D Model Extension
Our DReS-FL framework can be extended to more general cases in which clients can run s ( s ≥ 1 ) local SGD steps each round.
Denote the computation results after s local SGD steps in round t as ∆ (cid:101) w j ( s ; w ( t ) ) for j ∈ [ N ] .
Specifically, ∆ (cid:101) w j ( s = 1; w ( t ) ) ≜ (cid:101) g ( (cid:101) X
( I t )
j , (cid:101) Y ( I t )
j ; w ( t ) ) and ∆ (cid:101) w j ( s = 2; w ( t ) ) ≜ (cid:101) g ( (cid:101) X
( I t +1 )
j , (cid:101) Y
( I t +1 )
j ; w ( t ) − ηbK (cid:101) g
( (cid:101) X
( I t )
j , (cid:101) Y ( I t )
j ; w ( t ) )) .
By carefully selecting the learning rate η such that ηbK ∈ F p , the function ∆ (cid:101) w j ( s ; w ( t ) ) is still a polynomial in the finite field F p .
Therefore, the central server can recover the desired model update by polynomial interpolation at the cost of low dropout-resiliency caused by the high degree of ∆ (cid:101) w j ( s ; w ( t ) ) .
E Degree of Gradient in PINN
Given a data sample as ( x , y ) , the feedforward process of a PINN with L quadratic activation layers h is as follows:
[Equation]
where z ′ 0 ≜ x , z ′ l = h ( z l ) , and z l = W l z ′ l − 1 + b l for l ∈ 1: L + 1 .
z L +1 is the output of PINN, and the loss function is the squared error between y and z L +1 , i.e., ℓ = ∥
y − z L +1 ∥ 22 .
The notations
W l and b l correspond to the weight matrix and bias vector in PINN.
With the above preparation, we derive the gradients as follows:
[Equation]
where W l [ j ] is the j -th column in W l , and z ′ l − 1 [ j ] is the j -th element in the vector z ′
l − 1 .
According to the chain rule of gradient, the gradient of the loss function with respect to the weight W l [ j ] is
[Equation]
Note that the mappings from input z ′ 0 to z ′ l and z l +1 are polynomials with degree 2 l .
Therefore, the degree of the gradient
[Equation]