A CCELERATION OF F EDERATED L EARNING WITH A LLEVIATED F ORGETTING IN L OCAL T RAINING
Anonymous authors
Paper under double-blind review
A BSTRACT
Federated learning (FL) enables distributed optimization of machine learning models while protecting privacy by independently training local models on each client and then aggregating parameters on a central server, thereby producing an effective global model.
Although a variety of FL algorithms have been proposed, their training efficiency remains low when the data are not independently and identically distributed (non-i.i.d.) across different clients.
We observe that the slow convergence rates of the existing methods are (at least partially) caused by the catastrophic forgetting issue during the local training stage on each individual client, which leads to a large increase in the loss function concerning the previous training data at the other clients.
Here, we propose FedReg, an algorithm to accelerate FL with alleviated knowledge forgetting in the local training stage by regularizing locally trained parameters with the loss on generated pseudo data, which encode the knowledge of previous training data learned by the global model.
Our comprehensive experiments demonstrate that FedReg not only significantly improves the convergence rate of FL, especially when the neural network architecture is deep and the clients’ data are extremely non-i.i.d., but is also able to protect privacy better in classification problems and more robust against gradient inversion attacks.
I NTRODUCTION
Federated learning (FL) has emerged as a paradigm to train a global machine learning model in a distributed manner while taking privacy concerns and data protection regulations into consideration by keeping data on clients (Voigt & Von dem Bussche, 2017).
The main challenge faced in FL is how to reduce the communication costs (in training) without degrading the performance of the final resultant model, especially when the data on different clients are not independently and identically distributed (non-i.i.d.)
(Yuan & Ma, 2020; Wang et al., 2020b).
The most popular FL algorithm is
FedAvg (McMahan et al., 2017a).
In each training round of FedAvg, local training steps are separately performed at every client and the locally trained models are transferred to the server.
Then, the server aggregates the local models into a global model by simply averaging their parameters.
Although FedAvg succeeds in many applications, its training processes often diverge when the data are non-i.i.d., also known as heterogeneous, across the clients (Li et al., 2019; Zhao et al., 2018).
Several FL algorithms have been designed to improve FedAvg and tackle the heterogeneity issue mainly by reducing the difference between locally trained parameters (Li et al., 2018;
Karimireddy et al., 2020) or aggregating these parameters into different groups (Wang et al., 2020a; Yurochkin et al., 2019)
However, the performance of these methods is still far from satisfactory when a deep neural network architecture is employed (Rothchild et al., 2020; Li et al., 2021).
On the other hand, recent work in the literature (Geiping et al., 2020) shows that the transmission of trained model parameters does not ensure the protection of privacy.
Privacy-sensitive information can be recovered with gradient inversion attacks (Zhu et al., 2019).
Differential privacy (DP) (McMahan et al., 2017b; Abadi et al., 2016) is one of the most widely used strategies to prevent the leakage of private information.
However, when DP is incorporated into FL, the performance of the resulting model decays significantly (Jayaraman & Evans, 2019).
We observe that when the data are non-i.i.d.
across the clients, the locally trained models suffer from severe forgetting of the knowledge of previous training data at the other clients ( i.e. , the well-known catastrophic forgetting issue), perhaps due to the discrepancy between local data distributions and the global data distribution.
As shown in Figure 1 and supplementary Figure C.1, this forgetting issue leads to a large increase in the loss concerning these training data under the non-i.i d. setting, thereby slowing down the convergence speed.
In this work, we propose a novel algorithm, FedReg that reduces the communication costs in training by alleviating the catastrophic forgetting issue in the local training stage.
FedReg reduces knowledge forgetting by regularizing the locally trained parameters with generated pseudo data, which are obtained by using modified local data to encode the knowledge of previous training data as learned by the global model.
The potential conflict with the knowledge in the local data introduced by the pseudo data is dampened by the use of perturbed data, which are generated by making small perturbations to the local data, whose predictive values they help ensure.
The generation of the pseudo data and perturbed data only relies on the global model received from the server and the local data on the current client.
Thus, compared with FedAvg, no extra communication costs concerning the data on the other clients are incurred.
Our extensive experiments demonstrate the superiority of FedReg in accelerating the FL training process, especially when the neural network architecture is deep and the clients’ data are extremely heterogeneous.
Furthermore, the pseudo data can be further utilized in classification problems to defend against gradient inversion attacks.
More specifically, we show that with similar degree of protection of private information, the degradation in the performance of the global model learned by our method is much less than that learned by using FedAvg combined with DP.
Our contributions.
We demonstrate that when the data across clients are non-i.i.d., catastrophic forgetting in the local training stage is an important factor that slows down the FL training process.
We therefore propose a novel algorithm, FedReg, that accelerates FL by alleviating catastrophic forgetting with generated pseudo data.
We also perform extensive experiments to establish the superiority of FedReg.
We further show that in classification problems, the pseudo data generated in
FedReg can help protect private information from gradient inversion attacks with much less impact on the performance of the resultant model compared with DP.
R ELATED WORK 2. F EDERATED LEARNING
FL is proposed to address privacy concerns in a distributed learning environment.
In the FL paradigm, a client i keeps its local data D i on the client machine during the whole training process so that the server and other clients cannot directly access D i .
The objective is to find the parameter θ ∗ that minimizes the loss on the global data ∪ i ∈
C D
i , where C is the set of clients, i.e. ,
[Equation]
In this equation, L θ is the loss function with parameters θ , which can be cross-entropy or in some other custom-defined form.
FedAvg (McMahan et al., 2017a) and FedProx (Li et al., 2018) are the most widely used FL algorithms.
In FedAvg, in a training round t , the server sends the initial parameters θ ( t − 1) to a set of sampled clients C ( t ) .
Then, the parameters are updated independently for S epochs on each of these clients to minimize the loss on the local data, and the locally trained parameters θ ( t,i ) are then sent to the server.
The server aggregates the parameters by simply averaging over them and obtains the parameters θ ( t ) , i.e. , θ ( t ) = 1 K (cid:80) i ∈ C ( t ) θ ( t,i ) , where K is the number of sampled clients in round t .
FedAvg is unstable and diverge when the data are non-i.i.d.
across different clients (Li et al., 2018).
FedProx stabilizes FedAvg by including a proximal term in the loss function to limit the distance between θ ( t,i ) and θ ( t − 1) .
Although FedProx provides a theoretical proof of convergence, empirically it fails to achieve good performances when deep neural network architectures are used (Li et al., 2021).
SCAFFOLD (Karimireddy et al., 2020) assumes that the heterogeneity of data leads to a client-drift in gradients and correlates the drift with gradient correction terms.
As clients need to send extra information to the server, SCAFFOLD increases the risk of privacy leakage and doubles the communication burden compared with FedAvg.
Furthermore, the accuracy of the correction terms is highly correlated with the training history of the client.
Thus, the performance of SCAFFOLD decreases significantly when the number of clients is large, in which case each client is only sampled for few times and the estimation of the gradient correction terms is often inaccurate.
FedCurv (Shoham et al., 2019) aims to tackle the forgetting issue on non-i.i.d.
data with elastic weight consolidation (Kirkpatrick et al., 2017).
To achieve this, FedCurv needs to transfer Fisher information between the server and clients, which increases the communication costs to 2 .
5 times compared with FedAvg.
2. C ATASTROPHIC FORGETTING
Catastrophic forgetting occurs specifically when the neural network is trained sequentially on multiple tasks.
In this case, the optimal parameters for the current task might perform poorly on the objectives of previous tasks.
Many algorithms have been proposed to alleviate the forgetting issue.
Memory-based methods (Parisi et al., 2019) have achieved excellent performances in accommodating new knowledge while retaining previously learned experience.
Such memory-based methods, such as gradient episodic memory (GEM) (Lopez-Paz & Ranzato, 2017) and averaged gradient episodic memory (A-GEM)
(Chaudhry et al., 2018), store a subset of data from previous tasks and replay the memorized data when training on the current task.
For instance, A-GEM treats the losses on the episodic memories of previous tasks as inequality constraints in optimizing the objectives of current tasks and changes the model updates from the plain gradient g to g − w g ref , where g ref is the gradient computed from the loss on the memorized data and w is a non-negative weight.
Unfortunately, these memory-based techniques are not suitable for FL due to privacy concerns.
2. G RADIENT INVERSION ATTACKS
FL provides a privacy guarantee by keeping the users’ data on individual client machines locally and only sharing the model updates.
However, recent research has shown that data information can be recovered from the parameter updates (Geiping et al., 2020; Zhu et al., 2019) in the FedAvg framework by simply finding data with updates similar to the values returned from the client.
DP, which avoids privacy leakage by adding noise to training data (Sun et al., 2019) or model updates (Abadi et al., 2016; McMahan et al., 2017b), is the most widely used strategy to protect private information.
When adding noise to model updates, such as differentially private SGD (DPSGD)
(Abadi et al., 2016), the noise level in DP is controlled by the gradient norm bound C and the noise scale σ .
DP often causes a large performance degradation in the resultant model.
M ETHOD
Our main challenge is how to alleviate the forgetting of previously learned knowledge at each client without having to access data at the other clients in the local training stage.
For any data set D , we de- note the set of data near D within the Euclidean distance of δ as N ( D , δ ) = ∪ d
=( x , y ) ∈ D { ( x ′ , y ′ )
| 0 <
∥ x − x ′ ∥ ≤ δ } .
We assume that in round t on client i , (1) for all data d − ∈ ∪ j ∈ C / { i } D j local to the other clients, the global model with parameter θ ( t − 1) has a better feature representation than the local model with parameter θ ( t,i ) ; and (2) for any c > 0 , ∃ ϵ, δ > 0 such that if ∀ d ′ =
( x ′ , y ′ )
∈
N
( D i , δ ) , L θ ( t,i ) (( x ′ , f θ ( t − 1) ( x ′ )))
− L θ ( t − 1) (( x ′ , f θ ( t − 1) ( x ′ )))
≤ ϵ , then ∀ d − = ( x − , y − ) ∈ ∪ j ∈ C / { i } D j , L θ ( t,i ) (( x − , f θ ( t − 1) ( x − )))
− L θ ( t − 1) (( x − , f θ ( t − 1) ( x − ))) ≤ c , where f θ is the prediction function with parameters θ .
The assumption (2) guarantees that, in the local training stage, the range of changes in the predicted values of previous training data at the other clients can be controlled by constraining the changes in the predicted values of the local data near D i .
Based on these assumptions, we first generate pseudo data and then alleviate the catastrophic forgetting issue by regularizing the locally trained parameters with the loss on the pseudo data.
In other words, we hope that the pseudo data would achieve the same effect as the previous training data in constraining the local model.
Note that FedReg does not assume any particular form of the loss function, but the method of modifying gradients to enhance privacy protection (to be discussed below in Section 3.3) is currently only applicable to classification problems.
The pseudo code of FedReg is provided in Appendix D.
3. G ENERATION OF PSEUDO DATA AND PERTURBED DATA
With the above assumptions, the catastrophic forgetting issue can be alleviated by constraining the prediction values for data in N ( D , δ ) .
However, such a constraint is too strong and would hinder the learning of new knowledge from the local data.
Moreover, it is also computationally inefficient to enumerate all such data.
Observe that for a data point d ′ =
( x ′ , y ′ )
∈
N
( D i , δ ) , after locally training θ ( t,i ) with D i , if f θ ( t,i ) ( x ′ )
=
f θ ( t − 1) ( x ′ )
then the constraint on d ′ will not contribute to addressing the forgetting issue.
Therefore, to efficiently regularize the locally trained parameters, only data points with large changes in the predicted values should be used to alleviate knowledge forgetting.
Such data points are usually within a small distance to the data points in D i , but their predicted values given by θ ( t − 1) could be very different from the labels in D i .
Data satisfying these conditions can be easily obtained with the fast gradient sign method (Goodfellow et al., 2014) and we denote the data generated in this way as pseudo data D si .
More precisely, each data point ( x s , y s ) ∈ D si is iteratively generated from a data point ( x , y ) ∈ D i as follows:
[Equation]
where η s is the step size and E the number of steps.
Despite the relaxation in constraints achieved by the pseudo data generated above, some information conflicting with the knowledge embedded in the local data D i could possibly be introduced in D si due to the inaccuracy of the global model during the training process.
To further eliminate such conflicting information, perturbed data D pi = { ( x p , y p ) } with small perturbations on D i are iteratively generated as follows:
[Equation]
where the step size η p satisfies η p
≪
η s in order to make sure that the perturbed data is much closer to the local data than the pseudo data.
In our experiments [Equation] we take η p = 0 .
01 η s and E = 10 to reduce the complexity of tuning hyper-parameters [Equation]
3. A LLEVIATION OF CATASTROPHIC FORGETTING
With the pseudo data D si and perturbed data D pi generated above
[Equation]
we regularize θ ( t,i ) by requiring
[Equation]
where constraint (4) alleviates the catastrophic forgetting issue and constraint (5) eliminates conflicting information introduced in (4).
Constraint (5) also helps improve the robustness of the resultant model (Madry et al., 2018).
Expanding both sides of (4) and of (5) respectively with θ ( t,i ) β
= 0 .
5 (cid:0) θ ( t,i )
+ θ ( t − 1) (cid:1) so that the second-order term can be eliminated and ignoring higherorder terms (see Appendix A.1 for the details), we obtain
[Equation]
Due to the existence of potentially conflicting information in D si and D pi , directly finding θ ( t,i ) to satisfy the above two inequalities is mathematically ill-defined.
Besides, given the complex architectures of deep neural networks, it is computationally too challenging to solve either of the above inequalities.
Therefore, we approximate the optimal parameters θ ( t,i ) ∗ by solving the following constrained optimization problems:
[Equation]
By solving (8) and (9) (see Appendix A.2 for the detailed derivation), we obtain
[Equation]
i
[Equation]
Based on the generation process of the pseudo data, it is easy to note that (cid:80) d s ∈ D si L θ ( t − 1) ( d s ) =
min θ (cid:80) d s ∈ D si L θ ( d s ) , and thus the inequality sign in (4) can in fact be replaced by an equality sign.
In practice, however, we have observed that the training process would be more stable and result in non-negative w s and w p with the inequality formulation, probably due to errors introduced in the above approximation.
Finally, to prevent the gradient from converging to zero in a very small number of steps, slowly-updated parameters (Zhang et al., 2019b)
θ ( t,i ) γ [Equation]
with γ ∈ [0 , 1] are used to compute the gradient for D i , i.e. , g γ ( D i ) =
∇ θ ( t
[Equation]
In summary, taking into acount the above improvements, the local parameters are updated in each training step as:
[Equation]
3. M ODIFICATION OF GRADIENTS TO PROTECT PRIVACY
We further notice that in classification problems, the pseudo data can be used to modify the gradient g γ ( D i ) to enhance the protection of privacy.
If we denote D si ′ =
{ ( x , 1 n (cid:80) nj =1 e ( j ) )
| ( x , y ) ∈
D si } for an n -classification problem, where e ( j ) is the standard basis vector with a 1 at position j , then the data points in D si ′ are similar to those in D i , but they may have totally different labels.
Hence, it is reasonable to assume that g γ ( D i ) and g γ ( D si ′ ) contain similar semantic information but different classification information.
Since in the training of FL, only the classification information contributes to the model performance, removing the semantic information in g γ
( D i ) will enhance the privacy protection capability of FL without severely degrading the performance of the resultant model.
Based on this intuition, we presume that the components of g γ ( D i ) in the same (or roughly the same) direction of g γ ( D si ′ ) contain the semantic information and the other (orthogonal) components contain the classification information.
Thus, we compute a modified gradient (MG) to enhance the protection of privacy as
[Equation]
We refer to the version of FedReg in which the g γ ( D i ) term in (11) is replaced by ˜
g γ ( D i ) as FedReg with MG.
E XPERIMENTS
We compare the performance of FedFeg against the above-mentioned baseline FL algorithms including FedAvg, FedProx, FedCurv and SCAFFOLD as well as the well-known SGD algorithm (Bottou, 2012).
Note that SGD corresponds to a special case of FedAvg where the local training stage consists of only one step and all data on a client are used in a large single batch.
4. D ATA PREPARATION
FedReg is evaluated on MNIST (Deng, 2012), EMNIST (Cohen et al., 2017), CIFAR-10 and CIFAR100 (Krizhevsky et al., 2009), and CT images of COVID-19 (He, 2020).
To simulate a scenario for FL, the training data in each dataset are split into multiple clients in different ways and the performance of the trained model is evaluated on the test data.
The data preparation steps for each dataset are described below and more experimental details are provided in Appendix B.
MNIST The training data are split into 5,000 clients.
Each client has images either from only one class, named MNIST (one class), or from two classes, named MNIST (two classes).
The number of images per client follows a power law distribution (Li et al., 2018).
A 5-layer neural network with three layers of convolutional neural networks (CNN) and two layers of full connections (FC) is used.
EMNIST
The training data of the digits in EMNIST are split into 10,000 clients.
Each client owns 24 images from the same class.
The same model architecture used on MNIST is used here.
CIFAR-10 The training data are split into 10,000 clients.
Each client owns 5 images from random classes, named CIFAR-10 (uniform), or from only one class, named CIFAR-10 (one class).
A
ResNet-9 network with the Fixup initialization (Zhang et al., 2019a) is trained from scratch.
CIFAR-100 The training data are split into 50,000 clients and each client owns 1 image.
A ResNet9 network with the Fixup initialization is trained from scratch, named CIFAR-100 (ResNet-9), and a pre-trained transformer (Dosovitskiy et al., 2020) is fine-tuned, named CIFAR-100 (Transformer).
CT images related to COVID-
The dataset contains 349 CT images of COVID-19 patients with source information and 397 non-COVID-19 CT images without source information.
The data are split into training and test data in the same way provided in He (2020).
Then, the CT images of
COVID-19 patients in the training data are split to make sure that the images from the same source are assigned to the same client, and 32 clients are obtained in this way.
The non-COVID-19 images in the training data are distributed to each client following the proportion of COVID-19 images.
A
DenseNet-121 network (Huang et al., 2017) and a 10-layer neural network with 9 layers of CNNs and one layer of FC are trained on these CT images to distinguish the COVID-19 images from the non-COVID-19 ones.
[Table] [Table]
SCAFFOLD  -
[Table] [Table]
[Table] [Table]
[Table]
FedCurv0.
[Table]
4. C OMPARISON OF CONVERGENCE RATES
The results on MNIST and EMNIST are shown in Table 1.
FedReg required fewer communication rounds to converge compared with the baseline methods and achieved a higher final accuracy ( i.e. ,
[Table] [Table]
accuracy of the respective models after finishing all training rounds) under all the experimental settings, especially on EMNIST, where the number of clients is twice compared to MNIST, showing clearly the superiority of FedReg in dealing with non-i.i.d.
data when a shallow model is used and its robust scalability.
FedCurv also aims to allieviate the forgetting issue, but its performance is not significantly better than FedProx.
Note that due to its heavy reliance on the optimization history when estimating the correction term, in our experiments, SCAFFOLD performed worse than
FedAvg.
Table 2 illustrates that, on CIFAR-10, when the data are uniformly distributed, FedReg significantly outperformed all the baseline methods in both convergence speed (measured in communication rounds) and final model accuracy, demonstrating the superiority of FedReg in training deep neural networks.
When data across clients become heterogeneous and the images at each client are from the same class, the multiple local training steps in the baseline algorithms (other than SGD) failed to improve the performance of the locally trained model, resulting in a worse performance than that of SGD, while FedReg was still able to save about half of the communication costs compared with SGD.
A similar conclusion can be drawn when training the ResNet-9 network from scratch and fine-tuning the transformer model on CIFAR-100, as shown in Table 3, indicating the robustness of FedReg on various datasets and model architectures.
The results in distinguishing COVID-
CT images from non-COVID-19 images on a DenseNet-121 network, as shown in Table 3, further exhibits the superiority of FedReg in medical image processing applications when the model architecture is deep and complex.
To assess the advantage of FedReg on real-world data, it is compared with the other methods on a popular dataset Landmarks-User-160k (Hsu et al., 2020; Lai et al., 2021), where the data at each client consists of many classes, as shown in supplementary Table
C.1.
Although none of the methods did well (after 350 rounds), FedReg still performed the best.
The wall-clock time usages of the methods in each local training round on MNIST and EMNIST are given in supplementary Table C.2.
Note that since FedReg performs additional computation in each round, its local training requires a bit more time.
4. A LLEVIATION OF CATASTROPHIC FORGETTING
To prove that FedReg indeed alleviates the catastrophic forgetting, increases of the loss values concerning previous training data at the other clients are compared between FedReg and FedAvg.
As shown in Figure 2, the increases of the loss are significantly lower in FedReg than those in FedAvg, suggesting that although FedAvg and FedReg both forget some of the learned knowledge, the forgetting issue is less severe in FedReg.
To further explore the role of the generated pseudo data in the alleviation of forgetting, the values of the empirical Fisher information (Ly et al., 2017) in the pseudo data and previous training data are compared.
As shown in Figure 2, the values of the pseudo data and previous training data are significantly correlated.
Following the Laplace approximation (MacKay, 1992; Kirkpatrick et al., 2017), if we approximate the posterior distribution of the optimal parameters on a dataset by a normal distribution centered at θ ( t − 1) with a diagonal precision matrix, then the precision matrix can be approximated by the empirical Fisher information.
The similarity in the empirical Fisher information suggests that a model with high performance on the pseudo data has a relatively high probability to perform well on previous training data.
Therefore, regularizing the parameters with the pseudo data can help reduce the performance degradation on previous training data and alleviate the knowledge forgetting issue.
4. T HE IMPACT OF STEP SIZE IN GENERATING PSEUDO DATA
The impact of the hyper-parameter η s , which determines the distance between the pseudo data and the local data, is explored here.
Intuitively, when η s is too large, the distance between the generated pseudo data and the local data will be too large to effectively regularize the model parameters.
On the other hand, when η s is too small, the conflicting information from an inaccurate global model will slow down the convergence speed.
This intuition is empirically confirmed by the experiments presented in supplementary Figure C.2. On the EMNIST and CIFAR-10 (one class) datasets, when η s is too small, it takes more communication rounds to achieve the same accuracy.
As η s increases, the number of communication rounds decreases to a minimum and then bounces back, indicating that a less effective regularization on the model parameters in the local training stage has been reached.
4. P ROTECTION OF PRIVACY
The protection of private information is one of the most important concerns in the FL paradigm, especially in medical applications.
To illustrate the privacy security of FedReg with MG, gradient inversion attacks are used to recover information from updated parameters in classification problems.
Under a comparable degree of privacy protection, the resultant model performance is compared between FedReg with MG and the baseline methods with or without DPSGD (Abadi et al., 2016).
Using gradient inversion attacks, the quality of images recovered from the updated parameters in a local step is examined.
Note that when only one local training step is performed in each round, the updated parameters in FedAvg, FedProx, FedCurv, and SGD are all the same, leading to the same the images recovered from these methods.
We do not include SCAFFOLD here since its performance was significantly inferior to the other methods in the above experiments.
As shown in Figure 3, for some images randomly sampled from EMNIST and CIFAR-10, the quality of the images recovered from FedReg with MG is significantly worse than those recovered from the baseline methods, exhibiting a better privacy protection capability of FedReg with MG.
To compare the impact of privacy protection on model performance, we tune the noise level in DPSGD to generate images with similar quality as measured by PSNR (Hore & Ziou, 2010) and compare the performance of the resultant models.
The results are shown in Table 4.
It can be seen that the protection of private information with DP costs a large performance degradation in the baseline methods, while the performance of
FedReg with MG is degraded much less when maintaining a similar privacy protection level.
Note that the CT images used in the above experiments contain some privacy-sensitive information, including the date of birth of the patient and the date when the image was taken.
In the basic FedAvg, when a 10-layer neural network with 9 layers of CNNs and one layer of FC is used, such information can be easily recovered with gradient inversion attacks.
When DP is applied, although such time information could be protected, the resultant model suffers from severe performance degradation, as shown in Table 4.
In contrast, FedReg with MG is able to protect the sensitive time information with only mild model performance decay.
[Table]
DPSGD
[Table]
DPSGD
[Table]
DPSGD
[Table]
C ONCLUSIONS AND FUTURE WORK
In this work, we proposed a novel algorithm, FedReg, to accelerate the convergence speed of FL by alleviating the catastrophic forgetting issue in the local training stage.
Pseudo data are generated to carry knowledge about previous training data learned by the global model without incurring extra communication costs or accessing data provided at the other clients.
Our extensive experiments show that the generated pseudo data contain similar Fisher information as the previous training data at the other clients, and thus the forgetting issue can be alleviated by regularizing the locally trained parameters with the pseudo data.
The pseudo data can also be used to defend against gradient inversion attacks in classification problems with only mild performance decay on the resultant model compared with DP.
Although FedReg exhibits competitive performance in accelerating convergence speeds, its performance is influenced by hyper-parameters.
Automatic tuning of the hyperparameters could make FedReg easier to use.
Meanwhile, the question of how to modify gradients in FedReg to effectively protect privacy in regression problems remains to be explored.
Moreover, a rigorous proof of the convergence rate of FedReg is of theoretical interest.
We leave these questions to future work.
A S UPPLEMENTARY METHOD A. T AYLOR EXPANSION
Expanding both sides of the inequality (4) with θ ( t,i )
β
= θ
[Equation]
, we obtain (cid:88)
[Equation]
Substituting both sides of (4), dividing them by | D si | and ignoring all high order terms, then only the first order terms are left and the thus inequality (6) is obtained,
i
[Equation]
e. ,
[Equation]
The inequality (7) can be derived from (5) in a similar way.
A. S OLVING THE CONSTRAINT OPTIMIZATION
The constraint optimization problems in (8) and (9) can be solved by the Lagrange multiplier method.
More specifically, for the problem (8), i
[Equation]
e. ,
[Equation]
its Lagrangian can be written as
[Equation]
where α ≥ 0 .
The dual problem is
[Equation]
Setting the derivatives of L ( θ
[Equation]
α )
w.r
[Equation]
t. θ to zero
[Equation]
the value of θ ( t,i ) ′
minimizing L ( θ , α ) is obtained as
[Equation]
Thus
[Equation]
D ( α ) can be simplified as
[Equation]
Solving α ∗ = arg max D ( α ) , the value of α ∗ is
[Equation]
Thus, the value of θ ( t,i ) ′ is
[Equation]
The constraint optimization problem 9 can be solved similarly.
B E XPERIMENTAL DETAILS
In all the experiments, the learning rates for different methods are optimized by grid search, the optimal weight for the proximal term in FedProx is searched among { 1.0, 0.1, 0.01, 0.001 } , the optimal λ for FedCurv is searched among { 0.001, 0.0001, 0.00001 } , and the hyper-parameters ( γ and η s ) in FedReg are optimized by grid search as well.
The number of epochs in the local training stage is optimized in FedAvg and applied to the other methods.
B. C OMPUTATION OF loss ( t − 1) AND loss ( t )
Formally, the value of loss ( t − 1) for a client i ∈ C ( t − 1) is computed as
[Equation]
and the value of loss ( t ) is computed as
[Equation]
where θ ( t,j ) is the parameters sent from client j in round t .
B.
C OMPARISON OF CONVERGENCE RATES
The hyper-parameters in the experiments are listed below:
• MNIST.
Ten clients are sampled in each training round.
In the local training stage, the local data are processed in 20 epochs with a batch size of 10.
500 rounds are run in total so that all training data are utilized once on average.
The learning rate in all methods is 0 .
The weight for the proximal term in FedProx is 0 .
1 and λ in FedCurv is 10 − 4 .
In FedReg, γ = 0 .
5 and η s = 0 .
• EMNIST.
20 clients are sampled in each training round, and in the local training stage, the local data are processed in 20 epochs with a batch size of 24.
500 rounds are run in total so that all the training data are utilized once on average.
The learning rate in SCAFFOLD is 0 .
1 and in other methods, it is 0 .
The weight for the proximal term in FedProx is 0 .and
λ in FedCurv is 10 − 4 .
In FedReg, γ = 0 .
4 and η s = 0 .
• CIFAR-10.
Following the experimental settings in (Rothchild et al., 2020), 240 rounds are run in total with 100 clients sampled in each training round, and in the local training stage, the local data are processed in 40 epochs with a batch size of 5.
On CIFAR-10 (uniform), the learning rate is 0 . 05 in FedAvg, FedCurv and FedProx, 0 . 1 in SGD and FedReg, and 0 .
01 in SCAFFOLD respectively.
The weight for the proximal term in FedProx is 0 .and
λ in FedCurv is 10 − 4 .
In FedReg, γ = 0 .
4 and η s = 0 .
On CIFAR-10 (one class), the learning rate is 0 . 05 in FedAvg, FedCurv, FedProx and FedReg, 0 . 1 in SGD, and 0 .in SCAFFOLD, respectively.
The weight for the proximal term in FedProx is 0 .
001 and λ in FedCurv is 10 − 3 .
In FedReg, γ = 0 .
4 and η s = 0 .
01 . • CIFAR-100.
When ResNet-9 is adopted, 1,200 rounds are run in total with 100 clients sampled in each training round, and in the local training stage, the local data are processed in 10 epochs with a batch size of 1.
The learning rate is 0 . 05 in FedAvg, FedCurv and FedProx, and 0 .
1 in SGD and FedReg, respectively.
The weight for the proximal term in FedProx is 0 .
001 and λ = 10 − 3 in FedCurv.
In FedReg, γ = 0 .
25 and η s = 0 .
When the pre-trained transformer is adopted, 100 rounds are run in total with 500 clients sampled in
each training round, and in the local training stage, the local data are processed in 5 epochs with a batch size of 1.
The learning rate is 0 .
08 in FedCurv and 0 .
1 in other methods.
The weight for the proximal term in FedProx is 0 .
001 and λ = 10 − 5 in FedCurv.
In FedReg, γ = 0 .
02 and η s = 10 − 6 .
Note that as SCAFFOLD is too memory-consuming to run on
CIFAR-100 given our computational resources, the results of SCAFFOLD on CIFARare not reported.
• CT images related to COVID-19.
In our experiments, ten rounds are run in total, and in each round 10 clients are sampled.
In the local training stage, the local data are processed in 20 epochs.
The learning rate is 5 × 10 − 3 in FedAvg, FedCurv, FedProx and SCAFFOLD, 1 × 10 − 4 in SGD, and 5 × 10 − 4 in FedReg respectively.
The weight for the proximal term in FedProx is 0 .
1 and λ in FedCurv is 10 − 3 .
In FedReg, γ = 0 .
5 and η s = 10 − 6 .
Note that in batch-normalization (Ioffe & Szegedy, 2015), the values of running statistics are estimated using the parameters θ ( t,i ) , and the gradients g γ ( D i ) , g β ( D si ) and g β ( D pi ) are computed using the fixed running statistics instead of the values estimated from the batch data. • Landmarks-User-160k.
The training data in this dataset has 164,172 images of 2, landmarks from 1,262 users, and the test data contains 19,526 images.
Each user has images of 29 classes on average.
A DenseNet-121 network is trained on this dataset.rounds are run in total, and in each round 100 clients are sampled. In the local training stage, the local data is processed in 20 epochs. The learning rate is set as 0 . 1 in SGD, FedAvg, FedReg, and pFedGP (Achituve et al., 2021). As pFedGP is designed for personalized federated learning scenarios and trains an independent GP-tree for each client, to apply it to this dataset, a small global dataset is built by sampling 2,028 images from all training data, which covers all the classes contained in Landmarks-User-160k.
In each round, the deep kernel, which is the DenseNet-121 network without the classification layer, is trained on each client and aggregated in the server.
Then, a global GP-tree is trained on the sampled global sub-dataset to classify the images in the test data.
B. A LLEVIATION OF CATASTROPHIC FORGETTING
The hyper-parameters used here, except γ in FedReg, are the same as the values used in the previous section.
• Computation of loss increment.
To fairly compare the forgetting issue in FedReg and FedAvg, γ is set to be 1 . 0 in FedReg, and in each round, the client parameters are initialized with the parameters obtained in FedReg to avoid a large discrepancy between the values of loss ( t − 1) computed in FedReg and FedAvg.
For a client i ∈ C ( t − 1) , the loss increase is computed as loss ( t ) ( i ) − loss ( t − 1) ( i ) .
• The computation of Fisher information.
Following (Kunstner et al., 2019), in round t of client i ∈ C ( t ) , for the previous training data D ( t − ) = ∪ j ∈ C ( t − )
D j , where C ( t − ) = ∪ s ∈
[ t − 10 ,t − 1] C ( s ) , the empirical Fisher information is computed as fisher
[Equation]
[Equation]
Similarly for the pseudo data D s i , the empirical Fisher information is computed as
[Equation]
Note that considering the generation process of the pseudo data, the parameter values θ ( t,i ) used to compute Fisher information are the same local model parameters after a local training step
B. P ROTECTION OF PRIVATE INFORMATION
The target of gradient inversion attacks in Geiping et al.
(2020) and Zhu et al.
(2019) is to find
[Equation]
where ∆ θ (cid:0) θ ( t ) , · (cid:1) denotes the amounts (updates) that the parameters have changed from θ ( t ) with the corresponding data, TV ( d ) the total variation of d , and d true a data point on the client.
The distance function could be cosine distance Geiping et al.
(2020) or L 2 -distance Zhu et al.
Note that the method proposed in Zhu et al.
(2019) sets w = 0 .
To consider the worst case, in our attacking experiments, the parameters are only trained for one step with each data point, in which case the values of ∆ θ (cid:0) θ ( t ) , d (cid:1) obtained in FedAvg, FedCurv, FedProx and SGD are exactly the same.
32,000 attacking iterations are used in all experiments and both distance functions ( i.e. , cosine and L 2 ) are considered.
Adam (Kingma & Ba, 2014) optimizer is used, which performs better than L-BFGS empirically.
The learning rate and the weight for the total variation are optimized in (the basic) FedAvg and applied to the other methods.
The learning rate decaying scheme follows the method provided in Geiping et al.
More details concerning each dataset are discussed below.
• EMNIST.
The images to be recovered are randomly sampled.
The model architecture and hyper-parameters used in FedAvg, FedProx, FedCurv, SGD, and FedReg are the same as the ones used in section B.2.
In the baseline methods with DPSGD, C = 1 . 0 and σ = 0 . 01 , and other hyper-parameters are the same as the ones used in the corresponding baseline methods (without DPSGD).
In FedReg with MG, the learning rate and the value of γ are the same as the ones used in FedReg, and η s = 0 .
In the attacking algorithm, the initial learning rate is 1 . 0 , the distance function is L 2 -distance, and the weight for the total variation term is 10 − 8 .
• CIFAR-10.
The images to be recovered are randomly sampled.
The model architecture and hyper-parameters used in FedAvg, FedProx, FedCurv, SGD, and FedReg are the same as the ones used in section B.2.
In the baseline methods with DPSGD, C = 1 . 0 and σ = 0 . 05 , and the other hyper-parameters are the same as the ones used in the corresponding baseline methods (without DPSGD).
In FedReg with MG, the learning rate and the value of γ are the same as the one used in FedReg, and η s = 0 .
In the attacking algorithm, the initial learning rate is 1 . 0 , the distance function is cosine distance, and the weight for the total variation term is 10 − 6 .
• CT images.
Only the images containing time information are used here.
As DenseNetis too complex to be attacked, we use a ten-layer neural network with 9 layers of CNNs and one layer of FC instead.
10 rounds are run in total and in each round, 10 clients are sampled.
In the local training stage, the local data are processed in 10 epochs.
The learning rate is 0 .
005 in FedAvg, FedProx and FedCurv, and 0 .
001 in SGD, respectively.
The weight for the proximal term in FedProx is 0 .
01 and λ in FedCurv is 0 .
In FedReg, the learning rate is 0 .
γ = 0 .
5 and η s = 10 − 6 .
In the baseline methods with DPSGD, C = 1 . 0 and σ = 0 .
001 , and other hyper-parameters are the same as the ones used in the corresponding methods (without DPSGD).
In FedReg with MG, the learning rate and the values of γ and η s are the same as those in FedReg.
In the attacking algorithm, the initial learning rate is 1 . 0 , the distance function is L 2 -distance, and the weight for the total variation term is 10 − 8 .
C S UPPLEMENTARY RESULTS
[Table]
[Table]
D A LGORITHM
Input : K , T , { D i } i ∈ C , θ (0) , γ , η θ , η s , η p , S , E
Output : { θ ( t ) }
Tt = 1: for t = 1 , 2 , ..., T do 2:
Randomly sample K clients C ( t ) 3: for i ∈ C ( t ) do 4: θ ( t,i ) ←
θ ( t − 1) , generate D si and D pi with η s , η p and E 5: for s = 1 , 2 , ..., S do 6:
Compute g γ ( D i ) , θ ( t,i ) ← θ ( t,i ) − η θ g γ ( D i ) , θ ( t,i ) β ← 0 .
5 (cid:0) θ ( t,i )
+ θ ( t − 1) (cid:1) 7:
Compute g β
( D si )
, g β ( D pi ) , w s and w p , θ ( t,i ) ← θ ( t,i ) − w s g
β
( D si ) − w p g β
( D pi ) 8: end for 9:
Send θ ( t,i ) to the server 10: end for 11: θ ( t ) ← 1 K (cid:80) i ∈ C ( t ) θ ( t,i ) 12: end for 13: return { θ ( t ) } Tt =