FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learning
Abstract
Federated learning (FL) has recently attracted increasing attention from academia and industry, with the ultimate goal of achieving collaborative training under privacy and communication constraints.
Existing iterative model averaging based
FL algorithms require a large number of communication rounds to obtain a wellperformed model due to extremely unbalanced and non-i.i.d data partitioning among different clients.
Thus, we propose FedDM to build the global training objective from multiple local surrogate functions, which enables the server to gain a more global view of the loss landscape.
In detail, we construct synthetic sets of data on each client to locally match the loss landscape from original data through distribution matching.
FedDM reduces communication rounds and improves model quality by transmitting more informative and smaller synthesized data compared with unwieldy model weights.
We conduct extensive experiments on three image classification datasets, and results show that our method can outperform other
FL counterparts in terms of efficiency and model performance.
Moreover, we demonstrate that FedDM can be adapted to preserve differential privacy with
Gaussian mechanism and train a better model under the same privacy budget.
Introduction
Traditional machine learning methods are designed with the assumption that all training data can be accessed from a central location.
However, due to the growing data size together with the model complexity [11, 25, 31], distributed optimization [9, 12, 14] is necessary over different machines.
This leads to the problem of Federated Learning [19] (FL) – multiple clients (e g. mobile devices or local organizations) collaboratively train a global model under the orchestration of a central server (e.g. service provider) while the training data are kept decentralized and private.
Such a practical setting poses two primary challenges [17, 19, 34, 42, 43]: training data of the FL system are highly unbalanced and non-i.i.d.
across downstream clients and more efficient communication with fewer costs is expected because of unreliable devices with limited transmission bandwidth.
Most of the existing FL methods [19, 32, 34, 38, 50] adopt an iterative training procedure from
FedAvg [19], in which each round takes the following steps: 1)
The global model is synchronized with a selected subset of clients; 2) Each client trains the model locally and sends its weight or gradient back to the server; 3)
The server updates the global model by aggregating messages from selected ∗ Equal contribution.
clients.
This framework works effectively for generic distributed optimization while the difficult and challenging setting of FL, unbalanced data partition in particular, would result in statistical heterogeneity in the whole system [33] and make the gradient from each client inconsistent.
It poses a great challenge to the training of the shared model, which requires a substantial number of communication rounds to converge [50].
Although some improvements have been made over
FedAvg [19] including modifying loss functions [34], correcting client-shift with control variates [32] and the like, the reduced number of communication round is still considerable and even the amount of information required by the server rises [40].
In our paper, we propose a different iterative surrogate minimization based method, FedDM , referred to Fed erated Learning with iterative D istribution M atching.
Instead of the commonly-used scheme where each client maintains a locally trained model respectively and sends its gradient/weight to the server for aggregation, we take a distinct perspective at the client’s side and attempt to build a local surrogate function to approximate the local training objective.
By sending those local surrogate functions to the server, the server can then build a global surrogate function around the current solution and conduct the update by minimizing this surrogate.
The question is then how to build local surrogate functions that are informative and with a relative succinct representation.
Inspired by recent progresses in data condensation [39, 49] we build local surrogate functions by learning a synthetic dataset to replace the original one to approximate the objective.
It can be achieved by matching the original data distribution in the embedding space with the maximum mean discrepancy measurement (MMD) [10].
After the optimization of synthesized data, the client can transmit them to the server, which can then leverage the synthetic dataset to recover the global objective function for training.
Our method enables the server to have implicit access to the global objective defined by the whole balanced dataset from all clients, and thus outperforms previous algorithms involved in training a local model with unbalanced data in terms of communication efficiency and effectiveness.
We also demonstrate that our method can be adapted to preserve differential privacy under a modest budget, an important factor to the deployment of FL systems.
Our contributions are primarily summarized as follows:
• We propose FedDM, which is based on iterative distribution matching to learn a surrogate function.
It sends synthesized data to the server rather than commonly-used local model updates and improves communication efficiency and effectiveness significantly.
• We further analyze how to protect privacy of client’s data for our method and show that it is able to guarantee ( ϵ, δ ) -differential privacy with the Gaussian mechanism and train a better model under the same privacy budget.
• We conduct comprehensive experiments on three tasks and demonstrate that FedDM is better than its FL counterparts in communication efficiency as well as the final model performance.
Methodology
In this part, we first present the iterative surrogate minimization framework in Section 2.1, and then expand on the details of our implementation of FedDM in Section 2.2.
In addition, we discuss preserving differential privacy of our method through Gaussian mechanism in Section D.
2.1 Iterative surrogate minimization framework for federated learning
Neural network training can be formulated as solving the following finite sum minimization problem:
[Equation]
where w ∈ R d is the parameter to be optimized, D is the dataset and ℓ ( x i , y i ; w ) is the loss of the prediction on sample ( x i , y i ) ∈ D w.r.t.
w such as cross entropy.
We will abbreviate these terms as f ( w ) and ℓ i ( w ) for simplicity.
Eq.
equation 1 is typically solved by stochastic optimizers when training data are gathered in a single machine.
However, the scenario is different under the setting of federated learning with K clients.
In detail, each client k has access to its local dataset of the size n k with the set of indices
I k ( n k = |I
k | ), and we can rewrite the objective as f ( w ) =
K (cid:88) k
= n k n f k ( w ) where f k ( w ) = 1 n k (cid:88)
i ∈I k l i ( w ) .
(2)      6 Weight
L o ss
OriginalSurrogateTangent
Since information can only be communicated between the server and clients, previous methods [19, 32, 34, 38] train the global model by aggregation of local model updates, as introduced in Section 1.
However, as each client only sees local data which could be biased and skewed, the local updates is often insufficient to capture the global information.
Further, since local weight update consists limited information, it is hard for the server to obtain better joint update direction by considering higher order interactions between different clients.
We are motivated to leverage the surrogate function by the example in Figure 1.
Specifically, we synthesize a 1-D binary classification problem and learn a surrogate for the objective function.
We learn the surrogate function via distribution matching introduced in Section 2.2 around the weight of 0 .
Compared with the tangent line computed by the gradient, the surrogate function in orange matches the original one accurately and minimizing it leads to a satisfactory solution.
More details can be checked in Appendix C.
Thus, we hope to develop a novel scheme such that each client can send a local surrogate function instead of a single gradient or weight update to the server, so the server has a more global view to loss landscape to obtain a better update instead of pure averaging.
To achieve this goal, we propose to conduct federated learning with an iterative surrogate minimization framework.
At each communication round, let w r be the current solution, we build a surrogate training objective ˆ f r ( · ) to approximate the original training objective in the local area around w r , and then update the model by minimizing the local surrogate function.
The update rule can be written as w r +1 =
[Equation]
w ∈ B ρ ( w r )
ˆ f r ( w ) , where ˆ f r ( w )
[Equation]
f ( w ) , ∀ w
[Equation]
B ρ ( w r ) .
[Equation]
B ρ ( w r ) is a ρ -radius ball around w r .
Note that we do not expect to build a good surrogate function in the entire parameter space; instead, we only construct it near w r and obtain the update by minimizing the surrogate function within this space.
In fact, many optimization algorithms can be described under this framework.
For instance, if ˆ f r ( w ) = ∇ f ( w r ) T ( w − w r ) (based on the first-order Taylor expansion), then Eq.
equation 3 leads to the gradient descent update where ρ controls the step size.
To apply this framework in the federated learning setting, we consider the decomposition of Eq.
equation 2 and try to build surrogate functions to approximate each f k ( w ) on each client.
More specifically, each client aims to find
[Equation]
and send the local surrogate function ˆ f r,k ( · ) instead of gradient or weights to the server.
The server then form the aggregated surrogate function
[Equation]
and then use Eq.
equation 3 to obtain the update.
Again, if each ˆ f r,k is the Taylor expansion based on local data, it is sufficient for the client to send local gradient to the server, and the update will be equivalent to (large batch) gradient descent.
However, we will show that there exists other ways to build local approximations to make federated learning more communication efficient.
2.2 Local distribution matching
Inspired by recent progresses in data distillation [35, 39, 44, 48, 49], it is possible to learn a set of synthesized data for each client to represent original data in terms of the objective function.
Therefore, we propose to build local surrogate models based on the following approximation for the r -th round:
[Equation]
where S denotes the set of synthesized data and I S k is the corresponding set of indices.
Note that we aim to approximate f k only in a local region around w r instead of finding the approximation globally, which is hard as demonstrated in [39, 49].
To form the approximation function in Eq.
equation 6, we solve the following minimization problem:
[Equation]
where w is sampled from distribution P w , which is a Gaussian distribution truncated at radius ρ .
A different perspective at Eq.
equation 7 is that we can just match the distribution between the real data and synthesized ones given f k ( w ) and ˆ f r,k ( S ; w ) are just empirical risks.
A common way to achieve this is to estimate the real data distribution in the latent space with a lower dimension by maximum mean discrepancy (MMD) [49, 51]:
[Equation]
Here h w is the embedding function that maps the input into the hidden representation.
We use the empirical estimate of MMD in [49] since the underlying data distribution is inaccessible.
Furthermore, to make our approximation more accurate and effective, we match the outputs of the logit layer which corresponds exactly with the Eq.
equation 7, together with the preceding embedding layer:
[Equation]
where h w ( x ) again denotes intermediate features of the input while z w ( x ) ∈ R C represents the output of the final logit layer.
It should be emphasized that we learn synthesized data for each class respectively, which means samples in D and S belong to the same class.
For training, we adopt mini-batch based optimizers to make it more efficiently.
Specifically, a batch of real data and a batch of synthetic data are sampled randomly for each class independently by B D k c ∼ D k and B S k c ∼ S k .
We plug these two batches into Eq.
equation 9 to compute L c and L = (cid:80) C − 1 c =
L c .
S k can be updated with SGD by minimizing L for each client.
Then we aggregate all synthesized data from K clients at the server’s side:
[Equation]
Moreover, since synthesized data are trained based on a specific distribution around the current value of w , we need to iteratively synchronize the global weights with all the clients and obtain proper S according to the latest w for the next communication round.
Therefore, instead of transmitting parameters or gradients in previous FL algorithms, we propose federated learning with iterative distribution matching (FedDM) in Algorithm 1 following the steps below to train the global model: (a) At each communication round, for each client, we adopt Eq.
equation 9 as the objective function to train synthesized data for each class; (b) The server receives synthesized data and leverages them to update the global model; (c) The current weight is then synchronized with all the clients and a new communication rounds start by repeating step (a) and (b).
It should be noticed that through estimating the local objective, FedDM extracts richer information than existing model averaging based methods, and enables the server to explore the loss landscape from a more global view.
It reduces communication rounds significantly.
On the other hand, the explicit message uploaded to the server, or the number of float parameters, is relatively smaller.
This is especially true when training large neural network models, where the size of neural network parameters (and therefore gradient update) is much larger than the size of the input.
Take CIFAR as an example, when training data are distributed obeying Dir 10 (0 . 5) , the average number of classes per client (cpc) is 9 .
When we adopt the number of images per class (ipc) of 10 for the synthetic set, the total number of float parameters uploaded to the server is: the number of clients × cpc × ipc × image size = 10 × 9 × 10 × 3 × 32 × 32 ≈ 2 . 8 × 10 6 .
For those iterative model averaging model methods, the number of float parameters is equal to the product of weight size and the number of clients, which is 320010 × 10 ≈ 3 . 2 × 10 6 for ConvNet [39] and comparably larger than FedDM.
An extensive comparison is presented in Appendix E.
Experiments 3.1 Experimental setup
Datasets.
In this paper, we focus on image classification tasks, and select three commonly-used datasets: MNIST [1], CIFAR10 [5], and CIFAR100 [5].
We adopt the standard training and testing split.
Following commonly-used scheme [28], we simulate non-i.i.d.
data partitioning with Dirichlet distribution Dir K ( α ) , where K is the number of clients and α determines the non-i.i.d.
level, and allocate divided subsets to clients respectively.
A smaller value of α leads to more unbalanced data distribution.
The default data partitioning is based on Dir 10 (0 . 5) with 10 clients.
Furthermore, we also take into account different scenarios of data distribution, including Dir 10 (0 . 1) , Dir 10 (0 . 01) .
Results of Dir 50 (0 . 5) and Dir 10 (50) (i.i.d. scenario) can be found in Appendix G.2.
Baseline methods.
We compare FedDM with four representative iterative model averaging based methods: FedAvg [19], FedProx [34], FedNova [38], and SCAFFOLD [32].
We summarize the action of the client and the server, together with the transmitted message for all methods in Table 3.
3.2 Communication efficiency and convergence rate
We first evaluate our method in terms of communication efficiency and convergence rate on all three datasets on the default data partitioning Dir 10 (0 . 5) .
As we can see in Figure 2(a)-(c), our method
FedDM performs the best among all considered algorithms by a large margin on MNIST, CIFAR10, and CIFAR100.
Specifically, for CIFAR10, FedDM achieves 69 .
66 ± 0 .
13% on test accuracy while the best baseline SCAFFOLD only has 66 .
12 ± 0 .
17% after 20 communication rounds.
FedDM also has the best convergence rate and it significantly outperforms baseline methods within the initial few rounds.
Advantages of FedDM are more evident when we evaluate convergence as a function of the message size.
As mentioned in 2.2, FedDM requires less information per round.
Therefore, we can observe in Figure 2(d)-(e) that FedDM converges the fastest along with the message size.
Details of the message size of each method for different tasks are provided in Appendix E
[Figure]
In real-world applications, there are various extreme data distributions among clients.
To synthesize such non-i.i.d.
partitioning, we consider two more scenarios with Dir 10 (0 . 1) and Dir 10 (0 . 01) .
As mentioned, α → 0 implies each client holds examples from only one random class.
It can be seen in Table 1 that previous methods based on iterative model averaging are insufficient to handle these two challenging scenarios and their performance degrades drastically compared with Dir 10 (0 . 5) .
In contrast, FedDM performs consistently better and more robustly, since distribution matching enables it to approximate the global training objective more accurately.
[Table]
3.4 Analysis of FedDM
We analyze FedDM to investigate effects of hyperparameters such as ipc and network structure.
Besides
[Table]
we compare our method with a strong baseline of sending real images with the same size.
More extensive results are reported in Appendix G including visualization of learned synthetic data.
Effects of ipc.
Experiments are conducted on CIFAR10 with the distribution Dir 10 (0 . 5) with three different ipc values from [3 , 5 , 10] .
As the ipc increases, the performance gradually get better from 53 .
64 ± 0 .
24 ± 0 .
04% to 69 .
62 ± 0 .
However, in the meanwhile, more images per class indicates a heavier communication burden.
We need to trade off the model performance against the communication cost, and thus choose an appropriate ipc value based on the task.
[Figure]
T e s t
A cc u r a c y ( % )
CIFAR
FedAvgFedProxFedNovaSCAFFOLDFedDM
Different network structures.
Besides ConvNet, we evaluate FedDM under the default CIFAR10 setting on ResNet18.
It can be observed that our method works well even for this more complicated and larger model in Figure 3.
It should also be emphasized that for FL baseline methods, they have to transmit a larger amount of message while
FedDM maintains the original size.
This makes FedDM more efficient in larger networks.
Comparison with transmitting real images.
Our method is compared with REAL, which sends real images of the same size as FedDM (ipc = 10 ).
In particular, REAL achieves test accuracy of 68 . 66 ± 0 .
08% on CIFAR10 with the default setting, but cannot beat
FedDM with 69 .
62 ± 0 .
It indicates that our learned synthetic set can capture richer information of the whole dataset rather than just a few images.
Conclusions and limitations
In this paper we propose an iterative distribution matching based method, FedDM to achieve more communication-efficient federated learning.
By learning a synthetic dataset for each client to approximate the local objective function, the server can obtain the global view of the loss landscape better than just aggregating local model updates.
We also show that FedDM can preserve differential privacy with Gaussian mechanism.
However, there is still a trade-off between the size of the synthetic set and the final performance, especially for classification tasks with hundreds of clients or classes.
How to reduce the synthetic set to save communication costs can be a potential future direction.
A Related work
Federated Learning.
Federated learning [19, 42] has aroused heated discussion nowadays from both research and applied areas.
With the goal to train the model collaboratively, it incorporates the principles of focused data collection and minimization [42].
FedAvg [19] was proposed along with the concept of FL as the first effective method to train the global model under the coordination of multiple devices.
Since it is based on iterative model averaging, FedAvg suffers from heterogeneity in the FL system, especially the non-i i d. data partitioning which degrades the performance of the global model and adds to the burden of communication [33].
To mitigate the issue, some variants have been developed upon FedAvg including [32, 34, 38].
For instance, FedProx [34] modifies the loss function while FedNova [38] and SCAFFOLD [32] leverage auxiliary information to balance the distribution shift.
Apart from better learning algorithms with faster convergence rate, another perspective at improving efficiency is to reduce communication costs explicitly [23, 26, 29, 36, 41].
An intuitive approach is to quantize and sparsify the uploaded weights directly [36].
Efforts have also been made towards one-shot federated learning [24, 27, 40, 46], expecting to obtain a satisfactory model through only one communication round.
Differential Privacy.
To measure and quantify information disclosure about individuals, researchers usually adopt the state-of-the-art model, differential privacy (DP) [3, 7, 13].
DP describes the patterns of groups while withholding information about individuals in the dataset.
There are many scenarios in which DP guarantee is necessary [2, 4, 15, 21, 45].
For example, Abadi et.al [15] developed differentially private SGD (DP-SGD) which enabled training deep neural networks with non-convex objectives under a certain privacy budget.
It was further extended to settings of federated learning, where various techniques have been designed to guarantee user-level or local differential privacy [18, 20].
Recently, DP has been taken into consideration for hyperparameter tuning [45].
Dataset Distillation.
With the explosive growing of the size of training data, it becomes much more challenging and costly to acquire large datasets and train a neural network within moderate time [35, 44].
Thus, constructing smaller but still informative datasets is of vital importance.
The traditional way to reduce the size is through coreset selection [8, 30], which select samples based on particular heuristic criteria.
However, this kind of method has to deal with a trade-off between performance and data size [35, 39].
To improve the expressiveness of the smaller dataset, recent approaches consider learning a synthetic set of data from the original set, or data distillation for simplicity.
Along this line, different methods are proposed using meta-learning [22, 47], gradient matching [39, 48], distribution matching [49, 51], neural kernels [35, 44] or generative models [37].
B FedDM Algorithm
C Synthetic Binary Classification
We design a synthetic 1-D binary classification problem to better illustrate the advantage of learning a surrogate function for the training objective.
Specifically, we construct a dataset D s = { ( x i , y i ) | i = 1 , . . . , n } with n = 100 synthetic pairs in the following way:
[Equation]
p i <
[Equation]
[Equation] [Equation]
where p i is a random value sampled from Uniform (0 , 1) .
A prediction is made by ˆ y i
=
Sigmoid ( wx i ) with the weight w as the trainable parameter.
We use the binary cross entropy as the training objective:
[Equation]
Then we use n ′ = 20 randomly initialized examples { (˜ x j , ˜ y j ) | j = 1 , . .
, 20 } to match the objective around w = 0 as introduced in Section 2.2.
We plot the original objective, the surrogate function, and the tangent line at w = 0 obtained by the gradient in Figure 1.
D Differential privacy of FedDM
An important factor to evaluate a federated learning algorithm is whether it can preserve differential privacy.
Before analyzing our method, we first review some fundamentals of differential privacy.
Definition D.1 ( Differential Privacy [2] ) .
A randomized mechanism M : D → R with domain
D and range R satisfies ( ϵ, δ ) -differential privacy if for any two adjacent datasets D 1 , D 2 and any measurable subset S ⊆ R ,
[Equation]
In this paper, we focus on instance-level differential privacy, which indicates that D 1 and D 2 differ on a single element.
Typically, the randomized mechanism is applied to a query function of the dataset, f : D → X .
Without loss of generality, we assume that the output spaces R , X ⊆
R m .
A key quantity in characterizing differential privacy for various mechanisms is the sensitivity of a query [13] f : D → R m in a given norm ℓ p .
Formally this is defined as
[Equation]
Gaussian mechanism [13] is one simple and effective method to achieve ( ϵ, δ ) -differential privacy:
[Equation]
where Z
[Equation] [Equation]
It has been proved that under Gaussian mechanism, ( ϵ, δ ) -differential privacy is satisfied for the function f of sensitivity ∆ p if we choose σ ≥ (cid:113) 2 log 1 .
25 δ /ϵ
Differentially private SGD (DPSGD) [15] then applies Gaussian mechanism to deep learning optimization with hundreds of steps and demonstrates the following theorem:
Theorem D.1 ( Differential Privacy of DP-SGD ) .
There exist constants c 1 and c 2 so that given the sampling probability q and the number of steps T , for any ϵ < c 1 q 2 T , DP-SGD is ( ϵ, δ ) -differentially private for any δ > 0 if
[Equation]
We then prove that by leveraging DP-SGD to update the synthetic dataset which is initialized from random Gaussian noise, FedDM can preserve differential privacy of the original dataset.
We present this DP guarantee of FedDM in the theorem below:
Theorem D.2 ( Differential privacy of FedDM. ) .
Given the synthetic dataset S is initialized from random noise, FedDM trained with DP-SGD can guarantee ( ϵ, δ ) -differential privacy in a K-client federated learning system, with σ ≥ (cid:113) log( δ )
Tq 2 − ϵ or σ ≥ (cid:113) 2log(1 /σ )
ϵ
if Tq 2 ≤ ϵ/ 2 in each round.
The complete proof of Theorem D.2 is presented below.
According to Theorem D.1, using DP-SGD to train a neural network can protect differential privacy.
Back to FedDM, we first look at each client separately to investigate its differential privacy.
We show that the gradient of L c in equation 9 can be written as the average of individual gradients for each real example,
[Equation]
[Equation]
where ˜ g ( x i ) is the modified gradient for x i .
Recall the equation of L c , we have
[Equation]
L c are divided into two similar parts, L c,h and L c,z .
Then we take a look at the gradient of L c,h with respect to S k below:
[Equation]
Similarly, we have
[Equation]
Then the final gradient of L c is
[Equation]
It indicates that the synthetic set can be regarded as an equivalence to the network parameter in
DP-SGD, and leads to the conclusion that for each client k , Theorem D.1 holds during optimizing S k .
Furthermore, since the synthetic dataset is initialized from random noise, it would not leak privacy at the beginning of the optimization.
Next, to extend DP guarantee to a system with K clients, we use parallel composition [6]:
Theorem D.3 ( Parallel Composition [6] ) .
If there are K mechanisms M 1 , . .
, M K computed on disjoint subsets whose privacy guarantees are ( ϵ 1 , δ 1 ) , . . .
, ( ϵ K , δ K ) respectively, then any function of M 1 , . . .
, M K is (max i ϵ
i , max i δ i )
-differential private.
We can see that different clients maintain their own local datasets, which satisfies disjoint property.
Then this Gaussian mechanism is still ( ϵ, δ ) -differentially private for the whole system if each client satisfies ( ϵ, δ ) -differential privacy.
In addition, to quantify how much noise is required for each client, we can make use of the Tail bound in [15]:
[Equation]
Based on [15], α M ( λ ) ≤
Tq 2 λ 2 /σ 2 , without loss of generality, set λ
= σ 2 , it holds that δ ≤ exp( Tq 2 σ 2 − ϵσ 2 ) , and
σ ≥ (cid:113) log( δ )
Tq 2 − ϵ .
When Tq 2 ≤ ϵ/ 2 , we have σ ≥ (cid:113) 2log(1 /σ )
ϵ .
To guarantee differential privacy when leveraging the synthetic dataset for downstream tasks, we introduce the property of post-processing below:
Lemma D.1 ( Robustness to post-processing [13] ) .
Let M : D → R be a randomized mechanism that is ( ϵ, δ ) -differentially private.
If F : R → R ′ be an arbitrary deterministic or randomized mapping, F ( M ) is also ( ϵ, δ ) -differentially private.
Training a network on the synthetic dataset S is a post-processing operation, and Lemma D.1 ensures that any post-processing computation on S is differentially private as long as the generation of S satisfies ( ϵ, δ ) -differential privacy.
This property has also been deployed in similar applications such as [52], where the synthetic text generated from a DP-trained language model was used in various downstream tasks and still preserved privacy.
Finally, with Theorem D.1, Theorem D.3 and Lemma D.1, we complete the proof of Theorem D.2.
The whole procedure of FedDM integrated with DP-SGD is described in Algorithm 2.
Obtain the clipped gradient: ∇ S k L c ← ∇ S k L c / max (cid:16) 1 , ∥∇ S k L c ∥ 2 C (cid:17)
Add Gaussian noise: ∇ S k L c ← ∇ S k L c + 1 | B D k c | N (0 , σ 2 C 2 I ) Update S k ← S k − η c ∇ S k L
E Message size of different FL methods
In this section, we provide specific message size under different data partitioning of FedDM.
As discussed previously, the message size of all baseline methods are determined on the model size, while the message size varies from different scenarios and ipc values.
When there are 10 clients, we set ipc=10 for MNIST and CIFAR10, and ipc=5 for CIFAR100.
We present the results in Table 2.
It can be observed that FedDM are more advantageous for unbalanced data partitioning, such as Dir 10 (0 . 1) and Dir 10 (0 . 01) .
For the experiment of Dir 50 (0 . 5) on CIFAR10 with ConvNet, the message sizes of
FedDM and baselines are 3 .
1 × 10 6 and 1 . 6 × 10 7 respectively, where our method saves about 80% costs per round.
Moreover, if the underlying model are changed to ResNet-18 for Dir 10 (0 . 5) , then the number of parameters is about 1 .
1 × 10 8 .
F Implementation Details
Hyperparameters.
For FedDM, following [49], we select the batch size as 256 for real images, and update the synthetic set S k for T = 1 , 000 iterations with η c = 1 for each client in each communication round, and tune the number of images per class (ipc) within the range [3 , 5 , 10] .
Synthetic images are initialized as randomly sampled real images with corresponding labels suggested by [39, 49].
Considering the trade-off between communication efficiency and model performance,
[Table] [Table]
we choose ipc to be 10 for MNIST and CIFAR10, 5 for CIFAR100 when there are 10 clients.
The choice of radius ρ = 5 is discussed in Appendix G.4.
On the server’s side, the global model is trained with the batch size 256 for 500 epochs by SGD of η s = 0 .
For baseline methods 2 , we choose the same batch size of 256 for local training
[Table]
and tune the learning rate of SGD from [0 . 001 , 0 . 01 , 0 . 1] and local epoch from [5 , 10 , 15 , 20] .
In particular, we tune µ for FedProx in [0 . 01 , 0 . 1 , 1] .
For a fair comparison, all methods share the fixed number of communication rounds as
[Table]
and the same model structure ConvNet [39] by default.
A different network ResNet-18 [16] is evaluated as well.
All experiments are run for three times with different random seeds with one NVIDIA 2080Ti
GPU and the average performance is reported in the paper.
G Additional Experimental Results G.1 Performance with DP guarantee
As discussed in Section D, if the synthetic dataset is initialized from random noise, using DP-SGD in local training of FedDM can satisfy ( ϵ, δ ) -differential privacy, with σ ≥ (cid:113) 2log(1 /δ )
ϵ for any
Tq 2 ≤ ϵ/ 2 .
Such a mechanism also works on baseline methods with the same DP guarantee.
Therefore, our evaluation scheme just adopts the same level of Gaussian noise in DP-SGD given the specific budget ( ϵ, δ ) and then compares performance of different algorithms.
Specifically, we follow
FedDM with DP-SGD in Algorithm 2 with three noise levels from small ( σ = 1 ), medium ( σ = 3 ), to large ( σ = 5 ), and the gradient norm bound C = 5 .
S is initialized from N (0 , 1) to guarantee differential privacy.
We notice in Figure 4 that under the same DP guarantee, FedDM outperforms other FL counterparts in terms of convergence rate and final performance.
Moreover, the accuracy of FedDM does not drop significantly compared with all considered methods when the noise level increases from σ = 1 to σ = 5 , indicating that FedDM is most resistant to the perturbed optimization.
[Figure]
G.2 Learning curves of FL methods
We show a complete set of learning curves for all of our experiments.
Different data partitioning.
Here we present curves for different data partitioning.
We observe that FedDM still outperforms all other baselines under scenarios of Dir 10 (50) in Figure 8 which is almost an i
[Figure]
i
[Figure]
d. data partitioning, and Dir 50 (0 . 5) in Figure 9 which has more clients.
[Figure] [Figure]
A cc u r a c y (
[Figure]
A
[Figure]
r a c y
[Figure]
[Figure] [Figure]
[Figure] [Figure]
• Dir 50 (0 . 5)
[Figure]
Different noise levels.
Figure 10 displays learning curves of different σ .
[Figure]
Effects of ipc.
We show test accuracy curves to analyze effects of ipc in Figure
[Figure]
Performance on ResNet-18.
Detailed learning curves of test accuracy along with rounds and message size are shown in Figure
[Figure]
Transmitting real data.
We present a comparison with sending real images (REAL) in Figure 13.
[Figure]
G.3 Visualization of the synthetic dataset
By randomly picking a client under data partitioning of Dir 10 (0 . 5) , we provide the visualization of our synthetic dataset under different noise levels in Figure 14, 15, 16, and 17.
It can be observed clearly that even when there is no noise added to the gradient during optimization of the synthetic dataset, those images are still illegible from their original classes in Figure 14.
Furthermore, as σ increases, synthesized data become harder to recognize, which protects the client’s privacy successfully.
G.4 ρ -radius ball
It has been discussed in Section 2.1 that B ρ ( w r ) is a ρ -radius ball around w r .
Specifically,
[Equation]
In FedDM, we sample w based on a truncated Gaussian distribution below:
[Equation]
where we clip the sampled weight to guarantee that ∥ w − w r ∥ 2 ≤ ρ .
At the server’s side, when training the global model, we also clip the weight to the ρ -radius ball.
We conduct experiments to choose ρ from [3 , 5 , 10] and present the test accuracy after 20 communication rounds on CIFAR under the default Dir 10 (0 . 5) setting in Table 4.
We find that performance is similar and FedDM is not very sensitive to the choice of ρ .
ρ = 5 performs relatively the best
and we hypothesize that too small weight makes the optimization of the global model restricted and too big one adds to the difficulty of learning a surrogate function.
Based on results in Table 4, we select ρ = 5 for all our experiments.
[Table]