M ETA K NOWLEDGE C ONDENSATION FOR F EDERATED L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Existing federated learning paradigms usually extensively exchange distributed models at a central solver to achieve a more powerful model.
However, this would incur severe communication burden between a server and multiple clients especially when data distributions are heterogeneous.
As a result, current federated learning methods often require a large number of communication rounds in training.
Unlike existing paradigms, we introduce an alternative perspective to significantly decrease the communication cost in federate learning.
In this work, we first introduce a meta knowledge representation method that extracts meta knowledge from distributed clients.
The extracted meta knowledge encodes essential information that can be used to improve the current model.
As the training progresses, the contributions of training samples to a federated model also vary.
Thus, we introduce a dynamic weight assignment mechanism that enables samples to contribute adaptively to the current model update.
Then, informative meta knowledge from all active clients is sent to the server for model update.
Training a model on the combined meta knowledge without exposing original data among different clients can significantly mitigate the heterogeneity issues.
Moreover, to further ameliorate data heterogeneity, we also exchange meta knowledge among clients as conditional initialization for local meta knowledge extraction.
Extensive experiments demonstrate the effectiveness and efficiency of our proposed method.
Remarkably, our method outperforms the state-of-the-art by a large margin (from 74 .
07% to 92 .
95% ) on MNIST with a restricted communication budget ( i.e. , 10 rounds).
I NTRODUCTION
Most deep learning-based models are trained in a data-centralized manner.
However, in some cases, data might be distributed among different clients and cannot be shared.
To address this issue,
Federated Learning (FL) (Yang et al., 2019b;a; Kairouz et al., 2021) has been proposed to learn a powerful model without sharing private original data among clients.
In general, most prior FL works often require frequent model communications to exchange models between local clients and a global server, resulting in heavy communications burden caused by the decentralized data (Wu & Wang, 2021; Chencheng et al., 2022).
Therefore, it is highly desirable to obtain a powerful federated model with only a few communication rounds.
In this work, we propose a new meta knowledge-driven federated learning paradigm to achieve an effective yet communication-efficient model, thus significantly reducing communication costs.
Unlike prior works, we formulate federated learning in a new perspective, where representative information will be distilled from original data and sent to the server for model training.
On the client side, we extract representative information of original data and condense it into a tiny set of highly-compressed synthetic data, namely meta knowledge .
Furthermore, we develop two mechanisms, i.e. , dynamic weight assignment and meta knowledge sharing, in the condensation process to mitigate the data heterogeneity issue widely existing in decentralized data.
On the server side, we train our global model with meta knowledge uploaded from clients rather than simply averaging client models.
Specifically, we firstly distill the task-specific knowledge from private data on local clients and condense it as meta knowledge.
The meta knowledge condensation process is modeled as a bi-level optimization procedure under the federated learning setting: the inner-loop minimizes the training loss on meta knowledge to update a model; and the outer-loop minimizes the training loss on original data to update meta knowledge based on the updated model.
In the optimization process, we assign dynamic weights to each sample based on its training loss.
By dynamically adjusting the weight of each sample in training, we empower each sample to contribute adaptively to the current model.
Besides, to further mitigate heterogeneous data distributions among different clients, we design a meta knowledge sharing mechanism.
The meta knowledge sharing mechanism provides conditional initialization to our meta knowledge optimization procedure, thus promoting knowledge exchange among clients.
Since meta knowledge from various clients has been uploaded to the server, our model can be trained with such knowledge that better describes the overall distribution.
In contrast, previous methods that average local models on the server do not have such a merit.
Additionally, we impose a learnable conditional generator on the central model training to improve the training stability.
Specifically, the generator models the statistical distribution of the uploaded meta knowledge.
We generate synthetic samples by the modeled distribution, providing historical information to model update.
Note that, as meta knowledge contains the essential information of the original data as well as the corresponding class information, it can be used as normal training data for model training.
Then, our global model is trained with the uploaded and generated meta knowledge on the server side.
Consequently, we significantly reduce the impacts of data heterogeneity and decrease the communication rounds.
We conduct extensive experiments on various datasets, including MNIST (LeCun et al., 2010), SVHN (Netzer et al., 2011), CIFAR10 (Krizhevsky & Hinton, 2009), and CIFAR100 (Krizhevsky & Hinton, 2009).
The experimental results demonstrate the effectiveness and efficiency of our proposed method.
Specifically, our method outperforms competing works by a remarkable margin on all the datasets, especially when facing limited communication budgets ( i.e. , 10 communication rounds).
Overall, our key contributions are summarized as follows:
• We propose a new meta knowledge driven federated learning paradigm.
We transmit highly condensed meta knowledge extracted from various local clients to the global server for model training, thus significantly mitigating data heterogeneity issues in federated learning.
• We design a federated meta knowledge extraction method that can effectively encode local data for global model training.
Moreover, we propose a dynamic weight assignment scheme to promote informativeness of meta knowledge, and a knowledge sharing strategy to exchange meta knowledge among clients without leaking original data.
• We introduce a server-side conditional generator that models the statistical distribution of uploaded meta knowledge to stabilize the training process.
Benefiting from the extracted meta knowledge and learned statistical distribution, our model requires fewer communication rounds compared to competing methods while achieving superior performance.
M ETHODOLOGY 2. P ROBLEM D EFINITION
Federated Learning (FL) trains a model across a set of decentralized devices, i.e. , a set of clients and a server.
Suppose there is data D distributed on C clients, each of which has a local private training dataset D c
= { x c i , y c i } , 1 ≤ i ≤ n c and a weight value p c .
It is noted that D = ∪D c and C (cid:80) c
=1 p c = 1 .
Without loss of generality, we discuss a multi-class classification problem under a federated learning setting.
The learning target is formulated as follows:
[Equation]
where L c ( · ) is a local objective function that is optimized on the c - th client.
The loss function L c ( · ) is formulated as follows:
[Equation]
where ℓ ( · ) is a user-defined loss function ( e.g. , cross-entropy loss), and w denotes model parameters.
As most FL algorithms need to exchange locally trained models multiple times, communication burden is often non-negligible.
Though one-shot FL (Zhou et al., 2020) has been proposed to reduce communication cost, it suffers performance degradation.
2. F EDERATED L EARNING VIA M ETA K NOWLEDGE
We propose a new FL paradigm to solve the aforementioned limitations.
Specially, as shown in
Figure 1, our method conducts federated meta knowledge extraction (FMKE) on local clients and server-side central model training (CMT).
To mitigate the data heterogeneity issue in FMKE, we design two mechanisms, i.e. , dynamic weight assignment, and meta knowledge sharing.
To stabilize the central model training, we introduce a learnable constraint modeled by a conditional generator.
The technical details are provided in the following subsections.
2.2. F EDERATED M ETA K NOWLEDGE E XTRACTION ON C LIENTS
We design Federated Meta Knowledge Extraction (FMKE) to extract key information from decentralized data.
In the decentralized scenario, the original data D is distributed on a set of clients.
Each client has its private data D c ( D = ∪D c , 1 ≤ c ≤ C , where C is the client number), and a model downloaded from a server.
For simplifying the following discussion, we denote the downloaded model on the client c as w c .
On each active local client c , FKME distills key information from corresponding local private data
D c .
The distilled information is condensed as meta knowledge ˆ D c , which will be used to replace the original data D c in global model training on the server 1 .
The cardinality of ˆ D , i.e. , the size of extracted meta knowledge from all active clients, is much less than that of D .
The condensed meta knowledge ˆ D is highly compressed and representative.
To make the models trained on the original data and the meta knowledge achieve the comparable performance, we formulate the target function of FMKE as 2 :
[Equation] where η denotes a learning rate for updating w c by stochastic gradient descent (SGD). To solve the target function, we employ a bi-level optimization solution (Rajeswaran et al., 2019; Wang et al., 2018).
To be specific, the meta knowledge extraction is formulated as a nested optimization process: in the inner-loop, based on an initialized meta knowledge, a model is updated to minimize the training loss over the meta knowledge; in the outer-loop, given the updated model, the meta knowledge is renewed by minimizing the training loss over the original data.
The loopy formulation of our bi-level optimization process on client c is:
[Equation]
where L c ( · , D c ) denotes a loss function over the original data D c on client c , L c ( · , ˆ D c ) denotes a loss function over the meta knowledge ˆ D c on client c .
The inner-loop and outer-loop are implemented alternatingly and stochastic gradient descent is employed to update the model and meta knowledge.
At first, on an active client c , we update the model parameter by the following formulation:
[Equation]
where η denotes a learning rate for the inner-loop.
An updated model w ∗ is obtained in this inner-loop.
Secondly, given the updated model w ∗ , we evaluate it on original data D c and calculate the loss
L c ( w ∗ , D c ) .
Then the condensed meta knowledge ˆ D c can be updated by:
[Equation]
where α denotes a learning rate for the outer-loop, and the meta knowledge ˆ D c is initialized based on a uniform distribution ( i.e. , ˆ D cini ∼ U [ − 1 , +1] ).
An updated meta knowledge ˆ D c is obtained in this outer-loop.
The inner-loop and outer-loop are conducted in an alternatingly manner.
Due to the data heterogeneity issue widely existing in FL, the extracted meta knowledge on each client might be biased.
To mitigate the heterogeneity issue, we design two effective mechanisms in
FMKE, namely Dynamic Weight Assignment, and Meta Knowledge Sharing.
Dynamic Weight Assignment :
Concretely, we dynamically assign weights for each training sample in D .
As training progresses, the prediction confidence of each sample varies, making the samples contribute dynamically to the meta knowledge extraction.
Thus, it is beneficial to treat those samples in an adaptive manner, i.e. , based on their prediction output, empowering the samples with different confidence to make different contributions in the current training.
Specifically, we calculate a dynamic weight value ϕ ci for each sample D ci , e.g. , ( x ci , y ci ) , based on its prediction loss ℓ ( w c , x ci , y ci ) .
The formulation is defined as:
[Equation]
where τ is a hyper-parameter to smooth the result, D c
= {D ci } , 1 ≤ i ≤ N c , and N c denotes original data number on client c .
We assign the dynamic weight to each sample to update meta knowledge:
[Equation]
where L c ( w
[Equation]
Meta Knowledge Sharing :
As an optimization process, an appropriate initialization for ˆ D c in Eq.
5 is crucial to obtain a good final result.
In prior bi-level optimization works (Rajeswaran et al., 2019; Wang et al., 2018), the initialization value is randomly sampled from a constant distribution ( i.e. , ˆ D cini ∼ U [ − 1 , +1] ), namely unconditional initialization.
Due to the heterogeneity issue in FL, the extracted meta knowledge might become biased to the corresponding local data.
To solve those limitations, we implement a conditional initialization in Eq.
8 with a meta knowledge sharing mechanism.
Conditional initialization (Wang et al., 2020; Denevi et al., 2020) requires the initialization value is obtained from data characteristics rather than random generation.
To achieve conditional initialization, we design a simple yet effective strategy in extracting meta knowledge ˆ D c for client c at the current round t .
Concretely, in the process of initializing client c , we randomly select client c ′ and use its meta knowledge ˆ D c ′ t − 1 extracted in the previous round t − 1 as an initial value in Eq. 8.
Correspondingly, the initialization for ˆ D c changes from an unconditional manner to a conditional manner: ˆ D cini ← ˆ D c ′
t − 1 , c ′ ∼ randint [1 , C ] .
In this manner, the meta knowledge condensation for client c is mainly determined by the local data on client c as well as the knowledge extracted on another client c ′ , mitigating the heterogeneity issue significantly.
2.2. S ERVER - SIDE C ENTRAL M ODEL T RAINING
After conducting FMKE, we upload the condensed meta knowledge ˆ D from clients to a server.
On the server, the uploaded meta knowledge is used as normal training data to train a global model W G :
[Equation]
9) where ℓ ( . ) is a cross-entropy loss function as in Eq. 2, and ˆ D =
∪ ˆ D c , 1 ≤ c ≤ C .
To further ameliorate data biases among diverse clients, we introduce additional synthetic training samples into the central model training.
Those introduced training samples are from the same distribution of upload meta knowledge ˆ D .
Specifically, at first, we model the statistical distribution of uploaded meta knowledge ˆ D via a conditional generator, and then we sample additional data points based on the learned distribution.
Thus, sampled data would share the same distribution as ˆ D .
After the introduction of sampled synthetic data, we not only stabilize our training procedure but also achieve better performance.
To facilitate the discussion, we divide the model W G into a feature extractor F with parameter W F G and a classifier C with parameter W C G , in which W G =
( W F G , W C G ) .
Accordingly, we denote a latent representation as z = F ( W F G , x ) and a final prediction as y ′ =
C ( W C G , z ) .
The conditional generator
G maps a label y into a latent representation z ∼ G ( y, w G ) , and G is optimized by the objective:
[Equation]
where w G denotes the parameter of G .
The trained generator G models the distribution of uploaded meta knowledge ˆ D .
By sampling data from the distribution, we obtain a set of "pseudo" meta knowledge ˆ D pseu with corresponding labels.
The generated "pseudo" meta knowledge ˆ D pseu as well as uploaded ˆ D are utilized to train the global model by minimizing the following function:
[Equation]
where β is a parameter and determined by the cardinality fraction | ˆ D
[Equation]
| | ˆ D| .
Iterative Symbiosis Paradigm: After central model training, we broadcast the obtained global
W G and meta knowledge ˆ D to clients.
On each active client, the broadcasted model W G as well as meta knowledge ˆ D are used for a new round of FMKE.
FMKE and CMT collaborate with each other in an iterative symbiosis paradigm, benefiting each other increasingly as the learning continues.
The algorithm of FedMK training is illustrated in Alg. When the training finishes, the trained global model is directly used for inference.
Computational Complexity: FedMK includes two parts: federated meta knowledge extraction on clients and global model training on the server.
On clients, our method adopts a bi-level optimization to extract the meta-knowledge.
The bi-level optimization has a running-time complexity of O ( N × n ) (Fallah et al., 2020a), in which n denotes the meta knowledge size, N denotes the number of samples on the client.
It should be noted that the selected nodes can conduct FMKE in parallel.
On the server, the global model training in our method has a running-time complexity of O ( n ) .
In total, the overall running-time complexity of our method is O ( N × n ) . [Table]
E XPERIMENTS 3. D ATASETS
We evaluate our algorithm and compare to the key related works on four benchmarks: MNIST (LeCun et al., 2010), SVHN (Netzer et al., 2011), CIFAR10 (Krizhevsky & Hinton, 2009), and CIFAR (Krizhevsky & Hinton, 2009).
MNIST is a database of handwritten digits (0-9).
In MNIST, there are 50 , 000 images in the training set and 10 , 000 images in the test set, and their size is 1 × 28 ×pixels ( c × w × h ).
There are 10 classes in MNIST dataset.
SVHN is a real-world image dataset, in which there are 600 , 000 color images collected from house numbers in Google Street View images.
In SVHN, each image is of size 3 × 32 × 32 .
The class number of SVHN is as the same as MNIST.
CIFAR10 dataset consists of 60 , 000 color images, each of which has a size of 3 × 32 × 32 .
There are 50 , 000 images in the training set and 10 , 000 images in the testing set.
There are 10 classes in
CIFAR10.
CIFAR100 dataset consists of 60 , 000 color images, each of which has a size of 3 × 32 × 32 .
There are 50 , 000 images in the training set and 10 , 000 images in the testing set.
For each image in
CIFAR100, there are two kinds of labels, i.e. , fine label and coarse label.
We choose coarse labels and therefore we have 20 classes in the experiment on CIFAR100.
3. I MPLEMENTATION D ETAILS
We set the user number to 20 , and the active-user number to 10 .
We use 50% of the training set and distribute it on all clients.
All testing data is utilized for evaluation.
We use LeNet (LeCun et al., 1989) as the backbone for all methods: FedAvg (McMahan et al., 2017), FedProx (Li et al., 2020),
FedDistill (Seo et al., 2020), FedEnsemble (Zhu et al., 2021), FedGen (Zhu et al., 2021), and FedMK.
Dirichlet distribution Dir ( α ) is used to model data distributions.
Specifically, we test three different α values: 0 . 5 , 0 . 75 , and 1 . 0 , respectively.
We set the communication round number to 10 .
Learning with a small communication round number ( e.g. , 10 ) denotes learning under a limited communication budget.
For the methods conducting local model training on clients, i.e. , FedAvg, FedProx, FedDistill,
FedEnsemble, and FedGen, we set the local updating number to 20 , and the batch size number to 32 .
In our method, we set meta knowledge size for each datasets based on their different characteristics ( e.g. , the class number, sample size, etc): 20 per class for MNIST; 100 per class for SVHN; 100 per class for CIFAR10; 40 per class for CIFAR100.
We run three trials and report the mean accuracy performance (MAP).
3. C OMPARATIVE S TUDIES
Compared with Prior Works: We run all methods under limited communication budgets ( 10 rounds) on four datasets and report the results in Table 1.
As shown in Table 1, when communication budgets are limited, all prior works fail to construct a model with a good performance; on the contrary, our method can learn a model outperforming competing works by a remarkable margin.
For example, on
MNIST, our method achieves 93 .
63% ( α = 1 ) MAP, outperforming the competing works significantly.
On SVHN, our method achieves 75 .
85% ( α = 1 ) MAP in 10 rounds, while all competing works have not converged yet.
We believe that the superior performance of our method comes from those aspects: 1) unlike competing methods conducting local model training with local private data, our FedMK trains a global model based on meta-knowledge from all active clients.
Training on knowledge from all active clients makes our method less biased; 2) the designed two mechanisms, i.e. , dynamic weight assignment and meta knowledge sharing, make the meta knowledge extraction more stable, leading to a better performance than competing methods 3 .
Convergence rate comparisons: We compare the convergence rates of all methods and show the performance curves with respect to communication rounds in Figure 2.
As expected, our method [Table] achieves a high convergence speed compared to all competing methods.
On all datasets, our method achieves satisfactory performance in much fewer communication rounds.
The results shown in both Table 1 and Figure 2 demonstrate the effectiveness and efficiency of our proposed method.
Communication cost comparisons: We analyze the communication cost of our method on MNIST. In each round of FedMK, we need to upload and download the meta knowledge, and download a trained model. In the experiment, the meta image size is 20 per class, each of which is 4 bytes × 28 × 28 (sizeof(f32)* w * h ). Thus, in each round, the uploading cost of FedMK is 4 bytes × 28 × 28 ×× 10 = 627 .
2 K (meta knowledge), and the downloading cost is 627 .
2 K + 105 K × 10 = 1677 .
2 K (meta knowledge and a model).
In total, the communication cost for FedMK is ( 627 . 2 K+ 1 , 677 . 2 K) ×
(# rounds)= 16 M. As shown in Figure 2, to achieve comparable performance on MNIST, FedAvg has to run around 200 rounds.
In this case, the communication cost for FedAvg is 105 K × 10 × 2 ×(model size × #active node × 2 ×
#communication)=420M.
This indicates that to obtain models with good performance, FedAvg and its variants require much higher communication costs.
Impact of the meta knowledge size: We explore the impact of different meta knowledge sizes on four datasets.
We draw the performance comparisons in Figure 3.
It can be seen that the final performance changes as the meta knowledge size varies.
For example, on CIFAR10, the MAP score improves as the meta knowledge size increases.
Another finding is that the optimal meta knowledge sizes between different datasets might be different.
For example, we achieve the highest MAP score when setting the meta knowledge size to 20 on MNIST; while on the other three datasets, we obtain the best MAP scores using different meta knowledge sizes.
More discussions about why utilizing meta knowledge decreases the required communication round, and why increasing meta knowledge size will not necessarily improve final performance, can be found in the supplementary material.
Impact of active client numbers : We explore the different active client numbers.
We set the number to 5 , 7 , and 10 , respectively.
The performance curves are shown in Figure 4.
As shown in the figure, our method outperforms all competing works significantly with all different active client numbers.
Impact of communication rounds: We conduct an experiment on CIFAR10 to analyze the impact of communication rounds.
Specifically, we set the communication round number to 20 , 30 , andfor comparisons.
The comparison results are shown in Figure 5.
We demonstrate that our method consistently outperforms competing works by a remarkable margin.
Impact of node numbers: We conduct experiments on SVHN to analyze the impact of node numbers.
Specifically, we set the node number to 200 for comparisons.
Intuitively, when the number of user becomes larger, the number of data on each client will be sparse.
And thus, the learning will become [Figure] [Table] difficult, leading to a performance drop.
In our experiment, we find that when we increase the node number to 200 , our method ( 70 . 51% ) still outperforms the FedGen ( 66 . 31% ) by a remarkable margin.
Impact of designed mechanisms : We conduct a study on CIFAR10 and CIFAR100 to explore the impact of each designed mechanism.
We report the performance in Table 2, where "w/o Iter" means that we conduct local meta knowledge condensation and global model update only once 4 , "w/o
Sharing" means that we do not adopt meta knowledge sharing between clients, "w/o pseudo meta knowledge" means that there is no learnable constraint on the server, "w/o dynamic weights" means that we update ˆ D by treating all samples in D equally.
As shown in Table 2, there is a drastic performance degradation when there is no iterative collaboration between meta knowledge extraction and central model training ("w/o Iter").
Making the meta knowledge condensation and central model training running in an iterative manner improves the performance significantly, e.g. , on CIFAR10, MAP score increases from 28 .
15% to 45 .
Moreover, the two designed mechanisms, i.e. , the meta knowledge sharing between clients, dynamic weights assignment, significantly boost the MAP score.
Impact of the weight assignment mechanism: To explore the impact of different weight assignment mechanisms, we tested another weight assignment strategy targeting ameliorating the class imbalance problem. This strategy has been utilized in prior works (Tian et al., 2022a; Shao et al., 2021). To simplify the discussion, we call this strategy as Class Balance Weight.
Concretely, in the Class Balance Weight strategy, the weight of each sample during the training is determined by the occurrence rate of the corresponding class. We replace the Dynamic Weight Assignment with the Balance Class Weight, and conduct an experiment on MNIST.
We report the results in Table 3. As shown in the Table, we demonstrate that our designed Dynamic Weight Assignment outperforms the Class Balance Weight method significantly.
Evaluation on the pathological non-iid setting: We conduct an experiment on MNIST under the pathological non-iid setting (Huang et al., 2021).
In the experiment, we set the node number to 20 , the active node number to 10 , the number of classes on each client to 5 , α to 1 . 0 .
We compare all methods under a limited budget communication ( i.e. , 10 rounds).
As seen in Table 4, our method achieves better performance compared to the competing methods.
[Table]
Visualization of condensed meta knowledge We visualize the condensed meta knowledge coded from MNIST dataset.
As shown in Figure 6, it is difficult to infer the semantic information from the meta knowledge.
Furthermore, we conduct an experiment based on Deep Leakage (Zhu & Han, 2020) to explore the possibility of restoring original data from meta knowledge.
We report the results in the supplementary document, in which we demonstrate that it is hard to construct correspondence between entries in restored data and original data.
R ELATED W ORK
Federated Learning.
As a privacy-preserving solution, federated learning (Li et al., 2019) provides a new training manner to learn models over a collection of distributed devices.
In federated learning, the data is located on distributed nodes and not shared.
Learning a model without exchanging local data between nodes minimizes the risk of data leakage but increases the difficulty of training.
As a representative method, FedAvg (McMahan et al., 2017) is proposed to obtain a global model by aggregating local models trained on active clients.
To improve the local training, works such as (Li et al., 2020; Zhu et al., 2021;
Li et al., 2021; Tian et al., 2022b) design various strategies in their solutions.
All those methods require a large number of communication round to construct a model with satisfactory performance, resulting in heavy communication burden.
Recently, several methods (Fallah et al., 2020b; Dinh et al., 2020; Seo et al., 2020; Li et al., 2021;
Zhu et al., 2021; Fan & Huang, 2021;
Gou et al., 2021; Dai et al., 2022; Zhang et al., 2022) have been proposed to handle the data heterogeneity issue, which can be categorized into personalization-based methods and knowledge distillation-based methods.
The personalized FL methods (Fallah et al., 2020b; Dinh et al., 2020; Dai et al., 2022) aim to learn client-specific models while our goal is to obtain a global model across different clients.
Knowledge distillation based FL methods (Seo et al., 2020; Zhu et al., 2021; Zhang et al., 2022) propose to share knowledge between nodes, and thus the local training is regularized by local private data and knowledge on other clients.
Compact Data Representation.
Generally, prior works compressing a large scale data into a small set can be categorized into two main branches: data selection and data compression.
Data selection methods (Rebuffi et al., 2017; Castro et al., 2018; Aljundi et al., 2019; Sener & Savarese, 2018) select the most representative samples from the original data based on predefined criteria.
The selected samples are utilized to replace the original data for model training.
However, how to select an appropriate criterion based on the given data and task is not a trivial issue.
To overcome the aforementioned limitations, synthesizing new samples rather than selecting existing samples becomes a more preferable solution.
(Wang et al., 2018; Zhao et al., 2021; Zhao & Bilen, 2021; Cazenavette et al., 2022) design different solutions for generating synthetic data based on given datasets.
In those methods, the generated data can replace the original data in the model construction process.
However, those prior synthetic data generation works require data to be localized in a centralized manner.
It is not trivial to directly apply those methods under a federated learning setting.
C ONCLUSION
In this paper, we present a new federated learning paradigm driven by meta knowledge, dubbed FedMK, to obtain an effective and fast-converging model.
With the help of the proposed paradigm,
FedMK can train a powerful model even under a limited communication budget ( e.g. , 10 communication rounds), decreasing the communication cost significantly.
Moreover, our designed mechanisms, i.e. , meta knowledge sharing, dynamic weight assignment, and a learned constraint, collectively facilitate the central model training, benefiting FedMK outperforming all competing methods.