Factorized-FL: Personalized Federated Learning with Parameter Factorization & Similarity Matching
Abstract
In real-world federated learning scenarios, participants could have their own personalized labels which are incompatible with those from other clients, due to using different label permutations or tackling completely different tasks or domains.
However, most existing FL approaches cannot effectively tackle such extremely heterogeneous scenarios since they often assume that (1) all participants use a synchronized set of labels, and (2) they train on the same tasks from the same domain.
In this work, to tackle these challenges, we introduce Factorized-FL , which allows to effectively tackle label- and task-heterogeneous federated learning settings by factorizing the model parameters into a pair of rank-1 vectors, where one captures the common knowledge across different labels and tasks and the other captures knowledge specific to the task for each local model.
Moreover, based on the distance in the client-specific vector space, Factorized-FL performs selective aggregation scheme to utilize only the knowledge from the relevant participants for each client.
We extensively validate our method on both label- and domainheterogeneous settings, on which it outperforms the state-of-the-art personalized federated learning methods.
Introduction
Personalized Federated Learning (PFL) aims to utilize the aggregated knowledge from other clients while learning a client-specific model that is specialized for its own task and data distribution, rather than learning a universal, global model for tackling all local tasks [1, 16, 5, 28].
While various personalized federated learning approaches have shown success in alleviating the data heterogeneity problem, yet, they are also limited as they follow the common assumptions of the standard federated learning setting, that (1) all participants use a synchronized set of labels and (2) all clients tackle the same task from the same domain.
In many real-world scenarios the first assumption may not hold since the labels for the same task could be differently annotated depending on the user environment (Figure 2 Left).
For example, when working with the same set of semantic classes, the labels across multiple clients could have a completely different ordering of the classes.
At client 1, the label 1 may denote the “Ship" class, while at client 2, the label 10 may denote the same class.
Also, the same “Car" class may be given the label “Vehicle" or “SUV".
The second assumption severely limits the pool of devices or institutions that can participate in the collaborative learning process.
However, clients working on different tasks and domains may have similar classes, or a common underlying knowledge, that may be helpful for the local models being trained at other clients (Figure 2 Right).
Thus, it would be helpful if we can allow the local models with heterogeneous tasks and domains to communicate the common knowledge across them.
[Figure] [Figure]
However, federated learning under label and domain heterogeneity is a nontrivial problem, as most methods suffer from severe performance degeneration in such settings (Table 1).
We analyze this phenomenon in Figure 1.
Specifically, we train equally-initialized models on four different datasets and observe how different the gradient updates become as training goes on (we measure normalized L 2 distance between them).
Learning on two
MNIST partitions will yield the smallest difference between gradients (Figure 1 (a) and (b) Gray).
Interestingly, simply permuting the labels of the partition 2 makes the gradient updates to largely diverge from the original gradients, compared to those of the model trained with synchronized labels (Figure 1 (a) and (b) Red).
Moreover, learning on the completely different dataset (CIFAR-10) makes model gradients diverge more severely compared to learning on the dataset with permuted labels (Figure 1 (a) and (b) Blue).
The heterogeneity in the labels and domains tackled by each participant leads to severe performance degeneration when performing federated learning (Figure 1 (c)).
We measure performance on MNIST partition 1 while aggregating a model trained on the different dataset for every 3 epochs.
We observe averaged models suffer from crucial performance degeneration in both FL scenarios.
Note that the classifier layers are not shared when performing federated learning.
However, such label- and domainheterogeneous FL settings have been relatively overlooked.
Although few previous works [1, 16, 25] briefly introduced such extreme settings as motivations, they did not formally define the problems nor tackle such scenarios, and rather targeted standard non-IID scenarios.
We name this challenging problem as the Agnostic Personalized Federated Learning (APFL) problem, where participants with personalized labels or from multiple domains can collaboratively learn while benefiting each other.
An APFL problem has two critical challenges: (1) Label Heterogeneity for the discrepancy of the labels, due to the lack of a synchronized labeling scheme across the clients; and (2) Domain Heterogeneity for the discrepancy in the task and domains tackled by each participant.
To tackle these challenges, we propose a novel FL framework, dubbed Factorized-FL , which factorizes model parameters into basis vectors and aggregate them in the factorized parameter space.
This allows to factorize the model aggregation to take place in a semantic parameter basis space which is more robust to the use of different labels and data.
Also, the factorization results in the separation of the client-general and client-specific knowledge, and thus prevents the aggregation of incompatible knowledge across clients.
Moreover, to further alleviate the model from collapsing into a degenerate solution, we measure the task similarity across the clients using the factorized parameters, to allow selective aggregation of the knowledge among the relevant models that work on similar tasks or domains.
We extensively validate our method on both label- and domain-heterogeneous settings, and show that it significantly outperforms the current state-of-the-art methods with lower communication cost due to using factorized parameters.
Our contributions can be summarized as follows:
• We introduce the Agnostic Personalized Federated Learning (APFL) problem and study its two critical challenges, Label and Domain Heterogeneity.
• We propose Factorized-FL for tackling the APFL problem, which factorizes model parameters to reduce parameter dimensionality for alleviating knowledge collapse, and utilize them to measure task-level similarity for matching relevant clients.
• We validate our method in both label- and domain-heterogeneous scenarios and show our method outperforms the current state-of-the-art personalized federated learning methods.
Related Work
Federated learning Many federated learning algorithms have been proposed since the introduction of FedAvg [18], but, we specifically focus on works that aim to tackle the heterogeneity problems, e.g.
Non-IIDness.
Some studies focus on regularization methods [19, 15], correcting disparity between server and clients [26, 9], or contrastive learning [13].
Recently, many existing works also tackle architecture-level heterogeneity [24, 29, 3].
Personalized federated learning aims to improve the individual local models instead of the universal global model, via the mixture methods [17, 2, 7], meta-learning [5], partial network aggregation [1, 16], or hyper-networks [25].
Recent approaches aim to enhance personalization performance by avoiding to aggregate irrelevant knowledge that are not helpful.
Zhang et al.
[28] download the models from other clients to aggregate only the beneficial ones for the local task at each client.
Sattler et al.
[23], Duan et al.
[4] measure client-wise similarity by using the gradient updates.
Our method also measures client similarity but in a more efficient way, by utilizing 1D vectors from factorization.
Re-parameterization for federated learning Jeong et al.
[8], Yoon et al.
[27] decompose model parameters (using an additional set of parameters) to train them with different objectives, which do not reduce the dimensionality of model parameters.
Some approaches factorize high dimensional model parameters into low rank matrices.
Kone ˇ cn ` y et al.
[10] introduce structured update which model directly learns factorized parameter space, and Nam et al.
[20] propose to use the Hadamard product of low rank matrices to enhance communication efficiency.
Unlike prior works, we utilize rank-1 vectors factorization to tackle heterogeneity in domains and tasks, by allowing the 1D vectors to capture task-general and the client-specific knowledge.
Also, we minimize the information loss from factorization by utilizing sparse bias matrices.
Problem Definition
We begin with the formal definition of the conventional federated learning scenario, and then introduce our novel Agnostic Personalized Federated Learning (APFL) problem.
3.1 Preliminaries
Our goal is to solve a multi-class classification task under an FL setting.
Let f g be a global model (neural network) at the global server and F = { f k } Kk =1 be a set of K local neural networks, where K is the number of local clients.
D =
{ x i , y i } Ni =1 be a given dataset, where N is the number of instances, x i ∈ R W × H × D is the i th examples in a size of width W , height H , and depth D , with a corresponding target label y i ∈ { 1 , . . .
, C } for the C -way multi-class classification problem.
The given dataset
D is then disjointly split into K sub-partitions D k = { x k,i , y k,i } N k i =
1 s.t.
D =
(cid:83) Kk =1 D k , which are distributed to the corresponding local model f k .
Let R be the total number of the communication rounds and r denote the index of the r th communication round.
At the initial round r = 1 , the global model f g initializes the global weights θ (1) f g and broadcasts θ (1) f g to an arbitrary subset of local models that are available for training at round r , such that F ( r ) ⊂ F , |F ( r ) | =
K ( r ) , and K ( r ) ≤ K , where
K ( r ) is the number of available local models at round r .
Then the active local models f k ∈ F ( r ) perform local training to minimize loss L ( θ ( r ) k ) on the corresponding sub-partition P k and update their local weights θ ( r +1)
k ← θ ( r ) k − η ∇L ( θ ( r ) k ) , where θ ( r ) k is the set of weights for the local model f k at round r and L ( · ) is the loss function.
When the local training is done, the global model F collects and aggregates the learned weights θ ( r +1)
f g ← N k N (cid:80) K ( r ) i =1 θ ( r ) k
and then broadcasts newly updated weights to the local models available at the next round r + 1 .
These learning procedures are repeated until the final round R .
On the other hand, Personalized Federated Learning aims to adapt the individual local models f 1: K to their local data distribution D 1: K , to obtain specialized solution for each task at the local client, while utilizing the knowledge from other clients.
Thus merging the local knowledge for personalized FL is not necessarily done in the form of θ ( r +1)
f g [Equation] r
[Equation]
θ [Equation] r ) k , and the specific ways to utilize the knowledge from others depends on the specific algorithm, i.e. θ ( r +1) k
[Equation]
where ω ( · ) is weighing function [28].
[Figure]
𝐖
Conventional Factorization
Hadamard Product of Low Rank Matrices
[Figure]
3.2 Agnostic Personalized Federated Learning
Agnostic Personalized Federated Learning (APFL) is a scenario where any local participants from diverse domains with their own personalized labeling schemes can collaboratively learn, benefiting each other.
There exist two critical challenges that need to be tackled to achieve this objective: (1)
Label Heterogeneity and (2) Domain Heterogeneity.
Label Heterogeneity
This scenario assumes that the labeling schemes are not perfectly synchronized across all clients, as described in Section 1 and Figure 2 Left.
Most underlying setting for this scenario is the same as the conventional single-domain setting with synchronized labels that is described in Section 3.1, except that labels are arbitrarily permuted amongst clients.
The local data
D k for the local model f k is now defined as D k = { x k,i , φ k ( y k,i ) }
N k i =1 , where φ k ( · ) is a mapping function for the local model f k which maps a given class y k,i with a randomly permuted label p k,i =
φ k ( y k,i ) .
Let the j th layer out of L layers in the neural networks of local model f k be ℓ jk and the last layer ℓ
Lk be the classifier layer.
Since each client has differently permuted labels, the personalized classifiers ℓ L 1: K are no longer compatible to each other.
While we can merge the layers below the classifier in this setting, training with heterogeneous labels could still lead to large disparity in the local gradients even in the initial communication round, as described in Figure 1.
Domain Heterogeneity
This scenario presumes that local clients learn on their own dataset D , that are completely different from the datasets that are used at other clients, as described in Section 1 and
Figure 2 Right.
In this setting, K disjoint datasets D 1: K are assigned to the K local clients f 1: K , where D k = { x k,i , y k,i } N k i
=1 is the dataset assigned to the local model f k .
The number of target classes may differ across clients, such that y k,i ∈ { 1 , . . .
, C k } .
We assume complete disjointness across clients, such that there is no instance- and class-wise overlap across the datasets: ∅ = (cid:84)
Kk =1 D k and ∅ = (cid:84)
Kk =1 C k , where C k is a set of classes for client k .
Similarly to the label-heterogeneous scenario, the personalized classifiers ℓ L 1: K are no longer compatible to each other due to the heterogeneity in the data and the labels.
Hence, the aggregation is done for the layers before the classifier, but they will be also incompatible as the learned model weights will be largely different across domains.
Factorized Federated Learning 4.1 Factorization of Model Parameters
Wang et al.
[26] discussed that the conventional knowledge aggregation, that is often performed in a coordinate-wise manner, may have severe detrimental effects on the averaged model.
This is because the deep neural networks have extremely high-dimensional parameters and thus meaningful element-wise neural matching is not guaranteed when aggregating the weights across different models trained under diverse settings.
One naive solution to this problem is to factorize model parameters into lower dimensional space, i.e. low rank matrices, as shown in Figure 3 (Left).
Conventional approaches, such as SVD, Tucker, or Canonical Polyadic decomposition, however, factorize model parameters after training [12, 22] is done.
Thus, the dimensionality at the time of knowledge aggregation will remain the same as the unfactorized model.
Koneˇcn`y et al.
[10], Nam et al.
[20] pre-decompose model parameters to low rank matrices for FL scenarios.
While Koneˇcn`y et al.
[10] use naive low rank matrices, Nam et al.
[20] use two sets of low rank matrices to improve expressiveness and utilize them as global and local weights (Figure 3 (Middle)).
Unlike prior works, our approach utilizes rank-1 vectors to perform aggregation in the lowest subspace possible for compatibility, while effectively yet efficiently enhancing expressiveness with sparse bias matrices, as shown in Figure 3 (Right) and Figure 5 (Left).
Another crucial difference of our method from the previous factorization methods is that our rankvectors have distinct roles.
Our factorization will separate the common knowledge from the task- or domain-specific knowledge, since u could be thought as the bases (the common knowledge across clients) and v could be thought as the coefficients (client-specific information).
[Figure]
In Figure 4 (a) and (b), which show the experimental results with the factorized model, we observe that u trained on two datasets becomes closer to each other (MNIST Partition 1) while v (personalized filter coefficient) remain largely different as FL goes on.
With this observation, we further aggregate u while allowing v to be different across clients, to allow personalized FL.
Further, we use the client specific v for similarity matching, to identify relevant local models from other clients.
In the following paragraphs, we describe our factorization method in detail, for both fully-connected and convolutional layers.
Factorization of Fully-Connected Layers We assume that each local model f k has a set of local weights θ k across all layers; that is, θ k
= { W ik } Li =1 .
The dimensionality of the dense weight W ik for each fully connected layer is W ik ∈ R
I × O , where I and O indicate respective input and output dimensions.
We can reduce the I × O complexity by factorizing the high order matrix into the outer product of two vectors as follows:
[Equation]
However, such extreme factorization of the weight matrices may result in the loss of expressiveness in the parameter space.
Thus, we additionally introduce a highly sparse bias matrix µ to further capture the information not captured by the outer product of the two vectors as follows:
[Equation]
We initialize µ with zeros so that it can gradually capture the additional expressiveness that are not captured by u and v during training.
We can control its sparsity by the hyper-parameter for the sparsity regularizer described in 4.3.
Factorization of Convolutional Layers
The difference between the fully-connected and convolutional layers is that the convolutional layers have multiple kernels (or filters) such that
W ik ∈ R F × F ×
I × O , where F is a size of filters (we assume the filter size is equally paired for the simplicity).
To induce u to capture base filter knowledge and v to learn filter coefficient, it is essential to design u ∈ R F · F and v ∈ R I · O , but not in arbitrary ways, such as u ∈ R I · F and v ∈ R O · F or u ∈ R O and v ∈ R I · F · F .
We observe that performance is degenerated when the parameters are ambiguously factorized (Figure 7 (h)).
Our proposed factorization method for convolutional layers are as follows:
[Equation]
π ( · ) is the weight reshaping function.
Note that we reparameterize our model at initialization time .
Then we reconstruct and train full weights of each layer W 1: L k , while optimizing u 1: L k , v 1: L k , and µ 1: L k , respectively, during training phase.
4.2 Similarity Matching
Since we assume label- and domain-heterogeneous FL scenarios, aggregating the parameter bases across all clients may not be optimal, since some of them could be highly irrelevant.
Yoon et al.
[27] and Zhang et al.
[28] also demonstrate that avoiding aggregation of irrelevant models from other clients improves local model performance.
Yoon et al.
[27] achieve this goal by taking the weighted combination of task-specific weights from other clients, and Zhang et al.
[28] suggest downloading the models from other clients and evaluating their performance on a local validation set, at each client. However, since they require additional communication and computing cost at the local clients, we provide a more efficient yet effective approach to match models that are beneficial to each other.
× 𝐼 "𝑂 𝐹 " 𝐹 𝐮 𝐯 𝑻 𝐮 ×𝐯 𝑻 ⊕ µ 𝐼 "𝑂 𝐹 𝐹
Filter Base Filter Coefficients
Sparse Bias Matrix 12345 … 𝐼(𝑂 𝐹 " 𝐹 1 2 3 4𝐼(𝑂 ...
123… 𝐼&𝑂 Filter𝐮 𝐯𝐮×𝐯
R e c o n s t r u c t
e
d K e r n e l s
Filter
Kernel Reshaping 𝐹 𝐹 𝐹 " 𝐹 𝐼 "𝑂 … 0.
Factorized Parameter Space
C li e n t
C li e n t C li e n t
K
Similarity Matching 0.
𝐯 𝒇 𝟏 𝑳# 𝐯 𝒇 𝟐 𝑳# 𝐯 𝒇 𝑲 𝑳# ⊗ ⊗ ⊗ 𝐮 𝒇 𝑲 ⊕ 𝐯 𝒇 𝑲
𝝁 𝒇 𝑲 𝐮 𝒇 𝟐 𝐯 𝒇 𝟐 𝝁 𝒇𝐮 𝒇 𝟏 𝐯 𝒇
𝝁 𝒇
[Figure]
Efficient similarity matching Our method utilizes a factorized vector v k for measuring similarity across different models, at the central server.
Since v are devised to learn personalized coefficients, we assume that clients trained on similar tasks or domains will have similar v .
Specifically, we use v L − 1 k of the layer before the classifier layer for similarity matching.
The similarity matching function Ω( · ) , is defined as the cosine similarity between target client f k and the other clients { f i } Ki ̸
= k , as follows:
[Equation]
The similarity scores for those with the cosine similarity scores lower than the given threshold τ , are set to zero.
Our method is significantly more efficient than the similarity matching approaches which use full gradient updates for clustering clients [23, 4], as we only use single vector v L − 1 k from each client k that are obtained without additional cost.
Personalized weighted averaging We allow each local model to individually perform weighted aggregation of the model weights from other clients, utilizing their similarity scores:
[Equation]
where ϵ is a hyperparameter for scaling the similarity score σ i .
We always set σ k , the similarity score for itself, as 1 . 0 .
We name this model that aggregates only basis vector u k from each client k as Factorized-FL α .
This will result in encoding common knowledge across local models in u k which is especially suitable for domain-heterogeneous scenarios.
However, for label-heterogeneous scenarios, we empirically observed that aggregating all parameters results in better performance, as the factorization helps averaging them in a more compatible space.
We thus introduce an additional variant, namely Factorized-FL β , which aggregates not only the parameter bases across all clients, but also coefficient and bias terms as well.
Yet, note that Factorized-FL β require significantly larger communication overhead over Factorized-FL α .
4.3 Learning Objective
Now we describe our final learning objective.
Instead of utilizing the single term θ k for local weights of neural network f k , now let U k , V k , and M k be sets of u k , v k , and µ k of all layers in f k , s.t.
U k
= { u ik } Li =1 , V k
= { v ik } Li =1 , and M k = { µ ik } Li
=1 , then our local objective function is,
[Equation]
[Equation]
where L is the standard cross-entropy loss performed on all minibatch B ∈ D k .
We add the Lsparsity inducing regularization term to make the bias parameters highly sparse, controlling its effect with a hyperparameter λ sparsity .
As for the full training procedure, please see the pseudo-code of the algorithm in the supplementary file (Section B). [Table] [Table] [Figure]
[Table]
[Figure] [Table] [Figure] [Figure]
[Figure]
Experiment 5.1 Experimental Setup
Baselines We first consider well-known baseline FL methods, such as (1) FedAvg [18] and (2)
FedProx [14].
We evaluate our factorization technique with (3) pFedPara [20] which also uses kernel factorization technique for personalized FL scenarios.
Our similarity matching approach is compared to (4) Clustered-FL [23] and (5) FedFOMO [28], which measuring client-wise similarity or helpfulness.
(6) Per-FedAvg [5] is also used for evaluation as it shows great performance on heterogeneous federated learning scenarios. We also show local training model, (7) Stand-Alone , for the lower bound performance. Detailed information are described in Appendix (Section D). Datasets (1) Label Heterogeneous Scenario : we use CIFAR-10 [11] and SVHN [21] datasets and create four different partitions for each dataset, which are conventional IID and non-IID as well
1234567891011121314151617181920 Client 1234567891011121314151617181920 C li en t 1234567891011121314151617181920 Client 1234567891011121314151617181920 C li en t 1234567891011121314151617181920 Client 1234567891011121314151617181920 C li en t 1234567891011121314151617181920 Client 1234567891011121314151617181920 C li en t
[Figure]
1234567891011121314151617181920 C li en t 1234567891011121314151617181920 Client 1234567891011121314151617181920 C li en t 1234567891011121314151617181920 Client 1234567891011121314151617181920 C li en t 1234567891011121314151617181920 Client 1234567891011121314151617181920 C li en t 1234567891011121314151617181920 Client 1234567891011121314151617181920 C li en t 1234567891011121314151617181920 Client 1234567891011121314151617181920 C li en t
[Figure]
91011121314151617181920 Client
[Figure]
[Figure]
as permuted IID and permuted non-IID, whose labels are permuted and incompatible to each other.
We split the datasets into 20 partitions and then simply permuted the labels on the same partition for label permuted settings.
(2) Domain Heterogeneous Scenario : we use CIFAR-100 datasets [11] and create five sub-datasets grouped by 10 similar classes, such as Household Objects, Fruits&Foods,
Trees&Flowers, Transport, and Animals.
We assign 4 clients for each sub-dataset, i.e. Client 1to Household Objects, Client 5-8 to Fruits&Foods, Client 9-12 to Trees&Flowers, Client 13-16 to
Transport, and Client 17-20 to Animals.
We then permute the labels for all partitions to simulate further realistic scenarios.
Further descriptions are elaborated in Appendix (Section C).
5.2 Experimental Result
Label-heterogeneous FL As shown in Table 1 (Top), for the standard IID and Non-IID settings, all
FL methods obtain higher performance than the local training baseline ( Stand-Alone ), which confirms that the locally learned knowledge is beneficial to others, when the data and label distributions are homogeneous across clients.
However, when the labels are not synchronized across all clients (Permuted IID/Non-IID), all previous FL methods suffer from significant performance degeneration, even lower than that of the local training baseline.
Again, note that we do not share the classifier layers to ensure fairness across all algorithms in these permuted settings.
We conjecture that this is caused by the label permutation leading the local model to evolve a permuted set of features that are not coordinate-wise compatible to others when aggregated.
Contrarily, our method Factorized-FL α shows consistent performance regardless of whether the labels are permuted or not.
Factorized-FL β even largely outperforms all baselines with significantly superior performance.
Test accuracy curves over the communication round and transmission cost are in Appendix (Section E).
Domain -heterogeneous FL Table 1 (Bottom) shows the experimental results for the domain and label heterogeneous scenarios.
We observe that the conventional FL baselines, i.e. FedAvg , FedProx , fail to obtain better performance over purely local training baseline ( Stand-Alone ) due to the naive aggregation of extremely heterogeneous knowledge, which causes detrimental knowledge collapse.
FedFOMO and Clustered-FL shows slightly higher performance ( 1 - 2% p ) over Stand-Alone model, as they can avoid irrelevant clients when aggregating local knowledge.
The other personalized FL methods, i.e. Per-FedAvg and pFedPara , also show 1 - 2% p higher performance over Stand-Alone model as they are specialized for personalized FL scenarios.
However, on average, our method largely outperforms all baseline models even with the smallest communication costs ( Factorized-FL α ), as shown in Figure 6 (e).
In the figures, we plot the convergence rate of our Factorized-FL α & β over communication round (Top) and transmission cost (Bottom), compared to the baseline models.
Both models consistently obtain superior performance compared to the other models.
Particularly, unlike pFedPara which uses low rank matrices for knowledge sharing, Factorized-FL α only communicates with factorized vectors, such as U for base knowledge sharing and v L − 1 k for similarity matching and this allows our method to largely reduce the communication costs while showing the best performance as shown in Figure 6 (e) Bottom.
Effect of kernel factorization In Figure 7 (g), we perform an ablation study of our factorization method in the domain-heterogeneous scenario.
To clearly see the effectiveness of our factorization methods, we compare Factorized-FedAvg , a variant of Factorized-FL β without similarity matching, against FedAvg .
As shown, Factorized-FedAvg achieves higher performance over the original FedAvg model.
As the only difference between the two is whether kernel is factorized or not, this clearly demonstrates that our factorization method alone improves the model performance by alleviating knowledge collapse.
We further analyze the effect of the sparse bias matrix M .
When we remove M from the Factorized-FedAvg model, Factorized-FedAvg w/o µ in the figure, we observe large performance drop.
This shows that that the bias term is essential in compensating for the loss of expressiveness from the rather extreme factorization.
With only U and V , we use 90% less model parameters ( 0 . 27 M ) compared to the regular kernel model ( 2 . 574 M ).
Also, in Figure 7 (k), we observe that model factorized by u ∈ R F · F and v ∈ R I · O perform better than the model factorized by u ∈ R I · F and v ∈ R O · F .
This is because in the former case, the factorization will separate the base knowledge from the task-specific knowledge, as described in Figure 5 (Left), but in the latter case the factorization does not have such a natural interpretation.
We also show the scalability and reliability of our factorization method with different architectures, consisting of convolutional, fully-connected layers, batch normalization, and skip connections as shown in Figure 7 (k).
Effect of similarity matching To verify the effectiveness of our similarity matching method using the personalized factorized vector v , we visualize the inter-client similarity of u L − 1 f k and v L − 1 f k from the second last layer of 20 clients in a domain heterogeneous setting.
As shown in Figure 7 from (a) round 1 to (e) round 100, we observe u L − 1 f k (upper row) are indeed highly correlated with other clients (the darker color indicate higher similarity scores) while v L − 1 f k (bottom row) are relatively uncorrelated to each other, as expected as our assumption that U f k capture base knowledge across all clients and V f k capture personalized knowledge.
Further, we also observe that v L − 1 f k obtains higher similarity to that of the parameters trained on the same domains (but with permuted labels), i.e. Client 1-4, Client 5-8, Client 9-12, Client 13-16, and Client 17-20, showing that they are effective in capturing task- and domain-level similarities across models.
We further visualize the frequency of the client matching across clients in Figure (f), after 100 communication rounds.
With only a single vector parameter v L − 1 f k , our method finds which clients’s knowledge will be helpful to which clients.
For further analysis on our similarity matching, we compare it against Random and Worst Matching baselines under the multi domain scenario.
The random matching baseline randomly selects three arbitrary models to be aggregated at each round, while the worst matching baseline selects three most dissimilar models.
As shown in Table 7 (g), both baselines significantly suffer from the performance degeneration compared to our similarity matching algorithm, which shows its effectiveness.
Analysis on γ , τ , and λ sparsity
Figure 7 (h) and (i) show the analysis of the hyperparameters, γ and τ , on the domain-heterogeneous dataset.
As shown in Figure 7 (h), the performance slightly decreases with larger γ , since higher rank weight matrices may result in larger coordinate-wise incompatibility of the parameters, which could lead to knowledge collapse when performing aggregation.
Also, the lower τ s show the lower performance, as less similar knowledge can be aggregated more frequently.
In Figure 7 (j), we train Client 1 on CIFAR-10 IID Partition 1 for 20 epochs using both factorized and regular models.
As shown, our factorized model outperforms regular models (2 . 574 M ) even with 30% less parameters (1 . 784 M ) .
We further analyze the sparsity in Section E.2 of Appendix.
Conclusion
We introduced a realistic federated learning scenario where the labeling schemes are not synchronized across all participants (label heterogeneity) and the tasks and the domains tackled by each local model are different from those of others (domain heterogeneity).
We then proposed a novel method to tackle this problem, whose local model weights are factorized into the product of two vectors plus a sparse bias term.
We then aggregate only the first vectors, for them to capture the common knowledge across clients, while allowing the other vectors and the sparse bias term to be client-specific, accounting for label and domain heterogeneity.
Further, we use the client-specific vectors to measure the similarity scores across local models, which are then used for weighted averaging, for personalized federated learning of each local model.
Our method not only avoids knowledge collapse from aggregating incompatible parameters across heterogeneous models, but also significantly reduces the communication costs.
We validate our method on both label and domain heterogeneous settings, on which it largely outperforms relevant baselines.
A Limitations and Societal Impacts
Limitations Federated Learning (FL) is devised to protect the privacy of sensitive personal data via strictly avoiding sharing them with other users, institutions, and companies.
The common limitation in FL is that, even though almost all existing methods regard sharing locally learned parameters of gradients as safe, they still may contain some possibility to reproduce the local data distribution, which violates preservation of privacy.
Although our work alleviates this probability by significantly reducing the dimensionality of parameters and partially sharing them, it may be still sub-optimal in an aspect of security, which is an open research domain for federated learning.
Potential Societal Impact Federated learning is a collaborative learning framework where multiple clients can actively participate in the learning procedure anywhere and anytime.
Nowadays, numerous terminal devices, i.e. smartphones, smartwatches, IoT devices, etc, are utilized as participants of the collaborative learning scheme.
However, as the number of participants in the federated learning increases, the number of communications between them increases accordingly.
Such situations leading to the exhaustive communication costs may worsen the global environmental crisis , i.e. global warming.
Even though our method can alleviate such risks, since ours significantly reduces the communication costs by partially sharing the factorized vectors, it may be sub-optimal.
It is essential to focus on compressing the knowledge efficiently and effectively for federated learning that the massive number of participants collaborate.
B Factorized-FL Algorithms
In this section, we describe our pseudo-code algorithms for Factorized-FL α and Factorized-FL β in Algorithm 1 and 2.
Our Factorized-FL α has strength for not only reducing the dimensionality of model parameters by factorizing them into rank 1 vector spaces and the additional highly-sparse matrices, but also effectively learning client-general and task-specific knowledge, performing well on domain heterogeneous scenario.
Particularly, Factorized-FL α transmits a small portion of the models which are a set of u ( U ) and a single vector v L − 1 form the second last layer of neural networks, which significantly reduces communication costs while showing strong performance in label- and domain-heterogeneous scenarios, as shown in Section 5 in the main document.
C Dataset Configurations
In this section, we describe detailed configurations for datasets that we used in label- and domainheterogeneous scenarios.
C.1 Label Heterogeneous Scenario
We use CIFAR-10 and SVHN for the label-heterogeneous scenario.
We first split each dataset into train, validation, and test sets for CIFAR-10 ( 48 , 000 / 6 , 000 / 6 , 000 ) and SVHN ( 79 , 431 / 9 , 929 / 9 , 929 ).
We then split the train set into K local partitions P 1:20 ( K = 20 ) for IID partitions (all instances in each class are evenly distributed to all clients) or for the non-IID partitions (instances in each class are sampled from Dirichlet distribution with α = 0 . 5 ).
We further permute the labels for each class per local partition P k for permuted IID and permuted non-IID scenarios.
We use different random seed per client, i.e. fixed global seed + client id , for example, 1234 +
[Table] [Table]
for 13:
[Table]
for 14: Function
[Table] [Table]
) 15: θ
[Table]
←
[Table]
M
[Table]
16: for
[Table]
do 17: for minibatch
[Table]
do 18: θ
[Table]
← θ k
[Table] [Table]
[Table] [Table]
for 20: end for 21: return U k , v L −
[Table]
f k
[Table]
if
[Table] [Table]
5 Cat 5 9 0 7 9 3 8 8          8 Deer 6 2 6 9 6 5 2 1          9 Dog 4 1 4 8 5 6 6 6
Frog 9 3 2 1 2 0 3 2          1 Horse 0 7 9 4 3 7 0 3          6 Ship 1 8 7 6 7 4 5 7          2 Truck 7 6 8 2 0 1 7
SVHN
Digit 2 5 3 0 8 2 4 4          7 Digit 8 4 1 5 1 8 1 0          4 Digit 3 0 5 3 4 9 9 5          5 Digit 5 9 0 7 9 3 8 8          8 Digit 6 2 6 9 6 5 2 1          9 Digit 4 1 4 8 5 6 6 6          0 Digit 9 3 2 1 2 0 3
[Table]
[Table] [Table] [Table] for Client 1 and 1234 + 19 for Client 20.
We provide permutations of labels that we used for each dataset in Table 2.
C.2 Domain Heterogeneous Scenario
We use CIFAR-100 datasets ( 60 , 000 ) and create five sub-datasets grouped by 10 similar classes, such as Fruits&Foods ( 6 , 000 ), Transport ( 6 , 000 ), Household Objects ( 6 , 000 ), Animals ( 6 , 000 ),
Trees&Flowers ( 6 , 000 ).
We then split train ( 4 , 800 ), test ( 600 ), validation ( 600 ) sets for each subdatset.
To have 20 clients in total, we assign four clients per subdataset, and split each train set into 4 partitions, making a single partition contains 1 , 200 instances. Additionally, we further permute
[Table] [Table]
[Table] [Table]
[Table]
- - - - - - - - - - - -    - - - - Rocket - - - - - - - - - - - -    - - - - Streetcar - - - - - - - - - - - -    - - - - Tank - - - - - - - - - - - -    - - - - Tractor - - - - - - - - - - - -    - - - -
[Table]
- - - - - - - - - - - -    - - - - Bus - - - - - - - - - - - -    - - - - Motorcycle - - - - - - - - - - - -    - - - -
[Table]
[Table]
[Table]
[Table]
the labels for those 20 partitions to simulate more realistic scenarios where labeling schemes are not synchronized across all clients even in the same domain (sub-dataset).
We provide class division and label permutation information in Table 3.
D Training Details & Implementations
In this section, we provide detailed implementation and training details that are not described in the main document.
D.1 Baseline Models
Stand-Alone does not share their locally learned knowledge with other clients.
It shows the pure model performance on the data partitions.
In our extremely heterogeneous scenarios, where knowledge collapse may happen severely, which even deteriorates the local knowledge, this model shows the comparable performance amongst the existing models.
FedAvg [18] performs weighted aggregation of the model parameters, considering the size of the local training set.
This model is considered as the standard baseline of many federated learning algorithms.
FedProx [15] uses proximal regularization term for alleviating divergence between global parameters and the local parameters.
This model is devised to tackle data heterogeneity (also known as the non-iid problems) across the clients.
Clustered-FL [23] performs bi-partitioning process for the participants when certain conditions are satisfied (calculating the norm of the gradient updates and comparing it with given threshold values).
It continues to split the given pool of the clients into two novel clusters and performs knowledge aggregations for each cluster. pFedPara [20] uses the Hadamard product of two sets of low rank matrices and reconstructs the kernel parameters.
It shares one of the sets of low rank matrices with other clients and remains another set of low rank matrices for the personalization.
Per-FedAvg [5] adopts Model-Agnostic Meta Learning (MAML) [6] approach to federated learning algorithm to search the initial global model which participants can easily adapt to their local data by being trained with only one or a few steps of gradient descent.
FedFOMO [28] leverages other clients’ knowledge for improving their local models.
It downloads a few random models from other clients and validates them on their own validation set at each client.
When some parameters show better validation performance compared to their own local performance, it aggregates such helpful parameters with their own local parameters.
D.2 Training Details
As default, all training configurations are equally set across all models, unless otherwise stated to ensure stricter fairness.
We use ResNet-9 architecture as local backbone networks and train them on 32 × 32 sized images with 256 for batch size.
We apply data augmentations, i.e. cropping, flipping, jittering, etc, during training.
Optimizer that we used is Stochastic Gradient Descent (SGD).
We set 1 e- 3 for learning rate, 1 e- 6 for weight decay, and 0 .
9 for momentum.
For baseline models, we use the reported hyper-parameters as default, or we adjust hyper-parameters so that they show the best performance for fairness.
For ours and pFedPara , the model capacity is adjusted to around 90% - 99% of the original size, as we fairly compare with other methods that use full capacity ( 2 .
57 M number of parameters).
For ours, we use [ 5 e- 4 , 1 e- 3 ] for λ sparsity , [ 0 - 0 .
75 ] for τ , [ 1 , 20 ] for ϵ .
We use 8 GPUs (NVIDIA Titan Xp) for experiments.
D.3 ResNet-9 Architecture
[Table]
We use ResNet-9 architecture consisting of eight convolutional layers and one fully connected layer as a classifier, as described in Table 4.
We use max pooling with size 2 after Conv 5 and an adaptive max pooling after Conv 8 to make output width 1 for the following
FC layer.
The total number of parameters of the model is 2 .
57 M .
As we use
PyTorch framework for implementation and the default data type of tensor of the framework is 32-bits floating point, the model size can be calculated as 2 . 57 × 4 = 10 . 28 Mbytes.
D.4 Calculation of Communication Cost
We measure the communication cost by { ( P S 2 C + P C 2 S ) × 4 } byte × K × R , where P S 2 C is number of server-to-client transmitted parameters and P C 2 S is number of client-to-server transmitted parameters.
Depending on the FL algorithms, P S 2 C and P C 2 S are differently calculated.
For example, FedFOMO downloads a few random models from the server ( 10 by default as reported in the paper) but sends only a single local model to the server.
Our Factorized-FL α only sends the small portion of model parameters, U and v L − 1 , to server, while receiving a single set of U from server.
E Additional Experimental Results
[Table]
W · N
W · N
W
W pFedPara ( U + V ) 2 · N ( U + V ) 2 · N
U + V
U + V
FedFOMO
W · N + H · W · N val
W · N
W ( H + 1) · W
Factorized-FL α ( U · V + M ) · N ( U · V + M ) · N
U + v
U
Factorized-FL β
( U · V + M ) · N ( U · V + M ) · N
U + V + M
U + V + M
E.1 The Space-Time Analysis
We provide the approximated space-time complexity in Table 5, compared against those of the essential baselines.
Let W be the model size, N be the number of instances, H be the number of other models (for FedFOMO algorithms), U be the size of the rank-1 vectors (or matrices for pFedPara ) in
U , V be that of V , M be the that of M , v be the vector for the second last layer from V .For training procedure of FedFOMO , it requires validation step for H number of other model weights on their validation set at each local client.
Note that, for communication costs of our methods ( Fac.-FL ), ( U · V + M ) ≈ W and thus ( U + V + M ) <
W for both Client-to-Server (C2S) and Server-to-Client (S2C) communication costs.
Please also refer to the actual size of the data transmission for the communication cost in Table1.
E.2 Sparsity Analysis on FL Scenarios
In the main document, we show the effect of model size and sparsity controlled by λ sparsity for a single model.
In this section, we analyze it under federated learning scenario.
In Figure 8 (a), we show the performance over model size in domain heterogeneous scenarios.
As shown, our method shows superior performance even with around 65% of the model size over the baseline model that achieves the best performance ( Per-FedAvg ) amongst other baseline models.
With 50% sparsity, ours still shows competitive performance compared to Clustered-FL and FedAvg , while it starts being significantly degenerated when sparsity becomes over 50% . [Figure] sparsity
[Figure]
In Figure 8 (b), we show accuracy over communication costs.
Note that, in our method, the model size is not really related to the communication costs since we send very small portion of model parameters.
For example, even though we use almost full model size ( λ sparsity = 3 e - 4 ), our communication cost is significantly lesser than the other baseline models, as shown in the figure.
E.3 Test Accuracy Curves and Plots for Communication Costs
For label-heterogeneous FL scenario (Table 1 (Top), we provide all test accuracy curves over communication rounds and transmission costs for results of CIFAR-10 and SVHN with stardard
IID/non-IID and permuted IID/non-IID partitions in Figure 10.
For domain-heterogeneous FL scenario (Table 1 (Bottom)), we provide performance of 20 clients In Figure 11.
[Figure]
u r a c
y
[Figure]
u r a c y
[Figure]
A cc u r a c
y
[Figure] [Figure]
Cost [Gb
[Figure]
A cc u r a c
y
Standard IID (CIFAR
[Figure] [Figure]
Cost [Gb
[Figure]
A cc u r a c
y
Standard NonIID (CIFAR-10)
[Figure]
Permuted
[Figure]
Transmission Cost [Gb]
[Figure]
A cc u r a c
y
Standard IID (SVHN)
[Figure]
[Table]
[Figure]
[Table]
[Figure]
C li e n t C li e n t
C li e n t C li e n t
C li e n t C li e n t
C li e n t C li e n t
C li e n t
C li e n t0 C li e n t 1 1 C li
e
n t 1 2 C li e n t 1 3 C li e n t 1 4 C li e n t
C li e n t6 C li e n t 1 7 C li e n t 1 8 C li e n t 1 9 C li e n t
[Figure]