Factorized-FL: Personalized Federated Learning with Parameter Factorization & Similarity Matching
Abstract
In real-world federated learning scenarios, participants could have their own personalized labels incompatible with those from other clients, due to using different label permutations or tackling completely different tasks or domains.
However, most existing FL approaches cannot effectively tackle such extremely heterogeneous scenarios since they often assume that (1) all participants use a synchronized set of labels, and (2) they train on the same tasks from the same domain.
In this work, to tackle these challenges, we introduce Factorized-FL , which allows to effectively tackle label- and task-heterogeneous federated learning settings by factorizing the model parameters into a pair of rank-1 vectors, where one captures the common knowledge across different labels and tasks and the other captures knowledge speciﬁc to the task for each local model.
Moreover, based on the distance in the client-speciﬁc vector space, Factorized-FL performs a selective aggregation scheme to utilize only the knowledge from the relevant participants for each client.
We extensively validate our method on both label- and domain-heterogeneous settings, on which it outperforms the state-of-the-art personalized federated learning methods.
The code is available at https://github.com/wyjeong/Factorized-FL.
Introduction
Personalized Federated Learning (PFL) aims to utilize the aggregated knowledge from other clients while learning a client-speciﬁc model that is specialized for its own task and data distribution, rather than learning a universal, global model for tackling all local tasks [1, 16, 5, 28].
While various personalized federated learning approaches have shown success in alleviating the data heterogeneity problem, yet, they are also limited as they follow the common assumptions of the standard federated learning setting, that (1) all participants use a synchronized set of labels and (2) all clients tackle the same task from the same domain.
In many real-world scenarios, the ﬁrst assumption may not hold since the labels for the same task could be differently annotated depending on the user environment, i.e. language or nationality (Figure 2 Left).
In other words, when working with the same set of semantic classes, the labels across multiple clients could be completely different.
For example, the same “Car" class may be given the label “Vehicle" or “Auto".
The second assumption severely limits the pool of devices or institutions that can participate in the collaborative learning process.
However, clients working on different tasks and domains may have similar classes, or a common underlying knowledge, that may be helpful for the local models being trained at other clients (Figure 2 Right).
Thus, it would be helpful if we can allow the local models with heterogeneous tasks and domains to communicate the common knowledge across them.
[Figure] [Figure]
However, federated learning under label and domain heterogeneity is a nontrivial problem, as most methods suffer from severe performance degeneration in such settings (Table 1).
We analyze this phenomenon in Figure
[Figure] Speciﬁcally, we train equally-initialized models on four different datasets and observe how different the gradient updates become as training goes on (we measure normalized L 2 distance between them).
Learning on two
MNIST partitions will yield the smallest difference between gradients (Figure 1 (a) and (b) Gray).
Interestingly, simply permuting the labels of the partition 2 makes the gradient updates to largely diverge from the original gradients, compared to those of the model trained with synchronized labels (Figure 1 (a) and (b) Red).
Moreover, learning on the completely different dataset (CIFAR-10) makes model gradients diverge more severely compared to learning on the dataset with permuted labels (Figure 1 (a) and (b) Blue).
The heterogeneity in the labels and domains tackled by each participant leads to severe performance degeneration when performing federated learning (Figure 1 (c)).
We measure performance on MNIST partition 1 while aggregating a model trained on the different dataset for every 3 epochs.
We observe averaged models suffer from crucial performance degeneration in both FL scenarios.
Note that the classiﬁer layers are not shared when performing federated learning.
However, such label- and domainheterogeneous FL settings have been relatively overlooked.
Although few previous works [1, 16, 25] brieﬂy introduced such extreme settings as motivations, they did not formally deﬁne the problems nor tackle such scenarios, and rather targeted standard non-IID scenarios.
We name this challenging problem as the Agnostic Personalized Federated Learning (APFL) problem, where participants with personalized labels or from multiple domains can collaboratively learn while beneﬁting each other.
An APFL problem has two critical challenges: (1) Label Heterogeneity for the discrepancy of the labels, due to the lack of a synchronized labeling scheme across the clients; and (2) Domain Heterogeneity for the discrepancy in the task and domains tackled by each participant.
To tackle these challenges, we propose a novel FL framework, dubbed Factorized-FL , which factorizes model parameters into basis vectors and aggregate them in the factorized parameter space.
This allows to factorize the model aggregation to take place in a semantic parameter basis space which is more robust to the use of different labels and data.
Also, the factorization results in the separation of the client-general and client-speciﬁc knowledge, and thus prevents the aggregation of incompatible knowledge across clients.
Moreover, to further alleviate the model from collapsing into a degenerate solution, we measure the task similarity across the clients using the factorized parameters, to allow selective aggregation of the knowledge among the relevant models that work on similar tasks or domains.
We extensively validate our method on both label- and domain-heterogeneous settings, and show that it signiﬁcantly outperforms the current state-of-the-art methods with lower communication cost due to using factorized parameters.
Our contributions can be summarized as follows:
• We introduce the Agnostic Personalized Federated Learning (APFL) problem and study its two critical challenges, Label and Domain Heterogeneity.
• We propose Factorized-FL for tackling the APFL problem, which factorizes model parameters to reduce parameter dimensionality for alleviating knowledge collapse, and utilize them to measure task-level similarity for matching relevant clients.
• We validate our method in both label- and domain-heterogeneous scenarios and show our method outperforms the current state-of-the-art personalized federated learning methods.
Related Work
Federated learning Many federated learning algorithms have been proposed since the introduction of FedAvg [18], but, we speciﬁcally focus on works that aim to tackle the heterogeneity problems, e.g.
Non-IIDness.
Some studies focus on regularization methods [19, 15], correcting disparity between server and clients [26, 9], or contrastive learning [13].
Recently, many existing works also tackle architecture-level heterogeneity [24, 29, 3].
Personalized federated learning aims to improve the individual local models instead of the universal global model, via the mixture methods [17, 2, 7], meta-learning [5], partial network aggregation [1, 16], or hyper-networks [25].
Recent approaches aim to enhance personalization performance by avoiding to aggregate irrelevant knowledge that are not helpful.
Zhang et al.
[28] download the models from other clients to aggregate only the beneﬁcial ones for the local task at each client.
Sattler et al.
[23], Duan et al.
[4] measure client-wise similarity by using the gradient updates.
Our method also measures client similarity but in a more efﬁcient way, by utilizing 1D vectors from factorization.
Re-parameterization for federated learning Jeong et al.
[8], Yoon et al.
[27] decompose model parameters (using an additional set of parameters) to train them with different objectives, which do not reduce the dimensionality of model parameters.
Some approaches factorize high dimensional model parameters into low rank matrices.
Kone ˇ cn ` y et al.
[10] introduce structured update which model directly learns factorized parameter space, and Nam et al.
[20] propose to use the Hadamard product of low rank matrices to enhance communication efﬁciency.
Unlike prior works, we utilize rank-1 vectors factorization to tackle heterogeneity in domains and tasks, by allowing the 1D vectors to capture task-general and the client-speciﬁc knowledge.
Also, we minimize the information loss from factorization by utilizing sparse bias matrices.
Problem Deﬁnition
We begin with the formal deﬁnition of the conventional federated learning scenario, and then introduce our novel Agnostic Personalized Federated Learning (APFL) problem.
3.1 Preliminaries
Our goal is to solve a multi-class classiﬁcation task under an FL setting.
Let f g be a global model (neural network) at the global server and F = { f k } Kk =1 be a set of K local neural networks, where K is the number of local clients.
D =
{ x i , y i } Ni =1 be a given dataset, where N is the number of instances, x i 2 R W ⇥ H ⇥ D is the i th examples in a size of width W , height H , and depth D , with a corresponding target label y i 2 { 1 , . . .
, C } for the C -way multi-class classiﬁcation problem.
The given dataset
D is then disjointly split into K sub-partitions D k = { x k,i , y k,i } N k i =
1 s.t. D =
S
Kk =1 D k , which are distributed to the corresponding local model f k .
Let R be the total number of the communication rounds and r denote the index of the r th communication round.
At the initial round r = 1 , the global model f g initializes the global weights ✓ (1) f g and broadcasts ✓ (1) f g to an arbitrary subset of local models that are available for training at round r , such that F ( r ) ⇢ F , |F ( r ) | =
K ( r ) , and K ( r )  K , where
K ( r ) is the number of available local models at round r .
Then the active local models f k 2 F ( r ) perform local training to minimize loss L ( ✓ ( r ) k ) on the corresponding sub-partition P k and update their local weights ✓ ( r +1) k ✓ ( r ) k
⌘ r L ( ✓ ( r ) k ) , where ✓ ( r ) k is the set of weights for the local model f k at round r and L ( · ) is the loss function.
When the local training is done, the global model F collects and aggregates the learned weights ✓ ( r +1)
f g N k N P K ( r ) i =1 ✓ ( r ) k
and then broadcasts newly updated weights to the local models available at the next round r + 1 .
These learning procedures are repeated until the ﬁnal round R .
On the other hand, Personalized Federated Learning aims to adapt the individual local models f 1: K to their local data distribution D 1: K , to obtain specialized solution for each task at the local client, while utilizing the knowledge from other clients.
Thus merging the local knowledge for personalized FL is not necessarily done in the form of ✓ ( r +1)
f g
[Equation]
K
[Equation]
r )
i
[Equation]
1 ✓ ( r ) k , and the speciﬁc ways to utilize the knowledge from others depends on the speciﬁc algorithm, i.e. ✓ ( r +1) k
[Equation]
where ! ( · ) is weighing function [28]. = = = ⨁
[Figure]
" ⊺ # # ! " " "⊺ ! # " #⊺ $ % ⊺ Global Weights Local Weights
Global Vector Local Vector # Conventional Factorization
Hadamard Product of Low Rank Matrices
Local Bias
Rank 1 Vector + Highly Sparse Bias (Ours) (Filter Coefficient) (Filter Base) ⊙ ("×$) ("×$) ("×$) & " & # ("×&) ($×&) ($×&) ($×&) ($×&) ('×&) ('×() ($×()
[Figure]
3.2 Agnostic Personalized Federated Learning
Agnostic Personalized Federated Learning (APFL) is a scenario where any local participants from diverse domains with their own personalized labeling schemes can collaboratively learn, beneﬁting each other.
There exist two critical challenges that need to be tackled to achieve this objective: (1)
Label Heterogeneity and (2) Domain Heterogeneity.
Label Heterogeneity
This scenario assumes that the labeling schemes are not perfectly synchronized across all clients, as described in Section 1 and Figure 2 Left.
Most underlying setting for this scenario is the same as the conventional single-domain setting with synchronized labels that is described in Section 3.1, except that labels are arbitrarily permuted amongst clients.
The local data
D k for the local model f k is now deﬁned as D k = { x k,i , ' k ( y k,i ) }
N k i =1 , where ' k ( · ) is a mapping function for the local model f k which maps a given class y k,i with a randomly permuted label p k,i = ' k
( y k,i ) .
Let the j th layer out of L layers in the neural networks of local model f k be ` jk and the last layer ` Lk be the classiﬁer layer.
Since each client has differently permuted labels, the personalized classiﬁers ` L 1: K are no longer compatible to each other.
While we can merge the layers below the classiﬁer in this setting, training with heterogeneous labels could still lead to large disparity in the local gradients even in the initial communication round, as described in Figure 1.
Domain Heterogeneity
This scenario presumes that local clients learn on their own dataset D , that are completely different from the datasets that are used at other clients, as described in Section 1 and
Figure 2 Right.
In this setting, K disjoint datasets D 1: K are assigned to the K local clients f 1: K , where D k = { x k,i , y k,i } N k i
=1 is the dataset assigned to the local model f k .
The number of target classes may differ across clients, such that y k,i 2 { 1 , . . .
, C k } .
We assume complete disjointness across clients, such that there is no instance- and class-wise overlap across the datasets: ? =
T Kk =1 D k and ? =
T Kk =1 C k , where C k is a set of classes for client k .
Similarly to the label-heterogeneous scenario, the personalized classiﬁers ` L 1: K are no longer compatible to each other due to the heterogeneity in the data and the labels.
Hence, the aggregation is done for the layers before the classiﬁer, but they will be also incompatible as the learned model weights will be largely different across domains.
Factorized Federated Learning 4.1 Factorization of Model Parameters
Wang et al.
[26] discussed that the conventional knowledge aggregation, that is often performed in a coordinate-wise manner, may have severe detrimental effects on the averaged model.
This is because the deep neural networks have extremely high-dimensional parameters and thus meaningful element-wise neural matching is not guaranteed when aggregating the weights across different models trained under diverse settings.
One naive solution to this problem is to factorize model parameters into lower dimensional space, i.e. low rank matrices, as shown in Figure 3 (Left).
Conventional approaches, such as SVD, Tucker, or Canonical Polyadic decomposition, however, factorize model parameters after training [12, 22] is done.
Thus, the dimensionality at the time of knowledge aggregation will remain the same as the unfactorized model.
Koneˇcn`y et al.
[10], Nam et al.
[20] pre-decompose model parameters to low rank matrices for FL scenarios.
While Koneˇcn`y et al.
[10] use naive low rank matrices, Nam et al.
[20] use two sets of low rank matrices to improve expressiveness and utilize them as global and local weights (Figure 3 (Middle)).
Unlike prior works, our approach utilizes rank-1 vectors to perform aggregation in the lowest subspace possible for compatibility, while effectively yet efﬁciently enhancing expressiveness with sparse bias matrices, as shown in Figure 3 (Right) and Figure 5 (Left).
Another crucial difference of our method from the previous factorization methods is that our rankvectors have distinct roles.
Our factorization will separate the common knowledge from the task- or domain-speciﬁc knowledge, since u could be thought as the bases (the common knowledge across clients) and v could be thought as the coefﬁcients (client-speciﬁc information).
[Figure]
In Figure 4 (a) and (b), which show the experimental results with the factorized model, we observe that u trained on two datasets becomes closer to each other (MNIST Partition 1) while v (personalized ﬁlter coefﬁcient) remain largely different as FL goes on.
With this observation, we further aggregate u while allowing v to be different across clients, to allow personalized FL.
Further, we use the client speciﬁc v for similarity matching, to identify relevant local models from other clients.
In the following paragraphs, we describe our factorization method in detail, for both fully-connected and convolutional layers.
Factorization of Fully-Connected Layers We assume that each local model f k has a set of local weights ✓ k across all layers; that is, ✓ k = { W ik } Li =1 .
The dimensionality of the dense weight W ik for each fully connected layer is W ik 2 R
I ⇥ O , where I and O indicate respective input and output dimensions.
We can reduce the I ⇥ O complexity by factorizing the high order matrix into the outer product of two vectors as follows:
[Equation]
However, such extreme factorization of the weight matrices may result in the loss of expressiveness in the parameter space.
Thus, we additionally introduce a highly sparse bias matrix µ to further capture the information not captured by the outer product of the two vectors as follows:
[Equation]
We initialize µ with zeros so that it can gradually capture the additional expressiveness that are not captured by u and v during training.
We can control its sparsity by the hyper-parameter for the sparsity regularizer described in 4.3.
Factorization of Convolutional Layers
The difference between the fully-connected and convolutional layers is that the convolutional layers have multiple kernels (or ﬁlters) such that
W ik 2 R F ⇥ F ⇥ I ⇥ O , where F is a size of ﬁlters (we assume the ﬁlter size is equally paired for the simplicity).
To induce u to capture base ﬁlter knowledge and v to learn ﬁlter coefﬁcient, it is essential to design u 2 R F · F and v 2 R I · O , but not in arbitrary ways, such as u 2 R I · F and v 2 R O · F or u 2 R O and v 2 R I · F · F .
We observe that performance is degenerated when the parameters are ambiguously factorized (Figure 7 (h)).
Our proposed factorization method for convolutional layers are as follows:
[Equation]
⇡ ( · ) is the weight reshaping function.
Note that we reparameterize our model at initialization time .
Then we reconstruct and train full weights of each layer W 1: L k , while optimizing u 1: L k , v 1: L k , and µ 1: L k , respectively, during training phase.
4.2 Similarity Matching
Since we assume label- and domain-heterogeneous FL scenarios, aggregating the parameter bases across all clients may not be optimal, since some of them could be highly irrelevant.
Yoon et al.
[27] and Zhang et al.
[28] also demonstrate that avoiding aggregation of irrelevant models from other clients improves local model performance.
Yoon et al.
[27] achieve this goal by taking the weighted combination of task-speciﬁc weights from other clients, and Zhang et al.
[28] suggest downloading the models from other clients and evaluating their performance on a local validation set, at each client.
However, since they require additional communication and computing cost at the local clients, we provide a more efﬁcient yet effective approach to match models that are beneﬁcial to each other.
× ! " # $ " $ ! " ×" % ⊕ µ ! " # $ $
Filter Base Filter Coefficients
Sparse Bias Matrix 12345 … '() $ " $ 1 2 3 4'() ...
123… %&' Filter( ) ! (×) !
R e c o n s t r u c t
e
d K e r n e l s
Filter
Kernel Reshaping $ $ $ " $ ! "
Factorized Parameter Space
C li e n t
C li e n t C li e n t
K
Similarity Matching 0.
#$ ! ! " "#$ ! ! # "#$ ⊗ ⊗ ⊗ # !
[-𝝁 𝒇 𝑲 𝐮 𝒇 𝟐 𝐯 𝒇 𝟐 𝝁 𝒇𝐮 𝒇 𝟏 𝐯 𝒇 
 𝝁 𝒇-]# ⊕ ! !
[Figure]
Efﬁcient similarity matching Our method utilizes a factorized vector v k for measuring similarity across different models, at the central server.
Since v are devised to learn personalized coefﬁcients, we assume that clients trained on similar tasks or domains will have similar v .
Speciﬁcally, we use v L  1 k of the layer before the classiﬁer layer for similarity matching.
The similarity matching function ⌦ ( · ) , is deﬁned as the cosine similarity between target client f k and the other clients { f i } Ki 6 =
k , as follows:
[Equation]
The similarity scores for those with the cosine similarity scores lower than the given threshold ⌧ , are set to zero.
Our method is signiﬁcantly more efﬁcient than the similarity matching approaches which use full gradient updates for clustering clients [23, 4], as we only use single vector v L  1 k from each client k that are obtained without additional cost.
Personalized weighted averaging We allow each local model to individually perform weighted aggregation of the model weights from other clients, utilizing their similarity scores:
[Equation]
where ✏ is a hyperparameter for scaling the similarity score  i .
We always set  k , the similarity score for itself, as 1 . 0 .
We name this model that aggregates only basis vector u k from each client k as Factorized-FL ↵ .
This will result in encoding common knowledge across local models in u k which is especially suitable for domain-heterogeneous scenarios.
However, for label-heterogeneous scenarios, we empirically observed that aggregating all parameters results in better performance, as the factorization helps averaging them in a more compatible space.
We thus introduce an additional variant, namely Factorized-FL  , which aggregates not only the parameter bases across all clients, but also coefﬁcient and bias terms as well.
Yet, note that Factorized-FL  require signiﬁcantly larger communication overhead over Factorized-FL ↵ .
4.3 Learning Objective
Now we describe our ﬁnal learning objective.
Instead of utilizing the single term ✓ k for local weights of neural network f k , now let U k , V k , and M k be sets of u k , v k , and µ k of all layers in f k , s.t.
U k
= { u ik } Li =1 , V k
= { v ik } Li =1 , and M k = { µ ik } Li
=1 , then our local objective function is, min
[Equation]
where L is the standard cross-entropy loss performed on all minibatch B 2 D k .
We add the L 1 sparsity inducing regularization term to make the bias parameters highly sparse, controlling its effect with a hyperparameter  sparsity .
As for the full training procedure, please see the pseudo-code of the algorithm in the supplementary ﬁle (Section A).
[Table] [Table]
[Table] [Table] [Table]
[Table] [Table]
[Figure]
Experiment 5.1 Experimental Setup
Baselines We ﬁrst consider well-known baseline FL methods, such as (1) FedAvg [18] and (2)
FedProx [14].
We evaluate our factorization technique with (3) pFedPara [20] which also uses kernel factorization technique for personalized FL scenarios.
Our similarity matching approach is compared to (4) Clustered-FL [23] and (5) FedFOMO [28], which measuring client-wise similarity or helpfulness.
(6) Per-FedAvg [5] is also used for evaluation as it shows great performance on heterogeneous federated learning scenarios.
We also show local training model, (7) Stand-Alone , for the lower bound performance.
Detailed information are described in Appendix (Section C).
Datasets (1) Label Heterogeneous Scenario : we use CIFAR-10 [11] and SVHN [21] datasets and create four different partitions for each dataset, which are conventional IID and non-IID as well
[Figure]
as permuted IID and permuted non-IID, whose labels are permuted and incompatible to each other.
We split the datasets into 20 partitions and then simply permuted the labels on the same partition for label permuted settings.
(2) Domain Heterogeneous Scenario : we use CIFAR-100 datasets [11] and create ﬁve sub-datasets grouped by 10 similar classes, such as Household Objects, Fruits&Foods,
Trees&Flowers, Transport, and Animals.
We assign 4 clients for each sub-dataset, i.e. Client 1to Household Objects, Client 5-8 to Fruits&Foods, Client 9-12 to Trees&Flowers, Client 13-16 to
Transport, and Client 17-20 to Animals.
We then permute the labels for all partitions to simulate further realistic scenarios.
Further descriptions are elaborated in Appendix (Section B).
5.2 Experimental Result
Label-heterogeneous FL As shown in Table 1 (Top), for the standard IID and Non-IID settings, all
FL methods obtain higher performance than the local training baseline ( Stand-Alone ), which conﬁrms that the locally learned knowledge is beneﬁcial to others, when the data and label distributions are homogeneous across clients.
However, when the labels are not synchronized across all clients (Permuted IID/Non-IID), all previous FL methods suffer from signiﬁcant performance degeneration, even lower than that of the local training baseline.
Again, note that we do not share the classiﬁer layers to ensure fairness across all algorithms in these permuted settings.
We conjecture that this is caused by the label permutation leading the local model to evolve a permuted set of features that are not coordinate-wise compatible to others when aggregated.
Contrarily, our method Factorized-FL ↵ shows consistent performance regardless of whether the labels are permuted or not.
Factorized-FL  even largely outperforms all baselines with signiﬁcantly superior performance.
Test accuracy curves over the communication round and transmission cost are in Appendix (Section D).
Domain-heterogeneous FL Table 1 (Bottom) shows the experimental results for the domain and label heterogeneous scenarios.
We observe that the conventional FL baselines, i.e. FedAvg , FedProx , fail to obtain better performance over purely local training baseline ( Stand-Alone ) due to the naive aggregation of extremely heterogeneous knowledge, which causes detrimental knowledge collapse.
FedFOMO and Clustered-FL shows slightly higher performance ( 1 - 2% p ) over Stand-Alone model, as they can avoid irrelevant clients when aggregating local knowledge.
The other personalized FL methods, i.e. Per-FedAvg and pFedPara , also show 1 - 2% p higher performance over Stand-Alone model as they are specialized for personalized FL scenarios.
However, on average, our method largely outperforms all baseline models even with the smallest communication costs ( Factorized-FL ↵ ), as shown in Figure 6 (e).
In the ﬁgures, we plot the convergence rate of our Factorized-FL ↵ &  over communication round (Top) and transmission cost (Bottom), compared to the baseline models.
Both models consistently obtain superior performance compared to the other models.
Particularly, unlike pFedPara which uses low rank matrices for knowledge sharing, Factorized-FL ↵ only communicates with factorized vectors, such as U for base knowledge sharing and v L  1 k for similarity matching and this allows our method to largely reduce the communication costs while showing the best performance as shown in Figure 6 (e) Bottom.
Effect of kernel factorization In Figure 7 (g), we perform an ablation study of our factorization method in the domain-heterogeneous scenario.
To clearly see the effectiveness of our factorization methods, we compare Factorized-FedAvg , a variant of Factorized-FL  without similarity matching, against FedAvg .
As shown, Factorized-FedAvg achieves higher performance over the original FedAvg model.
As the only difference between the two is whether kernel is factorized or not, this clearly demonstrates that our factorization method alone improves the model performance by alleviating knowledge collapse.
We further analyze the effect of the sparse bias matrix M .
When we remove M from the Factorized-FedAvg model, Factorized-FedAvg w/o µ in the ﬁgure, we observe large performance drop.
This shows that that the bias term is essential in compensating for the loss of expressiveness from the rather extreme factorization.
With only U and V , we use 90% less model parameters ( 0 . 27 M ) compared to the regular kernel model ( 2 . 574 M ).
Also, in Figure 7 (k), we observe that model factorized by u 2 R F · F and v 2 R I · O perform better than the model factorized by u 2 R I · F and v 2 R O · F .
This is because in the former case, the factorization will separate the base knowledge from the task-speciﬁc knowledge, as described in Figure 5 (Left), but in the latter case the factorization does not have such a natural interpretation.
We also show the scalability and reliability of our factorization method with different architectures, consisting of convolutional, fully-connected layers, batch normalization, and skip connections as shown in Figure 7 (k).
Effect of similarity matching To verify the effectiveness of our similarity matching method using the personalized factorized vector v , we visualize the inter-client similarity of u L  1 f k and v L  1 f k from the second last layer of 20 clients in a domain heterogeneous setting.
As shown in Figure 7 from (a) round 1 to (e) round 100, we observe u L  1 f k (upper row) are indeed highly correlated with other clients (the darker color indicate higher similarity scores) while v L  1 f k (bottom row) are relatively uncorrelated to each other, as expected as our assumption that U f k capture base knowledge across all clients and V f k capture personalized knowledge.
Further, we also observe that v L  1 f k obtains higher similarity to that of the parameters trained on the same domains (but with permuted labels), i.e. Client 1-4, Client 5-8, Client 9-12, Client 13-16, and Client 17-20, showing that they are effective in capturing task- and domain-level similarities across models.
We further visualize the frequency of the client matching across clients in Figure (f), after 100 communication rounds.
With only a single vector parameter v L  1 f k , our method ﬁnds which clients’s knowledge will be helpful to which clients.
For further analysis on our similarity matching, we compare it against Random and Worst Matching baselines under the multi domain scenario.
The random matching baseline randomly selects three arbitrary models to be aggregated at each round, while the worst matching baseline selects three most dissimilar models.
As shown in Table 7 (g), both baselines signiﬁcantly suffer from the performance degeneration compared to our similarity matching algorithm, which shows its effectiveness.
Analysis on  , ⌧ , and  sparsity
Figure 7 (h) and (i) show the analysis of the hyperparameters,  and ⌧ , on the domain-heterogeneous dataset.
As shown in Figure 7 (h), the performance slightly decreases with larger  , since higher rank weight matrices may result in larger coordinate-wise incompatibility of the parameters, which could lead to knowledge collapse when performing aggregation.
Also, the lower ⌧ s show the lower performance, as less similar knowledge can be aggregated more frequently.
In Figure 7 (j), we train Client 1 on CIFAR-10 IID Partition 1 for 20 epochs using both factorized and regular models.
As shown, our factorized model outperforms regular models (2 . 574 M ) even with 30% less parameters (1 . 784 M ) .
We further analyze the sparsity in Section D.2 of Appendix.
Conclusion
We introduced a realistic federated learning scenario where the labeling schemes are not synchronized across all participants (label heterogeneity) and the tasks and the domains tackled by each local model are different from those of others (domain heterogeneity).
We then proposed a novel method to tackle this problem, whose local model weights are factorized into the product of two vectors plus a sparse bias term.
We then aggregate only the ﬁrst vectors, for them to capture the common knowledge across clients, while allowing the other vectors and the sparse bias term to be client-speciﬁc, accounting for label and domain heterogeneity.
Further, we use the client-speciﬁc vectors to measure the similarity scores across local models, which are then used for weighted averaging, for personalized federated learning of each local model.
Our method not only avoids knowledge collapse from aggregating incompatible parameters across heterogeneous models, but also signiﬁcantly reduces the communication costs.
We validate our method on both label and domain heterogeneous settings, on which it largely outperforms relevant baselines.
Limitations Federated Learning (FL) is devised to protect the privacy of sensitive personal data by strictly avoiding sharing them with other users, institutions, and companies.
The common limitation in FL is that, even though almost all existing methods regard sharing locally learned parameters of gradients as safe, they still may contain some possibility to reproduce the local data distribution, which violates the preservation of privacy.
Although our work alleviates this probability by signiﬁcantly reducing the dimensionality of parameters and partially sharing them, it may be still sub-optimal in an aspect of security, which is an open research domain for federated learning.
Potential Societal Impact Federated learning is a collaborative learning framework where multiple clients can actively participate in the learning procedure anywhere and anytime.
Nowadays, numerous terminal devices, i.e. smartphones, smartwatches, IoT devices, etc, are utilized as participants of the collaborative learning scheme.
However, as the number of participants in federated learning increases, the number of communications between them increases accordingly.
Such situations leading to the exhaustive communication costs may worsen the global environmental crisis , i.e. global warming.
Even though our method can alleviate such risks, since ours signiﬁcantly reduces the communication costs by partially sharing the factorized vectors, it is essential to develop further optimal solutions for compressing the knowledge in both efﬁcient and effective ways for federated learning that the massive number of participants collaborate.
[-5 Cat 5 9 0 7 9 3 8 8          8 Deer 6 2 6 9 6 5 2 1          9 Dog 4 1 4 8 5 6 6 6          
-]
[-Frog 9 3 2 1 2 0 3 2          1 Horse 0 7 9 4 3 7 0 3          6 Ship 1 8 7 6 7 4 5 7          2 Truck 7 6 8 2 0 1 7 
-]
[-Digit 2 5 3 0 8 2 4 4          7 Digit 8 4 1 5 1 8 1 0          4 Digit 3 0 5 3 4 9 9 5          5 Digit 5 9 0 7 9 3 8 8          8 Digit 6 2 6 9 6 5 2 1          9 Digit 4 1 4 8 5 6 6 6          0 Digit 9 3 2 1 2 0 3 
-]
Acknowledgments and Disclosure of Funding
This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT)
(No.2019-0-00075, Artiﬁcial
Intelligence Graduate School Program(KAIST)).
It was also results of a study on the “HPC Support”
Project, supported by the ‘Ministry of Science and ICT’ and NIPA.
[-n t 1 2 C li e n t 1 3 C li e n t 1 4 C li e n t 
-]
[-C li e n t6 C li e n t 1 7 C li e n t 1 8 C li e n t 1 9 C li e n t 
-]