Factorized-FL: Personalized Federated Learning with Parameter Factorization & Similarity Matching
Anonymous Author(s)
AffiliationAddress email
Abstract
In real-world federated learning scenarios, participants could have their own personalized labels which are incompatible with those from other clients, due to usingdifferent label permutations or tackling completely different tasks or domains.
However, most existing FL approaches cannot effectively tackle such extremelyheterogeneous scenarios since they often assume that (1) all participants use asynchronized set of labels, and (2) they train on the same tasks from the samedomain.
In this work, to tackle these challenges, we introduce Factorized-FL ,which allows to effectively tackle label- and task-heterogeneous federated learningsettings by factorizing the model parameters into a pair of rank-1 vectors, whereone captures the common knowledge across different labels and tasks and the othercaptures knowledge specific to the task for each local model.
Moreover, based onthe distance in the client-specific vector space, Factorized-FL performs selectiveaggregation scheme to utilize only the knowledge from the relevant participantsfor each client.
We extensively validate our method on both label- and domainheterogeneous settings, on which it outperforms the state-of-the-art personalizedfederated learning methods.
Introduction
Personalized Federated Learning (PFL) aims to utilize the aggregated knowledge from other clientswhile learning a client-specific model that is specialized for its own task and data distribution, ratherthan learning a universal, global model for tackling all local tasks [1, 16, 5, 28].
While variouspersonalized federated learning approaches have shown success in alleviating the data heterogeneityproblem, yet, they are also limited as they follow the common assumptions of the standard federatedlearning setting, that (1) all participants use a synchronized set of labels, and (2) all clients tackle thesame task from the same domain.
In many real-world scenarios, the first assumption may not hold since the labels for the same taskcould be differently annotated depending on the user environment (Figure 2 Left).
For example,when working with the same set of semantic classes, the labels across multiple clients could havea completely different ordering of the classes.
At client 1, the label 1 may denote the “Ship" class,while at client 2, the label 10 may denote the same class.
Also, the same “Car" class may be giventhe label “Vehicle" or “SUV".
The second assumption severely limits the pool of devices or institutions that can participate in thecollaborative learning process.
However, clients working on different tasks and domains may havesimilar classes, or a common underlying knowledge, that may be helpful for the local models beingtrained at other clients (Figure 2 Right).
Thus, it would be helpful if we can allow the local modelswith heterogeneous tasks and domains to communicate the common knowledge across them.
[Figure] [Figure]
However, federated learning under label and domain heterogeneity is a nontrivial problem, as most methods suffer from severe performance degeneration in such settings (Table 1).
We analyze this phenomenon in Figure 1.
Specifically, we train equally-initialized models on fourdifferent datasets and observe how different the gradientupdates become as training goes on (we measure normalized L 2 distance between them).
Learning on two
MNIST partitions will yield the smallest difference between gradients (Figure 1 (a) and (b) Gray).
Interestingly,simply permuting the labels of the partition 2 makes thegradient updates to largely diverge from the original gradients, compared to those of the modeltrained with synchronized labels (Figure 1 (a) and (b) Red).
Moreover, learning on the completelydifferent dataset (CIFAR-10) makes model gradients diverge more severely compared to learning onthe dataset with permuted labels (Figure 1 (a) and (b) Blue).
The heterogeneity in the labels and domains tackled by each participant leads to severe performancedegeneration when performing federated learning (Figure 1 (c)).
We measure performance on MNISTpartition 1 while aggregating a model trained on the different dataset for every 3 epochs.
We observeaveraged models suffer from crucial performance degeneration in both FL scenarios.
Note that theclassifier layers are not shared when performing federated learning.
However, such label- and domainheterogeneous FL settings have been relatively overlooked.
Although few previous works [1, 16, 25]briefly introduced such extreme settings as motivations, they did not formally define the problemsnor tackle such scenarios, and rather targeted standard non-IID scenarios.
We name this challenging problem as the Agnostic Personalized Federated Learning (APFL) problem,where participants with personalized labels or from multiple domains can collaboratively learn whilebenefiting each other.
An APFL problem has two critical challenges: (1) Label Heterogeneity for thediscrepancy of the labels, due to the lack of a synchronized labeling scheme across the clients; and( ) Domain Heterogeneity for the discrepancy in the task and domains tackled by each participant
To tackle these challenges, we propose a novel FL framework, dubbed Factorized-FL , whichfactorizes model parameters into basis vectors and aggregate them in the factorized parameter space.
This allows to factorize the model aggregation to take place in a semantic parameter basis space whichis more robust to the use of different labels and data.
Also, the factorization results in the separationof the client-general and client-specific knowledge, and thus prevents the aggregation of incompatibleknowledge across clients.
Moreover, to further alleviate the model from collapsing into a degeneratesolution, we measure the task similarity across the clients using the factorized parameters, to allowselective aggregation of the knowledge among the relevant models that work on similar tasks ordomains.
We extensively validate our method on both label- and domain-heterogeneous settings, andshow that it significantly outperforms the current state-of-the-art methods with lower communicationcost due to using factorized parameters.
Our contributions can be summarized as follows:
• We introduce the Agnostic Personalized Federated Learning (APFL) problem and study its twocritical challenges, Label and Domain Heterogeneity.•
We propose Factorized-FL for tackling the APFL problem, which factorizes model parameters toreduce parameter dimensionality for alleviating knowledge collapse, and utilize them to measuretask-level similarity for matching relevant clients.
• We validate our method in both label- and domain-heterogeneous scenarios and show our method outperforms the current state-of-the-art personalized federated learning methods.
Related Work
Federated learning Many federated learning algorithms have been proposed since the introductionof FedAvg [18], but, we specifically focus on works that aim to tackle the heterogeneity problems, e.g.
Non-IIDness.
Some studies focus on regularization methods [19, 15], correcting disparity betweenserver and clients [26, 9], or contrastive learning [13].
Recently, many existing works also tacklearchitecture-level heterogeneity [24, 29, 3].
Personalized federated learning aims to improve the individual local models instead of the universalglobal model, via the mixture methods [17, 2, 7], meta-learning [5], partial network aggregation [1,16], or hyper-networks [25].
Recent approaches aim to enhance personalization performance byavoiding to aggregate irrelevant knowledge that are not helpful.
Zhang et al.
[28] download themodels from other clients to aggregate only the beneficial ones for the local task at each client.
Sattleret al.
[23], Duan et al.
[4] measure client-wise similarity by using the gradient updates.
Our methodalso measures client similarity but in a more efficient way, by utilizing 1D vectors from factorization.
Re-parameterization for federated learning Jeong et al.
[8], Yoon et al.
[27] decompose modelparameters (using an additional set of parameters) to train them with different objectives, which donot reduce the dimensionality of model parameters.
Some approaches factorize high dimensionalmodel parameters into low rank matrices.
Kone ˇ cn ` y et al.
[10] introduce structured update whichmodel directly learns factorized parameter space, and Nam et al.
[20] propose to use the Hadamardproduct of low rank matrices to enhance communication efficiency.
Unlike prior works, we utilizerank-1 vectors factorization to tackle heterogeneity in domains and tasks, by allowing the 1D vectorsto capture task-general and the client-specific knowledge.
Also, we minimize the information lossfrom factorization by utilizing sparse bias matrices.
Problem Definition
We begin with the formal definition of the conventional federated learning scenario, and then introduceour novel Agnostic Personalized Federated Learning (APFL) problem.
3.1 Preliminaries
Our goal is to solve a multi-class classification task under an FL setting.
Let f g be a global model(neural network) at the global server and F = { f k } Kk =1 be a set of K local neural networks, where Kis the number of local clients.
D =
{ x i , y i } Ni =1 be a given dataset, where N is the number of instances,x i ∈ R W × H × D is the i th examples in a size of width W , height H , and depth D , with a correspondingtarget label y i ∈ { 1 , . . .
, C } for the C -way multi-class classification problem.
The given dataset
D is then disjointly split into K sub-partitions D k = { x k,i , y k,i } N k i =
1 s.t. D =
(cid:83) Kk =1 D k , which aredistributed to the corresponding local model f k .
Let R be the total number of the communicationrounds and r denote the index of the r th communication round.
At the initial round r = 1 , the globalmodel f g initializes the global weights θ (1) f g and broadcasts θ (1) f g to an arbitrary subset of local modelsthat are available for training at round r , such that F ( r ) ⊂ F , |F ( r ) | =
K ( r ) , and K ( r ) ≤ K , where
K ( r ) is the number of available local models at round r .
Then the active local models f k ∈ F ( r )perform local training to minimize loss L ( θ ( r ) k ) on the corresponding sub-partition P k and updatetheir local weights θ ( r +1)
k ← θ ( r ) k − η ∇L ( θ ( r ) k ) , where θ ( r ) k is the set of weights for the local modelf k at round r and L ( · ) is the loss function.
When the local training is done, the global model Fcollects and aggregates the learned weights θ ( r +1)
f g ← N k N (cid:80) K ( r ) i =1 θ ( r ) k
and then broadcasts newlyupdated weights to the local models available at the next round r + 1 .
These learning procedures arerepeated until the final round R .
On the other hand, Personalized Federated Learning aims to adapt the individual local models f 1: K totheir local data distribution D 1: K , to obtain specialized solution for each task at the local client, whileutilizing the knowledge from other clients.
Thus merging the local knowledge for personalized FLis not necessarily done in the form of θ ( r +1)
f g ← N k N
[Equation]
K ( r ) i =1 θ ( r ) k , and the specific ways to utilize theknowledge from others depends on the specific algorithm, i.e. θ ( r +1) k
[Equation]
,where ω ( · ) is weighing function [28].
[Figure]
𝐖
Conventional Factorization
Hadamard Product of Low Rank Matrices
[Figure]
3.2 Agnostic Personalized Federated Learning
Agnostic Personalized Federated Learning (APFL) is a scenario where any local participants fromdiverse domains with their own personalized labeling schemes can collaboratively learn, benefitingeach other.
There exist two critical challenges that need to be tackled to achieve this objective: (1)
Label Heterogeneity and (2) Domain Heterogeneity.
Label Heterogeneity
This scenario assumes that the labeling schemes are not perfectly synchronized across all clients, as described in Section 1 and Figure 2 Left.
Most underlying setting forthis scenario is the same as the conventional single-domain setting with synchronized labels that isdescribed in Section 3.1, except that labels are arbitrarily permuted amongst clients.
The local data
D k for the local model f k is now defined as D k = { x k,i , φ k ( y k,i ) }
N k i =1 , where φ k ( · ) is a mappingfunction for the local model f k which maps a given class y k,i with a randomly permuted labelp k,i =
φ k ( y k,i ) .
Let the j th layer out of L layers in the neural networks of local model f k be ℓ jk and thelast layer ℓ Lk be the classifier layer.
Since each client has differently permuted labels, the personalizedclassifiers ℓ L 1: K are no longer compatible to each other.
While we can merge the layers below theclassifier in this setting, training with heterogeneous labels could still lead to large disparity in thelocal gradients even in the initial communication round, as described in Figure 1.
Domain Heterogeneity
This scenario presumes that local clients learn on their own dataset D , thatare completely different from the datasets that are used at other clients, as described in Section 1 and
Figure 2 Right.
In this setting, K disjoint datasets D 1: K are assigned to the K local clients f 1: K ,where D k = { x k,i , y k,i } N k i
=1 is the dataset assigned to the local model f k .
The number of target classesmay differ across clients, such that y k,i ∈ { 1 , . . .
, C k } .
We assume complete disjointness acrossclients, such that there is no instance-wise and class-wise overlap across the datasets: ∅ = (cid:84)
Kk =
1 D kand ∅ = (cid:84) Kk =1 C k , where C k is a set of classes for client k . Similarly to the label-heterogeneousscenario described above, the personalized classifiers ℓ L 1: K are no longer compatible to each other dueto the heterogeneity in the data and the labels.
Hence, the aggregation is done for the layers beforethe classifier, but they will be also incompatible as the learned model weights will be largely differentacross domains.
Factorized Federated Learning
4.1 Factorization of Model Parameters
Wang et al.
[26] discussed that the conventional knowledge aggregation, that is often performedin a coordinate-wise manner, may have severe detrimental effects on the averaged model.
This isbecause the deep neural networks have extremely high-dimensional parameters and thus meaningfulelement-wise neural matching is not guaranteed when aggregating the weights across different modelstrained under diverse settings.
One naive solution to this problem is to factorize model parameters into lower dimensional space,i.e. low rank matrices, as shown in Figure 3 (Left).
Conventional approaches, such as SVD, Tucker,or Canonical Polyadic decomposition, however, factorize model parameters after training [12, 22]is done.
Thus, the dimensionality at the time of knowledge aggregation will remain the same asthe unfactorized model.
Koneˇcn`y et al.
[10], Nam et al.
[20] pre-decompose model parameters tolow rank matrices for FL scenarios.
While Koneˇcn`y et al.
[10] use naive low rank matrices, Namet al.
[20] use two sets of low rank matrices to improve expressiveness and utilize them as global and local weights (Figure 3 (Middle)).
Unlike prior works, our approach utilizes rank-1 vectors toperform aggregation in the lowest subspace possible for compatibility, while effectively yet efficientlyenhancing expressiveness with sparse bias matrices, as shown in Figure 3 (Right) and Figure 5 (Left).
Another crucial difference of our method from the previous factorization methods is that our rank-1vectors have distinct roles.
Our factorization will separate the common knowledge from the task- ordomain-specific knowledge, since u could be thought as the bases (the common knowledge acrossclients) and v could be thought as the coefficients (client-specific information).
[Figure]
In Figure 4 (a) and (b), which show the experimentalresults with the factorized model, we observe thatu trained on two datasets becomes closer to eachother (MNIST Partition 1) while v (personalized filter coefficient) remain largely different as FL goes on.
With this observation, we further aggregate u whileallowing v to be different across clients, to allow personalized FL.
Further, we use the client specific v forsimilarity matching, to identify relevant local modelsfrom other clients.
In the following paragraphs, wedescribe our factorization method in detail, for both fully-connected and convolutional layers.
Factorization of Fully-Connected Layers We assume that each local model f k has a set of localweights θ k across all layers; that is, θ k = { W ik } Li =1 .
The dimensionality of the dense weight W ikfor each fully connected layer is W ik ∈ R
I × O , where I and O indicate respective input and outputdimensions.
We can reduce the I × O complexity by factorizing the high order matrix into the outerproduct of two vectors as follows:
[Equation]
However, such extreme factorization of the weight matrices may result in the loss of expressiveness inthe parameter space.
Thus, we additionally introduce a highly sparse bias matrix µ to further capturethe information not captured by the outer product of the two vectors as follows:
[Equation]
We initialize µ with zeros so that it can gradually capture the additional expressiveness that are notcaptured by u and v during training.
We can control its sparsity by the hyper-parameter for thesparsity regularizer described in 4.3.
Factorization
of
Convolutional Layers
The difference between the fully-connected and convolutional layers is that the convolutional layers have multiple kernels (or filters) such that
W ik ∈ R F × F × I × O , where F is a size of filters (we assume the filter size is equally paired forthe simplicity).
To induce u to capture base filter knowledge and v to learn filter coefficient, it is essential to design u ∈ R F · F and v ∈ R I · O , but not in arbitrary ways, such as u ∈ R I · F and v ∈ R O · For u ∈ R O and v ∈ R I · F · F .
We observe that performance is degenerated when the parameters areambiguously factorized (Figure 7 (h)).
Our proposed factorization method for convolutional layersare as follows:
[Equation]
π ( · ) is the weight reshaping function.
Note that we reparameterize our model at initialization time .
Then we reconstruct and train full weights of each layer W 1: L k , while optimizing u 1: L k , v 1: L k , and µ 1: L k ,respectively, during training phase.
4.2 Similarity Matching
Since we assume label- and domain-heterogeneous FL scenarios, aggregating the parameter basesacross all clients may not be optimal, since some of them could be highly irrelevant.
Yoon et al.
[27]and Zhang et al.
[28] also demonstrate that avoiding aggregation of irrelevant models from otherclients improves local model performance.
Yoon et al.
[27] achieve this goal by taking the weightedcombination of task-specific weights from other clients, and Zhang et al.
[28] suggest downloading
× 𝐼 "𝑂 𝐹 " 𝐹 𝐮 𝐯 𝑻 𝐮 ×𝐯 𝑻 ⊕ µ 𝐼 "𝑂 𝐹 𝐹
Filter Base Filter Coefficients
Sparse Bias Matrix 12345 … 𝐼(𝑂 𝐹 " 𝐹 1 2 3 4𝐼(𝑂 ...
123… 𝐼&𝑂 Filter𝐮 𝐯𝐮×𝐯
R e c o n s t r u c t
e
d K e r n e l s
Filter
Kernel Reshaping 𝐹 𝐹 𝐹 " 𝐹 𝐼 "𝑂 … 0.
Factorized Parameter Space
C li e n t
C li e n t C li e n t
K
Similarity Matching 0.
𝐯 𝒇 𝟏 𝑳# 𝐯 𝒇 𝟐 𝑳# 𝐯 𝒇 𝑲 𝑳# ⊗ ⊗ ⊗ 𝐮 𝒇 𝑲 ⊕ 𝐯 𝒇 𝑲
{+𝝁 𝒇 𝑲 𝐮 𝒇 𝟐 𝐯 𝒇 𝟐 𝝁 𝒇𝐮 𝒇 𝟏 𝐯 𝒇 
 𝝁 𝒇+}
[Figure]
However, since they require additional communication and computing cost at the local clients, weprovide a more efficient yet effective approach to match models that are beneficial to each other.
Efficient similarity matching Our method utilizes a factorized vector v k for measuring similarityacross different models, at the central server.
Since v are devised to learn personalized coefficients,we assume that clients trained on similar tasks or domains will have similar v .
Specifically, we usev L − 1 k of the layer before the classifier layer for similarity matching.
The similarity matching functionΩ( · ) , is defined as the cosine similarity between target client f k and the other clients { f i } Ki ̸ = k , asfollows:
[Equation]
The similarity scores for those with the cosine similarity scores lower than the given threshold τ , areset to zero.
Our method is significantly more efficient than the similarity matching approaches whichuse full gradient updates for clustering clients [23, 4], as we only use single vector v L − 1 k from eachclient k that are obtained without additional cost.
Personalized weighted averaging We allow each local model to individually perform weightedaggregation of the model weights from other clients, utilizing their similarity scores:
[Equation]
where ϵ is a hyperparameter for scaling the similarity score σ i .
We always set σ k , the similarityscore for itself, as 1 . 0 .
We name this model that aggregates only basis vector u k from each client kas Factorized-FL α .
This will result in encoding common knowledge across local models in u kwhich is especially suitable for domain-heterogeneous scenarios.
However, for label-heterogeneousscenarios, we empirically observed that aggregating all parameters results in better performance, asthe factorization helps averaging them in a more compatible space.
We thus introduce an additionalvariant, namely Factorized-FL β , which aggregates not only the parameter bases across all clients,but also coefficient and bias terms as well.
Yet, note that Factorized-FL β require significantlylarger communication overhead over Factorized-FL α .
4.3 Learning Objective
Now we describe our final learning objective.
Instead of utilizing the single term θ k for local weightsof neural network f k , now let U k , V k , and M k be sets of u k , v k , and µ k of all layers in f k , s.t.
U k
= { u ik } Li =1 , V k = { v ik } Li =1 , and M k = { µ ik } Li =1 , then our local objective function is,
[Equation]
[Table] [Table]
[Figure]
[Figure]
[Figure]
where L is the standard cross-entropy loss performed on all minibatch B ∈ D k .
We add the L 1sparsity inducing regularization term to make the bias parameters highly sparse, controlling its effectwith a hyperparameter λ sparsity .
As for the full training procedure, please see the pseudo-code of thealgorithm in the supplementary file (Section B).
Experiment
5.1 Experimental Setup
Baselines We first consider well-known baseline FL methods, such as (1) FedAvg [18] and (2)
FedProx [14].
We evaluate our factorization technique with (3) pFedPara [20] which also useskernel factorization technique for personalized FL scenarios.
Our similarity matching approach iscompared to (4) Clustered-FL [23] and (5) FedFOMO [28], which measuring client-wise similarityor helpfulness.
(6) Per-FedAvg [5] is also used for evaluation as it shows great performance on
1234567891011121314151617181920 Client 1234567891011121314151617181920 C li en t 1234567891011121314151617181920 Client 1234567891011121314151617181920 C li en t 1234567891011121314151617181920 Client 1234567891011121314151617181920 C li en t 1234567891011121314151617181920 Client 1234567891011121314151617181920 C li en t
[Figure]
1234567891011121314151617181920 C li en t 1234567891011121314151617181920 Client 1234567891011121314151617181920 C li en t 1234567891011121314151617181920 Client 1234567891011121314151617181920 C li en t 1234567891011121314151617181920 Client 1234567891011121314151617181920 C li en t 1234567891011121314151617181920 Client 1234567891011121314151617181920 C li en t 1234567891011121314151617181920 Client 1234567891011121314151617181920 C li en t [Figure]
91011121314151617181920 Client
[Figure] [Figure] heterogeneous federated learning scenarios. We also show local training model, (7) Stand-Alone ,for the lower bound performance.
Detailed information are described in Appendix (Section D). Datasets (1) Label Heterogeneous Scenario : we use CIFAR-10 [11] and SVHN [21] datasetsand create four different partitions for each dataset, which are conventional IID and non-IID as wellas permuted IID and permuted non-IID, whose labels are permuted and incompatible to each other.
We split the datasets into 20 partitions and then simply permuted the labels on the same partition forlabel permuted settings.
(2) Domain Heterogeneous Scenario : we use CIFAR-100 datasets [11] andcreate five sub-datasets grouped by 10 similar classes, such as Household Objects, Fruits&Foods,
Trees&Flowers, Transport, and Animals.
We assign 4 clients for each sub-dataset, i.e. Client 1-4to Household Objects, Client 5-8 to Fruits&Foods, Client 9-12 to Trees&Flowers, Client 13-16 to
Transport, and Client 17-20 to Animals.
We then permute the labels for all partitions to simulatefurther realistic scenarios.
Further descriptions are elaborated in Appendix (Section C).
5.2 Experimental Result
Label-heterogeneous FL As shown in Table 1 (Top), for the standard IID and Non-IID settings, all
FL methods obtain higher performance than the local training baseline ( Stand-Alone ), which confirms that the locally learned knowledge is beneficial to others, when the data and label distributionsare homogeneous across clients.
However, when the labels are not synchronized across all clients(Permuted IID/Non-IID), all previous FL methods suffer from significant performance degeneration,even lower than that of the local training baseline.
Again, note that we do not share the classifier layersto ensure fairness across all algorithms in these permuted settings.
We conjecture that this is causedby the label permutation leading the local model to evolve a permuted set of features that are notcoordinate-wise compatible to others when aggregated.
Contrarily, our method Factorized-FL αshows consistent performance regardless of whether the labels are permuted or not.
Factorized-FLβ even largely outperforms all baselines with significantly superior performance.
Test accuracycurves over the communication round and transmission cost are in Appendix (Section E).
Domain-heterogeneous FL Table 1 (Bottom) shows the experimental results for the domain andlabel heterogeneous scenarios.
We observe that the conventional FL baselines, i.e. FedAvg , FedProx ,fail to obtain better performance over purely local training baseline ( Stand-Alone ) due to the naiveaggregation of extremely heterogeneous knowledge, which causes detrimental knowledge collapse.
FedFOMO and Clustered-FL shows slightly higher performance ( 1 - 2% p ) over Stand-Alone model,as they can avoid irrelevant clients when aggregating local knowledge.
The other personalized FLmethods, i.e. Per-FedAvg and pFedPara , also show 1 - 2% p higher performance over Stand-Alonemodel as they are specialized for personalized FL scenarios.
However, on average, our method largelyoutperforms all baseline models even with the smallest communication costs ( Factorized-FL α ),as shown in Figure 6 (e).
In the figures, we plot the convergence rate of our Factorized-FL α & βover communication round (Top) and transmission cost (Bottom), compared to the baseline models.
Both models consistently obtain superior performance compared to the other models.
Particularly,unlike pFedPara which uses low rank matrices for knowledge sharing, Factorized-FL α only communicates with factorized vectors, such as U for base knowledge sharing and v L − 1 k for similaritymatching and this allows our method to largely reduce the communication costs while showing thebest performance as shown in Figure 6 (e) Bottom.
Effect of kernel factorization In Figure 7 (g), we perform an ablation study of our factorizationmethod in the domain-heterogeneous scenario.
To clearly see the effectiveness of our factorization methods, we compare Factorized-FedAvg , a variant of Factorized-FL β without similaritymatching, against FedAvg .
As shown, Factorized-FedAvg achieves higher performance over theoriginal FedAvg model.
As the only difference between the two is whether kernel is factorized or not,this clearly demonstrates that our factorization method alone improves the model performance byalleviating knowledge collapse.
We further analyze the effect of the sparse bias matrix M .
Whenwe remove M from the Factorized-FedAvg model, Factorized-FedAvg w/o µ in the figure, weobserve large performance drop.
This shows that that the bias term is essential in compensating forthe loss of expressiveness from the rather extreme factorization.
With only U and V , we use 90% lessmodel parameters ( 0 . 27 M ) compared to the regular kernel model ( 2 . 574 M ).
Also, in Figure 7 (k), weobserve that model factorized by u ∈ R F · F and v ∈ R I · O perform better than the model
factorized byu ∈ R I · F and v ∈ R O · F .
This is because in the former case, the factorization will separate the baseknowledge from the task-specific knowledge, as described in Figure 5 (Left), but in the latter case thefactorization does not have such a natural interpretation.
We also show the scalability and reliabilityof our factorization method with different architectures, consisting of convolutional, fully-connectedlayers, batch normalization, and skip connections as shown in Figure 7 (k).
Effect of similarity matching To verify the effectiveness of our similarity matching method usingthe personalized factorized vector v , we visualize the inter-client similarity of u L − 1 f k and v L − 1 f k fromthe second last layer of 20 clients in a domain heterogeneous setting.
As shown in Figure 7 from(a) round 1 to (e) round 100, we observe u L − 1 f k (upper row) are indeed highly correlated with otherclients (the darker color indicate higher similarity scores) while v L − 1 f k (bottom row) are relativelyuncorrelated to each other, as expected as our assumption that U f k capture base knowledge acrossall clients and V f k capture personalized knowledge.
Further, we also observe that v L − 1 f k obtainshigher similarity to that of the parameters trained on the same domains (but with permuted labels),i.e.
Client 1-4, Client 5-8, Client 9-12, Client 13-16, and Client 17-20, showing that they are effectivein capturing task- and domain-level similarities across models.
We further visualize the frequency ofthe client matching across clients in Figure (f), after 100 communication rounds.
With only a singlevector parameter v L − 1 f k , our method finds which clients’s knowledge will be helpful to which clients.
For further analysis on our similarity matching, we compare it against Random and Worst Matchingbaselines under the multi domain scenario.
The random matching baseline randomly selects threearbitrary models to be aggregated at each round, while the worst matching baseline selects three mostdissimilar models.
As shown in Table 7 (g), both baselines significantly suffer from the performancedegeneration compared to our similarity matching algorithm, which shows its effectiveness. [Table] ( U · V + M ) · N ( U · V + M ) · N U + V + M U + V + M
The space-time analysis Weprovide the approximated spacetime complexity in Table 2, compared against those of the essential baselines. Let W be themodel size, N be the number ofinstances, H be the number ofother models (for FedFOMO algorithms), U be the size of the rank-1 vectors (or matrices for pFedPara )in U , V be that of V , M be the that of M , v be the vector for the second last layer from V .For trainingprocedure of FedFOMO , it requires validation step for H number of other model weights on theirvalidation set at each local client. Note that, for communication costs of our methods ( Fac.-FL ),( U · V + M ) ≈ W and thus ( U + V + M ) <
W for both Client-to-Server (C2S) and Server-to-Client(S2C) communication costs.
Please also refer to the actual size of the data transmission for thecommunication cost in Table1.
Figure 7 (h) and (i) show the analysis of the hyperparameters, γ andτ , on the domain-heterogeneous dataset.
As shown in Figure 7 (h), the performance slightly decreaseswith larger γ , since higher rank weight matrices may result in larger coordinate-wise incompatibilityof the parameters, which could lead to knowledge collapse when performing aggregation.
Also, the lower τ s show the lower performance, as less similar knowledge can be aggregated more frequently.
In Figure 7 (j), we train Client 1 on CIFAR-10 IID Partition 1 for 20 epochs using both factorizedand regular models.
As shown, our factorized model outperforms regular models (2 . 574 M ) evenwith 30% less parameters (1 . 784 M ) .
We further analyze the effect of the sparsity in Section E.1 of Appendix.
Conclusion
We introduced a realistic federated learning scenario where the labeling schemes are not synchronizedacross all participants (label heterogeneity) and the tasks and the domains tackled by each localmodel are different from those of others (domain heterogeneity).
We then proposed a novel federatedlearning framework to tackle this problem, whose local model weights are factorized into the productof two vectors plus a sparse bias term.
We then aggregate only the first vectors, for them to capturethe common knowledge across clients, while allowing the other vectors and the sparse bias term to beclient-specific, accounting for label and domain heterogeneity.
Further, we use the client-specificvectors to measure the similarity scores across local models, which are then used for weightedaveraging, for personalized federated learning of each local model.
Our method not only avoidsknowledge collapse from aggregating incompatible parameters across heterogeneous models, but alsosignificantly reduces the communication costs.
We validate our method on both label and domainheterogeneous settings, on which it largely outperforms relevant baselines.
We discuss the limitationsand potential societal impacts of our work in Appendix (Section A). A Limitations and Societal Impacts
Limitations Federated Learning (FL) is devised to protect the privacy of sensitive personal datavia strictly avoiding to share them with other users, institutions, and companies.
The commonlimitation in FL is that, even though almost all existing methods regard sharing locally learnedparameters of gradients as safe, they still may contain some possibility to reproduce the local datadistribution, which violates preservation of privacy.
Although our work alleviates this probability bysignificantly reducing the dimensionality of parameters and partially sharing them, it may be stillsub-optimal in an aspect of security, which is an open research domain for federated learning.
Potential Societal Impact Federated learning is a collaborative learning framework where multipleclients can actively participate in the learning procedure anywhere and anytime.
Nowadays, numerousterminal devices, i.e. smartphones, smartwatches, IoT devices, etc, are utilized as participants ofthe collaborative learning scheme.
However, as the number of participants in the federated learningincreases, the number of communications between them increases accordingly.
Such situation leadingto the exhaustive communication costs may worsen the global environmental crisis , i.e. globalwarming.
Even though our method can alleviate such risks, since ours significantly reduces thecommunication costs by partially sharing the factorized vectors, it is may be sub-optimal.
Wethink that it is essential to conduct independent research for compressing the information efficientlyand effectively for knowledge communication of federated learning that the massive number ofparticipants collaborate.
B Factorized-FL Algorithms
In this section, we describe our pseudo-code algorithms for Factorized-FL α and Factorized-FLβ in Algorithm 1 and 2.
Our Factorized-FL α has strength for not only reducing the dimensionalityof model parameters by factorizing them into rank 1 vector spaces and the additional highly-sparsematrices, but also effectively learning client-general and task-specific knowledge, performing wellon domain heterogeneous scenario.
Particularly, Factorized-FL α transmits a small portion ofthe models which are a set of u ( U ) and a single vector v L − 1 form the second last layer of neuralnetworks, which significantly reduces communication costs while showing strong performance inlabel- and domain-heterogeneous scenarios, as shown in Section 5 in the main document.
C Dataset Configurations
In this section, we describe detailed configurations for datasets that we used in label- and domainheterogeneous scenarios.
C.1 Label Heterogeneous Scenario
We use CIFAR-10 and SVHN for the label-heterogeneous scenario.
We first split eachdataset into train, validation, and test sets for CIFAR-10 ( 48 , 000 / 6 , 000 / 6 , 000 ) and SVHN( 79 , 431 / 9 , 929 / 9 , 929 ).
We then split the train set into K local partitions P 1:20 ( K = 20 ) for IIDpartitions (all instances in each class are evenly distributed to all clients) or for the non-IID partitions(instances in each class are sampled from Dirichlet distribution with α = 0 . 5 ).
We further permute thelabels for each class per local partition P k for permuted IID and permuted non-IID scenarios.
We use
[Table]
[Table]
for 13:
[Table]
for 14: Function
[Table]
[Table]
) 15: θ
[Table]
←
[Table]
M
[Table]
16: for
[Table]
do 17: for minibatch
[Table]
do 18: θ
[Table]
← θ k
[Table]
[Table]
[Table]
[Table]
for 20: end for 21: return U k , v L −
[Table]
f k
[Table]
if
[Table]
[Table]
{+5 Cat 5 9 0 7 9 3 8 8          8 Deer 6 2 6 9 6 5 2 1          9 Dog 4 1 4 8 5 6 6 6          
+}
{+Frog 9 3 2 1 2 0 3 2          1 Horse 0 7 9 4 3 7 0 3          6 Ship 1 8 7 6 7 4 5 7          2 Truck 7 6 8 2 0 1 7 
+}
SVHN
{+Digit 2 5 3 0 8 2 4 4          7 Digit 8 4 1 5 1 8 1 0          4 Digit 3 0 5 3 4 9 9 5          5 Digit 5 9 0 7 9 3 8 8          8 Digit 6 2 6 9 6 5 2 1          9 Digit 4 1 4 8 5 6 6 6          0 Digit 9 3 2 1 2 0 3 
+}
[Table]
different random seed per client, i.e. fixed global seed + client id , for example, 1234 + 0for Client 1 and 1234 + 19 for Client 20.
We provide permutations of labels that we used for eachdataset in Table 3.
C.2 Domain Heterogeneous Scenario
We use CIFAR-100 datasets ( 60 , 000 ) and create five sub-datasets grouped by 10 similar classes,such as Fruits&Foods ( 6 , 000 ), Transport ( 6 , 000 ), Household Objects ( 6 , 000 ), Animals ( 6 , 000 ),
Trees&Flowers ( 6 , 000 ).
We then split train ( 4 , 800 ), test ( 600 ), validation ( 600 ) sets for each subdatset.
To have 20 clients in total, we assign four clients per subdataset, and split each train set into
[Table]
[Table]
[Table]
[Table]
[Table]
- - - - - - - - - - - -    - - - - Rocket - - - - - - - - - - - -    - - - - Streetcar - - - - - - - - - - - -    - - - - Tank - - - - - - - - - - - -    - - - - Tractor - - - - - - - - - - - -    - - - -
[Table]
- - - - - - - - - - - -    - - - - Bus - - - - - - - - - - - -    - - - - Motorcycle - - - - - - - - - - - -    - - - -
[Table]
[Table]
[Table]
[Table]
[Table]
- - - - - - - - - - - - - - - -   5 Raccoon - - - - - - - - - - - - - - - -
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
- - - - - - - - - - - - - - - -    4 partitions, making a single partition contains 1 , 200 instances.
Additionally, we further permutethe labels for those 20 partitions to simulate more realistic scenarios where labeling schemes are notsynchronized across all clients even in the same domain (sub-dataset).
We provide class division andlabel permutation information in Table 4.
D Training Details & Implementations
In this section, we provide detailed implementation and training details that are not described in themain document.
D.1 Baseline Models
Stand-Alone does not share their locally learned knowledge with other clients.
It shows thepure model performance on the data partitions.
In our extremely heterogeneous scenarios, where knowledge collapse may happen severely, which even deteriorates the local knowledge, this modelshows the comparable performance amongst the existing models.
FedAvg [18] performs weighted aggregation of the model parameters, considering the size of thelocal training set.
This model is considered as the standard baseline of many federated learningalgorithms.
FedProx [15] uses proximal regularization term for alleviating divergence between global parameters and the local parameters.
This model is devised to tackle data heterogeneity (also known as thenon-iid problems) across the clients.
Clustered-FL [23] performs bi-partitioning process for the participants when certain conditionsare satisfied (calculating the norm of the gradient updates and comparing it with given thresholdvalues).
It continues to split the given pool of the clients into two novel clusters and performsknowledge aggregations for each cluster.pFedPara [20] uses the Hadamard product of two sets of low rank matrices and reconstructs thekernel parameters.
It shares one of the sets of low rank matrices with other clients and remainsanother set of low rank matrices for the personalization.
Per-FedAvg [5] adopts Model-Agnostic Meta Learning (MAML) [6] approach to federated learning algorithm to search the initial global model which participants can easily adapt to their local databy being trained with only one or a few steps of gradient descent.
FedFOMO [28] leverages other clients’ knowledge for improving their local models.
It downloads afew random models from other clients and validates them on their own validation set at each client.
When some parameters show better validation performance compared to their own local performance,it aggregates such helpful parameters with their own local parameters.
D.2 Training Details
As default, all training configurations are equally set across all models, unless otherwise stated toensure stricter fairness.
We use ResNet-9 architecture as local backbone networks and train them on32 × 32 sized images with 256 for batch size.
We apply data augmentations, i.e. cropping, flipping,jittering, etc, during training.
Optimizer that we used is Stochastic Gradient Descent (SGD).
We set1 e-
3 for learning rate, 1 e- 6 for weight decay, and 0 .
9 for momentum.
For baseline models, we usethe reported hyper-parameters as default, or we adjust hyper-parameters so that they show the bestperformance for fairness.
For ours and pFedPara , the model capacity is adjusted to around 90%- 99% of the original size, as we fairly compare with other methods that use full capacity ( 2 .
57 Mnumber of parameters).
For ours, we use [ 5 e- 4 , 1 e- 3 ] for λ sparsity , [ 0 - 0 .
75 ] for τ , [ 1 , 20 ] for ϵ .
We use8 GPUs (NVIDIA Titan Xp) for experiments.
D.3 ResNet-9 Architecture
[Table]
We use ResNet-9 architecture consisting of eight convolutional layers andone fully connected layer as a classifier,as described in Table 5.
We use maxpooling with size 2 after Conv 5 and anadaptive max pooling after Conv 8 tomake output width 1 for the following
FC layer.
The total number of parameters of the model is 2 .
57 M .
As we use
PyTorch
D.4 Calculation of Communication Cost
We measure the communication cost by { ( P S 2 C + P C 2 S ) × 4 } byte × K × R , where P S 2 C is number ofserver-to-client transmitted parameters and P C 2 S is number of client-to-server transmitted parameters.
Depending on the FL algorithms, P S 2 C and P C 2 S are differently calculated.
For example, FedFOMOdownloads few random models from server ( 10 by default as reported in the paper) but sendsonly single local model to server.
Our Factorized-FL α only sends the small portion of modelparameters, U and v L − 1 , to server, while receiving a single set of U from server.
E Additional Experimental Results
[Figure]
E.1 Sparsity Analysis on FL Scenarios
In the main document, we show the effect of model size and sparsity controlled by λ sparsity for asingle model.
In this section, we analyze it under federated learning scenario.
In Figure 8 (a), weshow the performance over model size in domain heterogeneous scenario.
As shown, our methodshow superior performance even with around 65% of the model size over the baseline model thatachieves the best performance ( Per-FedAvg ) amongst other baseline models.
With 50% sparsity,ours still shows competitive performance compared to Clustered-FL and FedAvg , while it startsbeing significantly degenerated when sparsity becomes over 50% .
[Figure]
In Figure 8 (b), we show accuracy over communication costs.
Note that, in our method, the model sizeis not really related to the communication costs since we send very small portion of model parameters.
For example, even though we use almost full model size ( λ sparsity = 3 e - 4 ), our communication cost issignificantly lesser than the other baseline models, as shown in the figure.
E.2 Test Accuracy Curves and Plots for Communication Costs
For label-heterogeneous FL scenario (Table 1 (Top), we provide all test accuracy curves overcommunication rounds and transmission costs for results of CIFAR-10 and SVHN with stardard
IID/non-IID and permuted IID/non-IID partitions in Figure 10.
For domain-heterogeneous FLscenario (Table 1 (Bottom)), we provide performance of 20 clients In Figure 11.
[Figure]
u r a c
y
[Figure]
u r a c y
[Figure]
A cc u r a c
y
[Figure]
[Figure]
Cost [Gb
[Figure]
A cc u r a c
y
Standard IID (CIFAR
[Figure]
[Figure]
Cost [Gb
[Figure]
A cc u r a c
y
Standard NonIID (CIFAR-10)
[Figure]
Permuted
[Figure]
Transmission Cost [Gb]
[Figure]
A cc u r a c
y
Standard IID (SVHN)
[Figure]
[Table]
[Figure]
[Table]
[Figure]
C li e n t
C li e n t
C li e n t
C li e n t
C li e n t
C li e n t
C li e n t
C li e n t
C li e n t
C li e n t0 C li e n t 1 1 C li
e
{+n t 1 2 C li e n t 1 3 C li e n t 1 4 C li e n t 
+}
{+C li e n t6 C li e n t 1 7 C li e n t 1 8 C li e n t 1 9 C li e n t 
+}
[Figure]