Original: O N H ETEROGENEOUSLY D ISTRIBUTED D ATA , S PARSITY M ATTERS
Revised: A CHIEVING P ERSONALIZED F EDERATED L EARNING WITH S PARSE L OCAL M ODELS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Federated learning (FL) isparticularly vulnerable to heterogeneously distributed data, since a common global model in FL may not adapt to the heterogeneous data distribution of each user.
Revised: Federated learning (FL) is vulnerable to heterogeneously distributed data, since a common global model in FL may not adapt to the heterogeneous data distribution of each user.

Original: To counter this issue, personalized FL (PFL) was proposed to produce dedicated local models for each individual user.
Revised: To counter this issue, personalized FL (PFL) was proposed to produce dedicated local models for each individual user.

Original: However, PFL is far from its maturity, because existing PFL solutions either demonstrate unsatisfactory generalization towards different model architectures or cost enormous extra computation and memory.
Revised: However, PFL is far from its maturity, because existing PFL solutions either demonstrate unsatisfactory generalization towards different model architectures or cost enormous extra computation and memory.

Original: In this work, we propose federated learning with personalized sparse mask (FedSpa), a novel personalized federated learning scheme that employs personalized sparse masks to customize sparse local models on the edge.
Revised: In this work, we propose federated learning with personalized sparse mask (FedSpa), a novel PFL scheme that employs personalized sparse masks to customize sparse local models on the edge.

Original: Instead of training fully dense PFL models, FedSpa only maintains a ﬁxed number of active parameters throughout training (aka sparseto-sparse training), which enables users’ models to achieve personalization withconsistently cheap communication, computation, and memory cost.
Revised: Instead of training an intact (or dense) PFL model, FedSpa only maintains a ﬁxed number of active parameters throughout training (aka sparse-to-sparse training), which enables users’ models to achieve personalization with cheap communication, computation, and memory cost.

Original: We theoretically show that with the rise of data heterogeneity, setting a higher sparsity of FedSpa may potentially result in a smaller error bound on its personalized models, which also coincides with our empirical observations.
Revised: We theoretically show that with the rise of data heterogeneity, setting a higher sparsity of FedSpa may potentially result in a better convergence on its personalized models, which also coincides with our empirical observations.

Original: Comprehensive experiments demonstrate that FedSpa signiﬁcantly saves communication and computation costs, while simultaneously achieves higher model accuracy and faster convergence speed against several state-of-the-art PFL methods.
Revised: Comprehensive experiments demonstrate that FedSpa signiﬁcantly saves communication and computation costs, while simultaneously achieves higher model accuracy and faster convergence speed against several state-of-the-art PFL methods.

Original:  INTRODUCTION
Revised:  INTRODUCTION

Original: Data privacy raises increasingly intensive concerns, and governments have enacted legislation to regulate the privacy intrusion behavior of mobile users, e.g., the General Data
Revised: Data privacy raises increasingly intensive concerns, and governments have enacted legislation to regulate the privacy intrusion behavior of mobile users, e.g., the General Data

Original: Protection Regulation (Voigt & Von dem Bussche, 2017).
Revised: Protection Regulation (Voigt & Von dem Bussche, 2017).

Original: Traditional distributed learning approaches, requiring massive users’ data to be collected and transmitted to a central server for training model,soon may no longer be realistic
Revised: Traditional distributed learning approaches, requiring massive users’ data to be collected and transmitted to a central server for training model,

Original:  under the increasingly stringent regulations on users’ private data.
Revised: soon may no longer be realistic under the increasingly stringent regulations on users’ private data.

Original: On this ground, federated learning (FL), a distributed training paradigm emerges as a successful solution to cope with privacy concerns, which allows multiple clients to perform model training within the local device without the necessity to exchange the data to other entities.
Revised: On this ground, federated learning (FL), a distributed training paradigm emerges as a successful solution to cope with privacy concerns, which allows multiple clients to perform model training within the local device without the necessity to exchange the data to other entities.

Original: In this way, the data privacy leakage problem could be potentially relieved.
Revised: In this way, the data privacy leakage problem could be potentially relieved.

Original: Despite the promising prospect several notorious issues are afﬂicting practical performance of FL:
Revised: Despite the promising prospect several notorious issues are afﬂicting practical performance of FL:

Original: • The  Non-IID data distribution makes the training process prohibitively difﬁcult, and the obtained global model may not ﬁt all the local data.
Revised: • The global model produced by weight average (or FedAvg and its non-personalized variants) exhibits unsatisfactory performance in a Non-IID data distribution setting.

Original: To alleviate this problem, the most popular idea is to integrate personalized features into the global model.
Revised: To alleviate this problem, the most popular idea is to integrate personalized features into the global model, and produce dedicated model for each local distribution.

Original: However, how to make this integration is an open problem that remains unresolved.
Revised: However, how to make this integration is an open problem that remains unresolved.

Original: Prior works on personalized FL (PFL) zero in this issue, but the existing methods either demonstrate weak generalization towards different model architectures (Arivazhagan et al., 2019), or require extra computation and storage (Li et al., 2021).
Revised: Prior works on personalized FL (PFL) zero in this issue, but the existing methods either demonstrate weak generalization towards different model architectures (Arivazhagan et al., 2019), or require extra computation and storage (Li et al., 2021).

Original: • The communication and training overhead is prohibitively high for both the FL and PFL .
Revised: • The communication and training overhead is prohibitively high for both the FL and PFL .

Original: Clients in FL/PFL responsible for model training are mostly edge-devices with limited computation capacity and low bandwidth, and may not be powerful enough to fulﬁll a modern machine learning
Revised: Clients in FL/PFL responsible for model training are mostly edge-devices with limited computation capacity and low bandwidth, and may not be powerful enough to fulﬁll a modern machine learning

Original: task with large deep neural networks.
Revised: task with large deep neural networks.

Original: Existing studies (Li et al., 2020; Vahidian et al., 2021) integrate model compression into FL/PFL to save communication and computation overhead.
Revised: Existing studies (Li et al., 2020; Vahidian et al., 2021) integrate model compression into FL/PFL to save communication and computation overhead.

Original: However, both methods embrace the technique of dense-to-sparse training, which still requires a large amount of communication at the beginning of training.
Revised: However, both methods embrace the technique of dense-to-sparse training, which still requires a large amount of communication at the beginning of training.

Original: In addition, how to effectively aggregate the dynamic sparse models is another challenging problem that remains mysterious.
Revised: In addition, how to effectively aggregate the dynamic sparse models is another challenging problem that remains unresolved.

Original: [Figure]
Revised: [Figure]

Original: In this work, we propose FedSpa (see Figure 2), which has two key features to counter the above two challenges: ( i ) FedSpa does not deploy a single global model, but allows each client to own its unique sparse model masked by a personalized mask, which successfully alleviates the Non-IID challenge.
Revised: In this work, we propose FedSpa (see Figure 2), which has two key features to counter the above two challenges: ( i ) FedSpa does not deploy a single global model, but allows each client to own its unique sparse model masked by a personalized mask, which successfully alleviates the Non-IID challenge.

Original: ( ii )
Revised: ( ii )

Original: FedSpa allows each client to train over an evolutionary sparse model with constant sparsity  throughout the whole federated training process, which consistently alleviates the computation overhead of clients.
Revised: FedSpa allows each client to train over an evolutionary sparse model with constant sparsity 1 throughout the whole federated training process, which consistently alleviates the computation overhead of clients.

Original: Besides, all the local models in FedSpa are sparse models, which requires a smaller amount of communication cost in each communication round.
Revised: Besides, all the local models in FedSpa are sparse models, which requires a smaller amount of communication cost in each communication round.

Original: Theoretically, we conclude that with the rise of Non-IID extent, setting a higher sparsity may result in a smaller error bound on the personalized models ofFedSpa, which potentially provides instruction on how to determine the sparsity of masks, the key hyperparameters of FedSpa.
Revised: Theoretically, we conclude that with the rise of Non-IID extent, setting a higher sparsity may result in a better convergence on the personalized models of FedSpa.

Original: Empirically, in the Non-IID setting, we demonstrate that FedSpa accelerates the convergence (respectively 76.2% and 38.1% less communication rounds to reach the best accuracy of FedAvg (McMahan et al., 2016) and Ditto (Li et al., 2021)), increases the ﬁnal accuracy (up to 21.9% and 4.4% higher accuracy than FedAvg and Ditto, respectively), reduces the communication overhead (50% less parameters communicated than the dense solutions) and lowers the computation (15.3% lower ﬂoating-point operations (FLOPs) than algorithms trained with fully dense model).
Revised: Empirically, in the Non-IID setting, we demonstrate that FedSpa accelerates the convergence (respectively 76.2% and 38.1% less communication rounds to reach the best accuracy of FedAvg (McMahan et al., 2016) and Ditto (Li et al., 2021)), increases the ﬁnal accuracy (up to 21.9% and 4.4% higher accuracy than FedAvg and Ditto, respectively), reduces the communication overhead (50% less parameters communicated than the dense solutions), and lowers the computation (15.3% lower ﬂoating-point operations (FLOPs) than algorithms trained with fully dense model).

Original: To the end, we summarize our contribution as:
Revised: To the end, we summarize our contribution as:

Original: • We present asuccinct and novel formulation of the sparse personalized FL, which is agnostic to all kinds of network architecturesand generalized from traditional FL by enforcing personalized sparse masks to a global model.
Revised: • We present a novel formulation of the sparse personalized FL (SPFL) problem, which can be applied to various network architectures by enforcing personalized sparse masks to a global model.

Original: • We propose a SGD-based solution, dubbed FedSpa, to solve the proposed problem.
Revised: • We propose a solution dubbed as FedSpa to solve the SPFL problem.

Original: By our novel design, FedSpa reduces the communication and computation overhead of the general FL solution.
Revised: By our novel design, FedSpa reduces the communication and computation overhead of the general FL solution.

Original: • Two sparse-to-sparse mask searching techniques are integrated as plugins of our solution.
Revised: • Two sparse-to-sparse mask searching techniques are integrated as plugins of our solution.

Original: To adapt our PFL training context, we modify the DST-based mask searching technique to enable a warm-start of the searching process, which achieves superior performance.
Revised: To adapt our PFL training context, we modify the DST-based mask searching technique to enable a warm-start of the searching process, which achieves superior performance.

Original: • We theoretically present FedSpa’s error bound of the ultimate PFL problem.
Revised: • We theoretically present the convergence analysis of the personalized models.

Original: Experimental results demonstrate the superiority of FedSpa and also coincides with the theoretical conclusion.
Revised: Experimental results demonstrate the superiority of FedSpa and also coincides with the theoretical conclusion – with the rise of data heterogeneity, setting a higher sparsity of FedSpa may potentially result in a better convergence on its personalized models

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Federated learning (FL) (McMahan et al., 2016) is seriously afﬂicted by the issue of heterogeneously distributed (or Non-IID) data.
Revised: Federated learning (FL) (McMahan et al., 2016) is seriously afﬂicted by the issue of heterogeneously distributed (or Non-IID) data.

Original: Personalized FL (PFL), initiated by recent literature (Li et al., 2021;
Revised: Personalized FL (PFL), initiated by recent literature (Li et al., 2021;

Original: Arivazhagan et al., 2019), is shown to be effective to counter this issue of FL.
Revised: Arivazhagan et al., 2019), is shown to be effective to counter this issue of FL.

Original: In this work, we propose an alternative yet effective way to enhance PFL with personalized sparse models.
Revised: In this work, we propose an alternative yet effective way to enhance PFL with personalized sparse models.

Original: 2. P ERSONALIZED F EDERATED L EARNING
Revised: 2. P ERSONALIZED F EDERATED L EARNING

Original: We categorize PFL into ﬁve genres.
Revised: We categorize PFL into ﬁve genres.

Original: Firstly, PFL via layer partition, e.g., FedPer (Arivazhagan et al., 2019), LG-FedAvg (Liang et al., 2020), FedRep (Collins et al., 2021), is to divide the global model layers into shared layers and personalized layers.
Revised: Firstly, PFL via layer partition, e.g., FedPer (Arivazhagan et al., 2019), LG-FedAvg (Liang et al., 2020), FedRep (Collins et al., 2021), is to divide the global model layers into shared layers and personalized layers.

Original: For the shared layers, weights average as in FedAvg is adopted, while for personalized layers, models are trained only locally and will not be exchanged with others.
Revised: For the shared layers, weights average as in FedAvg is adopted, while for personalized layers, models are trained only locally and will not be exchanged with others.

Original: Secondly, PFL viaproximal regularization, e.g., Ditto (Li et al., 2021), L2GD (Hanzely & Richt´arik, 2020) is to add a proximal term on the local model to force the local model and global model closely in the local model ﬁne-tuning stage. 
Revised: Secondly, PFL via regularization, e.g., Ditto (Li et al., 2021), L2GD (Hanzely & Richt´arik, 2020) is to add a proximal term on the local model to force the local model and global model closely in the local model ﬁne-tuning stage. Similarly, Sarcheshmehpour et al.

Original:  Thirdly, PFL via model interpolation, e.g., MAPPER (Mansour et al., 2020), APFL (Deng et al., 2020) achieves personalization by linearly interpolating the weights of the cluster (global) model and local model as the personalized model.
Revised: (2021); SarcheshmehPour et al. (2021) propose a total variation (TV) regularization to form the network lasso (nLasso) problem, and primal-dual methods adapted from (Jung, 2020) are proposed to solve the nLasso problems. Thirdly, PFL via model interpolation, e.g., MAPPER (Mansour et al., 2020), APFL (Deng et al., 2020) achieves personalization by linearly interpolating the weights of the cluster (global) model and local model as the personalized model.

Original: Fourthly, PFL via transfer learning, e.g., FedMD (Li & Wang, 2019),
Revised: Fourthly, PFL via transfer learning, e.g., FedMD (Li & Wang, 2019),

Original: FedSteg (Yang et al., 2020), and Fedhealth (Chen et al., 2020), is to either use model and domain-speciﬁc local ﬁne-tuning or knowledge distillation to adapt the global model into the personalized model.
Revised: FedSteg (Yang et al., 2020), and Fedhealth (Chen et al., 2020), is to either use model and domain-speciﬁc local ﬁne-tuning or knowledge distillation to adapt the global model into the personalized model.

Original: Finally, PFL via model compression, e.g., LotteryFL (Li et al., 2020) and Sub-FedAvg (Vahidian et al., 2021), achieves personalization via employing principle model compression techniques, such as weight pruning and channel pruning, over the shared global model.
Revised: Finally, PFL via model compression, e.g., LotteryFL (Li et al., 2020) and Sub-FedAvg (Vahidian et al., 2021), achieves personalization via employing principle model compression techniques, such as weight pruning and channel pruning, over the shared global model.

Original: 2. S PARSE D EEP N EURAL N ETWORKS
Revised: 2. S PARSE D EEP N EURAL N ETWORKS

Original: Sparse neural networkshave been extensively studied due to their potentiality to signiﬁcantly reduce parameter count, memory requirements, and computational costs, which can be classiﬁed into two genres: dense-to-sparse methods and sparse-to-sparse methods.
Revised: Methods to Sparsify neural networks can be classiﬁed into two genres: dense-to-sparse methods and sparse-to-sparse methods.

Original: Dense-to-sparse methods train from a dense model, and compress the model along the training process.
Revised: Dense-to-sparse methods train from a dense model, and compress the model along the training process.

Original: Iterative pruning, ﬁrst proposed by Frankle & Carbin (2018), shows promising performance in dynamically searching for a sparse yet accurate network.
Revised: Iterative pruning, ﬁrst proposed by Frankle & Carbin (2018), shows promising performance in dynamically searching for a sparse yet accurate network.

Original: Recently, sparse-to-sparse methods have been proposed to pursue training efﬁciency.
Revised: Recently, sparse-to-sparse methods have been proposed to pursue training efﬁciency.

Original: Among them, dynamic sparse training (DST) (Bellec et al., 2018; Evci et al., 2020; Liu et al., 2021) is the most successful technique that allows sparse networks, trained from scratch, to match the performance of their dense equivalents.
Revised: Among them, dynamic sparse training (DST) (Bellec et al., 2018; Evci et al., 2020; Liu et al., 2021) is the most successful technique that allows sparse networks, trained from scratch, to match the performance of their dense equivalents.

Original: Stemming from the ﬁrst work – sparse evolutionary training (Mocanu et al., 2018; Liu et al., 2020), DST has evolved as a class of sparse training methods absorbing many advanced techniques, e.g., weight redistribution (Mostafa & Wang, 2019; Dettmers & Zettlemoyer, 2019), gradient-based regrowth (Dettmers & Zettlemoyer, 2019;
Revised: Stemming from the ﬁrst work – sparse evolutionary training (Mocanu et al., 2018; Liu et al., 2020), DST has evolved as a class of sparse training methods absorbing many advanced techniques, e.g., weight redistribution (Mostafa & Wang, 2019; Dettmers & Zettlemoyer, 2019), gradient-based regrowth (Dettmers & Zettlemoyer, 2019;

Original: Evci et al., 2020), and extra weight exploration (Jayakumar et al., 2020; Liu et al., 2021).
Revised: Evci et al., 2020), and extra weight exploration (Jayakumar et al., 2020; Liu et al., 2021).

Original: Our work also achieves personalization via model compression.
Revised: Our work also achieves personalization via model compression.

Original: We emphasize that three main progresses are made towardsthe interpretable and SOTA compression-based PFL: ( i )
Revised: We emphasize that three main progresses are made towards SOTA compression-based PFL: ( i )

Original: We rigorously formulate the sparse personalized FL problem, ﬁlling the gap left by the prior works.
Revised: We rigorously formulate the sparse personalized FL problem, ﬁlling the gap left by the prior works.

Original: ( ii ) While prior works either vaguely describe their model aggregation as ”aggregating the Lottery Ticket Network via FedAvg” (Li et al., 2020), or ”taking the average on the intersection of unpruned parameters in the network” (Vahidian et al., 2021), we explicitly formulate the aggregation as averaging the sparse update from clients.
Revised: ( ii ) While prior works either vaguely describe their model aggregation as ”aggregating the Lottery Ticket Network via FedAvg” (Li et al., 2020), or ”taking the average on the intersection of unpruned parameters in the network” (Vahidian et al., 2021), we explicitly formulate the aggregation as averaging the sparse update from clients.

Original: ( iii ) Both the two prominent prior works use the idea of iterative pruning to prune the network fromfully dense to sparse.
Revised: ( iii ) Both the two prominent prior works use the idea of iterative pruning to prune the network from dense to sparse.

Original: We instead provide two sparse-to-sparse training alternatives to plug in our solution, which largely reduces the costs of communication at the beginning of the training process, and exhibits remarkable performance.
Revised: We instead provide two sparse-to-sparse training alternatives to plug in our solution, which largely reduces the costs of communication at the beginning of the training process, and exhibits remarkable performance.

Original:  P ROBLEM F ORMULATION
Revised:  P ROBLEM F ORMULATION

Original: We assume a total number of K clients within our FL system, and we consistently use k to index a speciﬁc client.
Revised: We assume a total number of K clients within our FL system, and we consistently use k to index a speciﬁc client.

Original: First, we give a preliminary introduction on the general FL and PFL problem.
Revised: First, we give a preliminary introduction on the general FL and PFL problem.

Original: General FL problem.
Revised: General FL problem.

Original: Let w ∈ R
Revised: Let w ∈ R

Original: d be the global weight.
Revised: d be the global weight.

Original: General FL takes the formulation as below
Revised: General FL takes the formulation as below

Original: [Equation]
Revised: [Equation]

Original: where D = D 1 ∪· · ·∪D K is the joint distribution of k local heterogeneous distributions, data ( x , y ) is uniformly sampled according to distribution D k wrt.
Revised: where D = D 1 ∪· · ·∪D K is the joint distribution of k local heterogeneous distributions, data ( x , y ) is uniformly sampled according to distribution D k wrt.

Original: client k , and L ( · ; · ) is the loss function.
Revised: client k , and L ( · ; · ) is the loss function.

Original: Ultimate PFL Problem.
Revised: Ultimate PFL Problem.

Original: Let { w k } be the personalized models.
Revised: Let { w k } be the personalized models.

Original: The ultimate PFL is deﬁned as:
Revised: The ultimate PFL is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: according to (Zhang et al., 2020; Hanzely et al., 2021).
Revised: according to (Zhang et al., 2020; Hanzely et al., 2021).

Original: The problem could be separately solved by individual client with no communication.
Revised: The problem could be separately solved by individual client with no communication.

Original: However, if the local data is insufﬁcient, poor performance could be observed by this direct solution, since the local models cannot be boosted by other clients.
Revised: However, if the local data is insufﬁcient, poor performance could be observed by this direct solution, since the local models cannot be boosted by other clients.

Original: Regularized PFL problem.
Revised: Regularized PFL problem.

Original: Regularized PFL can ease the heterogeneous challenges encountered by general FL, while escaping the curse of insufﬁcient samples encountered by the ultimate PFL problem.
Revised: Regularized PFL can ease the heterogeneous challenges encountered by general FL, while escaping the curse of insufﬁcient samples encountered by the ultimate PFL problem.

Original: Inspired by (Chen & Chao, 2021), we formulate the Regularized PFL problem as follows:
Revised: Inspired by (Chen & Chao, 2021), we formulate the Regularized PFL problem as follows:

Original: [Equation]
Revised: [Equation]

Original: where w k denote the personalized models and R ( · ) is the regularizer that enables information exchange between clients, making the problem tractable even the local samples are insufﬁcient.
Revised: where w k denote the personalized models and R ( · ) is the regularizer that enables information exchange between clients, making the problem tractable even the local samples are insufﬁcient.

Original: However, it remains controversial about how to deﬁne the regularizer.
Revised: However, it remains controversial about how to deﬁne the regularizer.

Original: Also, the gap between regularized
Revised: Also, the gap between regularized

Original: PFL problem (P3) and the ultimate PFL problem (P2) still remains unspeciﬁed in existing PFL study.
Revised: PFL problem (P3) and the ultimate PFL problem (P2) still remains unspeciﬁed in existing PFL study.

Original: In this work, our ultimate goal is to solve problem (P2), which requires information exchange between clients to ensure effective solution.
Revised: In this work, our ultimate goal is to solve problem (P2), which requires information exchange between clients to ensure effective solution.

Original: Below, instead of utilizing regularizer as in (P3), we alternatively propose a novel Sparse PFL (SPFL) problem to reach the same goal.
Revised: Below, instead of utilizing regularizer as in (P3), we alternatively propose a novel Sparse PFL (SPFL) problem to reach the same goal.

Original: Sparse PFL problem.
Revised: Sparse PFL problem.

Original: By introducing personalized masks into FL, we derive the SPFL problem as
Revised: By introducing personalized masks into FL, we derive the SPFL problem as

Original: [Equation]
Revised: [Equation]

Original: where m ∗ k ∈ { 0 , 1 } d is a personalized sparse binary mask for k -th client, (cid:12) denotes the Hadamard product for two given vectors.
Revised: where m ∗ k ∈ { 0 , 1 } d is a personalized sparse binary mask for k -th client, (cid:12) denotes the Hadamard product for two given vectors.

Original: Our ultimate goal is to ﬁnd a global model w , such that the personalized model for k -th client can be extracted from the global model by personalized mask m ∗ k , i.e., m ∗ k (cid:12) w .
Revised: Our ultimate goal is to ﬁnd a global model w , such that the personalized model for k -th client can be extracted from the global model by personalized mask m ∗ k , i.e., m ∗ k (cid:12) w .

Original: The element of m ∗ k being 1 means that the weight in the global model is active for k -th personalized model, otherwise, remains dormant.
Revised: The element of m ∗ k being 1 means that the weight in the global model is active for k -th personalized model, otherwise, remains dormant.

Original: Thus, the information exchange between all personalized models is enforced by a shared global model w .
Revised: Thus, the information exchange between all personalized models is enforced by a shared global model w .

Original: Compared with existing PFL algorithms, solving our SPFL problem (P4) does not sacriﬁce additional computation and storage overhead of clients, since we do not maintain both personalized local models and global model in clients as in (Li et al., 2021; Mansour et al., 2020).
Revised: Compared with existing PFL algorithms, solving our SPFL problem (P4) does not sacriﬁce additional computation and storage overhead of clients, since we do not maintain both personalized local models and global model in clients as in (Li et al., 2021; Mansour et al., 2020).

Original: On contrary, the solution to our problem could potentially lower the communication and computation overhead.
Revised: On contrary, the solution to our problem could potentially lower the communication and computation overhead.

Original: Moreover, SPFL problem (P4) is agnostic to  model architecture, since we do not make modelspeciﬁc separation of the public and personalized layer in (Arivazhagan et al., 2019; Liang et al., 2020; Collins et al., 2021), or domain-speciﬁc ﬁne-tuning in (Chen et al., 2020; Yang et al., 2020).
Revised: Moreover, SPFL problem (P4) can be applied to most of the model architectures without modelspeciﬁc hyper-parameter tuning, since we do not make model-speciﬁc separation of the public and personalized layer in (Arivazhagan et al., 2019; Liang et al., 2020; Collins et al., 2021), or domainspeciﬁc ﬁne-tuning in (Chen et al., 2020; Yang et al., 2020).

Original:  F ED S PA : A L OCAL -SGD BASED S OLUTION FOR SPFL
Revised:  F ED S PA : SOLUTION FOR SPFL

Original: In this section, we ﬁrst introduce our proposed FedSpa in Algorithm 1.
Revised: In this section, we ﬁrst introduce our proposed FedSpa in Algorithm 1.

Original: Then, we give two types of  sparse-to-sparse mask searching methods for FedSpa.
Revised: Then, we specify the update rule of global model, and two sparse-to-sparse mask searching methods that can be plug in the update process.

Original: At last, we give a theoretical analysis on evaluating the quality of the iterates of FedSpa with respect to the ultimate PFL problem (P2).
Revised: At last, we give a theoretical analysis on evaluating the quality of the iterates of FedSpa with respect to the ultimate PFL problem (P2).

Original: 4. SGD- BASED S OLUTION
Revised: 4. G LOBAL M ODEL U PDATE RULE FOR F ED S PA

Original: The proposed FedSpa to solve Problem (P4) is motivated by the basic idea of data-parallel SGD.

Original: Data Parallel-based SGD.
Revised: Data Parallel-based Update.

Original: Mimicking the data-parallel solution to solve Problem (P4), we have
Revised: We ﬁrst propose the following iterative update to solve problem (P4)

Original: [Equation]
Revised: [Equation]

Original:  where ξ k,t is a batch of data that is uniformly sampled from the k -th client’s local distribution D k , η is the learning rate, and ˜ w k,t
Revised: 1) where ξ k,t is a batch of data that is uniformly sampled from the k -th client’s local distribution D k , η is the learning rate, and ˜ w k,t

Original: = m ∗ k (cid:12) w t is the sparse weights sparsiﬁed by mask m ∗ k .
Revised: = m ∗ k (cid:12) w t is the sparse weights sparsiﬁed by mask m ∗ k .

Original: However, the optimal personalized masks { m ∗ k } are generally not accessible to us in the solution process.
Revised: However, the optimal personalized masks { m ∗ k } are generally not accessible to us in the solution process.

Original: Let m k,t be an intermediate surrogate personalized mask of m ∗ k , we rewrite Eq.
Revised: Let m k,t be an intermediate surrogate personalized mask of m ∗ k , we rewrite Eq.

Original: (1) as:
Revised: (1) as:

Original: [Equation]
Revised: [Equation]

Original: For our SGD-based solution, it is worth mentioned that: ( i ) Some coordinates of the model weights have been made zero before doing the forward process, i.e., not all the parameters have to be involved when calculating L ( ˜ w k,t ; ξ k,t ) .
Revised: For our proposed update rule, it is worth mentioned that: ( i ) Some coordinates of the model weights have been made zero before doing the forward process, i.e., not all the parameters have to be involved when calculating L ( ˜ w k,t ; ξ k,t ) .

Original: This means that the computation overhead in the forward process could be potentially saved.
Revised: This means that the computation overhead in the forward process could be potentially saved.

Original: ( ii )
Revised: ( ii )

Original: In the backward process, the stochastic gradient ∇ ˜ w k,t L ( ˜ w k,t ; ξ k,t ) is masked again by m k,t , which means that we do not need to backward the gradient for those sparse coordinates.
Revised: In the backward process, the stochastic gradient ∇ ˜ w k,t L ( ˜ w k,t [Equation] k,t ) is masked again by m k,t , which means that we do not need to backward the gradient for those sparse coordinates.

Original: Thus, the computation cost can be largely saved.
Revised: Thus, the computation cost can be largely saved.

Original: FL-adapted SGD.
Revised: FL-adapted Update.

Original: To save the communication overhead, we adopt local SGD (Stich, 2018) and partial participation toadapt our solution.
Revised: To save the communication overhead, we integrate the idea from local SGD (Stich, 2018) and partial participation to our solution.

Original: Let ˜ w k,t,τ denote the weights before doing ( τ + 1) -th step of local SGD and set ˜ w k,t, 

Revised: Let ˜ w k,t,τ denote the weights before doing ( τ +1) -th step of local SGD and set ˜ w k,t,

Original: = m k,t (cid:12) w t (i.e., thesparse local weights will be synchronized every N steps with the global weights).
Revised: = m k,t (cid:12) w t (i.e., the local weights will be synchronized every N steps with the global weights).

Original: Then, each client k ∈ S t updates its model as below:
Revised: Then, each client k ∈ S t updates its model as below:

Original: [Equation]
Revised: [Equation]

Original: where ξ k,t,τ is a batch of sampled data in τ -th step at round t .
Revised: where ξ k,t,τ is a batch of sampled data in τ -th step at round t .

Original: After the local training is ﬁnished, the models of participated clients are updated and aggregated to the global model in server as follows:
Revised: After the local training is ﬁnished, the models of participated clients are updated and aggregated to the global model in server as follows:

Original: [Equation]
Revised: [Equation]

Original: where S t is the set of clients selected to be participant in round t .
Revised: where S t is the set of clients selected to be participant in round t .

Original: According to Eq.
Revised: According to Eq.

Original: (3), the update synchronized to the server (i.e., ˜ w k,t, 0 − ˜ w k,t,N ), and the model distributed to clients (i.e., ˜ w k,t, 0 ) are all sparse with a constant sparsity.
Revised: (3), the update synchronized to the server (i.e., ˜ w k,t, 0 − ˜ w k,t,N ), and the model distributed to clients (i.e., ˜ w k,t, 0 ) are all sparse with a constant sparsity.

Original: Therefore, the communication overhead over synchronization could be largely saved.
Revised: Therefore, the communication overhead over synchronization could be largely saved.

Original: At last, we summarize our proposed FedSpa in Algorithm 1.
Revised: At last, we summarize our proposed FedSpa in Algorithm 1.

Original:  1: procedure S ERVER ’ S M AIN L OOP 2:
Revised: Input Training iteration T ; Learning rate η ; Local Steps N ; Random seed seed ; 1: procedure S ERVER ’ S M AIN L OOP 2:

Original: Randomly initialize global model w3: m k, 0 = Initialization( · ) for k =0 , 1 , . .
Revised: Randomly initialize global model w3: m k, 0 = Initialization( seed ) for k =0 , 1 , . .

Original: , K (cid:46) Initialize masks via a mask-searching solution 4: for t = 0 , 1 , . .
Revised: , K (cid:46) Initialize masks via a mask-searching solution 4: for t = 0 , 1 , . .

Original: , T − 1 do 5:
Revised: , T − 1 do 5:

Original: Uniformly sample a fraction of client into S t 6: for each client k / ∈ S t do 7: m k,t +

Revised: Uniformly sample a fraction of client into S t 6: for each client k / ∈ S t do 7: m k,t +

Original: = m k,t (cid:46) Inherit masks for round t + 1 if not chosen 8: for k ∈ S t do 9:
Revised: = m k,t (cid:46) Inherit masks for round t + 1 if not chosen 8: for k ∈ S t do 9:

Original: Send ˜ w k,t, 

Revised: Send ˜ w k,t,

Original: = m k,t (cid:12) w t to client k 10:
Revised: = m k,t (cid:12) w t to client k 10:

Original: Call Client k ’s main loop and recieve U k,t and m k,t + 11: w t +1 =
Revised: Call Client k ’s main loop and receive U k,t and m k,t + 11: w t +1 =

Original: w t − 1 S (cid:80) k ∈ S t U k,t (cid:46) Average and apply the update 12: procedure C LIENT ’ S M AIN L OOP 13: for τ = 0 , 1 , . .
Revised: w t − 1 S (cid:80) k ∈ S t U k,t (cid:46) Average and apply the update 12: procedure C LIENT ’ S M AIN L OOP 13: for τ = 0 , 1 , . .

Original: , N − 1 do 14:
Revised: , N − 1 do 14:

Original: Sample a batch of data ξ k,t,τ from local dataset 15: g k,t,τ ( ˜ w k,t,τ ) = ∇ ˜ w k,t,τ L ( ˜ w k,t,τ ; ξ k,t,τ ) 16: ˜ w k,t,τ +1 =
Revised: Sample a batch of data ξ k,t,τ from local dataset 15: [Equation] L ( ˜ w k,t,τ ; ξ k,t,τ ) 16: ˜ w k,t,τ +1 =

Original: ˜ w k,t,τ − η m k,t (cid:12) g k,t,τ ( ˜ w k,t,τ ) (cid:46)
Revised: ˜ w k,t,τ − η m k,t (cid:12) g k,t,τ ( ˜ w k,t,τ ) (cid:46)

Original: Local update with ﬁxed mask m k,t 17:
Revised: Local update with ﬁxed mask m k,t 17:

Original: U k,t = ˜ w k,t, 0 − ˜ w k,t,N 18: m k,t +1 =
Revised: U k,t = ˜ w k,t, 0 − ˜ w k,t,N 18: m k,t +1 =

Original: Next Masks( · ) (cid:46) Plug in a mask-searching solution to produce next masks 19:
Revised: Next Masks( · ) (cid:46) Plug in a mask-searching solution to produce next masks 19:

Original: Send back U k,t and m k,t +
Revised: Send back U k,t and m k,t +

Original:  M ASK  ECHNIQUE
Revised: 4. S PARSE - TO - SPARSE M ASK S EARCHING T ECHNIQUE

Original: The framework of FedSpa is extensible, which allows us to plug in arbitrary masking searching techniques.
Revised: The framework of FedSpa is extensible. Speciﬁcally, we use the mask surrogate m k,t in Eq. (3) to perform update, which allows us to plug in arbitrary mask searching techniques to determine the iterate process pf m k,t .

Original: In this work, we nominate two kinds of sparse-to-sparse training techniques: modiﬁed dynamic sparse training (DST) (Liu et al., 2021) and Random Static Masks (RSM), into FedSpa to reduce the computation and communication cost.
Revised: In this work, we nominate two kinds of sparse-to-sparse training techniques: modiﬁed dynamic sparse training (DST) (Liu et al., 2021) and Random Static Masks (RSM), into FedSpa to reduce the computation and communication cost.

Original: Modiﬁed DST for FedSpa.
Revised: Modiﬁed DST for FedSpa.

Original: Our modiﬁed DST solution (see Algorithm 2) for FL follows these procedures.
Revised: Our modiﬁed DST solution (see Algorithm 2) for FL follows these procedures.

Original: Firstly, randomly initialize the same mask for each client based on Erd ´ os-R ´ enyi Kernel (ERK)
Revised: Firstly, randomly initialize the same mask for each client based on Erd´os-R´enyi Kernel (ERK)

Original: (Evci et al., 2020).
Revised: (Evci et al., 2020).

Original: Secondly, after local training, each client prunes out a number of unpruned weights with the smallest magnitude, and the number of weights being pruned is determined by a decayed pruned rate.
Revised: Secondly, after local training, each client prunes out a number of unpruned weights with the smallest magnitude, and the number of weights being pruned is determined by a decayed pruned rate.

Original: Thirdly, recover the same amount of weights pruned in the last step.
Revised: Thirdly, recover the same amount of weights pruned in the last step.

Original: We follow the recovery process as in (Evci et al., 2020) by utilizing the gradient information to do the recovery.
Revised: We follow the recovery process as in (Evci et al., 2020) by utilizing the gradient information to do the recovery.

Original: By our DST method, the number of sparse weights (aka.
Revised: By our DST method, the number of sparse weights (aka.

Original: sparse volume ) remains a constant (i.e., β ) throughout the whole training process.
Revised: sparse volume ) remains a constant (i.e., β ) throughout the whole training process.

Original:  1: procedure I NITIALIZATION  2:
Revised: Input Initial pruning rate α 0 ; Set of Model layers J ; 1: procedure I NITIALIZATION ( seed ) 2:

Original: Randomly initialize m k, 0 using the same random seed.
Revised: Randomly initialize m k, 0 using the same random seed seed .

Original: 3: procedure N EXT M ASKS ( ˜ w k,t,N ) 4:
Revised: 3: procedure N EXT M ASKS ( ˜ w k,t,N ) 4:

Original: Decay α t using cosine annealing with initial pruned rate α5:
Revised: Decay α t using cosine annealing with initial pruning rate α5:

Original: Sample a batch of data and backward the dense gradient g ( ˜ w k,t,N ) 6: for layer j ∈ J do 7:
Revised: Sample a batch of data and backward the dense gradient g ( ˜ w k,t,N ) 6: for layer j ∈ J do 7:

Original: Update mask m ( j ) k,t + 12 by zeroing out α t -proportion of weights with magnitude pruning 8:
Revised: Update mask m ( j ) k,t + 12 by zeroing out α t -proportion of weights with magnitude pruning 8:

Original: Update mask m ( j ) k,t +1 via recovering weights with gradient information g ( ˜ w k,t,N ) 9:
Revised: Update mask m ( j ) k,t +1 via recovering weights with gradient information g ( ˜ w k,t,N ) 9:

Original: Return m k,t +
Revised: Return m k,t +

Original: Remark.
Revised: Remark.

Original: We highlight our main modiﬁcation over traditional DST techniques like Rigl (Evci et al., 2020) and Set (Mocanu et al.
Revised: We highlight our main modiﬁcation over traditional DST techniques like Rigl (Evci et al., 2020) and Set (Mocanu et al.

Original: to an FL context exist in two main aspects: ( i )
Revised: to an FL context exist in two main aspects: ( i )

Original: The pruning is performed individually by each client based on their local models, and the gradient used for weights recovery is derived using the client’s local training data.
Revised: The pruning is performed individually by each client based on their local models, and the gradient used for weights recovery is derived using the client’s local training data.

Original: ( ii ) Once the next masks are generated, existing DST solutions immediately apply them to the local model weight.
Revised: ( ii ) Once the next masks are generated, existing DST solutions immediately apply them to the local model weight.

Original: Indicated by Liu et al.
Revised: Indicated by Liu et al.

Original: (2021), by doing so, the recovered coordinate may need extra training steps to grow from 0 to a dense value.
Revised: (2021), by doing so, the recovered coordinate may need extra training steps to grow from 0 to a dense value.

Original: Our solution relieves this problem by applying the new mask on the global weights (which are dense), such that the recovered coordinates could have a dense initial value to warm-start.
Revised: Our solution relieves this problem by applying the new mask on the global weights (which are dense), such that the recovered coordinates could have a dense initial value to warm-start.

Original: Below, we incorporate Random Static Masks (RSM) into FedSpa, which is shown in Algorithm 3.

Original: 1: procedure I NITIALIZATION  2:
Revised: 1: procedure I NITIALIZATION ( seed ) 2:

Original: Randomly initializethe same m k, 0 using the same random seed  3: procedure N EXT M ASKS 4: m k,t [-+
-] = m k,t 5:
Revised: Randomly initialize m k, 0 using the same random seed seed 3: procedure N EXT M ASKS 4: m k,t [-+
-]+1 = m k,t 5:

Original: Return m k,t +
Revised: Return m k,t +

Original: RSM for FedSpa.
Revised: RSM for FedSpa.

Original: RSM  is basically ﬁxing m k,t for all k ∈
Revised: RSM (shown in Algorithm 3) is basically ﬁxing m k,t for all k ∈

Original: [ K ] to the same randomly initialized mask, which remains unchanged during the whole training session.
Revised: [ K ] to the same randomly initialized mask, which remains unchanged during the whole training session.

Original: This solution also ensures the same sparse volume for all the clients throughout the training process, and could alsopotentially reduce the computation and communication overhead as DST.
Revised: This solution also ensures the same sparse volume for all the clients throughout the training process, and could also reduce the computation and communication overhead as DST.

Original: Interestingly, within the setting of the homogeneous data distribution, we empirically show that RSM is more effective than DST(see Section 5) in FedSpa.
Revised: Interestingly, within the setting of the homogeneous data distribution, we empirically show that RSM is more effective than DST in FedSpa.

Original: 4. T HEORETICAL A NALYSIS
Revised: 4. T HEORETICAL A NALYSIS

Original: In this section, we shall introduce the error bound of personalized models produced by FedSpa.
Revised: In this section, we shall introduce the convergence property of personalized models produced by FedSpa.

Original: We ﬁrst give the following assumptions.
Revised: We ﬁrst give the following assumptions.

Original: Assumption 1 (Coordinate-wise bounded gradient dissimilarity) .
Revised: Assumption 1 (Coordinate-wise bounded gradient dissimilarity) .

Original: For any ˜ w ∈ R d , there exists a constant G ≥ 0 bounding the coordinate-wise gradient dissimilarity over all clients, i.e., (cid:13) 
Revised: For any ˜ w ∈ R d , there exists a constant G ≥ 0 bounding the coordinate-wise gradient dissimilarity over all clients, i.e., (cid:13) (cid:13)(cid:13) ∇ F k ( ˜ w ) − 1 K (cid:80) K k (cid:48) =1 ∇ F k (cid:48) ( ˜ w )

Original: [Equation]
Revised: [Equation]

Revised: ∞ ≤ G .

Original: Assumption 2 (Bounded variance) .
Revised: Assumption 2 (Bounded variance) .

Original: Assume that g k,t,τ ( ˜ w ):= ∇L ( ˜ w ; ξ k,t,τ ) is an unbiased estimator of ∇ F k ( ˜ w ) with bounded variance, i.e., E (cid:104) (cid:107) g k,t,τ ( ˜ w )
Revised: Assume that g k,t,τ ( ˜ w ):= ∇L ( ˜ w ; ξ k,t,τ ) is an unbiased estimator of ∇ F k ( ˜ w ) with bounded variance, i.e., E (cid:104) (cid:107) g k,t,τ ( ˜ w )

Original: −∇ F k ( ˜ w ) (cid:107) 

Revised: −∇ F k ( ˜ w ) (cid:107)

Original: (cid:105) ≤ σ 2 ,
Revised: (cid:105) ≤ σ 2 ,

Original: ∀ k, t, τ, ˜ w ∈ R d .
Revised: ∀ k, t, τ, ˜ w ∈ R d .

Original: Assumption 3 (L-smoothness) .
Revised: Assumption 3 (L-smoothness) .

Original: We assume L-smoothness over the client’s loss function, i.e., (cid:107)∇ F k ( ˜ w 1 ) − ∇ F k ( ˜ w 2 ) (cid:107) ≤ L (cid:107) ˜ w 1 − ˜ w 2 (cid:107) holds for arbitrary ˜ w 1 , ˜ w 2 ∈ R d .
Revised: We assume L-smoothness over the client’s loss function, i.e., (cid:107)∇ F k ( ˜ w 1 ) − ∇ F k ( ˜ w 2 ) (cid:107) ≤ L (cid:107) ˜ w 1 − ˜ w 2 (cid:107) holds for arbitrary ˜ w 1 , ˜ w 2 ∈ R d .

Original: Assumption 4 (Coordinate-wise bounded gradient) .
Revised: Assumption 4 (Coordinate-wise bounded gradient) .

Original: Suppose the coordinate-wise gradient of each client is upper-bounded, i.e., (cid:107)∇ ˜ w F
Revised: Suppose the coordinate-wise gradient of each client is upper-bounded, i.e.

Original: [Equation]
Revised: [Equation]

Original: Assumptions 2 and 3 are commonly used for characterizing the convergence of FL algorithms.
Revised: Assumptions 2 and 3 are commonly used for characterizing the convergence of FL algorithms.

Original: We modify the other two assumptions slightly from their counterparts in existing FL literature (Xu et al., 2021, see Assumptions 3 and 5) by replacing the L 2 norm with L ∞ norm.
Revised: We modify the other two assumptions slightly from their counterparts in existing FL literature (Xu et al., 2021, see Assumptions 3 and 5) by replacing the L 2 norm with L ∞ norm.

Original: These modiﬁcations are made to reveal the coordinate-wise gradient information in our analysis.
Revised: These modiﬁcations are made to reveal the coordinate-wise gradient information in our analysis.

Original: Theorem 1 (Error Bound towards Problem (P2)) .
Revised: Theorem 1 (Convergence of personalized models) .

Original: Given the above assumptions, suppose the learning rate satisﬁes η ≤ 1 16 LN , the optimal personalized masks m ∗ k ∈ { 0 , 1 } d and the evolutionary masks m k,t ∈ { 0 , 1 } d maintain the same sparse volume constraint, i.e., || 1 − m ∗ k[-|| 
-]
Revised: Given the above assumptions, suppose the learning rate satisﬁes η ≤ 1 16 LN , the optimal personalized masks m ∗ k ∈ { 0 , 1 } d and the evolutionary masks m k,t ∈ { 0 , 1 } d maintain the same sparse volume constraint, i.e., || 1 − m ∗ k[-|| 
-]

Original: {+|| 
+} = || 1 − m k,t || 

Revised: {+|| 
+} = || 1 − m k,t ||

Original: = β , the personalized sparse models { ˜ w k,t } generated via FedSpa exhibits the following error bound on solving the  ultimate PFL problem (P2):
Revised: = β , the personalized sparse models { ˜ w k,t } generated via FedSpa exhibits the following convergence property towards the optimal solution of ultimate PFL problem (P2):

Original: [Equation]
Revised: [Equation]

Original: where κ 
Revised: where κ = 12 − 150 N 3 η 3 L 3 − 15 N 2 η 2 L 2 − 5 NηL and ρ t = (25 N [Equation] η

Original: [Equation]
Revised: [Equation]

Original: (cid:124)
Revised: L 3 + 5 N 2 η 3 L 2)( σ 2 + 18 N Φ t ) + 4 N [Equation] 2 L + Nη 2 K

Original: [Equation]
Revised: [Equation]

Revised: , dist ( m

Original: [Table]
Revised: [Equation]

Original: , dist ( m 1 , m 2 ) is the hamming distancebetween masks m 1 and m 2 .
Revised: m [Equation] ) is the hamming distancebetween masks m 1 and m 2 .

Original: Below we give several comments on the above theorem.
Revised: Below, we give several comments on the above theorem.

Original: Our main focus of analysis is on the second and third non-vanished terms while communication round T → ∞. • Impact of sparse volume.
Revised: We focus our analysis on the on the second and third residuals, which do not vanish as communication round T → ∞ • Impact of sparse volume.

Original: The result in Eq.
Revised: The result in Eq.

Original: (5) is highly related to setting of sparse volume, i.e., β .
Revised: (5) is highly related to setting of sparse volume, i.e., β .

Original: The term βB 2 would vanish as β → 0 , while the term ( d − β ) G 2 would disappear as β → d .
Revised: The term βB 2 would vanish as β → 0 , while the term ( d − β ) G 2 would disappear as β → d .

Original: When the data is highly heterogeneous, i.e., gradient dissimilarity is drastic, G could be prohibitively large to dominate the error.
Revised: When the data is highly heterogeneous, i.e., gradient dissimilarity is drastic, G could be prohibitively large to dominate the error.

Original: Then setting the sparsity level to a relatively large value, which results in a lower ( d − β ) G 2 , would potentially give a smaller non-vanished error.
Revised: Then setting the sparsity level to a relatively large value, which results in a lower ( d − β ) G 2 , would potentially give a smaller non-vanished error.

Original: On contrary, when the data heterogeneity is mild, the error could be dominated by βB 2 .
Revised: On contrary, when the data heterogeneity is mild, the error could be dominated by βB 2 .

Original: Then setting an excessively high sparsity might even hurt the performance.
Revised: Then setting an excessively high sparsity might even hurt the performance.

Original: This conclusion is also evidenced by our experimental results (see Figure 1 in Section 1, where the optimal sparsity setting of FedSpa that achieves the highest accuracy is 0.5).
Revised: This conclusion is also evidenced by our experimental results (see Figure 1 in Section 1, where the optimal sparsity setting of FedSpa that achieves the highest accuracy is 0.5).

Original: • Error by evolutionary mask searching.
Revised: • Error by evolutionary mask searching.

Original: Recall that FedSpa uses evolutionary masksin Eq. (2) as a replacement of optimal masksin Eq. (1) to perform update.
Revised: Recall that FedSpa uses evolutionary masks as a surrogate of optimal masks to perform update on the global model w t (see Eq.

Original: This replacement may bring additional errors to the personalized models, as the hamming distance between the evolutionary masks and optimallocal masks (i.e., the term dist ( m k,t , m ∗ k ) ) exists in ourerror bound.
Revised: This replacement may bring additional errors to the convergence bound, as the hamming distance between the evolutionary masks and optimal masks (i.e., the term dist ( m k,t , m ∗ k ) ) exists in our bound.

Original: • Error by dissimilar evolutionary masks.
Revised: • Error by dissimilar evolutionary masks.

Original: The theoretical result on the term dist ( m k,t , m k (cid:48) ,t ) in Φ t indicates that the dissimilarity among the evolutionary masks may also play a role in optimizing the ultimate PFL problem (P2).
Revised: The theoretical result on the term dist ( m k,t , m k (cid:48) ,t ) in Φ t indicates that the dissimilarity among the evolutionary masks may also play a role in optimizing the ultimate PFL problem (P2).

Original: This term could be minimized if the evolutionary masks remain the same throughout the training process, and would dominate the error if the masks are too distinct.
Revised: This term could be minimized if the evolutionary masks remain the same throughout the training process, and would dominate the error if the masks are too distinct.

Original: This observation exactly motivates us to present the naive mask searching technique RSM in Algorithm 3, which forces all the clients to share the same random masks.
Revised: This observation exactly motivates us to present the naive mask searching technique RSM in Algorithm 3, which forces all the clients to share the same random masks.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we conduct extensive experiments on verifying the efﬁcacy of the proposed FedSpa.
Revised: In this section we conduct extensive experiments on verifying the efﬁcacy of the proposed FedSpa.

Original: Our implementation of FedSpa is based on an open-source FL simulator FedML (He et al., 2020).
Revised: Our implementation of FedSpa is based on an open-source FL simulator FedML (He et al., 2020).

Original: 5. E XPERIMENTAL S ETUP
Revised: 5. E XPERIMENTAL S ETUP

Original: Dataset.
Revised: Dataset.

Original: We evaluate the efﬁcacy of FedSpa on EMNIST-Letter (EMNIST-L henceforth), CIFAR10, and CIFAR100 datasets.
Revised: We evaluate the efﬁcacy of FedSpa on EMNIST-Letter (EMNIST-L henceforth), CIFAR10, and CIFAR100 datasets.

Original: We simulate the client’s data distribution on Non-IID and IID setting.
Revised: We simulate the client’s data distribution on Non-IID and IID setting.

Original: We simulate two groups of Non-IID settings via γ -Dirichlet distribution, named setting A and setting B.
Revised: We simulate two groups of Non-IID settings via γ -Dirichlet distribution, named setting A and setting B.

Original: Setting A and setting B respectively specify γ = 0 .
Revised: Setting A and setting B respectively specify γ = 0 .

Original: 1 for both EMNIST-L and CIFAR100, while specify γ = 0 .
Revised: 1 for both EMNIST-L and CIFAR100, while specify γ = 0 .

Original: 3 for CIFAR10.
Revised: 3 for CIFAR10.

Original: Details of our simulation setting are available in Appendix A.1.
Revised: Details of our simulation setting are available in Appendix B.1.

Original: Baselines 2 .
Revised: Baselines.

Original: We compare our proposed FedSpa with four baselines, including FedAvg (McMahan et al.
Revised: We compare our proposed FedSpa with four baselines, including FedAvg (McMahan et al.

Original: , Sub-FedAvg (Vahidian et al.
Revised: , Sub-FedAvg (Vahidian et al.

Original: , Ditto (Li et al.
Revised: , Ditto (Li et al.

Original: (2021)) and Local.
Revised: (2021)) and Local.

Original: We tune the hyper-parameters of the baselines to their best states.
Revised: We tune the hyper-parameters of the baselines to their best states.

Original: Speciﬁcally, the regularization factor of Ditto is set to 0.5.
Revised: Speciﬁcally, the regularization factor of Ditto is set to 0.5.

Original: The prune rate each round, distance threshold, and accuracy threshold of Fed-Subavg are ﬁxed to 0.05, 0.0001, 0.5, respectively.
Revised: The prune rate each round, distance threshold, and accuracy threshold of Fed-Subavg are ﬁxed to 0.05, 0.0001, 0.5, respectively.

Original: We ran 3 random seeds in our comparison.
Revised: We ran 3 random seeds in our comparison.

Original: Models and hyper-parameters.
Revised: Models and hyper-parameters.

Original: We use LeNet5 for EMNIST-L, VGG11 for CIFAR10, and
Revised: We use LeNet5 for EMNIST-L, VGG11 for CIFAR10, and

Original: ResNet18 for CIFAR100 in our experiment.
Revised: ResNet18 for CIFAR100 in our experiment.

Original: We use a SGD optimizer with weight decayed parameter 0.0005.
Revised: We use a SGD optimizer with weight decayed parameter 0.0005.

Original: The learning rate is initialized with 0.1 and decayed with 0.998 after each communication round.
Revised: The learning rate is initialized with 0.1 and decayed with 0.998 after each communication round.

Original: We simulate 100 clients in total, and in each round 10 of them are picked to perform local training (the setting follows McMahan et al.
Revised: We simulate 100 clients in total, and in each round 10 of them are picked to perform local training (the setting follows McMahan et al.

Original: For all the methods except Ditto, local epochs are ﬁxed to 5.
Revised: For all the methods except Ditto, local epochs are ﬁxed to 5.

Original: For Ditto, in order to ensure a fair comparison, each client uses 3 epochs for training of the local model, and 2 epochs for global model training.
Revised: For Ditto, in order to ensure a fair comparison, each client uses 3 epochs for training of the local model, and 2 epochs for global model training.

Original: The batch size of all the experiments is ﬁxed to 128.
Revised: The batch size of all the experiments is ﬁxed to 128.

Original: For FedSpa, the pruning rate (i.e., α t ) is decayed using cosine annealing with an initial pruned rate 0.5.
Revised: For FedSpa, the pruning rate (i.e., α t ) is decayed using cosine annealing with an initial pruned rate 0.5.

Original: The initial sparsity of layers is initialized by ERK with scale parameter 1.
Revised: The initial sparsity of layers is initialized by ERK with scale parameter 1.

Original: 5. M AIN P ERFORMANCE E VALUATION
Revised: [Equation] E VALUATION

Original: We ﬁx the dense ratio of FedSpa (DST), FedSpa (RSM), and the ﬁnal dense ratio of Fed-SubAvg both to 0.5 (i.e., 50% of parameters are pruned) in our main evaluation.
Revised: We ﬁx the dense ratio of FedSpa (DST), FedSpa (RSM), and the ﬁnal dense ratio of Fed-SubAvg both to 0.5 (i.e., 50% of parameters are pruned) in our main evaluation.

Original: Other hyper-parameters are ﬁxed as default.
Revised: Other hyper-parameters are ﬁxed as default.

Original: Figure 3 and Table 1 illustrate the training performance of different algorithms on three datasets.
Revised: Figure 3 and Table 1 illustrate the training performance of different algorithms on three datasets.

Original: We evaluate the performance based on the following metrics:
Revised: We evaluate the performance based on the following metrics:

Original: Final Accuracy.
Revised: Final Accuracy.

Original: In the Non-IID setting, we show that FedSpa (DST) achieves remarkable performance.
Revised: In the Non-IID setting, we show that FedSpa (DST) achieves remarkable performance.

Original: Speciﬁcally, in Non-IID setting B of CIFAR100, FedSpa (DST) achieves respectively 4.4%, 11.9% and 21.9% higher ﬁnal model accuracy, compared with Ditto, Sub-FedAvg and FedAvg.
Revised: Speciﬁcally, in Non-IID setting B of CIFAR100, FedSpa (DST) achieves respectively 4.4%, 11.9% and 21.9% higher ﬁnal model accuracy, compared with Ditto, Sub-FedAvg and FedAvg.

Original: FedSpa (DST) seems to achieve better performance as the FL tasks becoming difﬁcult (since better performance is observed in a higher Non-IID extent, and in datasets that are intrinsically more difﬁcult).
Revised: FedSpa (DST) seems to achieve better performance as the FL tasks becoming difﬁcult (since better performance is observed in a higher Non-IID extent, and in datasets that are intrinsically more difﬁcult).

Original: Interestingly, in the IID setting, we show that all the personalized solutions exhibit some extents of performance degradation, which become more signiﬁcant as the dataset becomes challenging.
Revised: Interestingly, in the IID setting, we show that all the personalized solutions exhibit some extents of performance degradation, which become more signiﬁcant as the dataset becomes challenging.

Original: The compression-based methods seem to be especially vulnerable in this setting.
Revised: The compression-based methods seem to be especially vulnerable in this setting.

Original: Our interpretation for this phenomenon is that: since the information exchange between clients would be limited by employing different sub-networks for training, the clients could not efﬁciently make an effective fusion on their models through parameter averaging.
Revised: Our interpretation for this phenomenon is that: since the information exchange between clients would be limited by employing different sub-networks for training, the clients could not efﬁciently make an effective fusion on their models through parameter averaging.

Original: This hypothesis is substantiated by our experiment on FedSpa (RSM), an alternative implementation of FedSpa, which forces all the masks to maintain the same sub-network.
Revised: This hypothesis is substantiated by our experiment on FedSpa (RSM), an alternative implementation of FedSpa, which forces all the masks to maintain the same sub-network.

Original: FedSpa (RSM) achieves commensurate performance with FedAvg in the IID setting, outperforming the personalized solutions. 
Revised: FedSpa (RSM) achieves commensurate performance with FedAvg in the IID setting, outperforming the personalized solutions. The reason FedSpa (DST)

Revised: [Table]

Original: The reason FedSpa (DST) performing better in Non-IID setting can also be explained by Theorem 1, which concludes that with proper sparsity, ahigher Non-IID extent may promise a smaller error bound to FedSpa.
Revised: performing better in Non-IID setting can also be explained by Theorem [Table] which concludes that in a Non-IID setting, setting proper sparsity for personalized models may promise a better convergence

Revised: [Table]

Original: Convergence.
Revised: Convergence.

Original: As shown in Table2, FedSpa achieves signiﬁcantly faster convergence, which potentially saves the communication rounds to train a model from scratch to a speciﬁc accuracy.
Revised: As shown in Table

Original: Training FLOPs and Communication.
Revised: [Table]

Original: From Table 1, FedSpa(DST) achieves 15.4% ∼ 42.9% lower FLOPs than the dense solutions (e.g., Ditto, FedAvg), 13.0% ∼ 28.2% lower communication
Revised: FedSpa achieves signiﬁcantly faster convergence

Original: [Table]
Revised: [Table]

Revised: which potentially saves the communication rounds to train a model from scratch to a speciﬁc accuracy

Original: [Table]
Revised: [Table]

Original: overhead than another model compression solution Sub-FedAvg

Original: [Table]
Revised: [Table]

Original:  and 50% lower communication than the dense solution.
Revised: Training FLOPs and Communication. From Table 1, FedSpa (DST) achieves 15.4% ∼ 42.9% lower FLOPs than the dense solutions (e.g., Ditto, FedAvg), 13.0% ∼ 28.2% lower communication overhead than another model compression solution Sub-FedAvg, and 50% lower communication than the dense solution.

Original: The edge of FedSpa (DST) stems from its training pattern – it is trained from a sparse model [Table] with constant sparsity throughout the training process.
Revised: The edge of FedSpa (DST) stems from its training pattern – it is trained from a sparse model, with constant sparsity throughout the training process.

Original: However, it is interesting to see that the training FLOPS of Fed-SubAvg is considerably lower than FedSpa, even under the same sparsity setting.
Revised: However, it is interesting to see that the training FLOPS of Fed-SubAvg is considerably lower than FedSpa, even under the same sparsity setting.

Original: This phenomenon stems from our ERK initialization, which is essential for the high performance of our solution, for which we will have a further discussion in Appendix A. [Table] 5. A BLATION S TUDY
Revised: This phenomenon stems from our ERK initialization, which is essential for the high performance of our solution, for which we will have a further discussion in Appendix B.4.

Original: We do extensive ablation studies to show the effect of different components and hyper-parameters of FedSpa.

Original: The detailed results are placed in Appendix A.4 due to limited space.

Original: Firstly, we show the impact of different sparsity of our solution (see Figure 4).

Original: Secondly, we evaluate how mask initialization impacts the performance (see Figure 5 and 6).

Original: Thirdly, we study whether gradient information points out the right direction to recover the weight coordinates (see Figure 7).

Original: Finally, we present an interesting ﬁnding on the dense global model trained by FedSpa (see Figure 8).

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this paper, we propose FedSpa, a personalized FL solution that enables sparse-to-sparse training and efﬁcient sub-model aggregation.
Revised: In this paper, we propose FedSpa, a personalized FL solution that enables sparse-to-sparse training and efﬁcient sub-model aggregation.

Original: As demonstrated by our experiments, FedSpa exhibits outstanding performance in the Non-IID setting, outperforming other existing solutions in terms of accuracy, convergence speed as well as communication overhead.
Revised: As demonstrated by our experiments, FedSpa exhibits outstanding performance in the Non-IID setting, outperforming other existing solutions in terms of accuracy, convergence speed as well as communication overhead.

Original: Additionally, we present theoretical analysis to evaluate the error bound of FedSpa towards the ultimate PFL problem.
Revised: Additionally, we present theoretical analysis to evaluate the error bound of FedSpa towards the ultimate PFL problem.

Original: Future direction includes designing new model aggregation solutions for the sparse sub-network, and new mask searching techniques speciﬁcally targeting on federated learning process.
Revised: Future direction includes designing new model aggregation solutions for the sparse sub-network, and new mask searching techniques speciﬁcally targeting on federated learning process.

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: For sake of reproducibility of our solution we make the following efforts: ( i )
Revised: For sake of reproducibility of our solution we make the following efforts: ( i )

Original: In Appendix A.1, we clearly state the data splitting method for IID and Non-IID data distribution.
Revised: In Appendix B.1, we clearly state the data splitting method for IID and Non-IID data distribution.

Original: ( ii )In Appendix
Revised: ( ii )

Original: A.2, the network models we used in our experiment are clearly described.
Revised: In Appendix B.2, the network models we used in our experiment are clearly described.

Original: ( iii ) In Appendix A.4, the proposed FedSpa with various hyperparameters during the implementation are also systemically tested to demonstrate its stability and superiority.
Revised: ( iii ) In Appendix B.4, the proposed FedSpa with various hyper-parameters during the implementation are also systemically tested to demonstrate its stability and superiority.

Original: ( iv ) In Appendix B, we give the self-contained proof of Theorem 1.
Revised: ( iv ) In Appendix C, we give the self-contained proof of Theorem 1.

Original: ( v ) At last, the source-code of FedSpa will be released upon acceptance
Revised: ( v ) At last, the source-code of FedSpa is enclosed in the supplementary material.

Original: A D ETAILED S ETTING AND A DDITIONAL E XPERIMENTS A. D ATA S PLITTING S ETTING
Revised: A C RITICAL C OMPONENTS OF F ED S PA (DST)

Revised: ERK initialization. In Algorithm 2, we use Erd´os-R´enyi Kernel (ERK) originally proposed by Evci et al.

Revised: (2020) to initialize the sparsity of each layer.

Revised: Speciﬁcally, the active parameters of the convolutional layer initialized by ERK is proportional to 1 n 1 − 1 + n l + w l + h l n l − 1 ∗ n l ∗ w l ∗ h l , where n l − 1 , n l w l and h l respectively specify number of input channels, output channels and kernel’s width and height in the l -th layer.

Revised: For the linear layer, the number of active parameters scale with 1 n l − 1 +n l n l − 1 ÷ n l where n l −and n l are the number of neurons in the l − 1 -th and l -th layer.

Revised: This initialization basically allows the layer with less parameters have more proportion of active parameters.

Revised: Cosine annealing.

Revised: Recall that we set the initial pruning rate as α 0 and gradually decay it towith cosine annealing Liu et al.

Revised: The update of pruning rate with cosine annealing can be formalized as: α t = 0 .

Revised: 5 × α 0 × (cid:16) 1 + cos (cid:16) t T − 1 π (cid:17)(cid:17) .

Revised: We perform this decay in order to ensure that the network (speciﬁcally, its active coordinates) would not experience drastic change on the later stage of training while ensuring that the mask searching is effective on the early stage of training.

Revised: B D ETAILED S ETTING AND A DDITIONAL E XPERIMENTS B. D ATA S PLITTING S ETTING

Original: In our implementation, we ﬁrst split the training data (60k pieces of data for CIFAR10 and CIFAR100, and 145.6k for EMNIST-L, respectively) to clients for IID setting and Non-IID setting.
Revised: In our implementation, we ﬁrst split the training data (60k pieces of data for CIFAR10 and CIFAR100, and 145.6k for EMNIST-L, respectively) to clients for IID setting and Non-IID setting.

Original: For the IID setting, data are uniformly sampled for each client.
Revised: For the IID setting, data are uniformly sampled for each client.

Original: For the Non-IID setting, we use γDirichlet distribution on the label ratios to ensure uneven label distributions among devices as (Hsu et al., 2019).
Revised: For the Non-IID setting, we use γDirichlet distribution on the label ratios to ensure uneven label distributions among devices as (Hsu et al., 2019).

Original: The lower the distribution parameter γ is, the more uneven the label distribution will be, and would be more challenging for FL.
Revised: The lower the distribution parameter γ is, the more uneven the label distribution will be, and would be more challenging for FL.

Original: After the initial splitting of training data, we samplepieces of testing data from the testing set to each client.
Revised: After the initial splitting of training data, we samplepieces of testing data from the testing set to each client.

Original: To simulate the personalized setting, each client’s testing data has the same proportion of labels as its training data.
Revised: To simulate the personalized setting, each client’s testing data has the same proportion of labels as its training data.

Original: Testing of the personalized model is performed by each client based on their personalized data, and the overall testing accuracy is calculated as the average of all the client’s testing accuracy.
Revised: Testing of the personalized model is performed by each client based on their personalized data, and the overall testing accuracy is calculated as the average of all the client’s testing accuracy.

Original: In our experiment, we simulate different Non-IID settings.
Revised: In our experiment, we simulate different Non-IID settings.

Original: For CIFAR10, Non-IID setting A and B respectively specify γ = 0 .
Revised: For CIFAR10, Non-IID setting A and B respectively specify γ = 0 .

Original: 5 and γ = 0 .
Revised: 5 and γ = 0 .

Original: For EMNIST-L and CIFAR100, since the number of the total labels are bigger 3 , we use smaller γ , wherein setting A and B respectively specify γ = 0 .
Revised: For EMNIST-L and CIFAR100, since the number of the total labels are bigger 3 , we use smaller γ , wherein setting A and B respectively specify γ = 0 .

Original: 2 and γ = 0 .
Revised: 2 and γ = 0 .

Original: A. N ETWORK A RCHITECTURES
Revised: B. N ETWORK A RCHITECTURES

Original: We follow the Caffe’s implementation of LeNet5 4 (LeCun et al., 2015), VGG11 (Simonyan &
Revised: We follow the Caffe’s implementation of LeNet5 4 (LeCun et al., 2015), VGG11 (Simonyan &

Original: Zisserman, 2014) and ResNet18 (He et al., 2016) to do the evaluation.
Revised: Zisserman, 2014) and ResNet18 (He et al., 2016) to do the evaluation.

Original: Suggested by Hsieh et al.
Revised: Suggested by Hsieh et al.

Original: (2020), DNNs with batch normalization layers (Ioffe & Szegedy, 2015) are particularly vulnerable to the Non-IID setting, suffering signiﬁcant model quality loss in the FL process.
Revised: (2020), DNNs with batch normalization layers (Ioffe & Szegedy, 2015) are particularly vulnerable to the Non-IID setting, suffering signiﬁcant model quality loss in the FL process.

Original: Following the recommendation from Hsieh et al.
Revised: Following the recommendation from Hsieh et al.

Original: (2020), we use group normalization (Wu & He, 2018) to substitute the original batch normalization layer in both ResNet18 and VGG11.
Revised: (2020), we use group normalization (Wu & He, 2018) to substitute the original batch normalization layer in both ResNet18 and VGG11.

Original: A. B ASELINE D ESCRIPTION
Revised: B. B ASELINE D ESCRIPTION

Original: Below, we give a brief introduction of the baselines compared in our evaluations:
Revised: Below, we give a brief introduction of the baselines compared in our evaluations:

Original: • FedAvg (McMahan et al., 2016) is the vanilla solution of FL.
Revised: • FedAvg (McMahan et al., 2016) is the vanilla solution of FL.

Original: It utilizes weights average to enable all the clients to collaboratively train a global model, which efﬁciently absorbs knowledge from personal data resided in clients.
Revised: It utilizes weights average to enable all the clients to collaboratively train a global model, which efﬁciently absorbs knowledge from personal data resided in clients.

Original: • Ditto (Li et al., 2021) is a personalized FL solution aiming to smooth the tension brought by the data heterogeneity problem of FL.
Revised: • Ditto (Li et al., 2021) is a personalized FL solution aiming to smooth the tension brought by the data heterogeneity problem of FL.

Original: Ditto achieves personalization via maintaining both the local models and global model.
Revised: Ditto achieves personalization via maintaining both the local models and global model.

Original: Speciﬁcally, within each round of iteration, each client ﬁrst trains the global model based on its local empirical loss (which shares the same procedure as FedAvg).
Revised: Speciﬁcally, within each round of iteration, each client ﬁrst trains the global model based on its local empirical loss (which shares the same procedure as FedAvg).

Original: After the global model is updated, each client additionally trains its local model based on a loss function involving its local empirical loss and the proximal term towards the global model.
Revised: After the global model is updated, each client additionally trains its local model based on a loss function involving its local empirical loss and the proximal term towards the global model.

Original: This local training phase is used to extract the global knowledge into each client’s local model.
Revised: This local training phase is used to extract the global knowledge into each client’s local model.

Original: Since each client has to maintain and train both local model and global model, Ditto might need extra computation and storage overhead to achieve its personalization.
Revised: Since each client has to maintain and train both local model and global model, Ditto might need extra computation and storage overhead to achieve its personalization.

Original: • Local is the direct solution to the ultimate PFL problem (P2).
Revised: • Local is the direct solution to the ultimate PFL problem (P2).

Original: Each client performs SGD based on its local data, and there is no communication between clients.
Revised: Each client performs SGD based on its local data, and there is no communication between clients.

Original: To mimic the FL setting, we sample 10 out of 100 clients to do the local update on its local model after every 5 epochs of training (same with the number of local epochs in a communication round that is performed by other solutions).
Revised: To mimic the FL setting, we sample 10 out of 100 clients to do the local update on its local model after every 5 epochs of training (same with the number of local epochs in a communication round that is performed by other solutions).

Original: For sake of consistency, we still use 1 communication round to represent 5 local epochs of Local in our evaluation.
Revised: For sake of consistency, we still use 1 communication round to represent 5 local epochs of Local in our evaluation.

Original: • Sub-FedAvg (Vahidian et al., 2021) is a prominent model compression-based PFL.
Revised: • Sub-FedAvg (Vahidian et al., 2021) is a prominent model compression-based PFL.

Original: Sub-FedAvg maintains personalized sub-networks for each client.
Revised: Sub-FedAvg maintains personalized sub-networks for each client.

Original: Training of Sub-FedAvg starts from a fully dense model, and this solution iteratively prunes out the parameters and channels as the training progresses.
Revised: Training of Sub-FedAvg starts from a fully dense model, and this solution iteratively prunes out the parameters and channels as the training progresses.

Original: Finally, the commonly shared parameters of each layer are removed, and only the personalized parameters that can represent the features of local 
Revised: Finally, the commonly shared parameters of each layer are removed, and only the personalized parameters that can represent the features of local data are kept.

Revised: Subsampling (Kone ˇ cn ` y et al., 2016) is a gradient-compression solution aiming to reduce the communication overhead of FL.

Original: data are kept
Revised: The local training procedure is the same with FedAvg. The difference is that Subsampling does not communicate the intact model for aggregation, but only communicates the sparse gradient update to the server for aggregation. Explicitly, in each round, the sparse gradient update is produced through element-wisely multiplying a random mask. Different from FedSpa, the randomized mask is independently generated in each round, and would only be used to compress the gradient when uploading the gradient update (which means in the model distribution phase, the model distributed would not be sparsiﬁed, and therefore would not save the downlink communication cost).

Original: A. A BLATION S TUDY
Revised: B. A BLATION S TUDY

Original: In this sub-section, we focus on the ablation study of FedSpa.
Revised: In this sub-section, we focus on the ablation study of FedSpa.

Original: Speciﬁcally, we study the impacts of dense ratio, different mask initialization methods, and the gradient-involved weight recovery procedure.
Revised: Speciﬁcally, we study the impacts of dense ratio, different mask initialization methods, and the gradient-involved weight recovery procedure.

Original: Additionally, we present an interesting observation on the performance of the global model trained by our personalized solution.
Revised: Additionally, we present an interesting observation on the performance of the global model trained by our personalized solution.

Original: Our ablation study is done with ResNet-18 on CIFAR100.
Revised: Our ablation study is done with ResNet-18 on CIFAR100.

Original: Impact of sparsity (aka.
Revised: Impact of sparsity (aka.

Original: sparse ratio).
Revised: sparse ratio).

Original: Fixing other components and hyper-parameters to the default value in our setup, we change the sparsity of FedSpa to 0.2, 0.4, 0.5, 0.6 and 0.8, to show its impact on the algorithm performance.
Revised: Fixing other components and hyper-parameters to the default value in our setup, we change the sparsity of FedSpa to 0.2, 0.4, 0.5, 0.6 and 0.8, to show its impact on the algorithm performance.

Original: Experimental results are available in Figure 4 and Table 3.
Revised: Experimental results are available in Figure 4 and Table 3.

Original: By our report, we observe that sparsity may impact learning performance under different data distribution settings.
Revised: By our report, we observe that sparsity may impact learning performance under different data distribution settings.

Original: For the IID setting, a higher sparsity seems to seriously degrade the training performance, while for the Non-IID setting, properly sparsifying the model may even enhance the ﬁnal accuracy, and with a higher Non-IID extent, the beneﬁt of sparsiﬁcation reinforces.
Revised: For the IID setting, a higher sparsity seems to seriously degrade the training performance, while for the Non-IID setting, properly sparsifying the model may even enhance the ﬁnal accuracy, and with a higher Non-IID extent, the beneﬁt of sparsiﬁcation reinforces.

Original: This observation surprisingly coincides with our theoretical conclusion given in Theorem 1, that setting the sparse volume to a proper value when gradient dissimilarity is large (i.e., under a high extent of
Revised: This observation surprisingly coincides with our theoretical conclusion given in Theorem 1, that setting the sparse volume to a proper value when gradient dissimilarity is large (i.e., under a high extent of

Original: Non-IID) could foster a better performance of personalized models.
Revised: Non-IID) could foster a better performance of personalized models.

Original: But too much sparsity, even in the highly non-iid setting (e.g. γ = 0 . 1 ) leads to performance degradation.
Revised: But too much sparsity, even in the highly non-iid setting (e.g. γ = 0 . 1 ) leads to performance degradation.

Original: On the contrary, while it is iid, the convergence could be dominated by the errors brought by sparsiﬁcation, and setting the mask to a higher sparsity could possibly enlarge these existing errors.
Revised: On the contrary, while it is iid, the convergence could be dominated by the errors brought by sparsiﬁcation, and setting the mask to a higher sparsity could possibly enlarge these existing errors.

Original: Another more intuitive interpretation for the impact of sparse ratio is from the perspective of information exchange.
Revised: Another more intuitive interpretation for the impact of sparse ratio is from the perspective of information exchange.

Original: Too much sparsiﬁcation may limit the information exchange between the local sparse models.
Revised: Too much sparsiﬁcation may limit the information exchange between the local sparse models.

Original: If all the clients maintain an extremely high sparsity, the intersected coordinates between clients’ local sparse models (or identically, their masks) would be small.
Revised: If all the clients maintain an extremely high sparsity, the intersected coordinates between clients’ local sparse models (or identically, their masks) would be small.

Original: Then the local update averaging process (see Eq.
Revised: Then the local update averaging process (see Eq.

Original: (4)), the only way to extract global knowledge into the local models, would not be effective.
Revised: (4)), the only way to extract global knowledge into the local models, would not be effective.

Original: On contrary, while the sparsity is set to an extremely low value, the personalized features of local models could be eliminated, since only limited coordinates in their models are different.[Table]
Revised: On contrary, while the sparsity is set to an extremely low value, the personalized features of local models could be eliminated, since only limited coordinates in their models are different.

Original: ERK vs. Uniform sparsity initialization.
Revised: ERK vs. Uniform sparsity initialization.

Original: Recall our mask initialization procedure in Algorithmthat the layer-wise sparsity is initialized by ERK.
Revised: Recall our mask initialization procedure in Algorithmthat the layer-wise sparsity is initialized by ERK.

Original: This in essence ensures that the layer-wise sparsity of a model is scaled with the number of parameters in a layer.
Revised: This in essence ensures that the layer-wise sparsity of a model is scaled with the number of parameters in a layer.

Original: Liu et al.
Revised: Liu et al.

Revised: (2021) conﬁrms the out-

Original: (2021) conﬁrms the outstanding effect of ERK initialization in improving overall training performance over the centralized training primitive, but it remains unexplored how it performs in our proposed distributed training framework.
Revised: [Table] standing effect of ERK initialization in improving overall training performance over the centralized training primitive [Table] but it remains unexplored how it performs in our proposed distributed training framework.

Original: Below, we show in Figure 5 how accuracy evolves with communication rounds under
Revised: Below [Table] we show in Figure 5 how accuracy evolves with communication rounds under

Original: ERK and Uniform 5 initialization.
Revised: ERK and Uniform 5 initialization.

Original: As shown, a drastic drop of accuracy is observed by replacing
Revised: As shown, a drastic drop of accuracy is observed by replacing

Original: ERK with Uniform, by which we conclude that ERK is an essential component for FedSpa (DST).
Revised: ERK with Uniform, by which we conclude that ERK is an essential component for FedSpa (DST) [Table] by ERK or Uniform.

Revised: Sparsity of FedSpa is ﬁxed to 0.5.

Original: However, though a signiﬁcant accuracy enhancement is observed, we note that integrating ERK may sacriﬁce potentially more FLOPS reduction.
Revised: However, though a signiﬁcant accuracy enhancement is observed, we note that integrating ERK may sacriﬁce potentially more FLOPS reduction.

Original: This observation can be found in Table 6, wherein our results show that initialization with Uniform can save 34.3% FLOPs of that with ERK.
Revised: This observation can be found in Table 6, wherein our results show that initialization with Uniform can save 34.3% FLOPs of that with ERK.

Original: [Table]
Revised: [Table]

Original: Different or same mask initialization.
Revised: Different or same mask initialization.

Original: Recall that based on the layer-wise sparsity calculated by
Revised: Recall that based on the layer-wise sparsity calculated by

Original: ERK, FedSpa uses the same random seed to initialize the mask, so as to make the mask exploration of all clients started from the same mask.
Revised: ERK, FedSpa uses the same random seed to initialize the mask, so as to make the mask exploration of all clients started from the same mask.

Original: In the following, we give another implementation that allows each client to share different masks in the beginning.
Revised: In the following, we give another implementation that allows each client to share different masks in the beginning.

Original: As shown in Figure 6, we surprisingly ﬁnd that for FedSpa (DST), maintaining different masks in initialization may slightly enhance its training performance in IID and Non-IID ( γ = 0 . 2 ) setting.
Revised: As shown in Figure 6, we surprisingly ﬁnd that for FedSpa (DST), maintaining different masks in initialization may slightly enhance its training performance in IID and Non-IID ( γ = 0 . 2 ) setting.

Original: We hypothesize that by different mask initialization, each client could more efﬁciently search for their optimal masks that better represents the features and labels of the personal data.
Revised: We hypothesize that by different mask initialization, each client could more efﬁciently search for their optimal masks that better represents the features and labels of the personal data.

Original: For FedSpa (RSM), compared with initialization using the same mask, different mask initialization may result in a drastic performance loss in the IID setting, and a signiﬁcant improvement in the NonIID setting.
Revised: For FedSpa (RSM), compared with initialization using the same mask, different mask initialization may result in a drastic performance loss in the IID setting, and a signiﬁcant improvement in the NonIID setting.

Original: With the same mask initialization of RSM, each client consistently trains based on the same sub-network, which completely eliminates personalization.
Revised: With the same mask initialization of RSM, each client consistently trains based on the same sub-network, which completely eliminates personalization.

Original: So this setting shares a similar performance with FedAvg – with satisfactory performance in IID setting and rather weak performance in Non-IID setting.
Revised: So this setting shares a similar performance with FedAvg – with satisfactory performance in IID setting and rather weak performance in Non-IID setting.

Original: On contrary, by initializing different masks in the beginning, FedSpa (RSM) reserves some degrees of personalization, since only the intersected coordinates in their local models are shared and updated by the information exchange (i.e., average) process.
Revised: On contrary, by initializing different masks in the beginning, FedSpa (RSM) reserves some degrees of personalization, since only the intersected coordinates in their local models are shared and updated by the information exchange (i.e., average) process.

Original: Consequently.
Revised: Consequently.

Original: FedSpa (RSM) with different mask initializations has a similar performance pattern with FedSpa (DST).
Revised: FedSpa (RSM) with different mask initializations has a similar performance pattern with FedSpa (DST).

Original: Another interesting observation is that FedSpa (RSM) with different mask initialization cannot outperform FedSpa (DST) in both the two groups of Non-IID settings.
Revised: Another interesting observation is that FedSpa (RSM) with different mask initialization cannot outperform FedSpa (DST) in both the two groups of Non-IID settings.

Original: This indicates that the DST mask searching process is effective to achieve a superior performance of FedSpa in Non-IID setting.
Revised: This indicates that the DST mask searching process is effective to achieve a superior performance of FedSpa in Non-IID setting.

Original: Weight recovery w/ or w/o gradient information.
Revised: Weight recovery w/ or w/o gradient information.

Original: Recall that in FedSpa (DST), we proposed to use gradient information to recover the pruned weights, which is empirically proven in (Evci et al., 2020) to outperform its random recovery counterpart in Set (Mocanu et al., 2018).
Revised: Recall that in FedSpa (DST), we proposed to use gradient information to recover the pruned weights, which is empirically proven in (Evci et al., 2020) to outperform its random recovery counterpart in Set (Mocanu et al., 2018).

Original: Speciﬁcally, for gradient information-based recovery, the weight coordinates with the top- α t magnitude of the gradient would be recovered, while for random recovery, the coordinates are recovered randomly.
Revised: Speciﬁcally, for gradient information-based recovery, the weight coordinates with the top- α t magnitude of the gradient would be recovered, while for random recovery, the coordinates are recovered randomly.

Original: To demonstrate the impact of the weight recovery method over FedSpa (DST), in Figure 7, we compare the gradient information-based recovery with random recovery.
Revised: To demonstrate the impact of the weight recovery method over FedSpa (DST), in Figure 7, we compare the gradient information-based recovery with random recovery.

Original: Our experimental result demonstrates that recovery with gradient information could slightly accelerate the convergence and enhance the ﬁnal accuracy in our FedSpa framework.
Revised: Our experimental result demonstrates that recovery with gradient information could slightly accelerate the convergence and enhance the ﬁnal accuracy in our FedSpa framework.

Original: Global model vs. Personalized model.
Revised: Global model vs. Personalized model.

Original: In our main experimental result, all the testings are conducted by clients based on their own personalized models.
Revised: In our main experimental result, all the testings are conducted by clients based on their own personalized models.

Revised: But it is interesting to evaluate whether

Original: But it is interesting to evaluate whether the global model trained by FedSpa itself could converge, or even could achieve commensurate performance with the global model trained by general FL solution (e.g. FedAvg).
Revised: [Table] the global model trained by FedSpa itself could converge [Table] or even could achieve commensurate performance with the global model trained by general FL solution (e.g. FedAvg).

Original: As demonstrated by Figure 8, we empirically ﬁnd that in the IID setting, the global model trained by FedSpa cannot recover the performance of that trained by FedAvg, and a considerable performance drop is also observed in the Non-IID setting.
Revised: As demonstrated by Figure [Table] we empirically ﬁnd that in the IID setting, the global model trained by FedSpa cannot recover the performance of that trained by FedAvg [Table]

Original:  Another observation is that the global model of FedSpa surprisingly maintains roughly the same performance as its personalized models in the IID setting, but conceivably suffers signiﬁcant performance loss in the Non-IID setting.
Revised: and a considerable performance drop is also observed in the Non-IID setting. Another observation is that the global model of FedSpa surprisingly maintains roughly the same performance as its personalized models in the IID setting [Table] but conceivably suffers signiﬁcant performance loss in the Non-IID setting.

Original: This corroborates our conclusion that sub-networks extracted from a global model may potentially outperform the full model, under the condition that the data distributions of clients are skewed (or heterogeneous). 
Revised: This corroborates our conclusion that sub-networks extracted from a global model may potentially outperform the full model, under the condition that the data distributions of clients are skewed (or heterogeneous). is ﬁxed to 0.5. Wall time.

Revised: Recall that we do an additional mask searching procedure in FedSpa (DST), which might possibly induce extra wall time on the local devices.

Revised: We show in Table 5 the wall time used for training and mask searching in one single local round.

Original: B P ROOF OF T HEOREM
Revised: The sparsity used for this experiment is ﬁxed to 0.5, while other parameters remain the default setting (see Section 5.1). We use one single 1080Ti to perform training for the GPU-based experiment, while the CPU-based experiment is conducted on an Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz with 8 cores. Our experimental results conﬁrm that the mask searching process only accounts for a small portion of wall time (approximately 5% − 20% ) for the entire computation time on the local devices. C P ROOF OF T HEOREM

Original: In this section, for sake of readability, we ﬁrst clarify the notations and reiterate several facts that we use in our proof.
Revised: In this section, for sake of readability, we ﬁrst clarify the notations and reiterate several facts that we use in our proof.

Original: Then we present several lemmas that are commonly used in the FL literature (see (Karimireddy et al., 2020; Xu et al., 2021)).
Revised: Then we present several lemmas that are commonly used in the FL literature (see (Karimireddy et al., 2020; Xu et al., 2021)).

Original: Later, several key lemmas are listed with exhaustive proof, and ﬁnally, the proof of our main theorem is given by exploiting the listed lemmas and facts.
Revised: Later, several key lemmas are listed with exhaustive proof, and ﬁnally, the proof of our main theorem is given by exploiting the listed lemmas and facts.

Original: B. N OTATIONS AND F ACTS
Revised: C. N OTATIONS AND F ACTS

Original: Throughout the proof, we assume (cid:80) τ equivalent to (cid:80) N − 1 τ =0 , (cid:80)
Revised: Throughout the proof, we assume (cid:80) τ equivalent to (cid:80) N − 1 τ =0 , (cid:80)

Original: k equivalent to (cid:80)
Revised: k equivalent to (cid:80)

Original: Kk =1 , and (cid:80) k,τ equivalent to (cid:80)
Revised: Kk =1 , and (cid:80) k,τ equivalent to (cid:80)

Original: Kk =1 (cid:80) N − 1 τ =

Revised: Kk =1 (cid:80) N − 1 τ =

Original: unless otherwise speciﬁed.
Revised: unless otherwise speciﬁed.

Original: In our proof, we reuse most of the notations from our problem formulation part.
Revised: In our proof, we reuse most of the notations from our problem formulation part.

Original: We use g k,t,τ ( ˜ w k,t,τ ) = ∇ ˜ w k,t,τ L ( ˜ w k,t,τ ; ξ k,t,τ ) to denote the stochastic gradient of client k in round t and at step τ .
Revised: We use g k,t,τ ( ˜ [Equation] k,t [Equation]

Revised: k

Revised: [Equation]

Revised: k

Revised: [Equation]

Revised: t

Revised: [Equation]

Revised: ) to denote the stochastic gradient of client k in round t and at step τ .

Original: Then as per our formulation in Section 4.1, we reiterate the following facts, which would be heavily used in our proof.
Revised: Then as per our formulation in Section 4.1, we reiterate the following facts, which would be heavily used in our proof.

Original: Fact 1 (Local step) .
Revised: Fact 1 (Local step) .

Original: As per Eq.
Revised: As per Eq.

Original: (3), one local step of client’s update can be formalized as follows:
Revised: (3), one local step of client’s update can be formalized as follows:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: ) where g k,t,τ ( ˜ w k,t,τ ) is the stochastic gradient over the sparse model weights ˜ w k,t,τ , and ˜ w k,t, 

Revised: where g k,t,τ ( ˜ w k,t,τ ) is the stochastic gradient over the sparse model weights ˜ w k,t,τ , and ˜ w k,t,

Original: = m k,t (cid:12) w t is the synchronized local weights at the beginning of a communication round.
Revised: = m k,t (cid:12) w t is the synchronized local weights at the beginning of a communication round.

Original: Fact 2 (Local update from client k ) .
Revised: Fact 2 (Local update from client k ) .

Original: The local update of clients can be formalized as follows:
Revised: The local update of clients can be formalized as follows:

Original: [Equation]
Revised: [Equation]

Original: Fact 3 (Server’s update) .
Revised: Fact 3 (Server’s update) .

Original: The server aggregates the sparse update by averaging, which can be formalized as follows:
Revised: The server aggregates the sparse update by averaging, which can be formalized as follows:

Original: [Equation]
Revised: [Equation]

Original: Fact 4 (Global and local loss) .
Revised: Fact 4 (Global and local loss) .

Original: The local loss of a client is denoted by F k ( ˜ w k ) , and is formulated as:
Revised: The local loss of a client is denoted by F k ( ˜ w k ) , and is formulated as:

Original: [Equation]
Revised: [Equation]

Original: where ˜ w k = m ∗ k (cid:12) w , and the global loss can be formalized as follows:
Revised: where ˜ w k = m ∗ k (cid:12) w , and the global loss can be formalized as follows:

Original: [Equation]
Revised: [Equation]

Original: and { m ∗ k } in the function are viewed as optimal personalized masks which satisﬁes the sparse volume constraint (cid:107) 1 − m ∗ k (cid:107) 

Revised: and { m ∗ k } in the function are viewed as optimal personalized masks which satisﬁes the sparse volume constraint (cid:107) 1 − m ∗ k (cid:107)

Original: = β for all k .
Revised: = β for all k .

Original: B. A UXILIARY L EMMAS
Revised: C. A UXILIARY L EMMAS

Original: In the following.
Revised: In the following.

Original: we shall present several common lemmas that are heavily used in the FL literature.
Revised: we shall present several common lemmas that are heavily used in the FL literature.

Original: Lemma 1 (Cauchy-Schwarz) .
Revised: Lemma 1 (Cauchy-Schwarz) .

Original: Assume arbitrary vector sequences { a k } k =1 ,...,K and { b k } k =1 ,...,K , Cauchy-Schwarz inequality implies:
Revised: Assume arbitrary vector sequences { a k } k =1 ,...,K and { b k } k =1 ,...,K , Cauchy-Schwarz inequality implies:

Original: [Equation]
Revised: [Equation]

Original: by taking b k = 1 , we also have:
Revised: by taking b k = 1 , we also have:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Lemma 2 (Separating mean and variance, Lemma B.3 (Xu et al., 2021)) .
Revised: Lemma 2 (Separating mean and variance, Lemma B.3 (Xu et al., 2021)) .

Original: Let { a 1 , . . .
Revised: Let { a 1 , . . .

Original: , a τ } be τ random vectors in R d .
Revised: , a τ } be τ random vectors in R d .

Original: Suppose that { a i − ξ i } form a martingale difference sequence, i.e.
Revised: Suppose that { a i − ξ i } form a martingale difference sequence, i.e.

Original: E [ a i − ξ i | a 1 , . .
Revised: E [ a i − ξ i | a 1 , . .

Original: , a i − 1 ] = 0 , and suppose that their variance is bounded by E (cid:104) (cid:107) a i − ξ i (cid:107) 2 (cid:105) ≤ σ 2 .
Revised: , a i − 1 ] = 0 , and suppose that their variance is bounded by E (cid:104) (cid:107) a i − ξ i (cid:107) 2 (cid:105) ≤ σ 2 .

Original: Then, the following inequality holds:
Revised: Then, the following inequality holds:

Original: [Equation]
Revised: [Equation]

Original: (cid:88) i =

Original: a
Revised: τσ

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   ≤ 2 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) τ (cid:88) i =

Original: ξ i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) + 2 τσ 2 .

Original: Lemma 3 (Relaxed triangle inequality, Lemma 3 (Karimireddy et al., 2020)) .
Revised: Lemma 3 (Relaxed triangle inequality, Lemma 3 (Karimireddy et al., 2020)) .

Original: Let v i and v j be vectors in R d .
Revised: Let v i and v j be vectors in R d .

Original: Then the following inequality holds true for any a > 0 :
Revised: Then the following inequality holds true for any a > 0 :

Original: [Equation]
Revised: [Equation]

Original: Lemma 4.
Revised: Lemma 4.

Original: For random vector v 1 satisfying E [ v 1 ] = 
Revised: For random vector v 1 satisfying E [ v 1 ] =

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original:   , and assume another random vector v 2 is independent with v 1 , we have:
Revised:  , and assume another random vector v 2 is independent with v 1 , we have:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: This completes the proof.
Revised: This completes the proof.

Original: B. K EY L EMMAS
Revised: C. K EY L EMMAS

Original: In this section, we present several important lemmas that would be used in our formal proof.
Revised: In this section, we present several important lemmas that would be used in our formal proof.

Original: All the presented claims are rigorously proved.
Revised: All the presented claims are rigorously proved.

Original: Lemma 5 (Smoothness of f ( w ) ) .
Revised: Lemma 5 (Smoothness of f ( w ) ) .

Original: Assume ˜
Revised: Assume ˜

Original: w k = m ∗ k (cid:12) w for any m ∗ k ∈ { 0 , 1 } d , we have Lsmoothness for f ( w ) = 1 K (cid:80) k
Revised: w k = m ∗ k (cid:12) w for any m ∗ k ∈ { 0 , 1 } d , we have Lsmoothness for f ( w ) = 1 K (cid:80) k

Original: F k ( ˜ w k ) , i.e., for any w 1 , w 2 ∈ R d , we have:(cid:107)∇
Revised: F k ( ˜ w k ) , i.e., for any w 1 , w 2 ∈ R d , we have:

Original: [Equation]
Revised: [Equation]

Original: (cid:107) .

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst equality holds by the Fact 4 and inequality (a) is due to Assumption 3.
Revised: where the ﬁrst equality holds by the Fact 4 and inequality (a) is due to Assumption 3.

Original: This completes the proof.
Revised: This completes the proof

Original:  Lemma 6 (Separating mean and variance of stochastic gradient) .
Revised: [Equation] Lemma 6 (Separating mean and variance of stochastic gradient) .

Revised: For m k

Revised: [Equation]

Revised: t ∈ {

Original: For m k,t ∈ { 0 , 1 } d , We have the following bounding for the expected average gradient:
Revised: [Equation] 1 } d [Equation] We have the following bounding for the expected average gradient:

Original: [Equation]
Revised: [Equation]

Original: where E t [ · ] denotes the expectation over all the randomness of round t .
Revised: where E t [ · ] denotes the expectation over all the randomness of round t .

Original: Proof.
Revised: Proof.

Original: We view 1 S m k,t (cid:12) g k,t,τ ( ˜ w k,t,τ ) for all k and τ as stochastic vectors.
Revised: We view 1 S m k,t (cid:12) g k,t,τ ( ˜ w k,t,τ ) for all k and τ as stochastic vectors.

Original: By the unbiasedness of stochastic gradient, we know their variance satisﬁes:
Revised: By the unbiasedness of stochastic gradient, we know their variance satisﬁes:

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality holds since m k,t ∈ { 0 , 1 } d and the last inequality is due to Assumption 2.
Revised: where the ﬁrst inequality holds since m k,t ∈ { 0 , 1 } d and the last inequality is due to Assumption [Equation]

Original: As per the variance given above, and directly apply Lemma 2, the claim immediately shows.
Revised: As per the variance given above, and directly apply Lemma 2, the claim immediately shows.

Original: Lemma 7.
Revised: Lemma 7.

Original: We have the following error bound by introducing personalized masks:
Revised: We have the following error bound by introducing personalized masks:

Original: [Equation]
Revised: [Equation]

Original: and,
Revised: and,

Original: [Equation]
Revised: [Equation]

Original:  where dist ( m 1 , m 2 ) is the hamming distance between two masks.
Revised: 21) where dist ( m 1 , m 2 ) is the hamming distance between two masks [Equation]

Original: Proof.
Revised: Proof.

Original: Let ∇ ( j ) F k ( ˜ w k,t,τ ) and ∇ ( j ) F k ( ˜ w k,t ) be the derivative over the j -th weight coordinates.
Revised: Let ∇ ( j ) F k ( ˜ w k,t,τ ) and ∇ ( j ) F k ( ˜ w k,t ) be the derivative over the j -th weight coordinates.

Original: Deﬁne the support of a vector v ∈ R d as supp ( v ) =
Revised: Deﬁne the support of a vector v ∈ R d as supp ( v ) =

Original: { j : v ( j ) (cid:54) = 0 } .
Revised: { j : v ( j ) (cid:54) = 0 } .

Original: The left hand side of the ﬁrst claim can be rewritten and bounded as follows:
Revised: The left hand side of the ﬁrst claim can be rewritten and bounded as follows:

Original: [Equation]
Revised: [Equation]

Original: k (

Original: [Equation]

Original: w

Original: [Equation]

Original: t

Original: [Equation]

Original: τ )

Original: [Equation]

Original: j )

Original: [Equation] (cid:107)+ (cid:88) j ∈ supp ( m k,t ) and j/ ∈ supp ( m ∗ k  )  (cid:107)∇ [Equation] j ) F k ( ˜ w k
Revised: where inequality ( a ) is true since we have (cid:107)∇ ( j ) F k ( ˜ w k,t,τ ) (cid:107) 2 ≤ B 2 and (cid:107)∇ ( j ) F k ( ˜ w k,t ) (cid:107) 2 ≤ B 2 by coordinate-wise bounded gradient Assumption 4.

Original: [Equation]

Original: t,τ ) (cid:107)

Original: [Equation]

Original: + (cid:88) j/ ∈ supp ( m k,t ) and j ∈ supp ( m

Original: [Equation]

Original: k ) (cid:107)∇

Original: [Equation]

Original: j

Original: [Equation]

Original: F k [Equation] ˜ [Equation]
Revised: Inequality (b) is due to the deﬁnition of hamming distance, dist ( v 1 , v 2 )

Original: k

Original: [Equation]

Original: t ) (cid:107)( a ) ≤ (cid:88)

Original: j ∈ supp ( m k,t ) ∩ supp

Original: [Equation]

Original: m ∗

Original: [Equation]

Original: ) (cid:107)∇

Original: [Equation]

Original: b

Original: [Equation] j ∈ supp [Equation] k
Revised: = (cid:80) j ∈ supp ( v 1 ⊕ v 2 ) .

Original: [Equation] [Equation] supp [Equation]
Revised: This shows our ﬁrst claim.

Original: ) (cid:107)∇ (

Original: [Equation]

Original: ∇

Original: [Equation]

Original: dist

Original: [Equation] ≤ (cid:88) j (cid:107)∇ ( j ) F k ( ˜ w k [Equation]
Revised: Following the same technique

Original: t

Original: [Equation]

Original: τ ) − ∇ ( j )

Original: [Equation]

Original: k ( ˜

Original: [Equation]

Original: k

Original: [Equation]
Revised: [Equation]

Original: t ) (cid:107) 2 + dist ( m k

Original: [Equation]

Original: t

Original: [Equation]

Original: m

Original: [Equation]

Original: k )

Original: B≤(cid:107)∇ F k ( ˜ w k,t,τ ) − ∇ F k ( ˜ w k,t ) (cid:107) 2 + dist ( m k,t , m ∗ k ) B(22) where inequality ( a ) is true since we have (cid:107)∇ ( j ) F k ( ˜ w k,t,τ ) (cid:107) 2 ≤ B 2 and (cid:107)∇ ( j ) F k ( ˜ w k,t ) (cid:107) 2 ≤ B 2 by coordinate-wise bounded gradient Assumption 4.

Original: Inequality (b) is due to the deﬁnition of hamming distance, dist ( v 1 , v 2 )

Original: = (cid:80) j ∈ supp ( v 1 ⊕ v 2 ) .

Original: This shows our ﬁrst claim.

Original: Following the same technique, we expand the left hand side of the second claim using Cauchy
Revised: we expand the left hand side of the second claim using Cauchy

Original: Schwarz inequality in Lemma 1, as follows:
Revised: Schwarz inequality in Lemma 1, as follows:

Original: [Equation]
Revised: [Equation]

Original: For the last inequality, we use the same technique as in the ﬁrst claim to bound the second term and the third term, and the ﬁrst term is bounded by Lemma 8.
Revised: For the last inequality, we use the same technique as in the ﬁrst claim to bound the second term and the third term, and the ﬁrst term is bounded by Lemma [Equation]

Original: Lemma 8.
Revised: Lemma 8.

Original: Let (cid:107) 1 − m (cid:107) 

Revised: Let (cid:107) 1 − m (cid:107)

Original: = β be the sparse volume of a mask m , and suppose m ∈ { 0 , 1 } d , by assumption 4, we have the following relation:
Revised: = β be the sparse volume of a mask m , and suppose m ∈ { 0 , 1 } d , by assumption 4, we have the following relation:

Original: [Equation]
Revised: [Equation]

Original: and,
Revised: and,

Original: [Equation]
Revised: [Equation]

Original: where d is the dimension of the models (or identically, the dimension of the mask m ).
Revised: where d is the dimension of the models (or identically, the dimension of the mask m ).

Original: Proof.
Revised: Proof.

Original: The proof of this lemma follows a similar technique in Lemma 7.
Revised: The proof of this lemma follows a similar technique in Lemma 7.

Original: Let ∇ ( j ) F k ( ˜ w k,t ) be the derivative over the j -th weight coordinate.
Revised: Let ∇ ( j ) F k ( ˜ w k,t ) be the derivative over the j -th weight coordinate.

Original: Deﬁne the support of a vector v ∈ R d as supp ( v ) =
Revised: Deﬁne the support of a vector v ∈ R d as supp ( v ) =

Original: { j : v ( j ) (cid:54) = 0 } .
Revised: { j : v ( j ) (cid:54) = 0 } .

Original: The left hand side (L.H.S) of our ﬁrst claim can be bounded by:
Revised: The left hand side (L.H.S) of our ﬁrst claim can be bounded by:

Original: [Equation]
Revised: [Equation]

Original: 26) where the last inequality is obtained from the sparsity constraint and the coordinate-wise gradient dissimilarity Assumption 1.
Revised: where the last inequality is obtained from the sparsity constraint and the coordinate-wise gradient dissimilarity Assumption 1.

Original: This shows our ﬁrst claim.
Revised: This shows our ﬁrst claim.

Original: For our second claim, we expand its L.H.S as follows:
Revised: For our second claim, we expand its L.H.S as follows:

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is due to Assumption 1 and Assumption 4.
Revised: where the last inequality is due to Assumption 1 and Assumption 4.

Original: It completes the proof.
Revised: It completes the proof.

Original: Lemma 9 (Drift towards Synchronized Point) .
Revised: Lemma 9 (Drift towards Synchronized Point) .

Original: For any t ∈ { 1 , . . .
Revised: For any t ∈ { 1 , . . .

Original: , T } , τ ∈ { 0 , . . .
Revised: , T } , τ ∈ { 0 , . . .

Original: , N } ,
Revised: , N } ,

Original: and learning rate satisﬁes η ≤ 1 16 LN , we have the following claim:
Revised: and learning rate satisﬁes η ≤ 1 16 LN , we have the following claim:

Original: [Equation]
Revised: [Equation]

Original: where Φ t = 1 K (cid:80) k
Revised: where Φ t = 1 K (cid:80) k

Original: (( d − β ) G 2 + 1 K
Revised: (( d − β ) G 2 + 1 K

Original: (cid:80) k (cid:48)
Revised: (cid:80) k (cid:48)

Original: B 2 ( dist ( m k,t , m k (cid:48) ,t ) + dist ( m k (cid:48) ,t , m ∗ k (cid:48) ))) and E t [ · ] denotes the expectation over all the randomness of round t .
Revised: B 2 ( dist ( m k,t , m k (cid:48) ,t ) + dist ( m k (cid:48) ,t , m ∗ k (cid:48) ))) and E t [ · ] denotes the expectation over all the randomness of round t .

Original: Proof.
Revised: Proof.

Original: We follow the basic techniques from (Reddi et al., 2020) to prove this lemma
Revised: We follow the basic techniques from (Reddi et al., 2020) to prove this lemma

Original: [Equation]
Revised: [Equation]

Original: We ﬁrst assume that:
Revised: We ﬁrst assume that:

Original: • O = m k,t (cid:12) g k,t,τ − 1 ( ˜ w k,t,τ − 1 ) − m k,t (cid:12) ∇ F k ( ˜ w k,t,τ − 1 ) • P
Revised: • O = m k,t (cid:12) g k,t,τ − 1 ( ˜ w k,t,τ − 1 ) − m k,t (cid:12) ∇ F k ( ˜ w k,t,τ − 1 ) • P

Original: = m k,t (cid:12) ∇ F k ( ˜ w k,t,τ − 1 ) − m k,t (cid:12) ∇ F k ( ˜ w k,t )
Revised: = m k,t (cid:12) ∇ F k ( ˜ w k,t,τ − 1 ) − m k,t (cid:12) ∇ F k ( ˜ w k,t )

Original: • Q = m k,t (cid:12) ∇ F k ( ˜ w k,t ) − ∇ f ( w t )
Revised: • Q = m k,t (cid:12) ∇ F k ( ˜ w k,t ) − ∇ f ( w t )

Original: We can expand ˜ w k,t,τ as follows:
Revised: We can expand ˜ w k,t,τ as follows:

Original: [Equation]
Revised: [Equation]

Original: where the last inequality follows from Lemma 3 and Lemma 4.
Revised: where the last inequality follows from Lemma 3 and Lemma 4.

Original: Explicitly, we use Lemma 4 to treat the stochastic term with E t (cid:107) O (cid:107) 2 , and then we use Lemma 3 with a = 2 N − 1 to separate the other four terms.
Revised: Explicitly, we use Lemma 4 to treat the stochastic term with E t (cid:107) O (cid:107) 2 , and then we use Lemma 3 with a = 2 N − 1 to separate the other four terms.

Original: Then we proceed by separately bounding the components in the above inequality.
Revised: Then we proceed by separately bounding the components in the above inequality.

Original: Bounding the second term:
Revised: Bounding the second term:

Original: [Equation]
Revised: [Equation]

Original: where the last inequality holds by Assumption 2.
Revised: where the last inequality holds by Assumption 2.

Original: Bounding the third term:
Revised: Bounding the third term:

Original: [Equation]
Revised: [Equation]

Original: Bounding the fourth term:
Revised: Bounding the fourth term:

Original: [Equation]
Revised: [Equation]

Original: where we use the second claim in Lemma 7 in the last inequality.
Revised: where we use the second claim in Lemma 7 in the last inequality.

Original: Let Φ
Revised: Let Φ

Original: [Equation]
Revised: [Equation]

Original: (cid:48) ))) .
Revised: (cid:48) ))) .

Original: The bound can be simpliﬁed as follows:
Revised: The bound can be simpliﬁed as follows:

Original: [Equation]
Revised: [Equation]

Original: Putting together: Plugging all the components into Eq.(29)
Revised: Putting together: Plugging all the components into Eq.(29)

Original: [Equation]
Revised: [Equation]

Original: the following result immediately follows:
Revised: the following result immediately follows:

Original: [Equation]
Revised: [Equation]

Original: where the last inequlity holds by our assumption η ≤ 1 16 LN .
Revised: where the last inequlity holds by our assumption η ≤ 1 16 LN .

Original: Unrolling the recursion
Revised: Unrolling the recursion

Original: [Equation]
Revised: [Equation]

Original: we obtain the following results:
Revised: we obtain the following results:

Original: [Equation]
Revised: [Equation]

Original: The last inequality holds since (cid:16) (1 + 1 N − 1 ) N − 1 (cid:17) ≤ 5 for N ≥ 1 .
Revised: The last inequality holds since (cid:16) (1 + 1 N − 1 ) N − 1 (cid:17) ≤ 5 for N ≥ 1 .

Original: This completes the proof.
Revised: This completes the proof.

Original: B. F ORMAL P ROOF
Revised: C. F ORMAL P ROOF

Original: We start our proof by expanding f ( w t +1 ) under its smoothness condition (see Lemma 5)
Revised: We start our proof by expanding f ( w t +1 ) under its smoothness condition (see Lemma 5)

Original: [Equation]
Revised: [Equation]

Original: which indicates that:
Revised: which indicates that:

Original: [Equation]
Revised: [Equation]

Original: E t (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
Revised: E t (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: (cid:88) k ∈ S
Revised: (cid:88) k ∈ S

Original: [Equation]
Revised: [Equation]

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: t (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:124) (cid:123)(cid:122) (cid:125) T
Revised: t (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:124) (cid:123)(cid:122) (cid:125) T

Original: [Equation]
Revised: [Equation]

Original: (36) where the last inequality holds since − ab ≤ 12 (( b − a ) 2 − a 2 )
Revised: (36) where the last inequality holds since − ab ≤ 12 (( b − a ) 2 − a 2 )

Original: [Equation]
Revised: [Equation]

Original: and E t [ · ] is the expectation over all the randomness in round t
Revised: and E t [ · ] is the expectation over all the randomness in round t

Original: [Equation]
Revised: [Equation]

Original: In the following
Revised: In the following

Original: [Equation]
Revised: [Equation]

Original: we shall separately bound T 1 and T 2 .
Revised: we shall separately bound T 1 and T 2 .

Original: Bounding T 1 :
Revised: Bounding T 1 :

Original: [Equation]
Revised: [Equation]

Original: where inequality (a) is due to Cauchy-Schwarz inequality (i
Revised: where inequality (a) is due to Cauchy-Schwarz inequality (i

Original: [Equation]
Revised: [Equation]

Original: e., Lemma 1), (b) follows from the ﬁrst claim in Lemma 7, and the last inequality holds by Assumption
Revised: e., Lemma 1), (b) follows from the ﬁrst claim in Lemma 7, and the last inequality holds by Assumption

Original: [Equation]
Revised: [Equation]

Original: Plugging the results of Lemma
Revised: Plugging the results of Lemma

Original: [Equation]
Revised: [Equation]

Original: we obtain that:
Revised: we obtain that:

Original: [Equation]
Revised: [Equation]

Original: Bounding T 2 :
Revised: Bounding T 2 :

Original: [Equation]
Revised: [Equation]

Original: where (a) is obtained as per Lemma 6.
Revised: where (a) is obtained as per Lemma 6.

Original: Bounding T 3 :
Revised: Bounding T 3 :

Original: T
Revised: T

Original: [Equation]
Revised: [Equation]

Original: where the last inequality holds since
Revised: where the last inequality holds since

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is obtained by the L-smoothness Assumption
Revised: where the last inequality is obtained by the L-smoothness Assumption

Original: [Equation]
Revised: [Equation]

Original: Bounding
Revised: Bounding

Original: T 5 :
Revised: T 5 :

Original: [Equation]
Revised: [Equation]

Original: where the last inequality holds since K − S S ( K − 1) ≤ 1 under the condition S ≥ 1 .
Revised: where the last inequality holds since K − S S ( K − 1) ≤ 1 under the condition S ≥ 1 .

Original: Inequality (a) is obtained by Assumption 3 and the second claim in Lemma 7 .
Revised: Inequality (a) is obtained by Assumption 3 and the second claim in Lemma 7 .

Original: Summing T 4 and T 5 , we have the following bounding for T 3 :
Revised: Summing T 4 and T 5 , we have the following bounding for T 3 :

Original: [Equation]
Revised: [Equation]

Original: Plugging Lemma 9 into the above inequality, we have:
Revised: Plugging Lemma 9 into the above inequality, we have:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Plugging T 3 into Inequality (39)
Revised: Plugging T 3 into Inequality (39)

Original: [Equation]
Revised: [Equation]

Original: we bound T 2 as follows:
Revised: we bound T 2 as follows:

Original: [Equation]
Revised: [Equation]

Original: Plugging T 2 and T 1 into R.H.S of Inequality (36) , we obtain that:
Revised: Plugging T 2 and T 1 into R.H.S of Inequality (36) , we obtain that:

Original: [Equation]
Revised: [Equation]

Original: Taking expectation over the randomness before round t towards both sides of the inequality
Revised: Taking expectation over the randomness before round t towards both sides of the inequality

Original: [Equation]
Revised: [Equation]

Original: it yields:
Revised: it yields:

Original: [Equation]
Revised: [Equation]

Original: where κ = 12 − 150 N 3 η 3 L 3 − 15 N 2 η 2 L 2 − 5 NηL and ρ t = (25 N 3 η 4 L 3 + 5 N 2 η 3 L 2)( σ 2 +
Revised: where κ = 12 − 150 N 3 η 3 L 3 − 15 N 2 η 2 L 2 − 5 NηL and ρ t = (25 N 3 η 4 L 3 + 5 N 2 η 3 L 2)( σ 2 +

Original: [Equation]
Revised: [Equation]

Original: By rearanging
Revised: By rearanging

Original: [Equation]
Revised: [Equation]

Original: and summing from t =
Revised: and summing from t =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: T −
Revised: T −

Original: [Equation]
Revised: [Equation]

Original: the following result immediately shows:
Revised: the following result immediately shows:

Original: [Equation]
Revised: [Equation]

Original: Now we bound the local gradient over the personalized sparse model. i
Revised: Now we bound the local gradient over the personalized sparse model

Original: [Equation]
Revised: [Equation]

Original: e., the error bound towards ultimate PFL problem (P2) :
Revised: Lemma 1 ≤

Original: [Equation]
Revised: [Equation]

Original: Lemma [Equation] [Equation] In inequality (a), we use the second claim in Lemma 8 to treat the ﬁrst term, and the second term is bounded using a similar technique as in the ﬁrst claim in Lemma 8.
Revised: In inequality (a), we use the second claim in Lemma 8 to treat the ﬁrst term, and the second term is bounded using a similar technique as in the ﬁrst claim in Lemma 8.

Original: Let Υ = 3( d − β ) G 2 + 3 βB 2 + 3 K 2 (cid:80) k
Revised: Let Υ = 3( d − β ) G 2 + 3 βB 2 + 3 K 2 (cid:80) k

Original: (cid:80) k (cid:48) dist ( m ∗ k , m ∗ k (cid:48) )
Revised: (cid:80) k (cid:48) dist ( m ∗ k , m ∗ k (cid:48) )

Original: B 2 and plugging inequality (48) into the above inequality, it further implies that:
Revised: B 2 and plugging inequality (48) into the above inequality, it further implies that:

Original: [Equation]
Revised: [Equation]

Original: This shows the claim.
Revised: This shows the claim.

Revised: D D ISCUSSION ON VARIOUS BOUNDS IN EXISTING STUDY

Revised: Under the non-convex setting, Theorem 1 bounds the squared gradient norm of the personalized iterates over the local loss.

Revised: Existing studies on PFL zero in various different kinds of convergence bound, and under different assumptions.

Revised: Below are some concrete examples:

Revised: • Li et al.

Revised: (2021) study the upper bound of E (cid:104) (cid:107) ˜ w k,T − w ∗ k (cid:107) 2 (cid:105) for any device k (see their Corollary 1), where w k,T is the personalized weights at iteration T and w ∗ k is the optimal weights under the client k ’s local loss.

Revised: Under the assumption of strongly convex, smoothness, bounded gradient, and gradient dissimilarity, as well as an additional assumption on the distance between personalized models and the optimal global model, it is theoretically proved that the proposed solution, named Ditto, achieves O (1 /T ) convergence rate.

Revised: Their derived convergence rate is on the same scale as the convergence rate of the global model produced by FedAvg.

Revised: However, it is important to note that an extra assumption on the distance between personalized models and the optimal global model is required to achieve this rate.

Revised: • Hanzely et al.

Revised: (2021) study the convergence of the proposed regularization problem (see their Theorem 4.5).

Revised: Speciﬁcally, they propose to bound E (cid:104) (cid:80) K k =1 (cid:107) ˜ w k,T − w ∗ k ( λ ) (cid:107) 2 (cid:105) where ˜ w k,T is the personalized model, and w ∗ k ( λ ) is the optimal model for their deﬁned regularization problem under the regularized hyper-parameter λ .

Revised: Under the strongly convex and smoothness assumption, the authors derive the bound as E (cid:104)(cid:80)

Revised: Kk =1 (cid:107) ˜

Revised: w k,T − w ∗ k ( λ ) (cid:107) 2 (cid:105) ≤ (cid:0) 1 − αµn (cid:1) T (cid:80)

Revised: Kk =1 (cid:107)

Revised: [Equation]

Revised: Further, by applying the variance reduction technique on their proposed L2GD solution, they eliminate the constant term (i.e., the second term) in the above bound.

Revised: However, their convergence analysis are made towards a special case of the regularized problem.

Revised: It remains unspeciﬁed about how and whether the personalized iteration produced by L2GD can achieve convergence towards the optimum for each client’s local loss (i.e., the optimum of the ultimate PFL problem (P2)).

Revised: In other words, the gap between ˜ w k,T and w ∗ k = min w k ˆ F k ( w k ) remains unknown.

Revised: • Under the non-convex setting, Deng et al.

Revised: (2020) study the upper bound of squared gradient norm, or formally, 1 T (cid:80)

Revised: Tt =1 E (cid:104) (cid:107)∇ F k ( ˜ w k,t ) (cid:107) 2 (cid:105) (see their Theorem 6) , where ˜ w k,t is the personalized model for client k and ∇ F k ( · ) is the gradient of a model over the local loss.

Revised: Under the smoothness and bounded variance conditions, they derive a bound with rate O ( 1 √ T ) to convergence.

Revised: Similar to our results, a non-vanished residual presents in their bound.

Revised: This non-vanished residual is related to: i) the gradient dissimilarity between clients’ local loss function; and (ii) the gradient discrepancy, The form of this non-vanished term is highly similar to our derived residual term (i.e., Υ in our Theorem 5), since both the gradient dissimilarity factor (i.e., G in Assumption 1) and gradient bound (i.e., B in Assumption 4) would present in the non-vanished residual.

Revised: Additionally, when the personalized factor is completely eliminated i.e., α k → 0 in their formulation or β → d in our formulation, the non-vanished residual (i.e., Γ in their formulation and Υ in ours) would both become signiﬁcant to dominate the bound.