Paper: SpaFL: Communication-Efficient Federated Learning with Sparse Models and Low Computational Overhead (ICLR-2024)

Reviewer comment (Theory):
Section 3.2.2 needs to be written more clear. During local training with e < E, does the thresholds not be updated? Equation (5) and (6) only work for the e=E? And how the equation (6) is derived?

Author response:
**A1**: Yes, when $e < E$, only model parameters are updated while thresholds are frozen. In the last epoch, i.e., $e = E$, we freeze model parameters and update thresholds using eq. (5) and (6). Hence, during $E-1$ epochs, model parameters can adapt to newly generated masks by recovering from pruning-induce noise. Then, in the last epoch, we give the thresholds performance feedback on the current sparse model. We believe that the explanation of the algorithm was not clear enough in the manuscript, and we will clarify it in the revised version. 

For the derivation of eq. (6), we define $S(\cdot)$ as a unit step function and $Q\_{k, ij}^{E-1, l} (t)) = |w\_{k, ij}^{E-1, l} (t)|  -  \tau_i^l(t)$. Then, we can derive eq. (6) as follows:

$$ \frac{\partial F\_k (\tilde{w}\_k^{E-1}(t),  \tau(t) )}{\partial \tau_i^l(t)} = \sum\_{j=1}^{n} \frac{ \partial \tilde{w}\_{k, ij}^{E-1} (t) } { \partial \tau\_i^l (t) }   \frac{\partial F\_k (\tilde{w}\_k^{E-1} (t),  \tau(t)) }{\partial \tilde{w}\_{k, ij}^{E-1} (t) } =  \sum\_{j=1}^{n\_{in}^l}  \frac{ \partial \tilde{w}\_{k, ij}^{E-1, l} (t) } { \partial Q\_{k, ij}^{E-1, l} (t) } \frac{ \partial Q\_{k, ij}^{E-1, l} (t) } { \partial \tau\_i^l (t) } g\_k( \tilde{w}\_k^{E-1}(t))\_{ij}^l $$
$$= \sum\_{j=1}^{n\_{in}^l} w\_{k, ij}^{E-1, l} (t) \frac{ \partial S(Q\_{k, ij}^{E-1, l} (t)) } { \partial Q\_{k, ij}^{E-1, l} (t) } (-1) g\_k( \tilde{w}\_k^{E-1}(t))\_{ij}^l  = -\sum\_{j=1}^{n\_{in}^l} w\_{k, ij}^{E-1, l} (t) g\_k( \tilde{w}\_k^{E-1}(t))\_{ij}^l ,$$
where the last equation results from the fact that we used the identity straight-through estimator for $\frac{ \partial S(Q\_{k, ij}^{E-1, l} (t)) } { \partial Q\_{k, ij}^{E-1, l} (t) }$ and $w$ and $\tau$ are constrained to be within
$[−1, 1]$ and $[0, 1]$ as described in Section. 3.2.1.

Reviewer comment (Experiments):
Experiment settings are not clear enough. The training dataset is split using Dirichlet sampling. For personalized FL, how are test datasets split and how the models are tested? Why all methods use the same learning rates? Maybe different methods have different best learning rates.

Author response:
**A3:** Test datasets are distributed using the same Dirichlet sampling used in distributing the training datasets. We calculated the accuracy by averaging each client’s model performance on its test datasets following [1, 2]. We averaged all the results over at least 10 different random seeds.  As stated by the reviewer, every method has different well working learning rates. But, we found that we didn't provide them in the manuscript. Please find the table for the learning rates of the baselines.  

|Algorithm| FMNIST | CIFAR-10 | CIFAR-100|
|:------:| :----: | :----:| :-----:|
|FedAvg| $lr = 0.001$ | $lr = 0.01$ | $lr = 0.1$|
|Sub-FedAvg|  $lr = 0.001$ | $lr = 0.001$ |$lr = 0.2$ |
|LG-FedAvg| $lr = 0.001$ | $lr = 0.005$ | $lr = 0.0015$|
|PruneFL| $lr = 0.01$ | $lr = 0.001$ | $lr = 0.05$|
|LotteryFL| $lr = 0.001$| $lr = 0.005$ | $lr = 0.01$|
|FedDST| $lr = 0.001$ | $lr = 0.01$ | $lr = 0.01$|
|FedPM| $lr = 0.15 $ |$lr = 0.1$ | $lr = 0.1$|

We will add this table to the revised manuscript. 

[1] Li, Tian, et al. "Ditto: Fair and robust federated learning through personalization." International Conference on Machine Learning. PMLR, 2021.

[2] Mugunthan, Vaikkunth, et al. "Fedltn: Federated learning for sparse and personalized lottery ticket networks." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

Reviewer comment (Theory):
The theoretical proof does not consider the data heterogeneity. Will the thresholds still converge under the data heterogeneity?

Author response:
**A4:**. We would like to note that we do consider the data heterogeneity in this paper, and we did not remove it from the theoretical analysis. However, the reason why the data heterogeneity does not appear explicitly in the proof is that we took a rather different approach compared to previous works [3] and [4]. In SpaFL, we do not have a global model and, thus, the clients only share thresholds and update them in the last epoch, i.e., e=E. The server generates global thresholds using the scheduled users’ gradients in the last epoch. Hence, in the proof, we showed that the sum of those gradients can be bounded based on the assumptions about the smoothness of the loss function and unbiasedness of the stochastic gradients of thresholds.  This approach eventually shows that global thresholds will converge. This result holds for data heterogeneity. 

[3] Karimireddy, Sai Praneeth, et al. "Scaffold: Stochastic controlled averaging for federated learning." International conference on machine learning. PMLR, 2020.

[4] Reddi, Sashank J., et al. "Adaptive Federated Optimization." International Conference on Learning Representations. 2020.

Reviewer comment (Experiments):
In experiments, E = 3 means local iterations = 3, or local epochs = 3? Local iteration = 3 seems to be too small. Could you find other references to support this setting? Because many FL works set this as epochs [1].  

[1] Communication-Efficient Learning of Deep Networks from Decentralized Data.  
[2] SCAFFOLD: Stochastic Controlled Averaging for Federated Learning.  
[3] Adaptive Federated Optimization.

Author response:
**A5:**  $E=3$ means that scheduled clients perform 3 local epochs using their datasets. We found that we used ‘iteration’ in Section 3. We will clarify this in the revised version. For this choice of local epochs, we followed the settings as in [2] and [5]

[5]: Isik, Berivan, et al. "Sparse Random Networks for Communication-Efficient Federated Learning." The Eleventh International Conference on Learning Representations. 2022.

Reviewer comment (Writing):
This study primarily concentrates on personalized federated learning, where no global model is trained. It would enhance clarity if the authors could differentiate between personalized federated learning (pFL) and federated learning (FL), as the paper references FL multiple times, which typically involves a global model.

Author response:
**A3:** We agree with the reviewer that we can improve the clarity of the manuscript by explicitly differentiating personalized federated learning (pFL) and federated learning (FL). We will revise the manuscript by mentioning that SpaFL considers a pFL scenario. This is a straightforward revision.

Reviewer comment (Reproducibility):
During the local training for parameters and thresholds, if the gradients are calculated for every weight and applied with the binary mask, then how does it help save the computation overhead? Or if the gradients are calculated w.r.t. the sparse weights, how is it achieved in practice?

Author response:
**A1:** During the backpropagation, for each layer, we calculate the gradients for the input activations and the gradients for the weights. In SpaFL, we are able to reduce the number of FLOPs to calculate the gradients for input activations by convolving sparse weight tensors with the gradients of the output activations. In particular, we sparsified the weight tensors before performing convolutions by detaching them from the computation graph. Below, we provide a detailed explanation about how we reduce the computation overhead of the backpropagation.

Consider a convolutional layer with an input tensor $𝑋 ∈ R^{𝑁×𝐶×𝑋×𝑌},$ parameter tensor $𝑊 ∈ R^{𝐹 ×𝐶×𝑅×𝑆}$, and output tensor $𝑂 ∈ R^{𝑁×𝐹 ×𝐻×𝑊}$. Here, the input tensor $X$ consists of $N$ number of samples, each of which has $X×Y$ dimension. The parameter tensor $W$ has $F$ filters of $C$ channels with kernel size $R × S$. Assume that $𝑊$ is pruned, and $𝜌$% of the parameters are active. The output tensor $O$ will have $F$ output channels with dimension $H × W$ for $N$ samples. We can calculate the number of FLOPs for the gradients of input activations by convolving the gradient of output activations with the parameter matrix $W$. Hence, it can be approximated to $𝜌 × 𝑁 × (𝐹 × 𝑅 × 𝑆) × 𝐶 × 𝑋 × 𝑌$. Since we used a sparse tensor $𝑊$ during the forward propagation, we can reduce the number of FLOPs for calculating the gradients of input activations by $𝜌$. Then, we can calculate the gradients of the weights by convolving the input activation with the output gradient. This can be approximately given by $(N × X × Y ) × F × C× R × S$. Since we did not sparsify the activations, there is no reduction of FLOPs in this calculation. Therefore, the total number of FLOPs for the backpropagation in SpaFL can be given by $(1 + 𝜌) × 𝑁 × 𝐹 × 𝑅 × 𝑆 × 𝐶 × 𝑋 × 𝑌$. Meanwhile, without sparsity, the number of FLOPs for the backpropagation will be $2 × 𝑁 × 𝐹 × 𝑅 × 𝑆 × 𝐶 × 𝑋 × 𝑌$. Hence, we can reduce the computing overhead by $\frac{1+𝜌}{2}$ in the backpropagation compared to a baseline that does not use sparse weights. As shown in Fig. 2 in the manuscript, model densities in SpaFL usually go below $10$% after 200 rounds. Hence, we can reduce the computation overhead in the backpropagation around $30$% compared to dense baselines during the training.

Reviewer comment (Experiments):
What happens if clients do not receive the server update due to unavailability? It is specifically important as the solution is designed for resource-constrained cross-device FL, where clients are only sometimes available.

Author response:
**A1:** We agree with the reviewer that clients may not always be able to receive global thresholds due to their constrained resources. To demonstrate that SpaFL is also compatible with intermittent availability, we did comparison between SpaFL and a baseline where only a small subset of clients can receive the server update. This baseline can capture the unavailability of some clients who cannot participate in the training at a given time. Below, we provide an empirical comparison between SpaFL and the baseline with the scheduling size of 10. 

| Algorithms | FMNIST | CIFAR-10 | CIFAR-100|
|:------:|:--------:|:-------:|:--------:|
|SpaFL|$90.31\pm0.35$ | $73.85\pm2.80$| $38.80\pm1.10$|
|SpaFL + unavailability|$90.24\pm0.08$| $73.45\pm3.19$| $38.65 \pm 0.43$|

For SpaFL, the communications costs are $1.0208$ Gbit, $2.4956$ Gbit, and  $0.7674$ Gbit, for FMNIST, CIFAR-10, and CIFAR-100, respectively. For the baseline, the communications costs are $0.1856$ Gbit, $0.4537$ Gbit, and  $0.1395$ Gbit, for FMNIST, CIFAR-10, and CIFAR-100, respectively. We can see that the performance loss of the baseline is very small. Moreover, the baseline used only $20$% of the communication costs used by SpaFL. Therefore, this result demonstrates the compatibility of SpaFL with a scenario where only a subset of clients is available and receives the server update.

Reviewer comment (Reproducibility):
How do non-participant clients update their model?

Author response:
**A2:** As demonstrated by response **A1**, intermittently non-participating clients can update their model when they become available and receive server update. If a client is not participating in the training at a given time, then we can proceed with the SpaFL algorithm with a subset of participating clients. However, if some clients do not participate in the training permanently, then they cannot update their model in SpaFL. However, we believe this is the case for most of FL algorithms and not unique to SpaFL.

Reviewer comment (Experiments):
Is there any global model available?

Author response:
**A3:** Since SpaFL personalizes clients' models, it does not have a global model. Instead, clients share global thresholds. Since global thresholds are aggregated parameter importance, clients can know which neuron/filter is globally important, thereby learning how to prune their model. 

However, SpaFL can also be extended to share some model parameters between clients. Clients can transmit their unpruned parameters to the server as done in [8,9,22,24,25,26] (references are from the manuscript) along with their thresholds. Then, the server can generate a global model by aggregating the received parameters and thresholds. Hence, we extend SpaFL to enable clients to share both global model and thresholds.

Reviewer comment (Experiments):
The author should include a comparison with prior works that adapt sparse learning in FL, such as [8,9,22,24,25,26] (references are from this paper). Some of these methods can reach high sparsities comparable to the 1% communication cost of SpaFL.

Author response:
**A4:** We agree with the reviewer that a comparison between SpaFL and works with sparse learning will be important to demonstrate the low communication and computing costs of SpaFL. Hence, we compare SpaFL with two sparse learning baselines [1, 2] as the reviewer suggested. The work in [1] adopted sparse learning in a personalized FL setting, and [2] trained a global sparse model across clients. Here, we would like to note that we already showed the performance of [2] in our manuscript. For [1], we fixed the density of models to 50% following their methodology. We used the same network architectures used in our manuscript for this experiment and set learning rates of [1] as 0.001, 0.01, and 0.1 for FMNIST, CIFAR-10, CIFAR-100, respectively. For CIFAR-100, we decayed the learning rate by 0.998 per each round. For other hyperparameters (e.g., batch size, local epoch $E$, scheduling size $K$), we used the same settings in our manuscript for each dataset.  Below, we provide an empirical comparison. 

| Algorithms | FMNIST | CIFAR-10 | CIFAR-100|
|:------:|:--------:|:-------:|:--------:|
|SpaFL|$\mathbf{90.31\pm0.35}$  | $\mathbf{73.85\pm2.80}$| $\mathbf{38.80\pm1.10}$|
|FedSpa [1]|$89.30\pm0.20$| $67.63\pm0.05$| $36.52 \pm 0.03$|
|FedDST [2] | $84.46\pm0.14$ | $60.18\pm0.03$ |$22.26 \pm 0.14$|

We also provide the amount of used computing and communications below. 'FLOP' is the number of used FLOPs and 'Comm' is the number of communicated bits between clients and a server during the training.

| Algorithms |       FMNIST (FLOP, Comm)         | CIFAR-10 (FLOP, Comm) | CIFAR-100 (FLOP, Comm)|
|:------:|:--------:|:-------:|:--------:|
|SpaFL|  $\mathbf{(2.72 \times10^{11}, 1.02 Gbit )} $| $\mathbf{(1.59\times10^{12}, 2.50 Gbit)}$ | $\mathbf{(2.84 \times10^{11}, 0.77 Gbit)}$ |
|FedSpa [1]|$(5.25\times10^{11}, 55.2 \text{Gbit})$| $(4.29 \times 10^{12}, 129  \text{Gbit}$| $(9.27 \times 10^{11},  10.2  \text{Gbit} )$|
|FedDST [2] | $(2.89\times10^{11}, 74.5 \text{Gbit})$ | $(3.44\times10^{12}, 139  \text{Gbit})$ |$(8.84\times10^{11}, 13.3  \text{Gbit})$|

Hence, we can see that SpaFL achieves better performance with less resources compared to both sparse learning baselines.

Reviewer comment (Reproducibility):
How does the server or clients control the density of the models.

Author response:
**A5:** In SpaFL, we control the density of models through the regularizer $\alpha$. As $\alpha$ increases, we enforce more sparsity into models. Unlike works in [8,9,22,24,25,26], which fix the density of models, SpaFL makes models as sparse as possible while balancing the performance and sparsity. This is a meaningful advantage because it enables clients to learn optimal sparse structures that can minimize the performance loss. We can see the improvements in performance and costs achieved by SpaFL in comparsion to the approaches in [8,9,22,24,25,26] from the response **A4**.

Reviewer comment (Theory):
Some aspects of the algorithm are not clearly explained. It is not clear why the regularizer in (4) is chosen over other options. It is also not clear why the second update of the client models in 3.2.4 is necessary because technically it should be possible to continue training the model with the new global thresholds as described in 3.2.2.

Author response:
**A1**:  The main reason for choosing the regularizer  $R=\exp(-||\tau||)$ in (4) is that the exponential function goes to zero asymptotically as thresholds increase. This is reasonable because it can penalize low thresholds without making them become extremely large. We can also use a similar regularizer that can give penalty for low thresholds without encouraging them to become a too large value. For example, a regularizer $R' = \frac{1}{||\tau|| + 0.1}$ satisfies such property. Below, we provide an empirical comparison between SpaFL and the baseline that uses $R'$ as below.
|Algorithm| FMNIST| CIFAR-10| CIFAR-100| 
|:-------------------:|:-------------------:|:------:|:------:|
|SpaFL|    $\boldsymbol{90.31\pm0.34}$    |  $\boldsymbol{73.85 \pm 2.80}$ | $\boldsymbol{38.80 \pm 1.10}$|
|Baseline| $89.92 \pm 0.25$| $73.62 \pm 3.42$ | $38.73 \pm 0.53$ | 

For the regularizer $R$, the achieved model densities are $5.36$%, $7.57$% and $7.38$% for FMNIST, CIFAR-10, and CIFAR-100, respectively. Meanwhile, with $R'$, the model densities are $4.85$%, $5.60$%, and $8.01$% for FMNIST, CIFAR-10, and CIFAR-100, respectively. We can see that both regularizers work well with SpaFL. Hence, SpaFL is compatible with any regularizers that can penalize the magnitude of thresholds asymptotically. 

**A2**: The main purpose of Section 3.2.4 is to update the model parameters by extracting the aggregated importance from global thresholds. By investigating the difference between two consecutive global thresholds and parameters, clients can deduce which parameters are globally important.  Although clients perform their training using the received global thresholds as described in Section 3.2.2, this additional update provides meaningful performance gains. We provide an empirical comparison between SpaFL and the baseline that does not use the update in 3.2.4 as below
|Algorithm| FMNIST| CIFAR-10| CIFAR-100| 
|:-------------------:|:-------------------:|:------:|:------:|
|SpaFL|    $\boldsymbol{90.31\pm0.34}$    |  $\boldsymbol{73.85 \pm 2.80}$ | $\boldsymbol{38.80 \pm 1.10}$|
|Baseline| $90.02 \pm 0.30$| $72.92 \pm 2.15$ | $38.06 \pm 0.80$ | 

We can see that the update in Section 3.2.4. can provide a clear improvement compared to the baseline by utilizing the aggregated parameter importance from global thresholds.

Reviewer comment (Evaluation):
The sparsification is unstructured and thus may not actually lead to reduction in computation cost or latency due to inefficient utilization of the hardware. Since there are already several works on sparse FL I believe it is important to now start considering hardware performance etc to truly differentiate from prior work.

Author response:
**A6**: Yes, SpaFL can be readily extended to perform structured pruning. Specifically, we can prune entire parameters connected to a filter/neuron.  In SpaFL, we defined a threshold for each filter/neuron and pruned each connected parameter if its magnitude is smaller than that of the threshold. This approach leads to unstructured sparsity.  We can endow SpaFL with structured sparsity by calculating the average magnitude of all parameters connected to their filter/neuron and pruning all of them if their average magnitude is smaller than their threshold. Hence, SpaFL can be extended to do filter/neuron-wise pruning. In fact, we have implemented SpaFL with structured sparsity following your comments, and you can see the performance in the following table. 

|Algorithm| FMNIST| CIFAR-10| CIFAR-100| 
|:-------:|:--------:|:------:|:------:|
|Structured SpaFL|    $\boldsymbol{89.71\pm0.44}$    |  $\boldsymbol{70.05 \pm 2.65}$ | $\boldsymbol{33.68 \pm 3.48}$|
|Sub-FedAvg| $89.29\pm0.69$ | $70.05 \pm 1.88$ | $31.05 \pm 1.12$|
|LotteryFL| $89.15 \pm 0.62$| $66.82 \pm 0.11$ | $28.90 \pm 0.22$|

The achieved model densities are 50.6%, 50%, and 49.3% for FMNIST, CIFAR-10, and CIFAR-100, respectively. Since it is well known that filter/neuron-wise pruning is less flexible than unstructured pruning, the results show accuracy loss compared to the original SpaFL. However, we can see that the structured SpaFL still outperforms unstructured baselines at the same model density 50%.

Reviewer comment (Experiments):
In addition to an intuitive justification for the regularizer in (4) and the second update in 3.2.4 can you also provide an empirical comparison with alternate regularizers?

Author response:
**A1**:  The main reason for choosing the regularizer  $R=\exp(-||\tau||)$ in (4) is that the exponential function goes to zero asymptotically as thresholds increase. This is reasonable because it can penalize low thresholds without making them become extremely large. We can also use a similar regularizer that can give penalty for low thresholds without encouraging them to become a too large value. For example, a regularizer $R' = \frac{1}{||\tau|| + 0.1}$ satisfies such property. Below, we provide an empirical comparison between SpaFL and the baseline that uses $R'$ as below.
|Algorithm| FMNIST| CIFAR-10| CIFAR-100| 
|:-------------------:|:-------------------:|:------:|:------:|
|SpaFL|    $\boldsymbol{90.31\pm0.34}$    |  $\boldsymbol{73.85 \pm 2.80}$ | $\boldsymbol{38.80 \pm 1.10}$|
|Baseline| $89.92 \pm 0.25$| $73.62 \pm 3.42$ | $38.73 \pm 0.53$ | 

For the regularizer $R$, the achieved model densities are $5.36$%, $7.57$% and $7.38$% for FMNIST, CIFAR-10, and CIFAR-100, respectively. Meanwhile, with $R'$, the model densities are $4.85$%, $5.60$%, and $8.01$% for FMNIST, CIFAR-10, and CIFAR-100, respectively. We can see that both regularizers work well with SpaFL. Hence, SpaFL is compatible with any regularizers that can penalize the magnitude of thresholds asymptotically.

Reviewer comment (Experiments):
Likewise can you also provide a comparison with a baseline which does not use the update in 3.2.4 but instead just directly continues training the model with the new global thresholds?

Author response:
**A2**: The main purpose of Section 3.2.4 is to update the model parameters by extracting the aggregated importance from global thresholds. By investigating the difference between two consecutive global thresholds and parameters, clients can deduce which parameters are globally important.  Although clients perform their training using the received global thresholds as described in Section 3.2.2, this additional update provides meaningful performance gains. We provide an empirical comparison between SpaFL and the baseline that does not use the update in 3.2.4 as below
|Algorithm| FMNIST| CIFAR-10| CIFAR-100| 
|:-------------------:|:-------------------:|:------:|:------:|
|SpaFL|    $\boldsymbol{90.31\pm0.34}$    |  $\boldsymbol{73.85 \pm 2.80}$ | $\boldsymbol{38.80 \pm 1.10}$|
|Baseline| $90.02 \pm 0.30$| $72.92 \pm 2.15$ | $38.06 \pm 0.80$ | 

We can see that the update in Section 3.2.4. can provide a clear improvement compared to the baseline by utilizing the aggregated parameter importance from global thresholds.

Reviewer comment (Theory):
Can you provide a derivation for (6) and (11)?

Author response:
**A3**:  To derive (6), we start from the gradient of the loss with respect to a threshold, as follows:

$$ \frac{\partial F\_k (\tilde{w}\_k^{E-1}(t),  \tau(t) )}{\partial \tau_i^l(t)} = \sum\_{j=1}^{n} \frac{ \partial \tilde{w}\_{k, ij}^{E-1} (t) } { \partial \tau\_i^l (t) }   \frac{\partial F\_k (\tilde{w}\_k^{E-1} (t),  \tau(t)) }{\partial \tilde{w}\_{k, ij}^{E-1} (t) }=  \sum\_{j=1}^{n\_{in}^l}  \frac{ \partial \tilde{w}\_{k, ij}^{E-1, l} (t) } { \partial Q\_{k, ij}^{E-1, l} (t) } \frac{ \partial Q\_{k, ij}^{E-1, l} (t) } { \partial \tau\_i^l (t) } g\_k( \tilde{w}\_k^{E-1}(t))\_{ij}^l $$
$$= \sum\_{j=1}^{n\_{in}^l} w\_{k, ij}^{E-1, l} (t) \frac{ \partial S(Q\_{k, ij}^{E-1, l} (t)) } { \partial Q\_{k, ij}^{E-1, l} (t) } (-1) g\_k( \tilde{w}\_k^{E-1}(t))\_{ij}^l = -\sum\_{j=1}^{n\_{in}^l} w\_{k, ij}^{E-1, l} (t) g\_k( \tilde{w}\_k^{E-1}(t))\_{ij}^l ,$$
where $Q\_{k, ij}^{E-1, l} (t)) = |w\_{k, ij}^{E-1, l} (t)|  -  \tau_i^l(t)$, $S(\cdot)$ is a unit step function, $\tilde{w} = w \cdot S(Q(w))$. The last equation results from the fact that we used the identity straight-through estimator and $w$ and $\tau$ is constrained to be within
$[−1, 1]$ and $[0, 1]$ as described in Section 3.2.1 and 3.2.2.

The derivation of eq. (11) is similar to the above. We start from the gradient of the loss with respect to a parameter $w$ as follows
$$ \frac{\partial F\_k (\tilde{w}\_k(t), \tau(t) )} {\partial w\_k(t)}  =  \frac{\partial \tilde{w}\_k(t)}{\partial w\_k(t)}  \frac{  F\_k (\tilde{w}\_k(t), \tau(t) ) } {\partial \tilde{w}\_k(t) }= \frac{\partial \tilde{w}\_k(t) }{ \partial Q\_k(t) } \frac{\partial Q\_k(t)}{\partial w\_k(t)} g\_k( \tilde{w}\_k(t) )$$
$$ = w\_k(t) \frac{\partial S(Q\_k(t))}{\partial Q\_k(t)} \frac{\partial |w\_k(t)|}{\partial w\_k(t)}  g\_k( \tilde{w}\_k(t) ) = |w\_k(t)| g\_k( \tilde{w}\_k(t) ).$$
This completes the derivation of eq. (11).

Reviewer comment (Presentation):
What is the model density of the baselines in Table 1? I do not see it presented in the table.

Author response:
**A5**: As described in Section 5.2, the target model density of the baselines that use pruning is set to 0.5 following the configurations therein.

Reviewer comment (Experiments):
Do you have any thoughts on how the sparsification approach described herein could be made structured and thus more hardware efficient?

Author response:
**A6**: Yes, SpaFL can be readily extended to perform structured pruning. Specifically, we can prune entire parameters connected to a filter/neuron.  In SpaFL, we defined a threshold for each filter/neuron and pruned each connected parameter if its magnitude is smaller than that of the threshold. This approach leads to unstructured sparsity.  We can endow SpaFL with structured sparsity by calculating the average magnitude of all parameters connected to their filter/neuron and pruning all of them if their average magnitude is smaller than their threshold. Hence, SpaFL can be extended to do filter/neuron-wise pruning. In fact, we have implemented SpaFL with structured sparsity following your comments, and you can see the performance in the following table. 

|Algorithm| FMNIST| CIFAR-10| CIFAR-100| 
|:-------:|:--------:|:------:|:------:|
|Structured SpaFL|    $\boldsymbol{89.71\pm0.44}$    |  $\boldsymbol{70.05 \pm 2.65}$ | $\boldsymbol{33.68 \pm 3.48}$|
|Sub-FedAvg| $89.29\pm0.69$ | $70.05 \pm 1.88$ | $31.05 \pm 1.12$|
|LotteryFL| $89.15 \pm 0.62$| $66.82 \pm 0.11$ | $28.90 \pm 0.22$|

The achieved model densities are 50.6%, 50%, and 49.3% for FMNIST, CIFAR-10, and CIFAR-100, respectively. Since it is well known that filter/neuron-wise pruning is less flexible than unstructured pruning, the results show accuracy loss compared to the original SpaFL. However, we can see that the structured SpaFL still outperforms unstructured baselines at the same model density 50%.