Original: Z ERO FL: E FFICIENT O N -D EVICE T RAINING FOR F EDERATED L EARNING WITH L OCAL S PARSITY
Revised: Z ERO FL: E FFICIENT O N -D EVICE T RAINING FOR F EDERATED L EARNING WITH L OCAL S PARSITY

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: When the available hardware cannot meet the memory and compute requirements to efﬁciently train high performing machine learning models, a compromise in either the training quality or the model complexity is needed.
Revised: When the available hardware cannot meet the memory and compute requirements to efﬁciently train high performing machine learning models, a compromise in either the training quality or the model complexity is needed.

Original: In Federated Learning (FL), nodes are orders of magnitude more constrained than traditional servergrade hardware and are often battery powered, severely limiting the sophistication of models that can be trained under this paradigm.
Revised: In Federated Learning (FL), nodes are orders of magnitude more constrained than traditional servergrade hardware and are often battery powered, severely limiting the sophistication of models that can be trained under this paradigm.

Original: While most research has focused on designing better aggregation strategies to improve convergence rates and in alleviating the communication costs of FL, fewer efforts have been devoted to accelerating on-device training.
Revised: While most research has focused on designing better aggregation strategies to improve convergence rates and in alleviating the communication costs of FL, fewer efforts have been devoted to accelerating on-device training.

Original: Such stage, which repeats hundreds of times (i.e. every round) and can involve thousands of devices, accounts for the majority of the time required to train federated models and, the totality of the energy consumption at the client side.
Revised: Such stage, which repeats hundreds of times (i.e. every round) and can involve thousands of devices, accounts for the majority of the time required to train federated models and, the totality of the energy consumption at the client side.

Original: In this work, we present the ﬁrst study on the unique aspects that arise when introducing sparsity at training time in FL workloads.
Revised: In this work, we present the ﬁrst study on the unique aspects that arise when introducing sparsity at training time in FL workloads.

Original: We then propose ZeroFL , a framework that relies on highly sparse operations to accelerate on-device training.
Revised: We then propose ZeroFL , a framework that relies on highly sparse operations to accelerate on-device training.

Original: Models trained with ZeroFL and 95% sparsity achieve up to 3% higher accuracy compared to competitive baselines obtained from adapting a state-of-the-art sparse training framework to the FL setting.
Revised: Models trained with ZeroFL and 95% sparsity achieve up to 2.3% higher accuracy compared to competitive baselines obtained from adapting a state-of-the-art sparse training framework to the FL setting.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Despite it being a relatively new subﬁeld of machine learning (ML), Federated Learning (FL) (McMahan et al., 2017; Reddi et al., 2021; Horvath et al., 2021) has become an indispensable tool to enable privacy-preserving collaboratively learning, as well as to deliver personalised models tailored to the end-user’s local data and context (Arivazhagan et al., 2019; Hilmkil et al., 2021; Cheng et al., 2021).
Revised: Despite it being a relatively new subﬁeld of machine learning (ML), Federated Learning (FL) (McMahan et al., 2017; Reddi et al., 2021; Horvath et al., 2021) has become an indispensable tool to enable privacy-preserving collaboratively learning, as well as to deliver personalised models tailored to the end-user’s local data and context (Arivazhagan et al., 2019; Hilmkil et al., 2021; Cheng et al., 2021).

Original: For example: next-word prediction (Hard et al., 2018), physical activity detection (Doherty et al., 2017), keyword spotting (Hard et al., 2020), among others.
Revised: For example: next-word prediction (Hard et al., 2018), physical activity detection (Doherty et al., 2017), keyword spotting (Hard et al., 2020), among others.

Original: Unlike standard centralised training, which normally takes place on the Cloud and makes use of powerful hardware (Hazelwood et al., 2018), FL is envisioned to run on commodity devices such as smartphones or IoT devices often running of batteries, which are orders of magnitude more restricted in terms of compute, memory and power consumption.
Revised: Unlike standard centralised training, which normally takes place on the Cloud and makes use of powerful hardware (Hazelwood et al., 2018), FL is envisioned to run on commodity devices such as smartphones or IoT devices often running of batteries, which are orders of magnitude more restricted in terms of compute, memory and power consumption (Qiu et al., 2021).

Original: This triplet of factors drastically limits the complexity of the ML models that can be trained on-device in a federated manner, ceiling their usefulness for the aforementioned applications as a result.
Revised: This triplet of factors drastically limits the complexity of the ML models that can be trained on-device in a federated manner, ceiling their usefulness for the aforementioned applications as a result.

Original: In order to adjust the memory and compute footprints of complex ML model to the FL setting, the research community has presented a number of approaches including: the use of distillation (Hinton et al., 2015) to enable the aggregation on the server side of heterogeneous model architectures (e.g. based on the compute capabilities of each device) that collaboratively train a single global model (Lin et al., 2020; Zhu et al., 2021); group knowledge transfer algorithm
Revised: In order to adjust the memory and compute footprints of complex ML model to the FL setting, the research community has presented a number of approaches including: the use of distillation (Hinton et al., 2015) to enable the aggregation on the server side of heterogeneous model architectures (e.g. based on the compute capabilities of each device) that collaboratively train a single global model (Lin et al., 2020; Zhu et al., 2021); group knowledge transfer algorithm

Original: (He et al., 2020); federated dropout, by which clients perform local training on a sub-model of the global model (Caldas et al., 2019), translates into lower overall communication costs and, enables better support for heterogeneous pools of clients regardless of their compute capabilities (Horvath et al., 2021); and, more generally, better aggregation strategies that enable faster convergence (Li et al., 2018;Wang et al., 2020; Reddi et al., 2021), reducing in this way overall device utilization (e.g. fewer local epochs) and number of communication rounds.
Revised: (He et al., 2020); federated dropout, by which clients perform local training on a sub-model of the global model (Caldas et al., 2019), translates into lower overall communication costs and, enables better support for heterogeneous pools of clients regardless of their compute capabilities (Horvath et al., 2021); and, more generally, better aggregation strategies that enable faster convergence (Li et al., 2018; Reddi et al., 2021), reducing in this way overall device utilization (e.g. fewer local epochs) and number of communication rounds.

Original: Other optimization techniques such as quantization and sparsity have been used in the context of
Revised: Other optimization techniques such as quantization and sparsity have been used in the context of

Original: FL but mostly as a way to reduce communication costs (Liu et al., 2021; Amiri et al., 2020; Shahid et al., 2021) but not to accelerate on-device training.
Revised: FL but mostly as a way to reduce communication costs (Liu et al., 2021; Amiri et al., 2020; Shahid et al., 2021) but not to accelerate on-device training.

Original: The use of sparse operations (e.g. convolutions) at training time has recently been shown to be an effective technique to accelerate training in centralised settings (Sun et al., 2017; Goli & Aamodt, 2020; Raihan & Aamodt, 2020).
Revised: The use of sparse operations (e.g. convolutions) at training time has recently been shown to be an effective technique to accelerate training in centralised settings (Sun et al., 2017; Goli & Aamodt, 2020; Raihan & Aamodt, 2020).

Original: The resulting models are as good or close to their densely-trained counterparts despite reducing by up to 90% their FLOPs budget and, resulting in an overall up to 3 .
Revised: The resulting models are as good or close to their densely-trained counterparts despite reducing by up to 90% their FLOPs budget and, resulting in an overall up to 3 .

Original: 3 × training speedup.
Revised: 3 × training speedup.

Original: Acceleration is achieved by performing sparse convolutions during the forward and/or backward pass, which requires at least one of the operands (i.e. inputs, weights, gradients) to be sufﬁciently sparse and, software and hardware support for such operations.
Revised: Acceleration is achieved by performing sparse convolutions during the forward and/or backward pass, which requires at least one of the operands (i.e. inputs, weights, gradients) to be sufﬁciently sparse and, software and hardware support for such operations.

Original: However, it is unclear how the different FL-speciﬁc challenges (i.e. data imbalance, stateless clients, periodic aggregation) will restrict the quality of the global model.
Revised: However, it is unclear how the different FL-speciﬁc challenges (i.e. data imbalance, stateless clients, periodic aggregation) will restrict the quality of the global model.

Original: This work considers the challenges and opportunities of inducing high levels of sparsity to accelerate training on-device for FL workloads, and provides the following contributions:
Revised: This work considers the challenges and opportunities of inducing high levels of sparsity to accelerate training on-device for FL workloads, and provides the following contributions:

Original: • The ﬁrst framework for Federated Learning that leverages sparsity as a mechanism to accelerate on-device training by inducing up to 95% sparse weights and activations.
Revised: • The ﬁrst framework for Federated Learning that leverages sparsity as a mechanism to accelerate on-device training by inducing up to 95% sparse weights and activations.

Original: This work considers three popular datasets: CIFAR-10 and FEMNIST for image classiﬁcation and, SpeechCommands for audio classiﬁcation.
Revised: This work considers three popular datasets: CIFAR-10 and FEMNIST for image classiﬁcation and, SpeechCommands for audio classiﬁcation.

Original: • A study on the unique aspects that arise when introducing sparsity at training time in FL: the degree of overlap between non-zero values decreases with layer-depth index and, the locations of zero-valued weights in the global model remain constant throughout most of the training rounds.
Revised: • A study on the unique aspects that arise when introducing sparsity at training time in FL: the degree of overlap between non-zero values decreases with layer-depth index and, the locations of zero-valued weights in the global model remain constant throughout most of the training rounds.

Original: Our discussion sets the foundations for future research in this area.
Revised: Our discussion sets the foundations for future research in this area.

Original: • A technique that alleviates the accuracy degradation when applying a state-of-the-art offthe-shelf sparsiﬁcation method to the FL domain.
Revised: • A technique that alleviates the accuracy degradation when applying a state-of-the-art offthe-shelf sparsiﬁcation method to the FL domain.

Original: ZeroFL achieves +2.1% and +3.7% higher accuracy than baselines when inducing 90% and 95% sparsity respectively.
Revised: ZeroFL achieves +2.3% and +1.5% higher accuracy than baselines when inducing 90% and 95% sparsity respectively.

Original: In addition, ZeroFL also leverages sparsity when transferring the local models to the central server reducing communication costs by 3 . 

Revised: In addition, ZeroFL also leverages sparsity when transferring the local models to the central server reducing communication costs by 3 .

Original: × while still outperforming competitive baselines.
Revised: × while still outperforming competitive baselines.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Pruning neural networks involves discarding parts of the model (e.g. individual weights or entire channels) that are irrelevant for solving the task at hand.
Revised: Pruning neural networks involves discarding parts of the model (e.g. individual weights or entire channels) that are irrelevant for solving the task at hand.

Original: This procedure generally produces a lightweight model representation more suitable for deployment on constrained devices with limited memory and compute budgets.
Revised: This procedure generally produces a lightweight model representation more suitable for deployment on constrained devices with limited memory and compute budgets.

Original: In this section we detail how different forms of pruning or sparsiﬁcation have been used to accelerate inference and, to a lesser extent, training.
Revised: In this section we detail how different forms of pruning or sparsiﬁcation have been used to accelerate inference and, to a lesser extent, training.

Original: We also discuss how these have been introduced to reduce communication costs in distributed and federated learning.
Revised: We also discuss how these have been introduced to reduce communication costs in distributed and federated learning.

Original: Unstructured pruning.
Revised: Unstructured pruning.

Original: Frameworks relying on unstructured pruning (Han et al.
Revised: Frameworks relying on unstructured pruning (Han et al.

Original: , 2015a;b; Guo et al., 2016; Molchanov et al., 2017) often achieve higher compression ratios at the expense of inference stages being as compute intensive in practice as those of the original model.
Revised: , 2015a;b; Guo et al., 2016; Molchanov et al., 2017) often achieve higher compression ratios at the expense of inference stages being as compute intensive in practice as those of the original model.

Original: This is because, assuming pruning has been homogeneously applied on the model, sparse operations can only be efﬁciently accelerated on supported hardware, such as modern GPUs (Wang, 2020;
Revised: This is because, assuming pruning has been homogeneously applied on the model, sparse operations can only be efﬁciently accelerated on supported hardware, such as modern GPUs (Wang, 2020;

Original: Zachariadis et al., 2020; Hong et al., 2018) or custom accelerators (Zhang et al., 2016;
Revised: Zachariadis et al., 2020; Hong et al., 2018) or custom accelerators (Zhang et al., 2016;

Original: Lu et al., 2019; Srivastava et al., 2020), for a sufﬁciently high sparsity ratio.
Revised: Lu et al., 2019; Srivastava et al., 2020), for a sufﬁciently high sparsity ratio.

Original: The lower the ratio, the less likely sparse operations would translate into measurable speedups.
Revised: The lower the ratio, the less likely sparse operations would translate into measurable speedups.

Original: In the case of CPUs, speedups due to sparse operations where one operand is unstructurally sparse are often only feasible at 90% sparsity ratios or higher (Hong et al., 2019; Wang, 2021).
Revised: In the case of CPUs, speedups due to sparse operations where one operand is unstructurally sparse are often only feasible at 90% sparsity ratios or higher (Hong et al., 2019; Wang, 2021).

Original: Structured pruning.
Revised: Structured pruning.

Original: Methods that apply structured pruning (He et al., 2018; 2017; Jian-Hao Luo &
Revised: Methods that apply structured pruning (He et al., 2018; 2017; Jian-Hao Luo &

Original: Lin, 2017; Yu et al., 2018; Molchanov et al., 2019; Wang et al., 2017), on the other hand, trade compression for acceleration potential.
Revised: Lin, 2017; Yu et al., 2018; Molchanov et al., 2019; Wang et al., 2017), on the other hand, trade compression for acceleration potential.

Original: These approaches modify the underlying computational graph by discarding entire channels, resulting in smaller but still dense convolution operations, or by removing the nodes all together if an entire layer is set to be removed by the chosen pruning strategy.
Revised: These approaches modify the underlying computational graph by discarding entire channels, resulting in smaller but still dense convolution operations, or by removing the nodes all together if an entire layer is set to be removed by the chosen pruning strategy.

Original: As a result, structured pruning frameworks are the preferred option when aiming to accelerate inference on general purpose hardware.
Revised: As a result, structured pruning frameworks are the preferred option when aiming to accelerate inference on general purpose hardware.

Original: A body of work across structured and unstructured pruning methods, attempts to induce structure in otherwise randomly sparse networks S. Gray & Kingma (2017); Ren et al.
Revised: A body of work across structured and unstructured pruning methods, attempts to induce structure in otherwise randomly sparse networks S. Gray & Kingma (2017); Ren et al.

Original: Wen et al.
Revised: Wen et al.

Original: ; Verelst & Tuytelaars (2020).
Revised: ; Verelst & Tuytelaars (2020).

Original: This is often referred to as block sparsity and consists in subdividing the matrix representations of inputs or weights into tiles (e.g. 16 × 16 tiles), and restrict the training in such a way that some tiles contain only zeros while the rest remain dense and real-valued.
Revised: This is often referred to as block sparsity and consists in subdividing the matrix representations of inputs or weights into tiles (e.g. 16 × 16 tiles), and restrict the training in such a way that some tiles contain only zeros while the rest remain dense and real-valued.

Original: Matrix-matrix multiplications following such a pattern can be accelerated at lower global sparsity ratios compared to those following unstructured sparsity Hoeﬂer et al.
Revised: Matrix-matrix multiplications following such a pattern can be accelerated at lower global sparsity ratios compared to those following unstructured sparsity Hoeﬂer et al.

Original: Other forms of constraining how sparsity occurs have been proposed, for example, a cache-aware reordering on the sparsity pattern of the weights Elsen et al.
Revised: Other forms of constraining how sparsity occurs have been proposed, for example, a cache-aware reordering on the sparsity pattern of the weights Elsen et al.

Original: This can be used to ensure high cache reuse on Cortex-A mobile CPUs, resulting in 2 .
Revised: This can be used to ensure high cache reuse on Cortex-A mobile CPUs, resulting in 2 .

Original: 4 × acceleration of MobileNets.
Revised: 4 × acceleration of MobileNets.

Original: Sparse training.
Revised: Sparse training.

Original: The majority of works making use of sparsity are envisioned for either model compression or to accelerate inference.
Revised: The majority of works making use of sparsity are envisioned for either model compression or to accelerate inference.

Original: Only recently, sparse operations have been considered to accelerate training.
Revised: Only recently, sparse operations have been considered to accelerate training.

Original: The work of Sun et al.
Revised: The work of Sun et al.

Original: (2017) presented a mechanism to induce high levels of sparsity in the gradients during backpropagation and, demonstrated large speedups when training
Revised: (2017) presented a mechanism to induce high levels of sparsity in the gradients during backpropagation and, demonstrated large speedups when training

Original: MLP-only models.
Revised: MLP-only models.

Original: More recently, Goli & Aamodt (2020) build upon the observation that gradients from consecutive batches are near identical.
Revised: More recently, Goli & Aamodt (2020) build upon the observation that gradients from consecutive batches are near identical.

Original: They present a framework to reuse a random sample of previously computed gradients and their thresholded difference w.r.t gradients from the current batch, resulting in a sparse tensor.
Revised: They present a framework to reuse a random sample of previously computed gradients and their thresholded difference w.r.t gradients from the current batch, resulting in a sparse tensor.

Original: Their framework accelerates training of CNNs by performing sparse convolutions during the backward pass at the cost of pre-computing partial gradients during forward pass.
Revised: Their framework accelerates training of CNNs by performing sparse convolutions during the backward pass at the cost of pre-computing partial gradients during forward pass.

Original: Closer to our work is SWAT (Raihan & Aamodt, 2020), a framework that relies on sparsiﬁed weights during inference and sparsiﬁed weights and activations for backward propagation.
Revised: Closer to our work is SWAT (Raihan & Aamodt, 2020), a framework that relies on sparsiﬁed weights during inference and sparsiﬁed weights and activations for backward propagation.

Original: Compared to the previous two frameworks, SWAT achieves superior performance in terms of training acceleration and accuracy retention at high sparsity ratios.

Original: Compression on communication.
Revised: Compression on communication.

Original: Kone ˇ cn ` y et al.
Revised: Kone ˇ cn ` y et al.

Original: (2016) proposed to restricts the updates of weight matrices to have a pre-speciﬁed structure in order to reduce the total communication cost.
Revised: (2016) proposed to restricts the updates of weight matrices to have a pre-speciﬁed structure in order to reduce the total communication cost.

Original: The structure can either be random or low-rank structure.
Revised: The structure can either be random or low-rank structure.

Original: ATOMO (Wang et al., 2018) introduced a generalised gradient decomposition and sparsiﬁcation technique, aiming to reduce the gradient sizes communicated upstream.
Revised: ATOMO (Wang et al., 2018) introduced a generalised gradient decomposition and sparsiﬁcation technique, aiming to reduce the gradient sizes communicated upstream.

Original: Han et al.
Revised: Han et al.

Original: (2020) proposed a different way of aggregation in the server, which instead of aggregating model weights, it aggregates the sparsiﬁed gradients after every local update step.
Revised: (2020) proposed a different way of aggregation in the server, which instead of aggregating model weights, it aggregates the sparsiﬁed gradients after every local update step.

Original: However, since the method requires to aggregate sparsiﬁed gradient after every step, it cannot beneﬁt from multiple local updates.
Revised: However, since the method requires to aggregate sparsiﬁed gradient after every step, it cannot beneﬁt from multiple local updates.

Original: Hence it might require extra communication rounds to reach the target performance.
Revised: Hence it might require extra communication rounds to reach the target performance.

Original: PruneFL
Revised: PruneFL

Original: Jiang et al.
Revised: Jiang et al.

Original: (2019) reduced both computation and communication overhead to minimize the overall training time by including an initial pruning at one selected client and further pruning as a part of FL process.
Revised: (2019) reduced both computation and communication overhead to minimize the overall training time by including an initial pruning at one selected client and further pruning as a part of FL process.

Original: Nevertheless, none of the aforementioned works explored the challenges of extending state-of-theart sparsiﬁcation methods to federated learning as a way to accelerate on-device training.
Revised: Nevertheless, none of the aforementioned works explored the challenges of extending state-of-theart sparsiﬁcation methods to federated learning as a way to accelerate on-device training.

Original: With
Revised: With

Original: ZeroFL, a framework speciﬁcally tailored to the FL setting, achieves better accuracy retention than with existing methods that remain exclusive to the centralised training paradigm.
Revised: ZeroFL, a framework speciﬁcally tailored to the FL setting, achieves better accuracy retention than with existing methods that remain exclusive to the centralised training paradigm.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: This section describes the state-of-the-art sparse training method SWAT (Raihan & Aamodt, 2020); the way we adapt it to the FL contexts; and the related challenges that would need be addressed.
Revised: This section describes the state-of-the-art sparse training method SWAT (Raihan & Aamodt, 2020); the way we adapt it to the FL contexts; and the related challenges that would need be addressed.

Original: 3. S PARSE W EIGHTS AND A CTIVATIONS T RAINING
Revised: 3. S PARSE W EIGHTS AND A CTIVATIONS T RAINING

Original: The SWAT framework embodies two strategies in the training process.
Revised: The SWAT framework embodies two strategies in the training process.

Original: During each forward pass, the weights are partitioned into active weights and non-active weights by a top-K (in magnitude) operator and only the active weights are used.
Revised: During each forward pass, the weights are partitioned into active weights and non-active weights by a top-K (in magnitude) operator and only the active weights are used.

Original: For the l th layer in the model, the layer maps the input activations a l − 1 onto feature maps
Revised: For the l th layer in the model, the layer maps the input activations a l − 1 onto feature maps

Original: o l using function f l : o l = f l ( a l − 1 , w l ) .
Revised: o l using function f l : o l = f l ( a l − 1 , w l ) .

Original: In this work we consider f l being the 3 × 3 convolution in the l -th layer.
Revised: In this work we consider f l being the 3 × 3 convolution in the l -th layer.

Original: In the backward pass, the gradient of input activations ( (cid:53) a l − 1 ) and the gradient of weights ( (cid:53) w l ) are calculated represented by functions G l and H l , as shown below:
Revised: In the backward pass, the gradient of input activations ( (cid:53) a l − 1 ) and the gradient of weights ( (cid:53) w l ) are calculated represented by functions G l and H l , as shown below:

Original: [Equation]
Revised: [Equation]

Original: Then in the backward pass, the retained layer inputs a l − 1 are also partitioned into active and nonactive by using the same top-K procedure.
Revised: Then in the backward pass, the retained layer inputs a l − 1 are also partitioned into active and nonactive by using the same top-K procedure.

Original: This results in full gradients and active weights being used in Eq.
Revised: This results in full gradients and active weights being used in Eq.

Original: 1, while full gradients and active activations are used in Eq.
Revised: 1, while full gradients and active activations are used in Eq.

Original: It is worth noticing that even weights and activations and sparsiﬁed in the forward and backward pass, the gradients generated through the training process are dense.
Revised: It is worth noticing that even weights and activations and sparsiﬁed in the forward and backward pass, the gradients generated through the training process are dense.

Original: Therefore, the resulting model is a dense.
Revised: Therefore, the resulting model is a dense.

Original: The compute cost of updating weights w l given a dense (cid:53) w l tensor is negligible compare to the savings due to performing the underlying convolutions in Eq.1&2, as this is essentially a weighted sum.
Revised: The compute cost of updating weights w l given a dense (cid:53) w l tensor is negligible compare to the savings due to performing the underlying convolutions in Eq.1&2, as this is essentially a weighted sum.

Original: 3. F ROM C ENTRALISED TO F EDERATED S PARSE T RAINING
Revised: 3. F ROM C ENTRALISED TO F EDERATED S PARSE T RAINING

Original: A direct adaptation of the SWAT framework to the FL setting could be done by framing each local training stage on a client as an instance of centralised training.
Revised: A direct adaptation of the SWAT framework to the FL setting could be done by framing each local training stage on a client as an instance of centralised training.

Original: However, one major difference between centralized training and FL is the notion of client statefulness .
Revised: However, one major difference between centralized training and FL is the notion of client statefulness .

Original: In a centralised scenario, each example in the training set is seen multiple times, once per epoch, allowing for the model to converge to a stable distribution of weights.
Revised: In a centralised scenario, each example in the training set is seen multiple times, once per epoch, allowing for the model to converge to a stable distribution of weights.

Original: This scenario is more suitable for sparsiﬁcation.
Revised: This scenario is more suitable for sparsiﬁcation.

Original: On the other hand, in a typical cross-device scenario, client’s availability is low and new data points are continuously being presented to the system as new clients participate in training rounds.
Revised: On the other hand, in a typical cross-device scenario, client’s availability is low and new data points are continuously being presented to the system as new clients participate in training rounds.

Original: This means that clients are likely to participate only once.
Revised: This means that clients are likely to participate only once.

Original: Such training behaviour inevitably leads to distributions of weights that change over time, making the application of sparsity inducing methods more difﬁcult as a result.
Revised: Such training behaviour inevitably leads to distributions of weights that change over time, making the application of sparsity inducing methods more difﬁcult as a result.

Original:  S PARSE T RAINING FOR F EDERATED L EARNING
Revised:  S PARSE T RAINING FOR F EDERATED L EARNING

Original: As a ﬁrst step, we conduct preliminaries investigations and measure SWAT’s performance when directly applied to FL without any adaptation.
Revised: As a ﬁrst step, we conduct preliminaries investigations and measure SWAT’s performance when directly applied to FL without any adaptation.

Original: This section describes the experimental protocol (Section 4.1), the obtained baseline results (Section 4.2) and a sparsiﬁcation effect analysis to highlight the weaknesses of this approach in Section 4.3.
Revised: This section describes the experimental protocol (Section 4.1), the obtained baseline results (Section 4.2) and a sparsiﬁcation effect analysis to highlight the weaknesses of this approach in Section 4.3.

Original: 4. E XPERIMENTAL S ETUP
Revised: 4. E XPERIMENTAL S ETUP

Original: While SWAT is used across the experiments as the standard sparsiﬁcation methodology, results also depend on various FL-speciﬁc hyper-parameters.
Revised: While SWAT is used across the experiments as the standard sparsiﬁcation methodology, results also depend on various FL-speciﬁc hyper-parameters.

Original: Federated learning is simulated with the Virtual
Revised: Federated learning is simulated with the Virtual

Original: Client Engine (VCE) of the Flower toolkit (Beutel et al., 2020) enabling us to scale to a large number of clients within in a single machine.
Revised: Client Engine (VCE) of the Flower toolkit (Beutel et al., 2020) enabling us to scale to a large number of clients within in a single machine.

Original: Datasets and hyper-parameters are detailed below.
Revised: Datasets and hyper-parameters are detailed below.

Original: Datasets.
Revised: Datasets.

Original: Experiments are conducted on two image classiﬁcation tasks of different complexity both in terms of the number of samples and classes: FEMNIST (Caldas et al., 2018) and CIFAR10 (Krizhevsky et al., 2009).
Revised: Experiments are conducted on two image classiﬁcation tasks of different complexity both in terms of the number of samples and classes: FEMNIST (Caldas et al., 2018) and CIFAR10 (Krizhevsky et al., 2009).

Original: FEMNIST is constructed by partitioning the data of the Extended
Revised: FEMNIST is constructed by partitioning the data of the Extended

Original: MNIST (Cohen et al., 2017) based on the writers of the digit-character.
Revised: MNIST (Cohen et al., 2017) based on the writers of the digit-character.

Original: We also include the Speech
Revised: We also include the Speech

Original: Commands dataset (Warden, 2018), where the task is to classify 1-second long audio clips.
Revised: Commands dataset (Warden, 2018), where the task is to classify 1-second long audio clips.

Original: Further details for these datasets can be found in the Appendix A.5.
Revised: Further details for these datasets can be found in the Appendix A.5.

Original: Data partitioning.
Revised: Data partitioning.

Original: We follow the latent Dirichlet allocation (LDA) partition method (Reddi et al., 2021; Yurochkin et al., 2019; Hsu et al., 2019) ensuring that each client is allocated the same number of training samples.
Revised: We follow the latent Dirichlet allocation (LDA) partition method (Reddi et al., 2021; Yurochkin et al., 2019; Hsu et al., 2019) ensuring that each client is allocated the same number of training samples.

Original: The level of heterogeneity is governed by the parameter α .
Revised: The level of heterogeneity is governed by the parameter α .

Original: As α → ∞ , partitions become more uniform (IID), and as α → 0 , partitions tend to be more heterogeneous.
Revised: As α → ∞ , partitions become more uniform (IID), and as α → 0 , partitions tend to be more heterogeneous.

Original: Our experimental evaluation considers α = 1 . 

Revised: Our experimental evaluation considers α = 1 .

Original: and α = 1000 .
Revised: and α = 1000 .

Original: Model Architecture.
Revised: Model Architecture.

Original: Following the convention for CIFAR-10, a ResNet-18 (He et al., 2016) architecture is instantiated in the client side and aggregated on the server.
Revised: Following the convention for CIFAR-10, a ResNet-18 (He et al., 2016) architecture is instantiated in the client side and aggregated on the server.

Original: We also make use of ResNetfor SpeechCommands.
Revised: We also make use of ResNetfor SpeechCommands.

Original: For FEMNIST, we employ the much smaller CNN ﬁrst proposed in (Caldas et al., 2018).
Revised: For FEMNIST, we employ the much smaller CNN ﬁrst proposed in (Caldas et al., 2018).

Original: Further details for these architectures can be found in Section A.6 in the Appendix.
Revised: Further details for these architectures can be found in Section A.6 in the Appendix.

Original: The models are trained with SGD, and all experiments imply one local epoch ( i.e. client epoch).
Revised: The models are trained with SGD, and all experiments imply one local epoch ( i.e. client epoch).

Original: An exponential decay deﬁned as η t = η start exp( tT log( η start /η end )) with η start , η end the starting and last learning rates respectively is applied at training time.
Revised: An exponential decay deﬁned as η t = η start exp( tT log( η start /η end )) with η start , η end the starting and last learning rates respectively is applied at training time.

Original: T represents the total number of FL rounds.
Revised: T represents the total number of FL rounds.

Original: Client partitioning.
Revised: Client partitioning.

Original: Following previous works, we propose to compose a pool of 100 clients with 10 active clients training concurrently in a given round (McMahan et al., 2017).
Revised: Following previous works, we propose to compose a pool of 100 clients with 10 active clients training concurrently in a given round (McMahan et al., 2017).

Original: We do this for all experiments except for FEMNIST, which comes pre-partitioned into 3597 clients.
Revised: We do this for all experiments except for FEMNIST, which comes pre-partitioned into 3597 clients.

Original: For this dataset we consider the setting of sampling 35 clients per round as in Caldas et al.
Revised: For this dataset we consider the setting of sampling 35 clients per round as in Caldas et al.

Original: Sparsity Ratios.
Revised: Sparsity Ratios.

Original: This work considers accelerating the convolutions involved during forward and backward propagation following a Top-K sparsity inducing mechanism at the weight level.
Revised: This work considers accelerating the convolutions involved during forward and backward propagation following a Top-K sparsity inducing mechanism at the weight level.

Original: As a  result, the expected sparse patternwould be unstructured, which can only be accelerated if tensors are sufﬁciently sparse.
Revised: As a [Figure] result, the expected sparse pattern unstructured, which can only be accelerated if tensors are sufﬁciently sparse.

Original: While sufﬁcient is mostly hardware-speciﬁc, for the target platforms often considered in FL (e.g. mobile CPUs and GPUs) we set a minimum sparsity ratio ( sp ) of 90%, above which acceleration can be achieved (Wang, 2021).
Revised: While sufﬁcient is mostly hardware-speciﬁc, for the target platforms often considered in FL (e [Figure] g. mobile CPUs and GPUs) we set a minimum sparsity ratio ( sp ) of 90%, above which acceleration can be achieved (Wang, 2021).

Original: We include in our study sp ∈ [0 . 7 , 0 . 9 , 0 .
Revised: We include in our study sp ∈ [0 . 7 , 0 . 9 , 0 .

Original: 95] in our initial evaluation.
Revised: 95] in our initial evaluation.

Original: We expect minimal accuracy drop for both IID and non-IID at 0.7 sparsity.
Revised: We expect minimal accuracy drop for both IID and non-IID at 0.7 sparsity.

Original: [Figure]

Original: 4. B ASELINES R ESULTS
Revised: 4. B ASELINES R ESULTS

Original: We begin by studying the effect of applying SWAT in both centralised learning and FL with the
Revised: We begin by studying the effect of applying SWAT in both centralised learning and FL with the

Original: CIFAR10 dataset, and the results are given in Fig.
Revised: CIFAR10 dataset, and the results are given in Fig.

Original: Applying SWAT in centralised training does not impact the validation accuracy much despite the high level of sparsity level, which is equivalent to the results from the original paper (Raihan & Aamodt, 2020) with a validation accuracy reaching 91.21% with a sparsity level of 95% against 93.32% for 70%.
Revised: Applying SWAT in centralised training does not impact the validation accuracy much despite the high level of sparsity level, which is equivalent to the results from the original paper (Raihan & Aamodt, 2020) with a validation accuracy reaching 91.21% with a sparsity level of 95% against 93.32% for 70%.

Original: We found sparse FL to be particularly sensitive to the learning rate and its scheduler.
Revised: We found sparse FL to be particularly sensitive to the learning rate and its scheduler.

Original: In particular, exponential decay annealing is crucial to reach relatively good performance.
Revised: In particular, exponential decay annealing is crucial to reach relatively good performance.

Original: It is also clear from the curves that a higher learning rate of 0 .
Revised: It is also clear from the curves that a higher learning rate of 0 .

Original: 2 reaches better accuracies in general than 0 .
Revised: 2 reaches better accuracies in general than 0 .

Original: As expected, however, FL offers lower levels of performance across all setups compared to centralised training.
Revised: As expected, however, FL offers lower levels of performance across all setups compared to centralised training.

Original: In addition, and conversely to centralised training, plots show consistent drops in the validation accuracy with the increase of the sparsity level.
Revised: In addition, and conversely to centralised training, plots show consistent drops in the validation accuracy with the increase of the sparsity level.

Original: It is worth noticing that the validation accuracy decreases by 4.60% and 2.78% while the sparsity level increases from 90% to 95% and 70% to 90% respectively for IID settings.
Revised: It is worth noticing that the validation accuracy decreases by 4.60% and 2.78% while the sparsity level increases from 90% to 95% and 70% to 90% respectively for IID settings.

Original: Then the validation accuracy drops by 8.3% and 1.84% when sparsity levels increase from 90% to 95% and 70% to 90% respectively for non-IID settings.
Revised: Then the validation accuracy drops by 8.3% and 1.84% when sparsity levels increase from 90% to 95% and 70% to 90% respectively for non-IID settings.

Original: This highlights an important degradation of performance in non-IID settings with very high levels of sparsity.
Revised: This highlights an important degradation of performance in non-IID settings with very high levels of sparsity.

Original: 4. S PARSIFICATION E FFECT A NALYSIS
Revised: 4. S PARSIFICATION E FFECT A NALYSIS

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 1, high levels of sparsity with FL ( e.g. ≥ 90 %) induce an signiﬁcant drop of accuracy that is more noticeable than for centralised training.
Revised: 1, high levels of sparsity with FL ( e.g. ≥ 90 %) induce an signiﬁcant drop of accuracy that is more noticeable than for centralised training.

Original: We aim at understanding this phenomenon to further reduce the gap between centralised and FL training by properly adapting sparsiﬁcation.
Revised: We aim at understanding this phenomenon to further reduce the gap between centralised and FL training by properly adapting sparsiﬁcation.

Original: As a ﬁrst step, we propose to investigate the behaviour of the neural weights from clients to clients under SWAT and FL with a sparsity ratio equals to 90 %.
Revised: As a ﬁrst step, we propose to investigate the behaviour of the neural weights from clients to clients under SWAT and FL with a sparsity ratio equals to 90 %.

Original: Indeed, an undocumented effect of SWAT occurs at inference time and may motivate an extension of the technique to work properly with FL.
Revised: Indeed, an undocumented effect of SWAT occurs at inference time and may motivate an extension of the technique to work properly with FL.

Original: During training, SWAT partitions the weights in two sets: active and non-active.
Revised: During training, SWAT partitions the weights in two sets: active and non-active.

Original: The former set is used as a sparsity map during any forward propagation ( i.e. both at training and inference times).
Revised: The former set is used as a sparsity map during any forward propagation ( i.e. both at training and inference times).

Original: In fact, a part of the weights are simply dropped from the neural network.
Revised: In fact, a part of the weights are simply dropped from the neural network.

Original: This implies that the neural network must remain sparsiﬁed when evaluating and inferring or the accuracy will drastically drops.
Revised: This implies that the neural network must remain sparsiﬁed when evaluating and inferring or the accuracy will drastically drops.

Original: For instance, a FL trained system on CIFAR10 that reaches an accuracy of 82 % on the validation set will drop to 10 % if the inference is done without sparsiﬁcation. 
Revised: For instance, a FL trained system on CIFAR10 that reaches an accuracy of 82 % on the validation set will drop to 10 % if the inference is done without sparsiﬁcation. [Figure]

Original: It empirically validates that sparse training has an impact on the internal representation of a neural network making it dependent on the sparsiﬁcation strategy during inference. 
Revised: It empirically validates that sparse training has an impact on the internal representation of a neural network making it dependent on the sparsiﬁcation strategy during inference. In practice

Revised: [Figure]

Original: In practice, the latter behaviour is explained by the fact that, during centralised training, sparsiﬁed weights tend to be always the same i.e. some parameters are simply discarded.
Revised: the latter behaviour is explained by the fact that, during centralised training, sparsiﬁed weights tend to be always the same i [Figure] e. some parameters are simply discarded.

Original: For FL, however, one may intuitively hypothesize that different clients will lead to different sparsiﬁcation maps during the local training; preventing the creation of a global federated sparsiﬁcation strategy.
Revised: For FL, however, one may intuitively hypothesize that different clients will lead to different sparsiﬁcation maps during the local training; preventing the creation of a global federated sparsiﬁcation strategy.

Original: To investigate[Figure] this, we propose to analyse the variations observed on the most active weights after aggregation on the server over the communication rounds.
Revised: To investigate this, we propose to analyse the variations observed on the most active weights after aggregation on the server over the communication rounds.

Original: Let us deﬁne the top-K weights as being the per-layer K % neural parameters with the highest norm.
Revised: Let us deﬁne the top-K weights as being the per-layer set of parameters with the highest norm.

Original: During each communication round, clients only send their top-K weights to the server while the remaining ones are set to zero.
Revised: During each communication round, clients only send their top-K weights to the server while the remaining ones are set to zero.

Original: After aggregation on the server side, the resulting weight matrix informs us on the level of overlapping non-zero parameters observed on the clients.
Revised: After aggregation on the server side, the resulting weight matrix informs us on the level of overlapping non-zero parameters observed on the clients.

Original: Indeed, every non-zero value obtained at a speciﬁc position will likely result in a non-zero value at the corresponding position in the aggregated weight matrix.
Revised: Indeed, every non-zero value obtained at a speciﬁc position will likely result in a non-zero value at the corresponding position in the aggregated weight matrix.

Original: For instance, if the number of non-sparse elements from the aggregated matrix is equivalent to the one of the clients ( K %)
Revised: For instance, if the number of non-sparse elements from the aggregated matrix is equivalent to the one of the clients ( K %)

Original: it means that all clients have the exact same top-K weights.
Revised: it means that all clients have the exact same top-K weights.

Original: With that in mind, we can deﬁne a non-zero ratio that is the number of non-zero parameters after aggregation divided by the the total number of elements in this layer.
Revised: With that in mind, we can deﬁne a non-zero ratio that is the number of non-zero parameters after aggregation divided by the the total number of elements in this layer.

Original: Thus, the higher this ratio is, the more different are the top-K weights sent from the clients.
Revised: Thus, the higher this ratio is, the more different are the top-K weights sent from the clients.

Original: Fig. 2 depicts this ratio for different CNN layers over the communication round of a CIFAR10 training FL procedure with K ∈ { 10 , 30 } %.
Revised: Fig.depicts this ratio for different CNN layers for a non-IID CIFAR-10 with K ∈ { 10 , 30 } %.

Original: The non-IID parameter α is set to 1 as this setup is most likely to occur in realistic conditions.

Original: First, a signiﬁcant overlap exists between clients across all CNN layers.
Revised: First, a signiﬁcant overlap exists between clients across all CNN layers.

Original: Indeed, the non-zero ratio almost never exceeds 0.40, meaning that at least 60% of the weights are not comprised in the top-K weights of the clients.
Revised: Indeed, the non-zero ratio almost never exceeds 0.40, meaning that at least 60% of the weights are not comprised in the top-K weights of the clients.

Original: This advocates for the fact that the most important weights for the current task tend to be the same across clients.
Revised: This advocates for the fact that the most important weights for the current task tend to be the same across clients.

Original: Then, we can see that the non-zero ratio does not seem to be signiﬁcantly impacted by changing K from 10 % to 30 % as it only slightly increases for most
Revised: Then, we can see that the non-zero ratio does not seem to be signiﬁcantly impacted by changing K from 10 % to 30 % as it only slightly increases for most

Original: CNN layers.
Revised: CNN layers.

Original: This is explained by the fact that while clients may have different top-10% weights, they tend to have the same top-30% parameters: a speciﬁc non-zero weight that is reported as a top-10% element from a single client out of the selected ones is most likely to be reported as non-zero as well by more of them when we increase to top-30% .
Revised: This is explained by the fact that while clients may have different top-10% weights, they tend to have the same top-30% parameters: a speciﬁc non-zero weight that is reported as a top-10% element from a single client out of the selected ones is most likely to be reported as non-zero as well by more of them when we increase to top-30% .

Original: In short, top-30% will gather most of information about weights that are considered as being important for the task while keeping the same level of sparsity.
Revised: In short, top-30% will gather most of information about weights that are considered as being important for the task while keeping the same level of sparsity.

Original: Second, we propose to examine the exact positions of the top-10% weights for certain CNN layers across a pool of 100 selected clients to better evaluate the overlap.
Revised: Second, we propose to examine the exact positions of the top-10% weights for certain CNN layers across a pool of 100 selected clients to better evaluate the overlap.

Original: Fig.
Revised: Fig.

Original: 3 shows the weight matrix recorded every 20 communication rounds after aggregation on the server.
Revised: 3 shows the weight matrix recorded every 20 communication rounds after aggregation on the server.

Original: Observations validate our intuition that most of zero and non-zero weights remain the same during the whole training.
Revised: Observations validate our intuition that most of zero and non-zero weights remain the same during the whole training.

Original: Based on this analysis, we hypothesis that the degradation of performance observed with high levels of sparsity for FL is due to the dilution of important information during the aggregation process.
Revised: Based on this analysis, we hypothesis that the degradation of performance observed with high levels of sparsity for FL is due to the dilution of important information during the aggregation process.

Original: For instance, a weight that would be only sent by a single client as part of its top-10% parameters would not be diluted with the noise of all the others clients.
Revised: For instance, a weight that would be only sent by a single client as part of its top-10% parameters would not be diluted with the noise of all the others clients.

Original: Conversely, this very same weight may be completely corrupted if we send the entire dense model for aggregation.
Revised: Conversely, this very same weight may be completely corrupted if we send the entire dense model for aggregation.

Original: Central server does: for t = 0 , ..., T − 1 do

Original: Server randomly selects K devices.

Original: for all k in K do

Original: Perform TrainLocally ( k, w t )

Original: Aggregation :

Original: If Top-K-Weight then w t +1 ← (cid:80) Kk =0 n k n w kt +

Original: If Diff on Top-K-Weight then w t +1 ← w t + (cid:80) Kk =0 n k n d kt +

Original: If Top-K Diff then w t +1 ← w t + (cid:80) Kk =0 n k n d kt +

Original: TrainLocally ( k, w t ) : for e = 1 , ..., E do

Original: Do local model training via SWAT with sparsity level sp .

Original: w e ← w e − 1 − η t (cid:53) F ( w e − 1 )

Original: Determine which weights to send for aggregation: If Top-K-Weight then return top 1 − sp + r mask weights.

Original: d kt +1 ← w E − w t

Original: If Diff on Top-K-Weight then return d kt +1 of top 1 − sp + r mask weights.

Original: If Top-K-Weights Diff then return top 1 − sp + r mask of d kt +1 .

Original:  Z ERO FL: LOCAL SPARSIFICATION OF UPLINK COMMUNICATION
Revised:  Z ERO FL: LOCAL SPARSIFICATION OF UPLINK COMMUNICATION

Original: Motivated by our previous analysis which suggests that not all weights are necessary to be transferred to the central server for aggregation, we propose ZeroFL : a method that applies local sparsiﬁcation before uplink communication.
Revised: Motivated by our previous analysis which suggests that not all weights are necessary to be transferred to the central server for aggregation, we propose ZeroFL : a method that applies local sparsiﬁcation before uplink communication.

Original: More precisely, we provide three strategies for local sparsiﬁcation to improve the performance of sparse training while reducing the communication cost at the same time.
Revised: More precisely, we provide three strategies for local sparsiﬁcation to improve the performance of sparse training while reducing the communication cost at the same time.

Original: By leveraging local sparsiﬁcation, ZeroFL reduces the uplink communication footprint, hence reducing the noise aggregated on the central server observed in Section 4.3.
Revised: By leveraging local sparsiﬁcation, ZeroFL reduces the uplink communication footprint, hence reducing the noise aggregated on the central server observed in Section 4.3.

Original: In particular, if some updates are only sent by few clients, we force others clients to send zero in that particular positions of updates following our three strategies.
Revised: In particular, if some updates are only sent by few clients, we force others clients to send zero in that particular positions of updates following our three strategies.

Original: After aggregation, the magnitude of these particular updates will be averaged with the uploaded values instead of being completely corrupted by the entire set of clients.
Revised: After aggregation, the magnitude of these particular updates will be averaged with the uploaded values instead of being completely corrupted by the entire set of clients.

Original: All three methods are summarized in Algorithm 1.
Revised: All three methods are summarized in Algorithm 1.

Original: Top-K-Weights.
Revised: Top-K-Weights.

Original: As shown in Sec.
Revised: As shown in Sec.

Original: 4.3, only top-K active weights are involved in the validation and inference stages, and an important part of these weights tend not to change during training.
Revised: 4.3, only top-K active weights are involved in the validation and inference stages, and an important part of these weights tend not to change during training.

Original: Therefore, the ﬁrst local sparsiﬁcation method is to sparsify the top-K weights on the client-side before sending them back to the central server for aggregation.
Revised: Therefore, the ﬁrst local sparsiﬁcation method is to sparsify the top-K weights on the client-side before sending them back to the central server for aggregation.

Original: Then, and even though the positions of top-K weights largely overlap between each clients, only sending exactly a number of weights corresponding to the sparsity level might be too restrictive and prevents natural variations in the weights.
Revised: Then, and even though the positions of top-K weights largely overlap between each clients, only sending exactly a number of weights corresponding to the sparsity level might be too restrictive and prevents natural variations in the weights.

Original: Hence, we introduce a parameter denoted as the mask ratio r mask , indicating the additional amount of weights that the selected clients will choose to send to the server after the local training.
Revised: Hence, we introduce a parameter denoted as the mask ratio r mask , indicating the additional amount of weights that the selected clients will choose to send to the server after the local training.

Original: Let sp be the sparsity level of the local training.
Revised: Let sp be the sparsity level of the local training.

Original: For instance, with sp = 0 . 9 , and after the local training process, each selected client sparsiﬁes their model by only keeping the top (1 − sp + r mask ) weights w.r.t their magnitude, while setting the rest to zero.
Revised: For instance, with sp = 0 . 9 , and after the local training process, each selected client sparsiﬁes their model by only keeping the top (1 − sp + r mask ) weights w.r.t their magnitude, while setting the rest to zero.

Original: In particular, if sp = 0 .
Revised: In particular, if sp = 0 .

Original: 9 and r mask = 0 .
Revised: 9 and r mask = 0 .

Original: 1 , the selected clients will send the top-20% to the central server for aggregation.
Revised: 1 , the selected clients will send the top-20% to the central server for aggregation.

Original: The model produced after aggregation is not guaranteed to be sparse.
Revised: The model produced after aggregation is not guaranteed to be sparse.

Original: If mask ratio equals to the sparsity level, the algorithm is degenerated to vanilla SWAT without local sparsiﬁcation.
Revised: If mask ratio equals to the sparsity level, the algorithm is degenerated to vanilla SWAT without local sparsiﬁcation.

Original: However, and as shown in Fig.2, the resulting model is most likely to be sparse.
Revised: However, and as shown in Fig.2, the resulting model is most likely to be sparse.

Original: Here, uplink communications are saved as only (1 − sp + r mask ) weights are sent as dense values.
Revised: Here, uplink communications are saved as only (1 − sp + r mask ) weights are sent as dense values.

Original: Diff on Top-K-Weights.
Revised: Diff on Top-K-Weights.

Original: The idea behind this method is to send local weight-updates for the top-K weights, rather than sending the top-K weights themselves.
Revised: The idea behind this method is to send local weight-updates for the top-K weights, rather than sending the top-K weights themselves.

Original: Given sp and r mask , we ﬁrst identify the weights that are the largest in magnitude by selecting the top (1 − sp + r mask ) .
Revised: Given sp and r mask , we ﬁrst identify the weights that are the largest in magnitude by selecting the top (1 − sp + r mask ) .

Original: The selected clients now send only the difference d of these top (1 − sp + r mask ) weights with respect to the originally received model w t as d t+1 = 
Revised: The selected clients now send only the difference d of these top (1 − sp + r mask ) weights with respect to the Central server does: for t = 0 , ..., T − 1 do Server randomly selects K devices. for all k in K do

Revised: Perform TrainLocally ( k, w t )

Revised: Aggregation :

Original:  w E − w t with w E the weights obtained after local training.
Revised: If Top-K-Weight then w t +1 ← (cid:80) Kk =0 n k n w kt + If Diff on Top-K Weight then w t +1 ← w t + (cid:80) Kk =0 n k n d kt +

Revised: If Top

Revised: K Diff then w t +1 ← w t + (cid:80)

Revised: Kk =0 n k n d kt +

Revised: TrainLocally ( k, w t ) : for e = 1 , ..., E do

Revised: Do local model training via SWAT with level sp .

Revised: w e ← w e −

Revised: Determine which weights to send for

Revised: : If return d return originally received model w t as d t +1 = w E − w t with w E the weights obtained after local training.

Original: The remaining differences are set to zero.
Revised: The remaining differences are set to zero.

Original: In this way, after the aggregation in the central server, the weights that are not in the top (1 − sp + r mask ) part will remain the same as during the previous round, while only the top (1 − sp + r mask ) part of the weights will be updated.
Revised: In this way, after the aggregation in the central server, the weights that are not in the top (1 − sp + r mask ) part will remain the same as during the previous round, while only the top (1 − sp + r mask ) part of the weights will be updated.

Original: Top-K-Weights Diff.
Revised: Top-K-Weights Diff.

Original: Conversely to Diff on Top-K-Weights, this strategy proposed to ﬁrst compute all the weights differences d
Revised: Conversely to Diff on Top-K-Weights, this strategy proposed to ﬁrst compute all the weights differences d

Original: and then only send the top (1 − sp + r mask ) of them to the server.
Revised: and then only send the top (1 − sp + r mask ) of them to the server.

Original: With this method, only highly moving weights will be considered.
Revised: With this method, only highly moving weights will be considered.

Original: All the aforementioned local sparsiﬁcation methods lead to substantial reductions in uplink communication costs.
Revised: All the aforementioned local sparsiﬁcation methods lead to substantial reductions in uplink communication costs.

Original: More precisely, total communications will be reduced by a factor of ( r mask − sp ) / 2 .
Revised: More precisely, total communications will be reduced by a factor of ( r mask − sp ) / 2 .

Original:  Z ERO FL: E XPERIMENTAL R ESULTS
Revised:  Z ERO FL: E XPERIMENTAL R ESULTS

Revised: We conducted extensive experiments on CIFAR10, Speech Commands (Warden, 2018) and FEMNIST (Caldas et al., 2018) datasets.

Original: We conducted extensive experiments CIFAR10 following the same experimental protocol as for baselines experiments.
Revised: The CIFAR10 experiments follow the same experimental protocol as for baselines experiments.

Original: The three local sparsiﬁcation strategies are compared with various mask ratio r mask =
Revised: The three local sparsiﬁcation strategies are compared with various mask ratio r mask =

Original: { 0 . 0 , 0 . 1 , 0 . 2 } and to vanilla SWAT without ZeroFL.
Revised: { 0 . 0 , 0 . 1 , 0 . 2 } and to vanilla SWAT without ZeroFL.

Original: Similarly to results obtained in Section 4.2, sparse training performs better with exponential learning rate decay.
Revised: Similarly to results obtained in Section 4.2, sparse training performs better with exponential learning rate decay.

Original: Hence, all setups are investigated with this scheduler with starting learning rates of 0 .
Revised: Hence, all setups are investigated with this scheduler.

Original: 1 and 0 . 2 . Table 1 reports thehighest test accuracies achieved within 700 communication rounds.
Revised: Table 1 reports the test accuracies achieved as well as the gain in communication cost when applying ZeroFL.

Original: First, it is worth noticing that all three local sparsiﬁcation methods with mask ratios higher than 0 .perform better than vanilla SWAT.
Revised: First, it is worth noticing that all three local sparsiﬁcation methods with mask ratios higher than 0 .perform better or similarly to vanilla SWAT.

Original: The biggest improvement for 90 % sparsity is achieved with theDiff on Top-K-Weights method with a mask ratio of 0 . 2 , which increases the test accuracy by 2 .1 %.
Revised: The biggest improvement for 90 % sparsity is achieved with the Top-K-Weights method with a mask ratio of 0 . 2 , which increases the test accuracy by 0 .

Original: The largest improvement for 95 % sparsity is achieved with the Top-K-Weights method with a mask ratio of 0 . 2 , and the it increases the test accuracy by 3 . 74 %.
Revised: The largest improvement for 95 % sparsity is achieved with the Top-K-Weights method with a mask ratio of 0 . 2 , and the it increases the test accuracy by 1 . 5 %.

Original: For SpeechCommands, we reported the performance at communication round 200 to show that
Revised: For SpeechCommands, we reported the performance at communication round 200 to show that

Original: ZeroFL gives faster convergences and higher accuracies with mask ratio higher than 0 , especially for the non-IID setting.
Revised: ZeroFL achieves faster convergences and higher accuracies with mask ratio higher than 0 , especially for the non-IID setting.

Original: More results can be found in appendix A.4, which demonstrates higher performance at round 300 when test accuracies are stabilized.
Revised: More results can be found in appendix A.4, which demonstrates higher performance at round 300 when test accuracies are stabilized.

Original: With ZeroFL, the performance can be improved by 5 . 5 % for 95 % sparsity with non-IID settings.
Revised: With ZeroFL, the performance can be improved by 2 . 3 % for 90 % sparsity in the non-IID setting.

Original: For FEMNIST we observe a similar trend when evaluating the different masking methods in ZeroFL: larger mask ratios result in better performing global models.
Revised: For FEMNIST we observe a similar trend when evaluating the different masking methods in ZeroFL: larger mask ratios result in better performing global models.

Original: However, sending the entire model from  client to server seems to be key to obtain good results.
Revised: However, sending the entire model from [Table] client to server seems to be key to obtain good results.

Original: We hypothesise this is because FEMNIST is a much more challenging dataset (62 classes) and the architecture used is relatively simple and less sensible to sparsiﬁcation as a result.[Table]
Revised: We hypothesise this is because FEMNIST is a much more challenging dataset (62 classes) and the architecture used is relatively simple and less sensible to sparsiﬁcation as a result.

Original: Overall [Table] performance improve with mask ratioscomprised between 0 and 0 . 2 , which indicates that there exist an optimal interval level.
Revised: Overall, performance improved with mask ratios between 0 and 0 . 2 , indicating that there exist an optimal interval level.

Original: Indeed, a mask ratio equal to the sparsity level degenerates the system to the vanilla SWAT, which obtains worse results.
Revised: Indeed, a mask ratio equal to the sparsity level degenerates the system to the vanilla SWAT, which obtains worse results.

Original: It also implies that there is a trade-off between communication cost and performance.
Revised: It also implies that there is a trade-off between communication cost and performance.

Original: By using mask ratio of 0 . 1 , each selected client performs local sparsiﬁcation with an effective sparsity level of 80 %, hence sending more data that for lower mask ratios values.
Revised: By using mask ratio of 0 . 1 , each selected client performs local sparsiﬁcation with an effective sparsity level of 80 %, hence sending more data that for lower mask ratios values.

Original: In Section A.1 we densely evaluate the impact of r mask ∈ [0 . 1 , 0 . 9] and, we observe no clear beneﬁt of choosing larger ratios over smaller ones (e.g. 0.1).
Revised: In Section A.1 we densely evaluate the impact of r mask ∈ [0 . 1 , 0 . 9] and, we observe no clear beneﬁt of choosing larger ratios over smaller ones (e.g. 0.1).

Original: We brieﬂy elaborate on this in the Appendix.
Revised: We brieﬂy elaborate on this in the Appendix.

Original: ZeroFL enables us to reduce the performance degradation observed with high levels of sparsity with
Revised: ZeroFL enables us to reduce the performance degradation observed with high levels of sparsity with

Original: FL while not completely alleviating it.
Revised: FL while not completely alleviating it.

Original: During communications, each weight matrix from the model is vectorized and transmitted using the Compressed Sparse Row (CSR) (Tinney & Walker, 1967) format.
Revised: During communication, each weight matrix from the model is vectorised and transmitted using the Compressed Sparse Row (CSR) (Tinney & Walker, 1967) format.

Original: Such representation requires exactly one integer index for each non-zero weight value in the model.
Revised: Such representation requires exactly one integer index for each non-zero weight value in the model.

Original: Table 1 shows the level of compression and reduction in communication for different level of mask ratios.
Revised: Table 1 shows the level of compression and reduction in communication for different level of mask ratios.

Original: Compared to the original 43MB dense model, uplink communications are reduced by factors of 7 . 5 × , 3 . 0 × and 1 .
Revised: Compared to the original 43MB dense model, uplink communications are reduced by factors of 7 . 5 × , 3 . 0 × and 1 .

Original: 9 × for mask ratios of 0 . 0 , 0 . 1 and 0 .
Revised: 9 × for mask ratios of 0 . 0 , 0 . 1 and 0 .

Original: 2 respectively, with a sparsiﬁcation ratio of 95 %.
Revised: 2 respectively, with a sparsiﬁcation ratio of 95 %.

Original: Communication savings are calculated as the size ratio between original and compressed models; considering both weights and indices in the CSR ﬁle.
Revised: Communication savings are calculated as the size ratio between original and compressed models; considering both weights and indices in the CSR ﬁle.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we consider the challenges of inducing high level of sparsity to accelerate on-device training for federated learning.
Revised: In this work, we consider the challenges of inducing high level of sparsity to accelerate on-device training for federated learning.

Original: We provide the ﬁrst framework to leverage sparsity as a mechanism to accelerate on-device training without the server imposing any restrictions.
Revised: We provide the ﬁrst framework to leverage sparsity as a mechanism to accelerate on-device training without the server imposing any restrictions.

Original: We study on the unique perspective that arise when introducing sparsity at training time, hence motivating us to propose innovative state-of-the-art off-the-shelve sparsiﬁcation methods to the FL domain ZeroFL .
Revised: We study on the unique perspective that arise when introducing sparsity at training time, hence motivating us to propose innovative state-of-the-art off-the-shelve sparsiﬁcation methods to the FL domain ZeroFL .

Original: The method achieves +2.1% and +3.7% higher accuracy while reducing 3.0 × uplink communication costs compared with competitive baselines for CIFAR10, and +5.5% and +2.6% higher accuracy for SpeechCommands.
Revised: The method achieves +1.5% accuracy while reducing 1.9 × uplink communication costs compared with competitive baselines for CIFAR10 at 95% sparsity, and +2.3% accuracy for non-IID SpeechCommands at 90% sparsity.

Original: Our ﬁndings call for further investigations on the device-oriented optimisation of federated learning to motivate realistic deployments of this training methodology.
Revised: Our ﬁndings call for further investigations on the device-oriented optimisation of federated learning to motivate realistic deployments of this training methodology.

Revised: A CKNOWLEDGEMENTS

Revised: This work was supported by the UK’s Engineering and Physical Sciences Research Council (EPSRC) with grants EP/M50659X/1 and EP/S001530/1 (the MOA project) and the European Research

Revised: Council (ERC) via the REDIAL project (Grant Agreement ID: 805194).

Revised: Part of this work was performed using HPC/AI resources from GENCI-IDRIS (Grant 2021-A0111012991)

Original: A A PPENDIX A. I MPACT OF r mask IN GLOBAL PERFORMANCE
Revised: A A PPENDIX A. I MPACT OF r mask IN GLOBAL PERFORMANCE

Original: We hypothesised in Section 6 that increasing the masking ratio, r mask , which is used to limit the amount of non-zero weights that would be uploaded to the central server after each client completes its local training, would yield better global model performance.
Revised: We hypothesised in Section 6 that increasing the masking ratio, r mask , which is used to limit the amount of non-zero weights that would be uploaded to the central server after each client completes its local training, would yield better global model performance.

Original: However, as it can be observed in
Revised: However, as it can be observed in

Original: Figure 4, test accuracy does not monotonically increase with r mask and lower values (i.e. 0.1) that enable large savings in upload communication could perform as good as with larger values (e.g. 0.6) after ﬁne-tuning.
Revised: Figure 4, test accuracy does not monotonically increase with r mask and lower values (i.e. 0.1) that enable large savings in upload communication could perform as good as with larger values (e.g. 0.6) after ﬁne-tuning.

Original: [Figure]
Revised: [Figure]

Original: A. CIFAR-10 WITH F ED A DAM STRATEGY
Revised: A. CIFAR-10 WITH F ED A DAM STRATEGY

Original: Table 2 shows the performance of the proposed masking methods in ZeroFL when evaluating
Revised: Table 2 shows the performance of the proposed masking methods in ZeroFL when evaluating

Original: CIFAR-10 on the non-IID setting using the FedAdam (Reddi et al., 2021) aggregation strategy.
Revised: CIFAR-10 on the non-IID setting using the FedAdam (Reddi et al., 2021) aggregation strategy.

Original: The results are inline with what was ﬁrst observed in Table 1 in Section 6: at large sparisty ratios (0.95) only ZeroFL making use of
Revised: The results are inline with what was ﬁrst observed in Table 1 in Section 6: at large sparisty ratios (0.95) only ZeroFL making use of

Original: [Table]
Revised: [Table]

Original: A. H EATMAP VISUALIZATIONS
Revised: A. H EATMAP VISUALIZATIONS

Original: As mentioned in Fig.
Revised: As mentioned in Fig.

Original: [Table]
Revised: [Table]

Original: a bigger version of heatmap for one of the CNN layer (layer 4) is shown in Fig.
Revised: a bigger version of heatmap for one of the CNN layer (layer 4) is shown in Fig.

Original: 5 for better reference.
Revised: 5 for better reference.

Original: Heatmap plots for SpeechCommands can be found in Fig.
Revised: Heatmap plots for SpeechCommands can be found in Fig.

Original: [Table]
Revised: [Table]

Original: which shows the similar results.
Revised: which shows the similar results.

Original: A. A DDITIONAL RESULTS FOR S PEECH C OMMANDS
Revised: A. A DDITIONAL RESULTS FOR S PEECH C OMMANDS

Original: Table 3 reports the performance of ZeroFL SpeechCommands after 300 communication rounds for both IID and non-IID settings.
Revised: Table 3 reports the performance of ZeroFL SpeechCommands after 300 communication rounds for both IID and non-IID settings.

Original: Combined with Table 1, the results show consistent better performance for non-IID settings for mask ratio higher than 0 .
Revised: Combined with Table 1, the results show consistent better performance for non-IID settings for mask ratio higher than 0 .

Original: The increase in accuracy at roundsin Table 1 shows faster convergence of ZeroFL compared with baselines, while improvement in accuracies at rounds 300 in Table 3 demonstrates the better performance of ZeroFL overall.
Revised: The increase in accuracy at roundsin Table 1 shows faster convergence of ZeroFL compared with baselines, while improvement in accuracies at rounds 300 in Table 3 demonstrates the better performance of ZeroFL overall.

Original: A. D ATASETS
Revised: A. D ATASETS

Original: This work considers two image classiﬁcation tasks of different complexity both in terms of the number of samples and classes: CIFAR

Revised: This work considers two image classiﬁcation tasks of different complexity both in terms of the number of samples and classes: CIFAR

Original: (Krizhevsky et al., 2009) and FEMNIST (Caldas et al., 2018) with 10 classes and 62 classes respectively.
Revised: (Krizhevsky et al., 2009) and FEMNIST (Caldas et al., 2018) with 10 classes and 62 classes respectively.

Original: The former is comprised of 60K 32 × 32 RGB images for training and 10K images for test.
Revised: The former is comprised of 60K 32 × 32 RGB images for training and 10K images for test.

Original: The latter, results in over 652K images for training and over 165K for test.
Revised: The latter, results in over 652K images for training and over 165K for test.

Original: FEMNIST images are 28 × 28 and grayscale.
Revised: FEMNIST images are 28 × 28 and grayscale.

Original: In both scenarios we randomly extract 10% out from the training set for validation.
Revised: In both scenarios we randomly extract 10% out from the training set for validation.

Original: This is done at the client level, i e., the validation
Revised: This is done at the client level, i e., the validation

Original: [Table]
Revised: [Table]

Original: set for each client is extracted from each client’s training partition.
Revised: set for each client is extracted from each client’s training partition.

Original: This is done to ensure that the validation set is representative of the underlying distribution
Revised: This is done to ensure that the validation set is representative of the underlying distribution.

Original: [Table] In addition [Table] we also perform analysis on the Speech Commands dataset (Warden, 2018) which consists of 65 K 1-second long audio clips of 30 keywords, with each clip consisting of only one keyword.
Revised: In addition, we also perform analysis on the Speech Commands dataset (Warden, 2018) which consists of 65 K 1-second long audio clips of 30 keywords, with each clip consisting of only one keyword.

Original: We train the model to classify the audio clips into one of the 10 keywords - “Yes”, “No”, “Up”,“Down”, “Left”, “Right”, “On”, “Off”, “Stop”, “Go”, along with “silence” ( i.e. no word spoken) and “unknown” word, representing the remaining 20 keywords from the dataset.
Revised: We train the model to classify the audio clips into one of the 10 keywords - “Yes”, “No”, “Up”,“Down”, “Left”, “Right”, “On”, “Off”, “Stop”, “Go”, along with “silence” ( i.e. no word spoken) and “unknown” word, representing the remaining 20 keywords from the dataset.

Original: The training set contains a total of 56 , 196 clips with 32 , 550 ( 57% ) samples from the “unknown” class and around 1800 samples ( 3 . 3% ) from each of the remaining classes, hence the dataset is naturally unbalanced.
Revised: The training set contains a total of 56 , 196 clips with 32 , 550 ( 57% ) samples from the “unknown” class and around 1800 samples ( 3 . 3% ) from each of the remaining classes, hence the dataset is naturally unbalanced.

Original: Similarly to (Zhang et al., 2018), each audio clip is preprocessed and 32 × 32 MFCC features are extracted and fed to the CNN backbone, a ResNet-18 as described in Section 4.1.
Revised: Similarly to (Zhang et al., 2018), each audio clip is preprocessed and 32 × 32 MFCC features are extracted and fed to the CNN backbone, a ResNet-18 as described in Section 4.1.

Original: A. M ODELS 
Revised: A. M ODELS AND H YPERPARAMETERS

Original: For ResNet-18, all 3 × 3 convolutional layers implement sparse training as discussed in Sec. 3.2.
Revised: For ResNet-18, all 3 × 3 convolutional layers implement sparse training as discussed in Sec. 3.2.

Original: As it is common with other optimizations, we leave the input layer unchanged (i.e. performing standard dense training).
Revised: As it is common with other optimizations, we leave the input layer unchanged (i.e. performing standard dense training).

Original: This architecture results in 11M parameters.
Revised: This architecture results in 11M parameters.

Original: For FEMNIST, we borrow the much smaller CNN ﬁrst proposed by Caldas et al.
Revised: For FEMNIST, we borrow the much smaller CNN ﬁrst proposed by Caldas et al.

Original: It is comprised of two 5 × 5 convolutional layers (each followed by a 2 × 2 maxpool laye) and two linear layers.
Revised: It is comprised of two 5 × 5 convolutional layers (each followed by a 2 × 2 maxpool laye) and two linear layers.

Original: We sparsify both convolutional layers and the ﬁrst linear layer but leave the ﬁnal layer unchanged (i.e. dense).
Revised: We sparsify both convolutional layers and the ﬁrst linear layer but leave the ﬁnal layer unchanged (i.e. dense).

Original: This architecture results in parameters 6.6M parameters, which the ﬁrst linear layer accounting for 97% of the memory footprint.
Revised: This architecture results in parameters 6.6M parameters, which the ﬁrst linear layer accounting for 97% of the memory footprint.

Revised: All our experiments make use of the following hyperparameters.

Revised: The start and end learning rate are 0 .

Revised: 1 and 0 .

Revised: 01 respectively for CIFAR10,