Original: Efficient and Light-Weight Federated Learning via Asynchronous Distributed Dropout
Revised: Efficient and Light-Weight Federated Learning via Asynchronous Distributed Dropout

Original: Anonymous Author(s)

Original: Abstract
Revised: Abstract

Original: We focus on dropout techniques for asynchronous distributed computations infederated learning (FL) scenarios.
Revised: We focus on dropout techniques for asynchronous distributed computations in federated learning (FL) scenarios.

Original: We propose AsyncDrop , a novel asynchronous
Revised: We propose AsyncDrop , a novel asynchronous

Original: FL framework with smart (i.e., informed/structured) dropout that achieves betterperformance compared to state of the art asynchronous methodologies, while resulting in less communication and training time costs.
Revised: FL framework with smart (i.e., informed/structured) dropout that achieves better performance compared to state of the art asynchronous methodologies, while resulting in less communication and training time costs.

Original: The key idea revolves aroundsub-models out of the global model, that take into account the device heterogeneity.
Revised: The key idea revolves around sub-models out of the global model, that take into account the device heterogeneity.

Original: We conjecture that such an approach can be theoretically justified.
Revised: We conjecture that such an approach can be theoretically justified.

Original: We implementour approach and compare it against other asynchronous baseline methods, byadapting current synchronous FL algorithms to asynchronous scenarios.
Revised: We implement our approach and compare it against other asynchronous baseline methods, by adapting current synchronous FL algorithms to asynchronous scenarios.

Original: Empirically, AsyncDrop significantly reduces the communication cost and training time,while improving the final test accuracy in non-i.i.d. scenarios.
Revised: Empirically, AsyncDrop significantly reduces the communication cost and training time, while improving the final test accuracy in non-i.i.d. scenarios.

Original:  Introduction
Revised:  Introduction

Original: Background on Federated Learning (FL).
Revised: Background on Federated Learning (FL).

Original: Federated Learning (FL) [21, 17, 15] is a distributedlearning protocol that has witnessed fast development the past demi-decade, as it deviates fromthe traditional distributed learning paradigms and allows the integration of small computing (edge)devices —such as smartphones [30], drones [25], and IoT devices [22]— in the learning procedure.
Revised: Federated Learning (FL) [21, 17, 15] is a distributed learning protocol that has witnessed fast development the past demi-decade, as it deviates from the traditional distributed learning paradigms and allows the integration of small computing (edge) devices —such as smartphones [30], drones [25] and IoT devices [22]— in the learning procedure.

Original: Such real-life, edge devices though are extremely heterogeneous [33]: they have drastically differentspecifications in terms of compute power, device memory and communication bandwidths.
Revised: Such real-life, edge devices though are extremely heterogeneous [33]: they have drastically different specifications in terms of compute power, device memory and communication bandwidths.

Original: Directlyapplying common synchronized FL algorithms –such as FedAvg and FedProx [17, 21] that requirefull model broadcasting and global synchronization– results often in a “stragglers” effect [23, 13, 31].
Revised: Directly applying common synchronized FL algorithms –such as FedAvg and FedProx [17, 21] that require full model broadcasting and global synchronization– results often in a “stragglers” effect [23, 13, 31].

Original: The dominant use of synchronous computations.
Revised: The dominant use of synchronous computations.

Original: One way to handle such issues is by utilizingasynchrony instead of synchrony in the learning process.
Revised: One way to handle such issues is by utilizing asynchrony instead of synchrony in the learning process.

Original: To explain the main differences, let usfirst set up the background.
Revised: To explain the main differences, let us first set up the background.

Original: In a synchronous distributed algorithm, a global model is usually storedat a central server and is broadcast periodically to all the participating devices.
Revised: In a synchronous distributed algorithm, a global model is usually stored at a central server and is broadcast periodically to all the participating devices.

Original: Then, each deviceperforms some local training steps on its own model copy, before the device sends the updated modelto the central server.
Revised: Then, each device performs some local training steps on its own model copy, before the device sends the updated model to the central server.

Original: Finally, the central server updates the global model by aggregating the receivedupdated model copies.
Revised: Finally, the central server updates the global model by aggregating the received updated model copies.

Original: This protocol is followed in most FL protocols, including the most well-knownbaselines like FedAvg [21], FedProx [17], FedNova [34] and SCAFFOLD [15].
Revised: This protocol is followed in most FL protocols, including the most well-known baselines like FedAvg [21], FedProx [17], FedNova [34] and SCAFFOLD [15].

Original: The main criticismagainst synchronous learning could be that it often results in heavy communication-/computationoverheads when mobile devices are used, long idle/waiting times for workers and stragglers effects.
Revised: The main criticism against synchronous learning could be that it often results in heavy communication-/computationoverheads when mobile devices are used, long idle/waiting times for workers and stragglers effects.

Original: Main dispute in asynchronous learning.
Revised: Main dispute in asynchronous learning.

Original: While asynchrony in computations seems a favorablealternative, the development of a asynchronous learning method, however, is often convoluted.
Revised: While asynchrony in computations seems a favorable alternative, the development of a asynchronous learning method, however, is often convoluted.

Original: Inthe past decade, HogWild!
Revised: In the past decade, HogWild!

Original: [24, 20] has emerged as a general asynchronous distributed methodology,and has been applied initially in basic ML problems like sparse linear/logistic regression [42, 41, 12].
Revised: [24, 20] has emerged as a general asynchronous distributed methodology, and has been applied initially in basic ML problems like sparse linear/logistic regression [42, 41, 12].

Original: Ideally, based on sparsity arguments, each edge device can independently update parts of the globalmodel –that overlap only slightly with the updates of other workers– in a lock-free fashion [24, 20].
Revised: Ideally, based on sparsity arguments, each edge device can independently update parts of the global model –that overlap only slightly with the updates of other workers– in a lock-free fashion [24, 20].

Original: This way, faster, more powerful edge workers do not suffer from idle waiting due to slower stragglers.
Revised: This way, faster, more powerful edge workers do not suffer from idle waiting due to slower stragglers.

Original: Yet, the use of asynchrony has been a topic of dispute in distributed neural network training [5, 2].
Revised: Yet, the use of asynchrony has been a topic of dispute in distributed neural network training [5, 2].

Original: Asynchronous training often inherently suffers from lower accuracy as compared to synchronized
Revised: Asynchronous training often inherently suffers from lower accuracy as compared to synchronized

Original: SGD, which results in the domination of synchronized SGD in neural network training [2].
Revised: SGD, which results in the domination of synchronized SGD in neural network training [2].

Original: Resurgence in asynchrony.
Revised: Resurgence in asynchrony.

Original: Recently, asynchronous methods have regained popularity, mainlydue to the interest in applying asynchronous motions within FL on edge devices.
Revised: Recently, asynchronous methods have regained popularity, mainly due to the interest in applying asynchronous motions within FL on edge devices.

Original: Yet, traditional off-the-self asynchronous algorithms still have issues, which might be exacerbated in the FL setting.
Revised: Yet, traditional off-the-self asynchronous algorithms still have issues, which might be exacerbated in the FL setting.

Original: As slower devices take longer local training time before updating the global model, this might resultin inconsistent update schedule of the global model, compared to that of faster devices.
Revised: As slower devices take longer local training time before updating the global model, this might result in inconsistent update schedule of the global model, compared to that of faster devices.

Original: This mighthave negative ramifications: i ) For FL on i.i.d.
Revised: This might have negative ramifications: i ) For FL on i.i.d.

Original: data, this will cause the gradient staleness problemand result in convergence rate decrease; and, ii ) on non-i.i.d.
Revised: data, this will cause the gradient staleness problem and result in convergence rate decrease; and, ii ) on non-i.i.d.

Original: data, this will result in significant dropin global model final accuracy.
Revised: data, this will result in significant drop in global model final accuracy.

Original: To overcome such problems, fresh proposals on asynchronous FL develop weighted global aggregationtechniques that take into consideration the heterogeneity of the devices [38, 3, 28]; yet, these methodsoften place a heavy computation/communication burden, as they rely on broadcasting the wholemodel’s updates to all the clients and/or the server.
Revised: To overcome such problems, fresh proposals on asynchronous FL develop weighted global aggregation techniques that take into consideration the heterogeneity of the devices [38, 3, 28]; yet, these methods often place a heavy computation/communication burden, as they rely on broadcasting the whole model’s updates to all the clients and/or the server.

Original: Other works monitor client speed to help trainingassignment [18, 1].
Revised: Other works monitor client speed to help training assignment [18, 1].

Original: Finally, recent efforts propose semi-asynchronous methods, where participatingdevices are selected and buffered in order to complete a semi-synchronous global update periodically[13, 37].
Revised: Finally, recent efforts propose semi-asynchronous methods, where participating devices are selected and buffered in order to complete a semi-synchronous global update periodically [13, 37].

Original: A thorough discussion on asynchronous methods in FL can be found in [39].
Revised: A thorough discussion on asynchronous methods in FL can be found in [39].

Original: What is different in this work?
Revised: What is different in this work?

Original: As most algorithms stem from directly adapting asynchrony insynchronous FL, one still needs to broadcast full model to all devices, following a data paralleldistributed protocol [9, 26], regardless of device heterogeneity .
Revised: As most algorithms stem from directly adapting asynchrony in synchronous FL, one still needs to broadcast full model to all devices, following a data parallel distributed protocol [9, 26], regardless of device heterogeneity .

Original: This inspire us to ask a key question:“Can we select sub-models out of the global model and send these to each device, taking into account the device heterogeneity?
Revised: This inspire us to ask a key question: “Can we select sub-models out of the global model and send these to each device, taking into account the device heterogeneity?

Original: Potentially, this could further reduce the negative effect of gradient staleness and overfitting problem in asynchronous FL.”
Revised: Potentially, this could further reduce the negative effect of gradient staleness and overfitting problem in asynchronous FL.”

Original: We answer this question affirmatively, by proposing a novel distributed dropout method for FL.
Revised: We answer this question affirmatively, by proposing a novel distributed dropout method for FL.

Original: Wedub our method AsyncDrop .
Revised: We dub our method AsyncDrop .

Original: Our approach assigns different sub-models to each device 1 ; empirically,such a strategy decreases the required time to converge to an accuracy level, while preservingfavorable final accuracy.
Revised: Our approach assigns different sub-models to each device 1 ; empirically, such a strategy decreases the required time to converge to an accuracy level, while preserving favorable final accuracy.

Original: This work reinstitutes the dispute between synchrony and asynchrony andfocuses on asynchronous computations should in heterogeneous distributed scenarios, as in FL.
Revised: This work reinstitutes the dispute between synchrony and asynchrony and focuses on asynchronous computations should in heterogeneous distributed scenarios, as in FL.

Original: Ouridea is based on the ideas of HogWild!
Revised: Our idea is based on the ideas of HogWild!

Original: [24, 20] –in terms of sparse sub-models– and Independent
Revised: [24, 20] –in terms of sparse sub-models– and Independent

Original: Subnetwork Training (IST) [40, 8, 19, 36] –where sub-models are deliberately created for distribution,in order to decrease both computational and communication requirements.
Revised: Subnetwork Training (IST) [40, 8, 19, 36] –where sub-models are deliberately created for distribution, in order to decrease both computational and communication requirements.

Original: Yet, we deviate from these works in the following sense: i ) To the best of our knowledge, we are notaware of any fully asynchronous FL implementations; the combination of HogWild!
Revised: Yet, we deviate from these works in the following sense: i ) To the best of our knowledge, we are not aware of any fully asynchronous FL implementations; the combination of HogWild!

Original: and IST ideaswas not stated and tested before this work.
Revised: and IST ideas was not stated and tested before this work.

Original: ii )
Revised: ii )

Original: While HogWild!-line of work provides optimizationguarantees, we are the first to consider the non-trivial, non-convex neural network setting and providetheoretical guarantees for convergence; such a result combines tools from asynchronous optimization,
Revised: While HogWild!-line of work provides optimization guarantees, we are the first to consider the non-trivial, non-convex neural network setting and provide theoretical guarantees for convergence; such a result combines tools from asynchronous optimization,

Original: Neural Tangent Kernel assumptions and focuses on convolutional neural networks, that deviate fromfully-connected layer simplified scenarios.
Revised: Neural Tangent Kernel assumptions and focuses on convolutional neural networks, that deviate from fully-connected layer simplified scenarios.

Original: Finally, iii ) we provide system-level algorithmic solutionsfor our approach, mirroring best-practices found during our experiments.
Revised: Finally, iii ) we provide system-level algorithmic solutions for our approach, mirroring best-practices found during our experiments.

Original: Overall, the contributionsof this work can be summarized as follows
Revised: Overall, the contributions of this work can be summarized as follows

Original:  • We consider and propose asynchronous distributed dropout for efficient large-scale FL.
Revised: : • We consider and propose asynchronous distributed dropout for efficient large-scale FL.

Original: Our focus ison non-trivial, non-convex ML models –as in neural network training– and our framework providesspecific engineering solutions how to handle these cases in practice.
Revised: Our focus is on non-trivial, non-convex ML models –as in neural network training– and our framework provides specific engineering solutions how to handle these cases in practice.

Original: • We theoretically characterize and support our proposal with rigorous and non-trivial convergencerate guarantees.
Revised: • We theoretically characterize and support our proposal with rigorous and non-trivial convergence rate guarantees.

Original: Currently, our conjecture assumes bounded delays; our future goal is to exploit recent developments on asynchronous optimization theory that drop such assumptions.
Revised: Currently, our conjecture assumes bounded delays; our future goal is to exploit recent developments on asynchronous optimization theory that drop such assumptions.

Original: Yet, ourtheory considers the harder case of neural network training, which is often omitted in existing –but
Revised: Yet, our theory considers the harder case of neural network training, which is often omitted in existing –but

Original: rather abstract– theory results.
Revised: rather abstract– theory results.

Original: • We provide specific implementation instances and share “best practices” for faster distributed FL neural network training in practice.
Revised: • We provide specific implementation instances and share “best practices” for faster distributed FL neural network training in practice.

Original: Our preliminary results include baseline asynchronous implementations of many synchronous methods (such as FedAvg, FedProx, and more), that are not existent currently, to the best of our knowledge.
Revised: Our preliminary results include baseline asynchronous implementations of many synchronous methods (such as FedAvg, FedProx, and more), that are not existent currently, to the best of our knowledge.

Original:  Problem Setup and Challenges
Revised:  Problem Setup and Challenges

Original: Federated Learning formulation .
Revised: Federated Learning formulation .

Original: We use the following notation for data representation: given90 a data source distribution D , { x i , y i } ∼ D represents a data point where x i is a data sample, and y i is its corresponding label (for classification purposes).
Revised: We use the following notation for data representation: given a data source distribution D , { x i , y i } ∼ D represents a data point where x i is a data sample, and y i is its corresponding label (for classification purposes).

Original: We use n to denote the total number of92 samples, unless otherwise stated.
Revised: We use n to denote the total number of samples, unless otherwise stated.

Original: Let S be the total number of clients in a distributed FL scenario.
Revised: Let S be the total number of clients in a distributed FL scenario.

Original: Each client i has its own local data D i such that the whole dataset satisfies D
Revised: Each client i has its own local data D i such that the whole dataset satisfies D

Original: = ∪ i D i , and usually94 D i ∩
Revised: = ∪ i D i , and usually D i ∩

Original: D j = ∅ , ∀ i ̸
Revised: D j = ∅ , ∀ i ̸

Original: = j .
Revised: = j .

Original: The goal of federated learning is to find a global model W that achieves good95 accuracy on all data D , by minimizing the following optimization problem:
Revised: The goal of federated learning is to find a global model W that achieves good accuracy on all data D , by minimizing the following optimization problem:

Original: [Equation]
Revised: [Equation]

Original:  D [Equation] where ℓ ( W , D i ) = 1 |D i | (cid:88) { x j ,y j }∈D i ℓ ( W , { x j , y j } )
Revised: i = ℓ ( W , D i ) (cid:41) , where ℓ ( W , D i ) = 1 |D i | (cid:88) { x j ,y j }∈D i ℓ ( W , { x j , y j } )

Original: With a slight abuse of notation, ℓ ( W , D i ) denotes the local loss function for user i , associated with alocal model W i (not indicated above), that gets aggregated with the models of other users.
Revised: With a slight abuse of notation, ℓ ( W , D i ) denotes the local loss function for user i , associated with a local model W i (not indicated above), that gets aggregated with the models of other users.

Original: Local datadistribution D i can be heterogeneous and follow a non-i.i.d.
Revised: Local data distribution D i can be heterogeneous and follow a non-i.i.d.

Original: distribution.
Revised: distribution.

Original: In the following sections,we consider both i.i.d.
Revised: In the following sections, we consider both i.i.d.

Original: case and non-i.i.d.
Revised: case and non-i.i.d.

Original: case.
Revised: case.

Revised: Algorithm

Original: Parameters : T iters, S clients, l local iters.
Revised: Parameters : T iters, S clients, l local iters.

Original: , W as current global model, W i as local model for i -th client, α ∈ (0 , 1) , η step size.
Revised: , W as current global model, W i as local model for i -th client, α ∈ (0 , 1) , η step size.

Original: ∞
Revised: ∞

Original: W ← randomly initialized global model.
Revised: W ← randomly initialized global model.

Original: //On
Revised: //On

Original: each client i asynchronously : for t = 0 , . .
Revised: each client i asynchronously : for t = 0 , . .

Original: , T − 1 do
Revised: , T − 1 do

Original: W i,t ← W //Train W i for l iters.
Revised: W i,t ← W //Train W i for l iters.

Original: via SGD for j = 1 , . . .
Revised: via SGD for j = 1 , . . .

Original: , l do
Revised: , l do

Original: W i,t ← W i,t − η ∂ L W
Revised: W i,t ← W i,t − η ∂ L W

Original: i,t end for //Write
Revised: i,t end for //Write

Original: local to global model
Revised: local to global model

Original: W ← (1 − α ) · W + α · W i,t end for
Revised: W ← (1 − α ) · W + α · W i,t end for

Original: Details of asynchronous training.
Revised: Details of asynchronous training.

Original: An abstractdescription of how asynchronous FL operates isprovided in Algorithm 1.
Revised: An abstract description of how asynchronous FL operates is provided in Algorithm 1.

Original: In particular, given anumber of server iterations T , each client i getsthe updated global model W t from the server,and further locally trains it using D i for a number of local iterations l . 2 Asynchronous federated learning assumes each client has differentcomputation power and communication bandwidth; this can be abstracted by different wallclock times required to finish a number of localtraining iterations.
Revised: In particular, given a number of server iterations T , each client i gets the updated global model W t from the server, and further locally trains it using D i for a number of local iterations l . 2 Asynchronous federated learning assumes each client has different computation power and communication bandwidth; this can be abstracted by different wallclock times required to finish a number of local training iterations.

Original: Thus, when client i has completed its round, the updated model is sharedwith the server to be aggregated, before the nextround of communication and computation startsfor client i .
Revised: Thus, when client i has completed its round, the updated model is shared with the server to be aggregated, before the next round of communication and computation starts for client i .

Original: This is different from classical synchronous federated learning, where the globalmodel is updated only when all participating clients finish certain local training iterations.
Revised: This is different from classical synchronous federated learning, where the global model is updated only when all participating clients finish certain local training iterations.

Original:  Asynchronous Distributed Dropout in the i.i.d. case
Revised:  Asynchronous Distributed Dropout in the i.i.d. case

Original: Intricacies of asynchronous FL.
Revised: Intricacies of asynchronous FL.

Original: Asynchronous motions often lead to inconsistent update schedulesof the global model and are characterized by gradient staleness and overfitting.
Revised: Asynchronous motions often lead to inconsistent update schedules of the global model and are characterized by gradient staleness and overfitting.

Original: While there are somerecent solutions [38, 3, 28, 18, 1, 13, 37], real-life FL applications include edge devices with limitedcommunication and computation capabilities that further exacerbate such issues(e.g., how often andfast they can connect with the central server, and how powerful as devices they are).
Revised: While there are some recent solutions [38, 3, 28, 18, 1, 13, 37], real-life FL applications include edge devices with limited communication and computation capabilities that further exacerbate such issues(e.g., how often and fast they can connect with the central server, and how powerful as devices they are).

Original: For instance,edge devices such as IoT devices or mobile phones [22], might only be able to communicate with theserver within short time windows, due to network conditions or user permission policy (e.g., a devicemight need to upload and download the updated model in a short time window).
Revised: For instance, edge devices such as IoT devices or mobile phones [22], might only be able to communicate with the server within short time windows, due to network conditions or user permission policy (e.g., a device might need to upload and download the updated model in a short time window).

Original: Such edge deviceshave limited computational capabilities, as compared to GPU-based systems.
Revised: Such edge devices have limited computational capabilities, as compared to GPU-based systems.

Original: Finally, existing semi-asynchronous methods [38, 3, 28, 18, 1, 13, 37] might require fast clients waituntil all other clients’ updates are completed, in order to receive the updated model for the next roundof local training.
Revised: Finally, existing semi-asynchronous methods [38, 3, 28, 18, 1, 13, 37] might require fast clients wait until all other clients’ updates are completed, in order to receive the updated model for the next round of local training.

Original: This might further result in nondeterministic long waiting time.(Distributed)
Revised: This might further result in nondeterministic long waiting time.

Original:  Dropout .
Revised: (Distributed) Dropout .

Original: Dropout [32, 29, 10, 4] is a widely-accepted regularization technique indeep learning.
Revised: Dropout [32, 29, 10, 4] is a widely-accepted regularization technique in deep learning.

Original: The procedure of Dropout is as follows: per training round, a random mask over theparameters is generated; this mask is used to nullify part of the neurons in the neural network forthis particular iteration.
Revised: The procedure of Dropout is as follows: per training round, a random mask over the parameters is generated; this mask is used to nullify part of the neurons in the neural network for this particular iteration.

Original: Variants of dropout include the drop-connect [32], multisample dropout [14],
Revised: Variants of dropout include the drop-connect [32], multisample dropout [14],

Original: Gaussian dropout [35], and the variational dropout [16].
Revised: Gaussian dropout [35], and the variational dropout [16].

Original: The idea of dropout has also been used in efficient distributed and/or federated learning scenarios.[11] introduces FjORD and the Ordered Dropout, a synchronous distributed dropout technique thatleads to ordered, nested representation of knowledge in models, and enables the extraction of lowerfootprint submodels without the need of retraining.
Revised: The idea of dropout has also been used in efficient distributed and/or federated learning scenarios. [11] introduces FjORD and the Ordered Dropout, a synchronous distributed dropout technique that leads to ordered, nested representation of knowledge in models, and enables the extraction of lower footprint submodels without the need of retraining.

Original: Such submodels are more suitable in client system heterogeneity, as they adapt submodel’s width to the client’s capabilities.
Revised: Such submodels are more suitable in client system heterogeneity, as they adapt submodel’s width to the client’s capabilities.

Original: Similar approaches include
Revised: Similar approaches include

Original: Nested Dropout [27] and HeteroFL [6].
Revised: Nested Dropout [27] and HeteroFL [6].

Original: Algorithm
Revised: Algorithm

Original: Parameters : T iters, S clients, l local iters.
Revised: Parameters : T iters, S clients, l local iters.

Original: , W as current global model, W i as local model for i -th client, α ∈ (0 , 1) , η step size.
Revised: , W as current global model, W i as local model for i -th client, α ∈ (0 , 1) , η step size.

Original: ∞
Revised: ∞

Original: W ← randomly initialized global model.
Revised: W ← randomly initialized global model.

Original: //On
Revised: //On

Original: each client i asynchronously : for t = 0 , . .
Revised: each client i asynchronously : for t = 0 , . .

Original: , T − 1 do
Revised: , T − 1 do

Original: Generate mask M i,t W i,t ← W t ⊙ M i,t //Train W
Revised: Generate mask M i,t W i,t ← W t ⊙ M i,t //Train W

Original: i,t for l iters.
Revised: i,t for l iters.

Original: via SGD for j = 1 , . . .
Revised: via SGD for j = 1 , . . .

Original: , l do
Revised: , l do

Original: W i,t ← W i,t − η ∂ L W
Revised: W i,t ← W i,t − η ∂ L W

Original: i,t end for //Write
Revised: i,t end for //Write

Original: local to global model
Revised: local to global model

Original: W [Equation] M [Equation]
Revised: W t +1 ← W t ⊙ ( M i,t ) c

Original: [Equation] end for
Revised: +((1 − α ) · W t + α · W i,t ) ⊙ M i,t end for

Original: Our proposal and main hypothesis .
Revised: Our proposal and main hypothesis .

Original: In this work, we focus on the asynchronous version of distributed dropout.
Revised: In this work, we focus on the asynchronous version of distributed dropout.

Original: We first study theoretically whether asynchrony provably works in non-trivial nonconvex scenarios – as in training convolutional neural networks – with randommasks that generate submodels for eachworker/participating device.
Revised: We first study theoretically whether asynchrony provably works in non-trivial nonconvex scenarios – as in training convolutional neural networks – with random masks that generate submodels for each worker/participating device.

Original: The algorithmwe focus on is described in Algorithm 2and is based on recent distributed protocols[40, 8, 19, 36].
Revised: The algorithm we focus on is described in Algorithmand is based on recent distributed protocols [40, 8, 19, 36].

Original: The main difference of Algorithm 2, compared to Algorithm 1, is thatthe former splits the model vertically periteration, where each submodel containsall layers of the neural network, but onlywith a (non-overlapping) subset of neuronsbeing active in each layer.
Revised: The main difference of Algorithm 2, compared to Algorithm 1, is that the former splits the model vertically per iteration, where each submodel contains all layers of the neural network, but only with a (non-overlapping) subset of neurons being active in each layer.

Original: Multiple local
Revised: Multiple local

Original: SGD steps can be performed without theworkers having to communicate.
Revised: SGD steps can be performed without the workers having to communicate.

Original: Theoretical Results .
Revised: Theoretical Results .

Original: We perform theoretical analysis on a one-hidden-layer convolutional neuralnetwork (CNN), and show the convergence with random filter dropout.
Revised: We perform theoretical analysis on a one-hidden-layer convolutional neural network (CNN), and show the convergence with random filter dropout.

Original: Consider a training dataset( X , y )
Revised: Consider a training dataset ( X , y )

Original: = { ( x i , y i ) } ni
Revised: = { ( x i , y i ) } ni

Original: =1 , where each x i ∈
Revised: =1 , where each x i ∈

Original: R ˆ d × p is an image and y i being its label.
Revised: R ˆ d × p is an image and y i being its label.

Original: Here, ˆ d is thenumber of input channels and p the number of pixels.
Revised: Here, ˆ d is the number of input channels and p the number of pixels.

Original: Let q denote the size of the filter, and let m bethe number of filters in the first layer.
Revised: Let q denote the size of the filter, and let m be the number of filters in the first layer.

Original: We let ˆ ϕ ( · ) denote the patching operator with ˆ ϕ ( x ) ∈ R q ˆ d × p[7].
Revised: We let ˆ ϕ ( · ) denote the patching operator with ˆ ϕ ( x ) ∈ R q ˆ d × p [7].

Original: Consider the first layer weight W ∈ R m × q ˆ d , and second layer (aggregation) weight a ∈ R m × p .
Revised: Consider the first layer weight W ∈ R m × q ˆ d , and second layer (aggregation) weight a ∈ R m × p .

Original: We assume that only the first layer weights W is trainable.
Revised: We assume that only the first layer weights W is trainable.

Original: The CNN model, trained on the meanssquared error, has the form:
Revised: The CNN model, trained on the means squared error, has the form:

Original: [Equation]
Revised: [Equation]

Original: where ζ abstractly represents all training parameters, f ( x , · ) denotes the output of the one-layer CNNfor input x , and L ( · ) is the loss function.
Revised: where ζ abstractly represents all training parameters, f ( x , · ) denotes the output of the one-layer CNN for input x , and L ( · ) is the loss function.

Original: Assumption 3.1 (Training Data) Assume that for all data points, we have ∥ x i ∥ F = q − 12 and| y i | ≤ C for some constant C .
Revised: Assumption 3.1 (Training Data) Assume that for all data points, we have ∥ x i ∥ F = q − 12 and | y i | ≤ C for some constant C .

Original: Moreover, for all i, i ′ we have x i ̸
Revised: Moreover, for all i, i ′ we have x i ̸

Original: = x i ′ .
Revised: = x i ′ .

Original: This assumption can be easily satisfied with data normalization.
Revised: This assumption can be easily satisfied with data normalization.

Original: For simplicity, let d :=
Revised: For simplicity, let d :=

Original: q ˆ d .
Revised: q ˆ d .

Original: Assumption 3.2 (Initialization) w 0 ,r ∼ N (cid:0) 0 , κ 2 I (cid:1) and a rj ∼ (cid:110) ± 1 p √ m (cid:111) for r ∈ [ m ] and j ∈ [ p ] .
Revised: Assumption 3.2 (Initialization) w 0 ,r ∼ N (cid:0) 0 , κ 2 I (cid:1) and a rj ∼ (cid:110) ± 1 p √ m (cid:111) for r ∈ [ m ] and j ∈ [ p ] .

Original: In an asynchronous scenario, we assume the neural network weights are updated with stale gradients:here, d t denotes the delay at training step t .
Revised: In an asynchronous scenario, we assume the neural network weights are updated with stale gradients: here, d t denotes the delay at training step t .

Original: Further, for all t iterations, we assume d t is bounded by aconstant E .
Revised: Further, for all t iterations, we assume d t is bounded by a constant E .

Original: Then, gradient descent motions with delays take the form:
Revised: Then, gradient descent motions with delays take the form:

Original: [Equation]
Revised: [Equation]

Revised: η

Revised: [Equation]

Revised: W t

Revised: [Equation]

Original: W t − d t ) , where d t ≤ E.
Revised: d t [Equation] , where d t ≤ E.

Original: Based on the above we make the following conjecture:
Revised: Based on the above we make the following conjecture:

Original: Conjecture 3.1 Let f ( · , · ) be a one-hidden-layer CNN with the second layer weight fixed.
Revised: Theorem 3.1 Let f ( · , · ) be a one-hidden-layer CNN with the second layer weight fixed.

Revised: Let u t abstractly represent the output of the model after t iterations, over the random selection of the masks.

Revised: Let ξ denote the dropout rate ( ξ = 1 dictates that all neurons are selected), and denote θ = 1 − (1 − ξ ) S the probability that a neuron is active in at least one subnetwork.

Original: Assumethe number of hidden neurons satisfies m = Ω (cid:16) max { n 4 K 2 λ 40 δ 2 max { n, d } , nλ 0 } (cid:17) and the step sizesatisfies η = O (cid:0) λ 0 n 2 (cid:1) .
Revised: Assume the number of hidden neurons satisfies m = Ω (cid:16) max { n 4 K 2 λ 40 δ 2 max { n, d } , nλ 0 } (cid:17) and the step size satisfies η = O (cid:0) λ 0 n 2 (cid:1) .

Revised: Let κ be a proper initialization scaling factor, and it is considered constant.

Revised: We use λto denote the smallest eigenvalue of the Neural Tangent Kernel matrix.

Original: Let Assumptions 1 and 2 be satisfied.
Revised: Let Assumptions 1 and 2 be satisfied.

Original: Then, we conjecture that:
Revised: Then, the following convergence rate guarantee is proved to be satisfied:

Original: [Equation]
Revised: [Equation]

Original:  Smart Partition in Asynchronous Dropout for FL
Revised:  Smart Partition in Asynchronous Dropout for FL

Original: Potential issues in asynchronous FL .
Revised: Potential issues in asynchronous FL .

Original: Despite theoretical support, asynchronous algorithms mightshow slower convergence rates and/or lower final model accuracies, mainly due to inconsistent globalmodel updates between slower devices and faster devices.
Revised: Despite theoretical support, asynchronous algorithms might show slower convergence rates and/or lower final model accuracies, mainly due to inconsistent global model updates between slower devices and faster devices.

Original: With non-i.i.d.
Revised: With non-i.i.d.

Original: data
Revised: data

Original: as in the case of FL,more frequent updates by faster devices could lead to global model overfitting over local data on thesame devices; this often results in significantly sub-optimal final model accuracy.
Revised: as in the case of FL, more frequent updates by faster devices could lead to global model overfitting over local data on the same devices; this often results in significantly sub-optimal final model accuracy.

Original: These facts suggesta more careful handling of model splitting and model distribution among heterogeneous workers.
Revised: These facts suggest a more careful handling of model splitting and model distribution among heterogeneous workers.

Revised: Algorithm 3 Smart Dropout for Asynchronous FL

Original: Parameters : T iters, S clients, l local iters.
Revised: Parameters : T iters, S clients, l local iters.

Original: , W as current global model, W i as local model for ith client, η g as global LR, q ( · ) weight score function, ψ ( i ) computes the computation capacity of ith worker, φ ( W , ψ ( i ) , q ( · )) is the Smart Dropout function that creates the mask, based on worker capacity ψ and score function q , α ∈ (0 , 1) .
Revised: , W as current global model, W i as local model for ith client, η g as global LR, q ( · ) weight score function, ψ ( i ) computes the computation capacity of ith worker, φ ( W , ψ ( i ) , q ( · )) is the Smart Dropout function that creates the mask, based on worker capacity ψ and score function q , α ∈ (0 , 1) .

Original: ∞
Revised: ∞

Original: W ← randomly initialized global model.
Revised: W ← randomly initialized global model.

Original: //On
Revised: //On

Original: each client i asynchronously : for t = 0 , . .
Revised: each client i asynchronously : for t = 0 , . .

Original: , T − 1 do //For
Revised: , T − 1 do //For

Original: i -th fastest worker, φ ( · ) drops weights with i -th largest q ( · ) score
Revised: i -th fastest worker, φ ( · ) drops weights with i -th largest q ( · ) score

Original: Generate mask M i,t = φ ( W t , ψ ( i ) , q ( · ))
Revised: Generate mask M i,t = φ ( W t , ψ ( i ) , q ( · ))

Original: W i,t ← W t ⊙ M i,t //Train W
Revised: W i,t ← W t ⊙ M i,t //Train W

Original: i,t for l iters.
Revised: i,t for l iters.

Original: via SGD for j = 1 , . . .
Revised: via SGD for j = 1 , . . .

Original: , l do
Revised: , l do

Original: W i,t ← W i,t − η g ∂ L W
Revised: W i,t ← W i,t − η g ∂ L W

Original: i,t end for if i -th client is fastest then
Revised: i,t end for if i -th client is fastest then

Original: Update η g end if //Write
Revised: Update η g end if //Write

Original: local to global model
Revised: local to global model

Original: W t +1 ← W t ⊙ ( M [Equation]
Revised: W t +1 ← W t ⊙ ( M i,t ) c

Original: [Equation]

Original: [Equation]

Original: +((

Original: [Equation] ) · W [Equation] + α · W i,t ) ⊙ M i,t //Update score q q ( W jt +1 ) 
Revised: +((1 − α ) · W t + α · W i,t ) ⊙ M i,t //Update score q q ( W jt +1 ) =

Original: = (cid:13)(cid:13)(cid:13) [Table] [Table]
Revised: (cid:13)(cid:13)(cid:13) W jt +1 − W j 0 (cid:13)(cid:13)(cid:13) 1 , ∀ j ∈ J end for

Original: Smart Dropout : improved asynchronous solution .
Revised: Smart Dropout : improved asynchronous solution .

Original: We propose to resolveinconsistency in global model updates by“balancing" the contribution from differentdevices with a Smart Dropout method.
Revised: We propose to resolve inconsistency in global model updates by “balancing" the contribution from different devices with a Smart Dropout method.

Original: Briefly, Smart Dropout assigns differentweights to different devices, based onthe updated rate of the weights, and thecomputation power of the devices.
Revised: Briefly, Smart Dropout assigns different weights to different devices, based on the updated rate of the weights, and the computation power of the devices.

Original: An algorithmic description is provided in
Revised: An algorithmic description is provided in

Original: Algorithm 3.
Revised: Algorithm 3.

Original: We assign model weights thatare less updated to the faster devices; thisway, we guarantees that such weights willbe updated more frequently than randomweight assignment.
Revised: We assign model weights that are less updated to the faster devices; this way, we guarantees that such weights will be updated more frequently than random weight assignment.

Original: Similarly, we assignmodel weights that are updated more often– and, thus, converging faster– to the slowerdevices.
Revised: Similarly, we assign model weights that are updated more often – and, thus, converging faster– to the slower devices.

Original: The premise behind such a protocol is that all weights, eventually, will beupdated/will converge with a similar rate.
Revised: The premise behind such a protocol is that all weights, eventually, will be updated/will converge with a similar rate.

Original: In the i.i.d.
Revised: In the i.i.d.

Original: case, this would reduce the gradient staleness: slower devices take fewersteps to converge on the assigned weights;this is compensated by assigning “fastconverging weights”, that were assignedto faster devices in the previous iterations.
Revised: case, this would reduce the gradient staleness: slower devices take fewer steps to converge on the assigned weights; this is compensated by assigning “fastconverging weights”, that were assigned to faster devices in the previous iterations.

Original: In the non-i.i.d.
Revised: In the non-i.i.d.

Original: case, such a strategy wouldreduce any overfitting over the local datain faster devices: The weights previouslyupdated the most by the faster devices willnot be assigned to the fast devices in thenear-future iterations.
Revised: case, such a strategy would reduce any overfitting over the local data in faster devices: The weights previously updated the most by the faster devices will not be assigned to the fast devices in the near-future iterations.

Original: Smart Dropout : its ingredients .
Revised: Smart Dropout : its ingredients .

Original: The above are encapsulated with a weight score function q ( · ) ,a device-capacity score function ψ ( · ) and the mask generator function φ ( · ) in Algorithm 3.
Revised: The above are encapsulated with a weight score function q ( · ) , a device-capacity score function ψ ( · ) and the mask generator function φ ( · ) in Algorithm 3.

Original: Thefunction q ( · ) quantifies the update speed of each weight.
Revised: The function q ( · ) quantifies the update speed of each weight.

Original: One simple score can be the norm ofweight change: Given predefined grouping of weights in set J , define W jt for j ∈ J as the j -thweight/group of weights of global network at the t -th update.
Revised: One simple score can be the norm of weight change: Given predefined grouping of weights in set J , define W jt for j ∈ J as the j -th weight/group of weights of global network at the t -th update.

Original: Then, the score function is selected as:q ( W jt )
Revised: Then, the score function is selected as: q ( W jt )

Original: = (cid:13)(cid:13)(cid:13) W
Revised: = (cid:13)(cid:13)(cid:13) W

Original: jt − W j 0 (cid:13)(cid:13)(cid:13) 1 , i.e., the score function measures how far from the initial values the j -thgroup of weights has moved.
Revised: jt − W j 0 (cid:13)(cid:13)(cid:13) 1 , i.e., the score function measures how far from the initial values the j -th group of weights has moved.

Original: In our experiments, we have tested grouping of weights J by filters andby layers.
Revised: In our experiments, we have tested grouping of weights J by filters and by layers.

Original: Further, we have some measure of the computation capacity of each device ψ ( i ) , and orderthe devices in descending order such that the first device is the fastest and last the slowest.
Revised: Further, we have some measure of the computation capacity of each device ψ ( i ) , and order the devices in descending order such that the first device is the fastest and last the slowest.

Original: The mask generator function φ ( W , ψ ( i ) , q ( · )) generates masks that nullify weights in W , basedon the score function q over the weights, and the device capacity list in ψ ( · ) .
Revised: The mask generator function φ ( W , ψ ( i ) , q ( · )) generates masks that nullify weights in W , based on the score function q over the weights, and the device capacity list in ψ ( · ) .

Original: The Smart Dropoutstrategy is that for the i -th fastest worker, we drop weights with i -th largest q score.
Revised: The Smart Dropout strategy is that for the i -th fastest worker, we drop weights with i -th largest q score.

Original:  Experiments
Revised:  Experiments

Original: We focus on fully asynchronous FL using CNNs.
Revised: We focus on fully asynchronous FL using CNNs.

Original: Yet, we highlight that, with proper tuning, our ideasare model agnostic by design.
Revised: Yet, we highlight that, with proper tuning, our ideas are model agnostic by design.

Original: To simulate the heterogeneous edge devices, we implement devices as different processes, while we manually insert different time delays to simulate different computenodes and communication bandwidths.
Revised: To simulate the heterogeneous edge devices, we implement devices as different processes, while we manually insert different time delays to simulate different compute nodes and communication bandwidths.

Original: In these preliminary results, we focus only on non-i.i.d.scenarios.
Revised: In these preliminary results, we focus only on non-i.i.d. scenarios.

Original: We consider the difficult case where data distribution will be correlated to computationpower, which results in a more heterogeneous model update during training.
Revised: We consider the difficult case where data distribution will be correlated to computation power, which results in a more heterogeneous model update during training.

Original: In all experiments,we use 104 devices/users in total and activate 8 of them at any moment.
Revised: In all experiments, we use 104 devices/users in total and activate 8 of them at any moment.

Original: In all Smart Dropoutexperiments, we use 25% dropout rate.
Revised: In all Smart Dropout experiments, we use 25% dropout rate.

Original: Baselines .
Revised: Baselines .

Original: We compare asynchronous implementations of five algorithms: i ) asynchronous FedAvg;ii ) asynchronous FedAvg with weighted aggregation based on gradient staleness; iii ) asynchronous
Revised: We compare asynchronous implementations of five algorithms: i ) asynchronous FedAvg; ii ) asynchronous FedAvg with weighted aggregation based on gradient staleness; iii ) asynchronous

Original: FjORD; iv ) asynchronous FedProx; and iv ) FedBuff.
Revised: FjORD; iv ) asynchronous FedProx; and iv ) FedBuff.

Original: These baselines are straightforward implementations of synchronous versions (except from FedBuff).
Revised: These baselines are straightforward implementations of synchronous versions (except from FedBuff).

Original: Preliminary results .
Revised: Preliminary results .

Original: We only highlight key points in our preliminary results; more findings andfurther discussion will be included during the workshop.
Revised: We only highlight key points in our preliminary results; more findings and further discussion will be included during the workshop.

Original: Focusing on the non-i.i.d.
Revised: Focusing on the non-i.i.d.

Original: case for CIFAR10(Table 1), we observe that our proposal gets the maximum accuracy compared to the rest of thealgorithms, while being stable: notice the high variability in values coming from different runs of ourcode (indicated with red colored text ).
Revised: case for CIFAR (Table 1), we observe that our proposal gets the maximum accuracy compared to the rest of the algorithms, while being stable: notice the high variability in values coming from different runs of our code (indicated with red colored text ).

Original: At the same time, our algorithm is faster and requires lesscommunication over the course of the execution.
Revised: At the same time, our algorithm is faster and requires less communication over the course of the execution.

Original: Similar observations can be made for the case of non-i.i.d.
Revised: Similar observations can be made for the case of non-i.i.d.

Original: CIFAR100 (Table 2).
Revised: CIFAR100 (Table 2).

Original: Here, observethat FedBuff achieves the best maximum accuracy.
Revised: Here, observe that FedBuff achieves the best maximum accuracy.

Original: However, again the achieved models are notconsistent over various runs of the algorithm; our algorithm shows a stable performance, withinreach of the high variance by FedBuff.
Revised: However, again the achieved models are not consistent over various runs of the algorithm; our algorithm shows a stable performance, within reach of the high variance by FedBuff.

Original: At the same time, our algorithms shows great performance interms of acceleration in execution and required communication bandwidth.
Revised: At the same time, our algorithms shows great performance in terms of acceleration in execution and required communication bandwidth.

Original: This behavior is furtherhighlighted in Figure 1.
Revised: This behavior is further highlighted in Figure 1.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]