Original: Cross-device Federated Architecture Search
Revised: Cross-device Federated Architecture Search

Original: Abstract
Revised: Abstract

Original: Federated learning (FL) has recently gained considerable attention due to its abilityto learn on decentralised data while preserving client privacy.
Revised: Federated learning (FL) has recently gained considerable attention due to its ability to learn on decentralised data while preserving client privacy.

Original: However, it alsoposes additional challenges related to the heterogeneity of the participating devices,both in terms of their computational capabilities and contributed data.
Revised: However, it also poses additional challenges related to the heterogeneity of the participating devices, both in terms of their computational capabilities and contributed data.

Original: Meanwhile,
Revised: Meanwhile,

Original: Neural Architecture Search (NAS) has been successfully used with centraliseddatasets, producing state-of-the-art results in constrained or unconstrained settings.
Revised: Neural Architecture Search (NAS) has been successfully used with centralised datasets, producing state-of-the-art results in constrained or unconstrained settings.

Original: However, such centralised datasets may not be always available.
Revised: However, such centralised datasets may not be always available for training.

Original: Most recentwork at the intersection of NAS and FL attempts to alleviate this issue in a crosssilo federated setting, which assumes homogeneous compute environment withdatacenter-grade hardware.
Revised: Most recent work at the intersection of NAS and FL attempts to alleviate this issue in a cross-silo federated setting, which assumes homogeneous compute environments with datacenter-grade hardware.

Original: In this paper we explore the question of whether wecan design architectures of different footprints in a cross-device federated setting,where the device landscape, availability and scale are very different.
Revised: In this paper we explore the question of whether we can design architectures of different footprints in a cross-device federated setting, where the device landscape, availability and scale are very different.

Original: To this end,we design our system, FedorAS , to discover and train promising architecturesin a resource-aware manner when dealing with devices of varying capabilitiesholding non-IID distributed data.
Revised: To this end, we design our system, FedorAS , to discover and train promising architectures in a resource-aware manner when dealing with devices of varying capabilities holding non-IID distributed data.

Original: We present empirical evidence of its effectivenessacross different settings, spanning across three different modalities (vision, speech,text), and showcase its better performance compared to state-of-the-art federatedsolutions, while maintaining resource efficiency.
Revised: We present empirical evidence of its effectiveness across different settings, spanning across three different modalities (vision, speech, text), and showcase its better performance compared to state-of-the-art federated solutions, while maintaining resource efficiency.

Original:  Introduction
Revised:  Introduction

Original: As smart devices become omnipresent where we live, work and socialise, the ML-powered servicesthat these provide grow in sophistication.
Revised: As smart devices become omnipresent where we live, work and socialise the ML-powered services that these provide grow in sophistication.

Original: This ambient intelligence has undoubtedly been sustainedby recent advances in Deep Learning (DL) across a multitude of tasks and modalities.
Revised: This ambient intelligence has undoubtedly been sustained by recent advances in Deep Learning (DL) across a multitude of tasks and modalities.

Original: Parallelto this race for state-of-the-art performance in various in DL benchmarks, mobile and embeddeddevices also became more capable to accommodate new DNN designs [31], some even integratingaccelerators to their SoCs (e g. NPUs) to efficiently run DL workloads [2].
Revised: Parallel to this race for state-of-the-art performance in various in DL benchmarks, mobile and embedded devices also became more capable to accommodate new Deep Neural Network (DNN) designs [36], some even integrating specialised accelerators to their System-On-Chips (SoC) (e g. NPUs) to efficiently run DL workloads [3].

Original: These often come invarious configurations in terms of their compute/memory capabilities and power envelopes [3] andco-exist in the wild as a rich multi-generational ecosystem ( system heterogeneity ) [63].
Revised: These often come in various configurations in terms of their compute/memory capabilities and power envelopes [4] and co-exist in the wild as a rich multi-generational ecosystem ( system heterogeneity ) [75].

Original: These devicesbring intelligence through users’ interactions, also innately heterogeneous amongst them, leading tonon-independent or identically distributed (non-IID) data in the wild ( data heterogeneity ).
Revised: These devices bring intelligence through users’ interactions, also innately heterogeneous amongst them, leading to non-independent or identically distributed (non-IID) data in the wild ( data heterogeneity ).

Original: Powered by the recent advances in SoCs’ capabilities and motivated by privacy concerns [59] over thecustody of data, Federated Learning (FL) [48] has emerged as a way of training on-device on user datawithout it ever directly leaving the device premises.
Revised: Powered by the recent advances in SoCs’ capabilities and motivated by privacy concerns [71] over the custody of data, Federated Learning (FL) [57] has emerged as a way of training on-device on user data without it ever directly leaving the device premises.

Original: However, FL training has largely been focusedon the weights of a static global model architecture, assumed to be runnable by every participatingclient [34].
Revised: However, FL training has largely been focused on the weights of a static global model architecture, assumed to be runnable by every participating client [39].

Original: Not only may this not be the case, but italso can  lead to subpar performance of the overalltraining process in the presence of stragglers or biases in the case of consistently dropping certainlow-powered devices.
Revised: Not only may this not be the case, but it can also lead to subpar performance of the overall training process in the presence of stragglers or biases in the case of consistently dropping certain low-powered devices.

Original: On the opposite end, more capable devices might not fully take advantage oftheir data if the deployed model is of reduced capacity to ensure all devices can participate [44].
Revised: On the opposite end, more capable devices might not fully take advantage of their data if the deployed model is of reduced capacity to ensure all devices can participate [51].

Original: Parallel to these trends, Neural architecture search (NAS) has become the de facto mechanism toautomate the design on DNNs that can meet the requirements (e.g. latency, model size) for these torun on resource-constrained devices.
Revised: Parallel to these trends, Neural Architecture Search (NAS) has become the de facto mechanism to automate the design of DNNs that can meet the requirements (e.g. latency, model size) for these to run on resource-constrained devices.

Original: The success of NAS can be partly attributed to the fact that theseframeworks are commonly run in datacenters, where high-performing hardware and/or large curated [Figure]
Revised: The success of NAS can be partly attributed to the fact that these frameworks are commonly run in datacenters, where high-performing hardware and/or large curated datasets [42, 24, 21, 38, 60, 60] are available.

Original: datasets [37, 20, 17, 33, 50, 50] are available. However, this also imposes two major limitations on thecurrent NAS approaches: i) privacy , i.e. these methods were often not designed to work in situationswhen user’s data must remain on-device; and, consequently, ii) tail data non-discoverability , i.e.they might never be exposed to infrequent or time/user-specific data that exist in the wild but notnecessarily in centralized datasets.
Revised: However, this also imposes two major limitations on current NAS approaches: i) privacy , i.e. these methods were often not designed to work in situations when user’s data must remain on-device; and, consequently, ii) tail data non-discoverability , i.e. they might never be exposed to infrequent or time/user-specific data that exist in the wild but not necessarily in centralized datasets.

Original: On top of these, the whole cost is born by the provider and separateon-device modelling/profiling needs to be done in the case of hardware-aware NAS [21, 58, 39],which has mainly focused on inference performance hitherto.
Revised: On top of these, the whole cost is born by the provider and separate on-device modelling/profiling needs to be done in the case of hardware-aware NAS [26, 70, 44], which has mainly focused on inference performance hitherto.

Original: Motivated by the aforementioned phenomena and limitations of the existing NAS methods, wepropose FedorAS , a system that performs NAS over heterogeneous devices holding heterogeneousdata in a resource-aware and federated manner.
Revised: Motivated by the aforementioned phenomena and limitations of the existing NAS methods, we propose FedorAS , a system that performs NAS over heterogeneous devices holding heterogeneous data in a resource-aware and federated manner.

Original: To this direction, we cluster clients in tiers basedon their capabilities and design supernet comprisingof operations covering the whole spectrum ofcompute complexities.
Revised: To this direction, we cluster clients into tiers based on their capabilities and design supernet comprising operations covering the whole spectrum of compute complexities.

Original: This supernet acts both as search space and a weight-sharing backbone.
Revised: This supernet acts both as search space and a weight-sharing backbone.

Original: Uponfederation, the supernet is only partially and stochastically shared to clients, respecting their computational and bandwidth capabilities.
Revised: Upon federation, it is only partially and stochastically shared to clients, respecting their computational and bandwidth capabilities.

Original: In turn, we leverage resource-aware one-shot path sampling[23] to facilitate lightweight on-device NAS.
Revised: In turn, we leverage resource-aware one-shot path sampling [28] and adapt it to facilitate lightweight on-device NAS.

Original: This way, networks in a given search space are not onlydeployed in a resource-aware manner, but also trained as such, by tuning the downstream communication (i.e. the subspace explored by each client) and computation (i.e. FLOPs of sampled paths) tomeet the device’s training budget.
Revised: In this way, networks in a given search space are not only deployed in a resource-aware manner, but also trained as such, by tuning the downstream communication (i.e. the subspace explored by each client) and computation (i.e. FLOPs of sampled paths) to meet the device’s training budget.

Original: Once federated training of the supernetwork has completed, usablepretrained networks can be extracted even before performing fine-tuning or personalising per device,thus minimising the number of extra on-device training rounds to achieve competitive performance.
Revised: Once federated training of the supernetwork has completed, usable pretrained networks can be extracted even before performing fine-tuning or personalising per device, thus minimising the number of extra on-device training rounds to achieve competitive performance.

Original: In summary, in this work we make the following contributions
Revised: In summary, in this work we make the following contributions:

Original: • We propose a system for resource efficient federated NAS that can be applied in cross-device settings, where partial device participation, device and data heterogeneity are innate characteristics.•
Revised: • We propose a system for resource efficient federated NAS that can be applied in cross-device settings, where partial device participation, device and data heterogeneity are innate characteristics.

Original:  We implement a system called FedorAS (Federated nAS) that leverages a server-residentsupernet enabling weight sharing for efficient knowledge exchange between clients, withoutdirectly sharing common model architectures with one another.•
Revised: • We implement a system called FedorAS (Federated nAS) that leverages a server-resident supernet enabling weight sharing for efficient knowledge exchange between clients, without directly sharing common model architectures with one another.

Original:  We propose a novel aggregation method named OPA (OPerator Aggregation) for weighingupdates from multiple “single-path one-shot” client updates in a frequency-aware manner.• We empirically evaluate the performance and convergence of our system under IID and non-IIDsettings across different datasets, tasks and modalities, spanning different device distributionsand compare our system’s behaviour against state-of-the-art FL techniques.  
Revised: • We propose a novel aggregation method named OPA (OPerator Aggregation) for weighing updates from multiple “single-path one-shot” client updates in a frequency-aware manner. • We empirically evaluate the performance and convergence of our system under IID and non-IID settings across different datasets, tasks and modalities, spanning different device distributions and compare our system’s behaviour against state-of-the-art FL techniques.  Background & Motivation Federated Learning.

Revised: A typical FL pipeline is comprised of three distinct stages: given a global model initialised on a central server, ω ( t =0)

Original: The FedorAS Framework
Revised: g , i) the server randomly samples k clients out of the available K ( k ≪ K for cross-device ; [11] k = K for cross-silo setting) and sends them the current state of the global model; ii) those k clients perform training on-device using their own data partition, D i , for a number of epochs and send the updated models, ω ( t ) i , back to the server after local training is completed; finally, iii) the server aggregates these models and a new global model, ω ( t +1) g , is obtained. This aggregation can be implemented in different ways [57, 67, 52]. For example, in FedAvg [57] each update is weighted by the relative amount of data on each client: ω ( t +1)

Revised: g = (cid:80) ki =

Revised: | D i |

Revised: [Table]

Revised: Stages ii) and iii) repeat until convergence.

Revised: The quality of the global model ω g can be assessed: on the global test set; by evaluating the fit of the ω g to each participating client’s data ( D i ) and derive fairness metrics [53]; or, by evaluating the adaptability of the ω g to each client’s data or new data these might generate over time, this is commonly referred to as personalised FL [27, 50].

Revised: Contrary to traditional distributed learning, cross-device FL performs the bulk of the compute on a highly heterogeneous [39] set of devices in terms of their compute capabilities, availability and data

Revised: [Figure]

Revised: distribution.

Revised: In such scenarios a trade-off between model capacity and client participation arises: larger architectures might result in more accurate models which may only be trained on a fraction of the available devices; on the other hand, deploying smaller footprint networks could target more devices – and thus more data – for training, but these might be of inferior quality (gap in Fig. 1)

Revised: Neural Architecture Search.

Revised: NAS is usually defined as a bi-level optimisation problem:

Revised: [Equation]

Revised: where A is a finite (discrete) set of architectures to search from (a search space ), L is a loss function, ω a are weights of a model with architecture a , and D { v,t } are validation and training datasets, respectively.

Revised: The main challenge of NAS comes directly from the fact that in order to assess quality of different architectures ( outer optimisation ), we have to obtain their optimal weights which entitles conducting full training ( inner optimisation ).

Revised: There exist multiple approaches to speed up NAS [62, 55, 25, 17, 26, 80, 1, 56, 68, 28, 58].

Revised: More relevant to our work are those utilising the concept of a supernet [16, 8]; where a single model that encapsulates all possible architectures from the search space is created and trained.

Revised: Specifically, a supernet is constructed by defining an operator that incorporates all candidate operations (the set of which we denote by O l ) for each searchable layer l .

Revised: A common choice is to define it as a weighted sum of candidates’ individual outputs y l

Revised: = (cid:80) o ∈

Revised: O l α ( o ) l ∗ o ( y l − 1 ) , where factors { α ( o ) l } o ∈ O l of each layer l can be defined differently for different methods (e.g, continuous parameters [55, 17, 25] or random one-hot vectors [28]).

Revised: Importantly for us, methods that use sparse weighting factors can avoid executing operations associated with zero weights, saving both memory and compute [17, 28].

Revised: After a supernet has been constructed and trained, searching for a ⋆ is usually performed by either investigating architectural parameters [55, 25, 17], or using zero-th order optimisation methods to directly solve the outer loop of Eq.

Revised: 1 while approximating ω ∗ a with weights taken from the supernet (thus avoiding the costly inner optimisation)

Revised: The final model can then be retrained in isolation using either random initialisation or weights from the supernet as a starting point.

Revised: Challenges of Federated NAS.

Revised: As highlighted before, devices in the wild exhibit different compute capabilities and can hold non-IID distributed data, resulting in system and data heterogeneity.

Revised: In the context of NAS, system heterogeneity has a particularly significant effects, as we might no longer be able to guarantee that any model from our search space can be efficiently trained on all devices.

Revised: This inability can be attributed to insufficient compute power, limited network bandwidth or unavailability of the client at hand.

Revised: Consequently, some of the models might be deemed worse than others not because of their worse ability to generalise, but because they might not be exposed to the same subsets of data as others – as shown in Fig.

Revised: 1, where we show models of different footprint trained across client of varying capabilities under constrained ( tier-aware ) and full ( tier-unaware ) participation.

Original: FedorAS is a resource-aware Federated NAS framework that combines the best of both worlds:learning from clients across all tiers and yielding models tailored to each tier that benefit from thiscollective knowledge. 
Revised: FedorAS is a resource-aware Federated NAS framework that combines the best of both worlds: learning from clients across all tiers and yielding models tailored to each tier that benefit from this collective knowledge. [Figure]

Original: Workflow.
Revised: Workflow.

Original: FedorAS ’s workflow consists of three stages (Fig. 1): (i) supernet training , (ii) modelsearch and validation and (iii) model fine-tuning .
Revised: FedorAS ’ workflow consists of three stages (Fig. 2): i) supernet training , ii) model search and validation and iii) model fine-tuning .

Original: First, we train the supernet in a resource-aware andfederated manner ( Stage 1 , Sec. 2.1).
Revised: First, we train the supernet in a resource-aware and federated manner ( Stage 1 , Sec. 3.1).

Original: We then search for models from the supernet with the goal offinding the best architecture per tier ( Stage 2 , Sec. 2.2).
Revised: We then search for models from the supernet with the goal of finding the best architecture per tier ( Stage 2 , Sec. 3.2).

Original: Models are effectively sampled, validatedon a global validation set and ranked per tier.
Revised: Models are effectively sampled, validated on a global validation set and ranked per tier.

Original: These architectures and their associated weights actas initialisation to the next phase, where each model is fine-tuned in a per-tier manner ( Stage 3 .
Revised: These architectures and their associated weights act as initialisation to the next phase, where each model is fine-tuned in a per-tier manner ( Stage 3 .

Original: Sec. 2.3).
Revised: Sec. 3.3).

Original: The end goal of our system is to have the best possible model per each cluster of devices.
Revised: The end goal of our system is to have the best possible model per each cluster of devices.

Original: Design rationale.
Revised: Design rationale.

Original: We build our system around the concept of a supernet to facilitate weight-sharingbetween architectures of various footprints.
Revised: We build our system around the concept of a supernet to facilitate weight-sharing between architectures of various footprints.

Original: Operations in the supernet are samplable from paths(i.e. models) of different footprint.
Revised: Operations in the supernet are samplable from paths (i.e. models) of different footprint.

Original: As such, while normally large models would not be directlytrained on data of low-tier clients, our design allows for indirectly sharing this knowledge through theassociation of the same operation to different paths.
Revised: As such, while normally large models would not be directly trained on data of low-tier clients, our design allows for indirectly sharing this knowledge through the association of the same operation to different paths.

Original: In order to ensure efficient training of a supernet,we chose to base our approach on SPOS [23] as its training procedure is one of the most lightweight,introducing very little overhead in terms of memory consumption and/or required floating-pointoperations, especially compared to [47].
Revised: To ensure efficient training of a supernet, we chose to base our approach on SPOS [28] and adapt it (see Eq. 4) since its training procedure is lightweight, introducing very little overhead in terms of memory consumption and/or required floating-point operations, especially compared to [55].

Original: Last, we opted for clustering devices into “tiers” accordingto their computational capabilities and search for an architecture for each tier as a balance betweenhaving one architecture to fit all needs [25] andhaving a different architecture per client [49].
Revised: Last, we opted for clustering devices into “tiers” based on their computational capabilities and search for an architecture for each tier as a balance between having one model to fit all needs [30] and a different architecture per client [59].

Original: 2.1 SuperNet Training
Revised: 3.1 SuperNet Training

Original: 0 Search space & models.
Revised: 0 Search space & models.

Original: First, we define the search space in terms of a range of operators optionsper layer in the network that are valid choices to form a network.
Revised: First, we define the search space in terms of a range of operators options per layer in the network that are valid choices to form a network.

Original: This search space resides as a wholeon the server and is only partially communicated to participating clients of a training round to keepcommunication overheads minimal.
Revised: This search space resides as a whole on the server and is only partially communicated to participating clients of a training round to keep communication overheads minimal.

Original: Specific models (i.e. paths) consist of selections of one operatoroption per layer, sampled in a device local iteration.1 Subspace sampling.
Revised: Specific models (i.e. paths) consist of selections of one operator option per layer, sampled in a single-path-one-shot manner on-device per local iteration. 1 Subspace sampling.

Original: It is obvious that communicating the whole space of operators along withthe associated weights to each device becomes quickly intractable, especially bearing in mind thatcommunication is usually a primary bottleneck in Federated Learning [34, 44].
Revised: It is obvious that communicating the whole space of operators along with the associated weights to each device becomes quickly intractable, especially bearing in mind that communication is usually a primary bottleneck in Federated Learning [39, 51].

Original: To this direction,
Revised: To this direction,

Original: FedorAS adopts a uniform parameter size budget, B Φ comm , and samples 1 the search space for operatorsuntil this limit is hit (Eq. 1).
Revised: FedorAS adopts a uniform parameter size budget, B Φ comm , and samples 2 the search space for operators until this limit is hit (Eq. 2).

Original: Setting the limit to half the size of a typical network deployed for a taskworked sufficiently well in our experiments and in fact accelerated convergence (Sec. 3.4).
Revised: Setting the limit to half the size of a typical network deployed for a task worked sufficiently well in our experiments and in fact accelerated convergence (Sec. 4.4).

Original: [Equation]
Revised: [Equation]

Original: where L is the number of layer in the supernet, O l the candidate and ˆ O l the selected operations inlayer l , 1 a unit vector of valid operations and Φ comm a measure of DNN size (e.g. #parameters).
Revised: where L is the number of layer in the supernet, O l the candidate and ˆ O l the selected operations in layer l , 1 a unit vector of valid operations and Φ comm a measure of DNN size (e.g. #parameters).

Original: In terms of sampling strategies, we experimented with uniform operator sampling, which we found towork sufficiently well and provide uniform coverage over the search space.
Revised: In terms of sampling strategies, we experimented with uniform operator sampling, which we found to work sufficiently well and provide uniform coverage over the search space.

Original: It is also worth noting thata different subspace (not necessarily mutex) is selected for each participating client in each round.2 Client-side sampling & local training.
Revised: It is also worth noting that a different subspace (not necessarily mutex) could be selected for each participating client in a round. 2 Client-side sampling & local training.

Original: Participating clients receive the subspace sampled on theserver, { ˆ O l } l ∈ [0; L ] , from which they sample a single operator on every layer.
Revised: Participating clients receive the subspace sampled on the server, { ˆ O l } Ll =0 , from which they sample a single operator on every layer.

Original: This constitutes a pathalong the supernet ( p L ) representing a single model.
Revised: This constitutes a path along the supernet ( p L ) representing a single model.

Original: For every batch, clients sample paths that do notsurpass the assigned training budget B Tier Φ train .
Revised: For every batch, clients sample paths that do not surpass the assigned training budget B Tier Φ train .

Original: Throughout this work, we consider Φ train ( · ) to be a costfunction that counts the FLOPs of a given operator.
Revised: Throughout this work, we consider Φ train ( · ) to be a cost function that counts the FLOPs of a given operator.

Original: This FLOPs limit is defined per tier so that anetwork does not exceed the capabilities of the target device.
Revised: This FLOPs limit is defined per tier so that a network does not exceed the capabilities of the target device.

Original: Path sampling is performed randomlyacross the valid (locally available) operations of each layer to ensure uniform coverage of the searchspace (Eq. 2). Alternative strategies, such as probabilistic sampling are left as future work.
Revised: Our goal is to sample valid paths uniformly, to ensure systematic coverage of the entire (sub) search space during training:

Original: [Equation]
Revised: [Equation]

Revised: However, realising Eq.

Revised: 3 efficiently is not a trivial task.

Original: After having sampled the path, a model  is created and gets a forward and backward pass from theinput sample at hand, in  a single-path one-shot manner [23].
Revised: Originally [28], the authors considered a naive approach of repeatedly sampling a path until it fits the given budget, which results in non-negligible overhead if the probability of finding a model under the threshold is low. Were we to employ such a method, the most restricted devices, for which the set of eligible models is the smallest, would be the ones burdened with the largest overhead. Therefore, we propose a greedy approximation in which operations are selected sequentially. Specifically, in order to obtain a path p

Original: {+L = { o i } Li =
+}
Revised: {+L = { o i } Li =
+}

Original: The number of samples passing througheach operator are kept and communicated back to  the server, along with the updates, for properlyweighting updated parameters upon aggregation, as we will see next.
Revised: we sample operations layer-by-layer, according to a random permutation σ , in such a way that the i -th operation is chosen from the candidates for layer σ ( i ) whose total overhead would not violate the constraint: [Equation] We can ensure that Eq. 4 can always obtain a valid architecture without resampling if layers have

Original: Operator updates coming fromsingle clients are dropped for privacy reasons.3 Aggregation with OPA.
Revised: Identity among their candidate operations and prioritising the selection of those which do not. After having sampled the path, a model is instantiated and a single optimization step using a single batch of data is performed. The number of samples passing through each operator are kept and communicated back to the server, along with the updates, for properly weighting updated parameters upon aggregation, as we will see next. 3 Aggregation with OPA.

Original: An operator gets stochastically exposed to clients data.
Revised: An operator gets stochastically exposed to clients data.

Original: This stemsfrom subspace sampling and client-side path sampling .
Revised: This stems from subspace sampling and client-side path sampling .

Original: As such, naively aggregating updates in anequi-weighed manner (Plain-Avg) or total samples residing  a client (FedAvg [48]) would not reflectthe training experience during a round.
Revised: As such, naively aggregating updates in an equiweighed manner (Plain-Avg) or total samples residing on a client (FedAvg [57]) would not reflect the training experience during a round.

Original: For this reason, we propose OPA, OPerator Aggregation,an aggregation method that weighs updates based on the relative experience of an operator acrossall clients that have updated that operator.
Revised: For this reason, we propose OPA, OPerator Aggregation, an aggregation method that weights updates based on the relative experience of an operator across all clients that have updated that operator.

Original: Concretely,  for each operation we  track the number ofexamples updating its weights for eachclient. Then, upon aggregation, for searchable operation o l ,we normalise over all the updates of that specific operator across clients.
Revised: Concretely, our method is generalisation of FedAvg where normalisation is performed independently for each layer, rather than collectively for full models. In order to enable that, we keep track of how many examples were used to update each searchable operation o l , independently on each client, and later use this information to weight updates.

Original: Formally put:
Revised: Formally:

Original: [Equation]
Revised: [Equation]

Original: where ω ( t ) g ( · ) are global weights, ω ( t ) i ( · ) are local weights of client
Revised: where ω ( t ) g ( · ) are global weights, ω ( t ) i ( · ) are local weights of client

Original: i at global step t and | D (i, o l , t ) |is the number of samples having backpropagated through an operator o l for client i in round t .
Revised: i at global step t , | D ( t ) i,o l | is the number of samples having backpropagated through an operator o l for client i in round t , and C ( t ) o l is the set of clients i s.t.

Revised: | D ( t ) i,o l | > 0 .

Revised: Updates to ω ( t ) g happen only if | C ( t )

Original: 2.2 Model Search & Validation
Revised: o l | > 1 in order to protect the privacy of single clients. Finally, if an operation is always selected, then Eq. 5 recovers FedAvg, which means we can effectively use OPA throughout the model and not only for searchable layers. 3.2 Model Search & Validation

Original: After training the supernet, a search phase is implemented to discover the best trained architectureper device tier.
Revised: After training the supernet, a search phase is implemented to discover the best trained architecture per device tier.

Original: Models are sampled with NSGA-II [19] and evaluated on a global validation set.
Revised: Models are sampled with NSGA-II [23] and evaluated on a global validation set.

Original: Ourrationale behind selecting the NSGA-II algorithm for our search is the fact that it is multi-objectiveand allows us for efficient space exploration with the goal of optimising for model size in a specifictier and accuracy.
Revised: The rationale behind selecting the NSGA-II algorithm for our search is the fact that it is multi-objective and allows us for efficient space exploration with the goal of optimising for model size in a specific tier and accuracy.

Original: Other search algorithms can be used in place of NSGA-II, based on the objective athand.
Revised: Other search algorithms can be used in place of NSGA-II, based on the objective at hand.

Original: At the end of this stage, we have a model architecture per tier, already encompassing knowledgeacross devices of different tiers, which serves as initialisation for further training. 
Revised: At the end of this stage, we have a model architecture per tier, already encompassing knowledge across devices of different tiers, which serves as initialisation for further training. [Table] [Figure] [Table]

Revised: [Figure]

Revised: [Table]

Original: 2.3 Fine-tuning Phase
Revised: [Figure] [Table] [Figure] 3.3 Fine-tuning Phase

Original: Subsequently, we move to train each of the previously produced models in a federated manner acrossall eligible clients.
Revised: Subsequently, we move to train each of the previously produced models in a federated manner across all eligible clients.

Original: This means that architectures falling in a specific tier in terms of footprint can betrained across clients belonging to that tier and above.
Revised: This means that architectures falling in a specific tier, in terms of footprint, can be trained across clients belonging to that tier and above.

Original: Our hypothesis here is that compared to regular
Revised: Our hypothesis here is that compared to regular

Original: FL where one needs to trade off capacity of participation, FedorAS allows for better generalisationdue to knowledge sharing through its supernet structure.
Revised: FL where one needs to trade off capacity of participation, FedorAS allows for better generalisation due to knowledge sharing through its supernet structure.

Original: Here we use conventional FedAvg andpartial client participation per round, again with the goal of training a single network per tier.
Revised: Here we use conventional FedAvg and partial client participation per round, again with the goal of training a single network per tier

Original:  Evaluation
Revised:  Evaluation

Original: This section provides a thorough evaluation of FedorAS across different tasks to show its performanceand generality.
Revised: This section provides a thorough evaluation of FedorAS across different tasks to show its performance and generality.

Original: Extra results ( e.g. convergence, fairness) can be found in  the Appendix.
Revised: First, we compare FedorAS to existing approaches in the context of federated NAS in the cross-device and cross silo setting. Next, we we draw from the broader FL literature and showcase our technique’s performance gains compared to homogeneous and heterogeneous federated solutions (Sec. 4.2). This can be traced back to the benefits of supernet weight sharing ; as such we, subsequently, quantify its contribution by comparing it to FedorAS with randomly initialised networks trained on eligible clients in a federated way (Sec. 4.3) without weight sharing. Last, we showcase the contribution of specific components of our system through an ablation study (Sec. 4.4) and also the performance and behaviour of alternative search methods (Sec. 4.5).

Original: 3.1 Experimental Setup
Revised: 4.1 Experimental Setup

Original: Datasets & Baselines.
Revised: Datasets & Baselines.

Original: Datasets are summarised in Tab. 1.
Revised: Datasets are summarised in Tab. 1.

Original: More information in Appendix D.
Revised: More information in Appendix D.

Original: [Figure]

Original: [Figure]

Original: Search space.
Revised: Search space.

Original: The adopted search spaces are specific to the dataset and task at hand, both in terms ofsize and type of operators.
Revised: The adopted search spaces are specific to the dataset and task at hand, both in terms of size and type of operators.

Original: In general, we assume our highest tier of model sizes to coincide witha typical network footprint used for that task in the FL literature (e.g. ResNet-18 for CIFAR-10).
Revised: In general, we assume our highest tier of model sizes to coincide with a typical network footprint used for that task in the FL literature (e.g. ResNet-18 for CIFAR-10).

Original: Nevertheless, there may be operators in the space that cannot be sampled in some tiers due to theirfootprint.
Revised: Nevertheless, there may be operators in the space that cannot be sampled in some tiers due to their footprint.

Original: FedorAS sets the communication budget B Φ comm to be half the size of the supernet.
Revised: FedorAS sets the communication budget B Φ comm to be half the size of the supernet.

Revised: The full set of available operators per task is provided in the Appendix D.3.

Original: Clusters definition.
Revised: Clusters definition.

Original: In our experiments, we cluster clients based on the amount of operations theycan sustain per round of training (#FLOPs), for simplicity.
Revised: In our experiments, we cluster clients based on the amount of operations they can sustain per round of training (#FLOPs), for simplicity.

Original: Other resources (e.g. #parameters, energyconsumption) can be used in place or in conjunction with FLOPs.
Revised: Other resources (e.g. #parameters, energy consumption) can be used in place or in conjunction with FLOPs.

Original: More details in Appendix D.4.
Revised: More details in Appendix D.4.

Original: 3.2 Performance evaluation
Revised: 4.2 Performance evaluation

Original: Federated NAS.
Revised: Federated NAS.

Original: We start the evaluation by comparing our system with existing works in the federated
Revised: We start the evaluation by comparing our system with existing works in the federated

Original: NAS domain.
Revised: NAS domain.

Original: Specifically, we find two systems that perform federated NAS, namely FedNAS [25]and SPIDER [49] and compare in the cross-device and cross-silo settings.
Revised: Specifically, we find two systems that perform federated NAS, namely FedNAS [30] and SPIDER [59] and compare in the cross-device and cross-silo settings.

Original: In the former setting,we adapt FedNAS to support partial participation and compare their technique to FedorAS underthe same cross-device settings in CIFAR-10. 
Revised: In the former setting, we adapt FedNAS to support partial participation and compare their technique to FedorAS under the same cross-device settings in CIFAR-10. Results are shown in Tab. 2, showing FedorAS performing 1.04% better than FedNAS on CIFAR-10 α =1 and 48.7% on CIFAR-10 α =0 . 1 , for the same training memory footprint.

Original: Results are shown in Tab. 2.
Revised: Further details can be found in Appendix D.6.

Original: At the same time, whilerunning in cross-silo setting is outside the scope of FedorAS , we have adapted our experimentalsetting to match that of FedNAS and SPIDER.
Revised: At the same time, while running in cross-silo setting is not the main focus of FedorAS , we have adapted our experimental setting to match that of FedNAS and SPIDER.

Original: Results are shown in Tab. 3.
Revised: Results are shown in Tab. 3 with FedorAS performing +11.6%

Revised: and -1.3% than the baselines, respectively, on the test set of their selected settings.

Original: [Table] 
Revised: [Table] [Table]

Revised: Comparison with Random Search.

Original: {+We further compare our search with a random search baseline – which is a naive way of running FL NAS in our experimental setting (the same search space, etc.) – and find that FedorAS performs significantly better at an average of +5.11pp, +3.24pp, +12.84pp, +9.27pp, +0.25p across tiers for CIFAR-
+}
Revised: {+We further compare our search with a random search baseline – which is a naive way of running FL NAS in our experimental setting (the same search space, etc.) – and find that FedorAS performs significantly better at an average of +5.11pp, +3.24pp, +12.84pp, +9.27pp, +0.25p across tiers for CIFAR-
+}

Original: [Table]
Revised: α = { 1000 , 1 , 0 . 1 } , CIFAR-100 and Shakespeare, respectively. Only in the case of SpeechCommands did our search result in -0.87pp of accuracy on average. We suspect this is an artifact of insufficiently-long fine-tuning which means models might not have converged (suggested by a high variance). Detailed results are provided in Appendix E.

Original: Federated Learning baselines.
Revised: Federated Learning baselines.

Original: Next, we compare the performance of FedorAS with differentfederated baselines, including homogeneous [56] and system heterogeneous systems [28, 38, 52].
Revised: Next, we compare the performance of FedorAS with different federated baselines, including homogeneous [67] and system heterogeneous frameworks [33, 43, 63].

Original: Inthe former setting, we compare with results from [56] on CIFAR100 LDA= 0 .
Revised: In the former setting, we compare with results from [67] on CIFAR100 α =0 .

Original: FedorAS performs 1pp (percentage point) better than the FedAvg baseline, at 45.84%, but at a fraction of the cost 2 .
Revised: FedorAS performs 1 percentage point (pp) better than the FedAvg baseline, at 45.84%, but at a fraction of the cost 3 .

Original: Atthe same time, training end-to-end the resulting FedorAS architecture from scratch – in a similarhomogeneous setting as the baseline– results in 11.43pp higher accuracy at 52.5%, again at a lowercost.
Revised: Simultaneously, retraining the discovered model from scratch using random initialisation under the same training setting as the baseline results in 11.43 pp higher accuracy than the best FedAdam (63.94% vs 52.50%), showcasing the quality of FedorAS -produced bespoke architectures.

Original: Comparisons with heterogeneous baselines  aresummarised in Tab. 4 and also partly depicted in 
Revised: With respect to heterogeneous baselines (ZeroFL [63], FjORD[33]), we see that FedorAS consistently leads to models with higher accuracy across tiers that are in the respective clients’ computational budget (Tab. 4). At the same time, we depict how FedorAS performs compared to FjORD and randomly selected architectures trained naively in a resource-aware manner in Fig. One can clearly see that the degrees of architectural freedom that our solution offers leads to significantly better accuracy per resource tier. Notably, we perform 15.20% and 12.58% better on average than FjORD on CIFAR-10 and Shakespeare, respectively, while still respecting client eligibility.

Original: Fig.
Revised: While model accuracy may not seem to scale proportionally to their size, we attribute this to the limited participation eligibility of clients, an innate trait of system heterogeneous landscape. Normally, this can cause performance gaps due to limited exposure to federated data (Fig. 1). We argue that FedorAS is able to bridge this gap (+0.03 points (p) avg), by means of weight sharing and OPA, +1.72 p more effectively than FjORD (avg Tier 4 vs Tier 1 gap across datasets).

Revised: Additional results.

Revised: Additional results on the convergence and fairness of FedorAS as well as alternative client allocation [43] to clusters are provided in Appendix G, H and I, respectively.

Original: [Table]
Revised: [Table]

Original: 3.3 Supernet weight-sharing with FedorAS
Revised: 4.3 Supernet weight-sharing with FedorAS

Revised: Here

Original: In this section, we evaluate the impact of weight sharing through FedorAS ’ supernet.
Revised: [Table] we evaluate the impact of weight sharing through FedorAS ’ supernet.

Original: To this end,we compare the performance of FedorAS ’ models to the same architectures models, but trainedend-to-end in a federated manner, across all four datasets.
Revised: We compare the performance of FedorAS ’ models to the same architectures models

Original: Results are presented in Tab. 5.
Revised: [Table]

Original: The standard setting across datasets is that we first train with FedorAS across the three stagesdescribed in Sec. 2. Subsequently, we take the output architectures from our system, randomlyinitialise them and train end-to-end in a federated manner,where each model trains across all eligibleclients (tier and above).
Revised: but trained end-to-end in a federated manner, across all four datasets.

Original: With this comparison, we aim to showcase that FedorAS 
Revised: With this comparison [Table] we aim to showcase that FedorAS [Table]

Original:  not only comes upwith better architectures, but it also effectively transfers knowledge between tiers through its supernetstructure and OPA aggregation scheme.
Revised: [Table] [Table] not only comes up with better architectures, but it also effectively transfers knowledge between tiers through its supernet structure and OPA aggregation scheme.

Original: Indeed, we can see in all cases models belonging to the most2 1.11 vs 0.16 GFLOPS, 11.4M vs 1.62M parameters, 4000 vs 850 global rounds
Revised: In order to accomplish that, we first train with FedorAS across the three stages described in Sec. 3 ( supernet-init ). Subsequently, we take the output architectures from our system, randomly initialise them and train end-to-end in a federated manner, where each model trains across all eligible clients ( rand-init ).

Original: [Table]
Revised: Results are presented in Tab. 5.

Original: restricted tiers (least amount of capable clients) benefit from being initialised from a supernet.
Revised: Indeed, models benefit from being initialised from a supernet across cases; this means that weightsharing mitigates both the limited model capacity of the lower-tier and the limited data exposure of large models.

Original: Theimprovement is further magnified as non-IID-ness increases, leadingto increases of up to 15 p.p. overrandom initialisation.
Revised: The accuracy improvement is further magnified as non-IID-ness increases, leading up to +15 pp over rand-init .

Original: Interestingly, we also see that in many cases low-tier models seem to benefit;we attribute this to the fact that, by being a part of the supernet, their weights are also updated in acontext of much larger models which allows them to eventually reach better minima.

Original: We presentresults on different tasks of the same dataset  in Appendix G.
Revised: Results on different tasks of the same dataset presented in Appendix F.

Original: 3.4 Ablationand sensitivity study
Revised: 4.4 Ablation study

Original: Here we measure the contribution of specific components of FedorAS to the quality and performanceof these models.
Revised: Next, we measure the contribution of specific components of FedorAS to the quality and performance of these models.

Original: To this direction, we firstly compare the performance of our system’s aggregationcompared to naive averaging.
Revised: To this direction, we firstly compare the performance of our system’s aggregation compared to naive averaging.

Original: Subsequently, we measure the impact of sending smaller subspacesto clients for supernet training.
Revised: Subsequently, we measure the impact of sending smaller subspaces to clients for supernet training.

Original: We provide indicative results  CIFAR-10(non-IID, α =1.0).
Revised: We provide indicative results on CIFAR-10 α =0 .

Original: Last,we assess the impact of differently sized validation sets to the quality of the models, as well as thebehaviour of FedorAS to different clients to tiers allocation.

Original: OPA vs. naive averaging.
Revised: OPA vs. naive averaging.

Original: OPA compared tonaive FedAvg leeds to increased accuracy by +1.2,+0.9, +1.

Revised: OPA compared to FedAvg leads to increased accuracy by +1.2, +0.9, +1.

Original: and +0.9 pp for tiers 1-4, respectively.
Revised: and +1.78 pp for tiers 1-4, respectively.

Original: More details in Appendix F.
Revised: More details in Appendix G.

Original: Subspace sampling size.
Revised: Subspace sampling size.

Original: FedorAS samples the supernet before communicating it to a client.
Revised: FedorAS samples the supernet before communicating it to a client.

Original: For
Revised: For

Original: CIFAR-10 ( α = 1 . 0 ) and when B Φ comm is set to 22.3M parameters (i.e. allowing sending wholesupernet), FedorAS yields 85.48%, 86.73%, 86.50% average accuracies across tiers for full, 1 / 2 and1 / 4 of the size of the search space, respectively.
Revised: CIFAR-10 ( α = 1 . 0 ) and when B Φ comm is set to 22.3M parameters (i.e. allowing sending whole supernet), FedorAS yields 85.48%, 86.73%, 86.50% average accuracies across tiers for full, 1 / 2 and 1 / 4 of the size of the search space, respectively.

Original: Thus, not only is the overhead of communicationreduced with FedorAS , but convergence to the same level of accuracy is faster.
Revised: Thus, not only is the overhead of communication reduced with FedorAS , but convergence to the same level of accuracy is faster.

Original: We hypothesise thatthis is due a better balance in the exploration vs. exploitation trade-off.
Revised: We hypothesise that this is due a better balance in the exploration vs. exploitation trade-off.

Original: More details in Appendix F.
Revised: More details in Appendix G.

Original: Validation set size.
Revised: 4.5 Evaluation of the Search Phase

Original: In order to assess the quality of models during search (stage 2),  we need a proxydataset to evaluate different paths and rank them.
Revised: In order to assess the quality of models during search (Stage 2), so far we have needed a proxy dataset to evaluate different paths and rank them.

Original: We consider this as a set of examples that eachapplication provider has centrally to measure the quality of their deployed models in productionsettings.
Revised: We consider this as a set of examples that each application provider has centrally to measure the quality of their deployed models in production settings. Validation set size.

Original:  To gauge the sensitivity of the end models to the underlying distribution of the validationset, we sample down the validation set, as a held-out portion of the clients datasets, to 20% and 50%of the original size.
Revised: Nevertheless, the size and representativeness of the centralised dataset might affect the quality of the search. To gauge the sensitivity of the end models to the underlying distribution of the validation set, we sample down the validation set, as a held-out portion of the clients datasets, to 20% and 50% of the original size.

Original: We find no noticeable impact on the quality of the final models.
Revised: We find no noticeable impact on the quality of the final models.

Original: Only in the caseof SpeechCommands, we notice an 1pp drop on average when scaling down to 20%, which can beattributed to the imbalanced nature of the dataset.

Original: Detailed results are presented inthe Appendix H.
Revised: Detailed results are presented in Appendix J.1.

Original: Different allocation of clients to tiers.
Revised: Federated search.

Original: To showcase that FedorAS can work efficiently underrealistic settings, we also adopted the client capabilities from Oort [38]. We witnessed similar scalingin the quality of  the end models.
Revised: There may be also cases where no such dataset can be centralised. To this direction, we test whether our search can operate under a federated setting with partial participation in order to faithfully rank the quality of models stemming from the supernet.

Original: Detailed results are presented in the Appendix I.
Revised: In this setting, we have implemented a federated version of NSGA-II.

Original: Conclusion
Revised: Instead of candidate models being evaluated on the same validation set, they are stochastically evaluated on sampled federated client datasets [ ? ]. We hypothesise it is possible to maintain faithful ranking of models compared to centralised evaluation if enough clients are leveraged to evaluate models, at the cost of increased communication cost. Furthermore, we expect the overall cost of achieving robust evaluation to increase as non-IID-ness and #clients increase, and the instantaneous cost of sending models to decrease over time, as NSGA-II converges to well-performing models (i.e. decreased diversity of models). Fig.

Original: In this work, we have presented FedorAS , a system that performs resource-efficient federated NASin the cross-device setting.{+4 shows results for [Figure] CIFAR
 α =0 .+}
Revised: {+4 shows results for [Figure] CIFAR
 α =0 .+}

Original: We have shown that our technique leads to better results across modalitiesand tasks compared to existing federated solutions.
Revised: 1 , with extra results and details presented in Appendix J.2. Our experiments support the aforementioned conjectures.

Original: Specifically, FedorAS is the first method toenable cross-device NAS offering significantly lower overhead compared to existing federated NAStechniques while at the same time achieving state-of-the-art performance compared to heterogeneous FL solutions by means of effective weight sharing and flexible resource-aware training.
Revised: Noticeably, even under highly non-IID settings, we can attain faithful FE at a reasonable cost (4 rounds to Kendall- τ > 0 . 8 for the results presented in Fig. 4)

Original: A  Related Work
Revised:  Related Work

Original: Data and system heterogeneity.
Revised: Data and system heterogeneity.

Original: Traditionally, works on heterogeneous FL havebeen focused ontackling the statistical data heterogeneity [57, 43, 29, 22, 45] or minimising the upstream communication cost [40, 36, 61, 24, 4] of sending updates from client to servers, as the primary bottleneckin the federated training process.
Revised: Traditionally, works on heterogeneous FL have focused on tackling the statistical data heterogeneity [69, 49, 34, 27, 53] or minimising the upstream communication cost [45, 41, 73, 29, 5] of sending updates from client to servers, as the primary bottleneck in the federated training process.

Original: However, it has become apparent that computational disparitybecomes an equally important barrier for contributing knowledge in FL.
Revised: However, it has become apparent that computational disparity becomes an equally important barrier for contributing knowledge in FL.

Original: As such, lately, there hasbeen a line of work focusing on this very problem, where the discrepancy between the dynamicsof different clients affects the convergence rate or fairness of the deployed system.
Revised: As such, lately, there has been a line of work focusing on this very problem, where the discrepancy between the dynamics of different clients affects the convergence rate or fairness of the deployed system.

Original: Specifically,such solutions draw from efficient ML and attempt to dynamically alter the footprint of localmodels my means of structured (PruneFL [32]), unstructured (Adaptive Federated Dropout [10], LotteryFL [41]) or importance-based pruning (FjORD [28]), quantisation (AQFL [1]), low-rankfactorisation (FedHM [64]), sparsity-inducing training (ZeroFL [53]) or distillation (GKT [26]).
Revised: Specifically, such solutions draw from efficient ML and attempt to dynamically alter the footprint of local models my means of structured (PruneFL [37]), unstructured (Adaptive Federated Dropout [13], LotteryFL [46]) or importance-based pruning (FjORD [33]), quantisation (AQFL [2]), low-rank factorisation (FedHM [76]), sparsityinducing training (ZeroFL [64]) or distillation (GKT [31]).

Original: However, each approach has limitations, either because they involve extra training overhead [26] andresidence of multiple DNN copies in memory [1], or because they require specialised hardware forperformance gains ([53, 10]).
Revised: However, each approach has limitations, either because they involve extra training overhead [31] and residence of multiple DNN copies in memory [2], or because they require specialised hardware for performance gains ([64, 13]).

Original: Last, some of the architectural changes proposed may not offer thedegrees of freedom that NAS exposes [28, 64, 41].
Revised: Last, some of the architectural changes proposed may not offer the degrees of freedom that NAS exposes [33, 76, 46] or may have a different optimisation objective altogether (i.e. personalisation [46]).

Original: Federated NAS.
Revised: Federated NAS.

Original: The concept of performing NAS in a federated setting has been visited before in theliterature [25, 49, 65, 66, 46].
Revised: The concept of performing NAS in a federated setting has been visited before in the literature [30, 59, 77, 78, 54].

Original: However, current solutions vary greatly in their approaches and goals,with most of them being applied and applicable to the cross-silo setting, where full participationand small number of participants are expected.
Revised: However, current solutions vary greatly in their approaches and goals, with most of them being applied and applicable to the cross-silo setting, where full participation and small number of participants are expected.

Original: Specifically, one of the first works in the area was FedNAS [25], which adopts a DARTS-based approach and aims to find a globally good model for allclients, to be personalised at a later stage.
Revised: Specifically, one of the first works in the area was FedNAS [30], which adopts a DARTS-based approach and aims to find a globally good model for all clients, to be personalised at a later stage.

Original: This approach requires the whole supernet to be transmittedand kept in memory for architectural updates, which leads to excessive requirements  that make itlargely inapplicable for cross-device setups spanning many clients of limited capabilities partiallyparticipating in training.
Revised: This approach requires the whole supernet to be transmitted and kept in memory for architectural updates, which leads to excessive requirements (Fig. 2) that make it largely inapplicable for cross-device setups spanning many clients of limited capabilities partially participating in training.

Original: To mitigate this requirement, [65] proposes an RL-based approach for 
Revised: To mitigate this requirement, [77] proposes an RL-based approach for cross-silo FL-based NAS.

Original: FL-based NAS, where they sample a series of models to be trained on each round on clients Despitethe intention, their three-phase process (pre-train, search, train) still incurs significant overheads.
Revised: Despite the intention, it still incurs significant overheads due to RL-based model sampling convergence and single model training per client.

Original: Asomewhat different approach is adopted by HAFL [46], which leverages graph hypernetworks as ameans to generate outputs of a model.
Revised: A somewhat different approach is adopted by HAFL [54], which leverages graph hypernetworks as a means to generate outputs of a model.

Original: While interesting, performance and scalability are not on parwith current state-of-the-art.
Revised: While interesting, performance and scalability are not on par with current state-of-the-art.

Original: On the front of personalisation, FedPNAS [5] searches for architectureswith a shared based component across clients and a personalised component.
Revised: On the front of personalisation, FedPNAS [6] searches for architectures with a shared based component across clients and a personalised component.

Original: However, this work isonly aimed at IID vision tasks and involves a meta-step for personalisation, which increases trainingoverheads significantly.
Revised: However, this work is only aimed at IID vision tasks and involves a meta-step for personalisation, which increases training overheads significantly.

Original: At the other extreme for cross-silo personalised FL, SPIDER [49] aims atfinding personalised architectures per client.
Revised: At the other extreme for cross-silo personalised FL, SPIDER [59] aims at finding personalised architectures per client.

Original: It requires the whole space on-device and federationis accomplished through a second static on-device network.
Revised: It requires the whole space on-device and federation is accomplished through a second static on-device network.

Original: These non-negligible overheads makeporting this approach to the cross-device setting non-trivial.
Revised: These non-negligible overheads make porting this approach to the cross-device setting non-trivial.

Original: In contrast, FedorAS brings Federated NAS to the cross-device setting and is designed with resourceefficiency and system heterogeneity in mind.
Revised: In contrast, FedorAS brings Federated NAS to the cross-device setting and is designed with resource efficiency and system heterogeneity in mind.

Original: This way, communication and computation cost is keptwithin the defined limits, respecting the runtime on constrained devices.
Revised: This way, communication and computation cost is kept within the defined limits, respecting the runtime on constrained devices.

Original: Crucially, our system doesnot assume full participation of clients and is able to efficiently exchange knowledge through thesupernet weight sharing and generalise across different modalities and granularity of tasks.  
Revised: Crucially, our system does not assume full participation of clients and is able to efficiently exchange knowledge through the supernet weight sharing and generalise across different modalities and granularity of tasks.  Conclusion

Revised: In this work we have presented FedorAS , a system that performs resource-efficient federated NAS in the cross-device setting.

Revised: Our method offers significantly lower overhead compared to existing federated NAS techniques, while achieving state-of-the-art performance compared to heterogeneous

Revised: FL solutions, by means of effective weight sharing and flexible resource-aware training

Original: B Broader Impact
Revised: A Introduction

Revised: B

Revised: D Detailed Experimental Setup

Revised: D.1 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Revised: D.2 Datasets . .

Revised: D.3 Search Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Revised: D. client assignment .

Original: {+D.
+}
Revised: {+D.
+}

Revised: Hyperparameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Revised: D. device Federated evaluation . . . . . . . . . . . . . . . . . . . . . .

Revised: D.7 Baselines . .

Revised: Sampling

Revised: Fairness the

Revised: Phase

Revised: J.

Revised: Sensitivity to size of validation set . . . . . . . . . . . . . . . . . . . . . . . . .

Revised: Federated Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Revised: A Introduction

Revised: This Appendix extends the content of the main paper by providing support material to the study presented in Sec. 3, 4 and, additional insights about FedorAS .

Revised: Concretely, the Appendix is divided into three main blocks of extra content:

Revised: • Impact and Limitations.

Revised: We concisely present the broad impact and limitations of our work as well as future research directions in Sec.

Revised: B, C. • Experimental Setup.

Revised: Sec.

Revised: D provides details on the libraries used to build FedorAS , the datasets considered for experiments as well as the hyperparameters used to obtain the results presented in Section 4.

Revised: Crucially, we provide a detailed description of the search spaces utilised in for each dataset domain, how tiers are defined and, a concise description of each baseline method included in this work.

Revised: • Additional Experiments.

Revised: Sec.

Revised: E through H study different aspects of FedorAS such as: i) learning multiple tasks using the supernet in Sec.

Revised: F; or ii) the convergence behaviour in Sec.

Revised: G; iii) the fairness aspect of FedorAS ; or under new scenarios altogether, such as: iv) an alternative procedure to assign clients to tiers in Section I; or iv) alternative search methods in Sec.

Revised: J.

Revised: Overall, the following content substantially extends what is already presented in the main text.

Revised: B Broader Impact

Original: Our system, FedorAS , performs federated and resource-aware NAS in the cross-device setting.
Revised: Our system, FedorAS , performs federated and resource-aware NAS in the cross-device setting.

Original: Despite the benefits illustrated compared to centralised NAS and cross-silo FL solutions, running
Revised: Despite the benefits illustrated compared to centralised NAS and cross-silo FL solutions, running

Original: Neural Architecture Search is still a resource-demanding process, in terms of compute, memoryand network bandwidth.
Revised: Neural Architecture Search is still a resource-demanding process, in terms of compute, memory and network bandwidth.

Original: While FedorAS ’s target devices can be of significantly lower TDP (i.e.smartphones and IoT devices vs. server-grade GPUs) – with consequences on the overall energyconsumption of training [53] – they are resources not directly owned by the operator.
Revised: While FedorAS ’s target devices can be of significantly lower TDP (i.e. smartphones and IoT devices vs. server-grade GPUs) – with consequences on the overall energy consumption of training [64] – they are resources not directly owned by the operator.

Original: As such, specialcare needs to be taken with respect to how many device resources are leveraged at any point in, so asnot hinder the usability of the device or invoke monetary costs to the user [35].
Revised: As such, special care needs to be taken with respect to how many device resources are leveraged at any point in, so as not hinder the usability of the device or invoke monetary costs to the user [40].

Original: C Limitations & Future Work
Revised: C Limitations & Future Work

Original: Despite the challenges address by FedorAS , our prototype has certain limitations.
Revised: Despite the challenges addressed by FedorAS , our prototype has certain limitations.

Original: First and foremost,we have opted to cluster devices (i.e. in tiers) based on their FLOPS.
Revised: First and foremost, we have opted to cluster devices (i.e. in tiers) based on their FLOPS.

Original: While it is perfectly normal todivide clusters based on other criteria (e.g. memory, latency, energy) or in a multi-objective manner,we have kept it simple.
Revised: While it is perfectly normal to divide clusters based on other criteria (e.g. memory, latency, energy) or in a multi-objective manner, we have kept it simple.

Original: Moreover, one can opt for biased sampling of i) clients participating in around, ii) the subspace they get communicated and iii) the paths sampled from that subspace, weopted for uniform sampling for all of the above for simplicity, uniform coverage of the search spaceand fairness in participation.
Revised: Moreover, one can opt for biased sampling of i) clients participating in a round, ii) the subspace they get communicated and iii) the paths sampled from that subspace, we opted for uniform sampling for all of the above for simplicity, uniform coverage of the search space and fairness in participation.

Original: We leave the exploration of such strategies as future work.
Revised: We leave the exploration of such strategies as future work.

Original: Last but notleast, we have considered privacy-enhancing techniques, such as Differential Privacy [12] or Secure
Revised: Last but not least, we have considered privacy-enhancing techniques, such as Differential Privacy [15] or Secure

Original: Aggregation [9, 6], as orthogonal to our scheme.
Revised: Aggregation [12, 7], as orthogonal to our scheme.

Original: Combining Federated NAS with such strategies canexpose interesting trade-offs of exploration-exploitation-privacy budgets that could be explored in thefuture.
Revised: Combining Federated NAS with such strategies can expose interesting trade-offs of exploration-exploitation-privacy budgets that could be explored in the future.

Original: D Detailed Experimental SetupD.1 Implementation
Revised: D Detailed Experimental Setup D.1 Implementation

Original: FedorAS was implemented on top of the Flower (v0.18) [8] framework and PyTorch (v1.11.0) [51].
Revised: FedorAS was implemented on top of the Flower (v0.18) [10] framework and PyTorch (v1.11.0) [61].

Original: We run all our experiments on a private cloud cluster in a simulated manner, across four iterationseach and report averages and standard deviations.
Revised: We run all our experiments on a private cloud cluster in a simulated manner, across four iterations each and report averages and standard deviations.

Original: D.2 Datasets
Revised: D.2 Datasets

Original: We partition CIFAR-10/100 following the Latent Dirichlet Allocation (LDA) partitioning [30],with each client receiving approximately equisized training sets.
Revised: We partition CIFAR-10/100 following the Latent Dirichlet Allocation (LDA) partitioning [35], with each client receiving approximately equisized training sets.

Original: For CIFAR-10, we considerα ∈ { 1000 , 1 . 0 , 0 . 1 } configurations, while for CIFAR-100, we adopt α = 0 .
Revised: For CIFAR-10, we consider α ∈ { 1000 , 1 . 0 , 0 . 1 } configurations, while for CIFAR-100, we adopt α = 0 .

Original: 1 as in [56].
Revised: 1 as in [67].

Original: Theremaining datasets come naturally partitioned.
Revised: The remaining datasets come naturally partitioned.

Original: CIFAR-10/100.
Revised: CIFAR-10/100.

Original: The CIFAR-10 datasets contains 10k and 50k 32x32 RGB images in its test andtraining sets respectively comprising ten classes.
Revised: The CIFAR-10 datasets contains 10k and 50k 32x32 RGB images in its test and training sets respectively comprising ten classes.

Original: The goal is to classify these images correctly.
Revised: The goal is to classify these images correctly.

Original: Similarly, CIFAR-100 follows an identical partitioning but this time across 100 classes ( fine lebels) or20 superclasses ( coarse labels).
Revised: Similarly, CIFAR-100 follows an identical partitioning but, this time, across 100 classes ( fine lebels) or 20 superclasses ( coarse labels).

Original: Both CIFAR datasets have a uniform coverage across their classes.
Revised: Both CIFAR datasets have a uniform coverage across their classes.

Original: SpeechCommands.
Revised: SpeechCommands.

Original: The Speech Commands datasets [62] is comprised of 105,829, 16KHz 1-secondlong audio clips of a spoken word (e.g. "yes", "up", "stop") and the task is to classify these in a 12or 35 classes setting.
Revised: The Speech Commands datasets [74] is comprised of 105,829, 16KHz 1-second long audio clips of a spoken word (e.g. "yes", "up", "stop") and the task is to classify these in aor 35 classes setting.

Original: The datasets comes pre-partitioned into 35 classes and in order to obtain the12-classes version, the standard approach [7, 16, 60] is to keep 10 classes of interest (i.e. "yes", "no","up", "down", "left", "right", "on", "off", "stop", "go"), place the remaining 25 under the "unknown"class and, introduce a new class "silence" where no spoken word appear is the audio clip.
Revised: The datasets comes pre-partitioned into 35 classes and in order to obtain the 12-classes version, the standard approach [9, 20, 72] is to keep 10 classes of interest (i.e. "yes", "no", "up", "down", "left", "right", "on", "off", "stop", "go"), place the remaining 25 under the "unknown" class and, introduce a new class "silence" where no spoken word appear is the audio clip.

Original: In thiswork we consider SpeechCommandsV2, the most recent version of this dataset.
Revised: In this work we consider SpeechCommandsV2, the most recent version of this dataset.

Original: The dataset spansthree disjoint set of speakers: 2112 form the training set, 256 for validation and, 250 for testing.
Revised: The dataset spans three disjoint set of speakers: 2112 form the training set, 256 for validation and, 250 for testing.

Original: In FedorAS , the supernet training phase makes uses of the 2112 clients in the training partitiononly.
Revised: In FedorAS , the supernet training phase makes uses of the 2112 clients in the training partition only.

Original: The results are obtained by measuring the performance of the discovered models on the dataof the 250 clients comprising the test set.
Revised: The results are obtained by measuring the performance of the discovered models on the data of the 250 clients comprising the test set.

Original: Data is not uniformly distributed and some clients havemore data than others.
Revised: Data is not uniformly distributed and some clients have more data than others.

Original: This dataset is processed by first extracting MFCC [18] features from eachaudio clip [67, 7].
Revised: This dataset is processed by first extracting MFCC [22] features from each audio clip [79, 9].

Original: Across our experiments we extract 40 MFCC features from a MelSpectrogramwhere each audio signal is first sub-sampled down to 8KHz and then sampled using 40ms wide timewindows strided 20ms appart.
Revised: Across our experiments we extract 40 MFCC features from a MelSpectrogram where each audio signal is first sub-sampled down to 8KHz and then sampled using 40ms wide time windows strided 20ms appart.

Original: This results in 1-second audio clip being transformed into a 1 × 40 × 51input that can be passed to a CNN.
Revised: This results in 1-second audio clip being transformed into a 1 × 40 ×input that can be passed to a CNN.

Original: Shakespeare.
Revised: Shakespeare.

Original: This dataset is built [14, 48] from The Complete Works of William Shakespeare andpartitioned in such a way data that from each role in the play is assigned to a client.
Revised: This dataset is built [18, 57] from The Complete Works of William Shakespeare and partitioned in such a way data that from each role in the play is assigned to a client.

Original: This results in atotal of 1,129 partitions where the average number of samples per device is 3.7K and the standarddeviation 6.2K samples.
Revised: This results in a total of 1,129 partitions where the average number of samples per device is 3.7K and the standard deviation 6.2K samples.

Original: This makes Shakespeare a relatively imbalanced dataset.
Revised: This makes Shakespeare a relatively imbalanced dataset.

Original: The task is tocorrectly predict the next character in a dialog given the previously seen characters in the sentence.
Revised: The task is to correctly predict the next character in a dialog given the previously seen characters in the sentence.

Original: C on v ( 3 )
Revised: The vocabulary considered has 86 English characters as well as four special tokens: start and end of sentence, padding and out-of-vocabulary tokens. D.3 Search Spaces We assume that the largest model in the search space is the maximal size that any of the clients can handle. The minimum cost is set to the fixed cost of the network (i.e. cost of non-searchable components).

Original: O pe r  a t i on 
Revised: Given this range, and number of clusters C , we define the first and last clusters at prct l , prct r percentiles of a randomly sampled set of models from the search space and linearly scale the C − 2 clusters in-between. All supernet search spaces are depicted in Fig. 5 and described below. C on v ( 3 )

Original: O pe r a t i on O pe r a t i on 
Revised: O pe r a t i on O pe r a t i on O pe r a t i on

Original: O pe r a t i on
Revised: O pe r a t i on

Original: D S C on v ( 3 )
Revised: D S C on v ( 3 )

Original: C on v ( 2 )
Revised: C on v ( 2 )

Original: G l oba l poo li ng
Revised: G l oba l poo li ng

Original: L i nea r repeated 4 times
Revised: L i nea r repeated 4 times

Original: O pe r a t i on
Revised: O pe r a t i on

Original: S k i p S k i p
Revised: S k i p S k i p

Original: O pe r a t i on
Revised: O pe r a t i on

Original: E m bedd
Revised: E m bedd

Original: i ng
Revised: i ng

Original: L i nea rc= c
Revised: L i nea r

Original: =same c=prev*1.5s=
Revised: [Figure] c=same c=prev*1.5s=

Original: C on v ( 3 )
Revised: C on v ( 3 )

Original: O pe r a t i on O pe r a t i on
Revised: O pe r a t i on O pe r a t i on

Original: G l oba l poo li ng
Revised: G l oba l poo li ng

Original: L i nea r repeated 2 times c= c
Revised: L i nea r [Figure] c

Original: =same
Revised: =same

Original: C on v ( 1 )
Revised: C on v ( 1 )

Original: A v g .
Revised: A v g .

Original: P oo l ( 3 ) c=prev*1.5s=
Revised: P oo l ( 3 ) c=prev*1.5s=

Original: O pe r a t i on c=same
Revised: O pe r a t i on c=same

Original: C on v ( 1 )
Revised: C on v ( 1 )

Original: A v g .
Revised: A v g .

Original: P oo l ( 3 ) c=prev*1.5s= repeated 2 times d= d=
Revised: P oo l ( 3 ) c=prev*1.5s= repeated 2 times d= d=

Original: The vocabulary considered has 86 English characters as well as four special tokens: start and end of

Original: [Figure]

Original: sentence, padding and out-of-vocabulary tokens.

Original: [Figure]

Original: D.3 Search Spaces

Original: [Figure]

Original: We assume that the largest model in the search space is the maximal size that any of the clientscan handle.

Original: The minimum cost is set to the fixed cost of the network (i.e. cost of non-searchablecomponents).

Original: Given this range, and number of clusters C , we define the first and last clusters atprct l , prct r percentiles of a randomly sampled set of models from the search space and linearly scalethe C − 2 clusters in-between.

Original: CIFAR-10 and CIFAR-100.
Revised: CIFAR-10 and CIFAR-100.

Original: We use a ResNet-like search space similar to the one used in, forexample, Cai et al.
Revised: We use a ResNet-like search space similar to the one used in, for example, Cai et al.

Original: Specifically, our model is a feedforward model with a fixed ( i.e. nonsearchable) stem layer followed by a sequence of 3 searchable blocks, each followed by a fixedreduction block.
Revised: Specifically, our model is a feedforward model with a fixed ( i.e. nonsearchable) stem layer followed by a sequence of 3 searchable blocks, each followed by a fixed reduction block.

Original: A standard block consists of 4 searchable layers organized again in a simplefeedforward manner.
Revised: A standard block consists of 4 searchable layers organized again in a simple feedforward manner.

Original: Operations within a standard block preserve shape of their inputs, but areallowed to change dimensions of intermediate results.
Revised: Operations within a standard block preserve shape of their inputs, but are allowed to change dimensions of intermediate results.

Original: On the other hand, the goal of reductionblocks is to reduce spatial resolution ( 2 × in each dimension) and increase number of channels ( 1 . 5 × ).
Revised: On the other hand, the goal of reduction blocks is to reduce spatial resolution ( 2 × in each dimension) and increase number of channels ( 1 . 5 × ).

Original: Reduction blocks are fixed throughout and consists of a depthwise separable convolution 3 × 3 , withthe first part performing spatial reduction and the second increasing the number of channels, and astandard 2 × 2 convolution applied to a residual link.
Revised: Reduction blocks are fixed throughout and consists of a depthwise separable convolution 3 × 3 , with the first part performing spatial reduction and the second increasing the number of channels, and a standard 2 × 2 convolution applied to a residual link.

Original: The sequence of blocks is finished with a globalaverage pooling per-channel and a classification layer outputting either 10 or 20/100 logits, whichis the only difference between the two models.
Revised: The sequence of blocks is finished with a global average pooling per-channel and a classification layer outputting either 10 or 20/100 logits, which is the only difference between the two models.

Original: Each convolution operation is followed by a batchnormalization (BN) and ReLU.
Revised: Each convolution operation is followed by a batch normalization (BN) and ReLU.

Original: In our experiments we considered the following candidate operations:
Revised: In our experiments we considered the following candidate operations:

Original: • standard convolution 1 × 1 with BN and ReLU• depthwise separable convolution 3 × 3 with expansion ratio (controlling the number ofintermediate channels) set to { 0 . 5 , 1 , 2 } , with each expansion ratio being an independentchoice in our search space• MobileNet-like block consisting of a convolution with kernel size k and expansion ratioe , followed by a Squeeze-and-Excitation layer, followed by a 1 × 1 convolution reverting the channels expansion, we considered { ( k = 1 , e = 2) , ( k = 3 , e = 0 . 5) , ( k = 3 , e =1) , ( k = 3 , e = 2) } • identity operation
Revised: • standard convolution 1 × 1 with BN and ReLU • depthwise separable convolution 3 × 3 with expansion ratio (controlling the number of intermediate channels) set to { 0 . 5 , 1 , 2 } , with each expansion ratio being an independent choice in our search space • MobileNet-like block consisting of a convolution with kernel size k and expansion ratio e , followed by a Squeeze-and-Excitation layer, followed by a 1 × 1 convolution reverting the channels expansion, we considered { ( k = 1 , e = 2) , ( k = 3 , e = 0 . 5) , ( k = 3 , e = 1) , ( k = 3 , e = 2) } • identity operation

Original: The stem layer was set to a 3 × 3 convolution outputting 64 channels.
Revised: The stem layer was set to a 3 × 3 convolution outputting 64 channels.

Original: Speech Commands.
Revised: Speech Commands.

Original: The model follows the one used for the CIFAR datasets but is made morelightweight – to roughly match what can be found in the literature – by reducing stem channels to 16and including only 1 (resp. 2) searchable operations in the first (resp.
Revised: The model follows the one used for the CIFAR datasets but is made more lightweight – to roughly match what can be found in the literature – by reducing stem channels toand including only 1 (resp. 2) searchable operations in the first (resp.

Original: last) two blocks.
Revised: last) two blocks.

Original: Additionally,reduction block only includes a single 1 × 1 convolution, that changes the number of channels,followed by a 3 × 3 average pooling that reduces spatial dimensions.
Revised: Additionally, reduction block only includes a single 1 × 1 convolution, that changes the number of channels, followed by a 3 × 3 average pooling that reduces spatial dimensions.

Original: We also include additionalcandidate operations:
Revised: We also include additional candidate operations:

Original: • standard convolution 3 × • depthwise separable convolution with kernel size 1 and expansion ratio
Revised: • standard convolution 3 ×• depthwise separable convolution with kernel size 1 and expansion ratio

Original: All other operations from the CIFAR model are also included.
Revised: All other operations from the CIFAR model are also included.

Original: Shakespeare.
Revised: Shakespeare.

Original: We base our design on the model used by Horvath et al.
Revised: We base our design on the model used by FjORD.

Original: Specifically, the modelis a recurrent network that begins with a fixed embedding layer outputting 8-dimensional featurevector per each input character.
Revised: Specifically, the model is a recurrent network that begins with a fixed embedding layer outputting 8-dimensional feature vector per each input character.

Original: This is then followed by a searchable operation #1 that increasesdimensionality from 8 to 128; in parallel, we have a searchable skip connection #1 operation whoseoutput is added to the output of the operation
Revised: This is then followed by a searchable operation #1 that increases dimensionality from 8 to 128; in parallel, we have a searchable skip connection #1 operation whose output is added to the output of the operation

Original: Later there is another searchable operation #2 withits own skip connection #2, both keeping the hidden dimension at 128.
Revised: Later there is another searchable operation #2 with its own skip connection #2, both keeping the hidden dimension at 128.

Original: Again, their outputs are addedand passed to the final classification layer.
Revised: Again, their outputs are added and passed to the final classification layer.

Original: Candidate operations for each of the four searchable layers are mainly the same, with minor adjustments made to make sure a valid model is always constructed.
Revised: Candidate operations for each of the four searchable layers are mainly the same, with minor adjustments made to make sure a valid model is always constructed.

Original: These include:
Revised: These include:

Original: • an LSTM layer

Original: [27],• a GRU layer [15],• a LiGRU layer with tanh activation [54], • a QuasiRNN layer [11], • a simple Linear layer (no recurrent connection) followed by a sigmoid activation,• a 1D convolution with kernel 5 spanning time dimension (looking at the current characterand 4 previous ones), followed by a sigmoid activation (no normalisation), • identity operation (only included in later operation, after feature dimension has been increased to 128), 
Revised: • an LSTM layer [32], • a GRU layer [19], • a LiGRU layer with tanh activation [65], • a QuasiRNN layer [14], • a simple Linear layer (no recurrent connection) followed by a sigmoid activation, • a 1D convolution with kernel 5 spanning time dimension (looking at the current character and 4 previous ones), followed by a sigmoid activation (no normalisation), • identity operation (only included in later operation, after feature dimension has been increased to 128), • zero operation, outputting zeros of the correct shape (only included in skip connection layers).

Original: • zero operation, outputting zeros of the correct shape (only included in skip connectionlayers).

Original: For LiGRU and QuasiRNN we used implementations provided by the Speechbrain project [55], afterminor modifications.
Revised: For LiGRU and QuasiRNN we used implementations provided by the Speechbrain project [66], after minor modifications.

Original: For others, we used standard operations from PyTorch.
Revised: For others, we used standard operations from PyTorch.

Original: All operations wereused as unidirectional.
Revised: All operations were used as unidirectional.

Original: We did not use any normalisation throughout the model.
Revised: We did not use any normalisation throughout the model.

Original: D.4 Tiers: definition and client assignment
Revised: D.4 Tiers: definition and client assignment

Original: With FedorAS the discovery and training of architectures happens in a tier-aware fashion as afederated process.
Revised: With FedorAS the discovery and training of architectures happens in a tier-aware fashion as a federated process.

Original: In this work we considered splitting each search space along the FLOPs dimensions(but other splits are possible, e.g: energy, peak-memory usage, etc – or a combination of these).
Revised: In this work we considered splitting each search space along the FLOPs dimensions (but other splits are possible, e.g: energy, peak-memory usage, etc – or a combination of these).

Original: Fig. 4illustrates the span in terms of model parameters and FLOPs of each search space presented in D.3and the split along the FLOPs dimensions for each of them.
Revised: Fig.illustrates the span in terms of model parameters and FLOPs of each search space presented in D. and the split along the FLOPs dimensions for each of them.

Original: These search spaces vary considerably interms of size and span, which motivated us to use different number of tiers or client-to-tier assignmentstrategies.
Revised: These search spaces vary considerably in terms of size and span, which motivated us to use different number of tiers or client-to-tier assignment strategies.

Original: Tab 6 shows the the FLOPs ranges considered for each tier, the number of models in thosesub-spaces exposed to FedorAS as well as the ratio of models in the entire search space that fall ontoeach tier.
Revised: Tab. 6 shows the the FLOPs ranges considered for each tier, the number of models in those sub-spaces exposed to FedorAS as well as the ratio of models in the entire search space that fall onto each tier.

Original: [Table]
Revised: [Table]

Original: Identifying FLOPs ranges for each tier.
Revised: Identifying FLOPs ranges for each tier.

Original: Regardless of the dataset, we follow a common approachto divide the FLOPs dimension that each dataset spans.
Revised: Regardless of the dataset, we follow a common approach to divide the FLOPs dimension that each dataset spans.

Original: The main aim is to finely control how manymodels fall onto the smallest/largest tier, given that these are considerably sparse regions of theentire search space.
Revised: The main aim is to finely control how many models fall onto the smallest/largest tier, given that these are considerably sparse regions of the entire search space.

Original: If we were to evenly split the FLOPs dimension, almost no architecture wouldfall onto the largest tier.
Revised: If we were to evenly split the FLOPs dimension, almost no architecture would fall onto the largest tier.

Original: To simplify the process of defining where each tier’s boundaries lie, wefollow these steps which require two hyperparameters: (1) we construct a long array of FLOPs frommodels sampled from the search space and this array is then sorted from lowest to highest; then,(2) we normalise this array of FLOPs and compute the cumulative sum of such normalised array;finally, (3) we identify the lower limit of the highest tier as maximum FLOPs value found in the firstρN, ρ ∈ [0 , 1] , elements of the array where N is the number of samples taken (we found 100K towork well).
Revised: To simplify the process of defining where each tier’s boundaries lie, we follow these steps which require two hyperparameters: (1) we construct a long array of FLOPs from models sampled from the search space and this array is then sorted from lowest to highest; then, (2) we normalise this array of FLOPs and compute the cumulative sum of such normalised array; finally, (3) we identify the lower limit of the highest tier as maximum FLOPs value found in the first ρN, ρ ∈ [0 , 1] , elements of the array where N is the number of samples taken (we found 100K to work well).

Original: Essentially, we find the FLOPs for which the approximated PDF over the FLOPS of agiven search space doesn’t surpass ρ .
Revised: Essentially, we find the FLOPs for which the approximated PDF over the FLOPS of a given search space doesn’t surpass ρ .

Original: Once that FLOPs value is identified, the FLOPs range is evenlysplit for the tiers below and the higher limit for the largest tier becomes the maximum FLOPs a modelin the search space can have.
Revised: Once that FLOPs value is identified, the FLOPs range is evenly split for the tiers below and the higher limit for the largest tier becomes the maximum FLOPs a model in the search space can have.

Original: This is the approach for CIFAR-10 and CIFAR-100 as illustrated in
Revised: This is the approach for CIFAR-10 and CIFAR-100 as illustrated in

Original: Fig.
Revised: Fig.

Original: 4 (a) and (b).
Revised: 6 (a) and (b).

Original: For SpeechCommands and Shakespeare we introduce a similar approach to give awider range to the smallest tier.
Revised: For SpeechCommands and Shakespeare we introduce a similar approach to give a wider range to the smallest tier.

Original: We refer to ρ L and ρ H as the PDF ratios to identify the boundariessplitting Tier1&2 and Tier3&4 respectively.
Revised: We refer to ρ L and ρ H as the PDF ratios to identify the boundaries splitting Tier1&2 and Tier3&4 respectively.

Original: The concrete values for each dataset as well as the
Revised: The concrete values for each dataset as well as the

Original: FLOPs values for each tier boundary are shown in Tab 6.
Revised: FLOPs values for each tier boundary are shown in Tab. 6.

Original: Clients to tiers assignment.
Revised: Clients to tiers assignment.

Original: For CIFAR-10/100 clients are uniformly assigned to a cluster of devicesor tier.
Revised: For CIFAR-10/100 clients are uniformly assigned to a cluster of devices or tier.

Original: For these datasets we consider four tiers, so each ends up containing 25% the clients resultingin 25 clients for CIFAR-10 and 125 clients for CIFAR-100.
Revised: For these datasets we consider four tiers, so each ends up containing 25% the clients resulting in 25 clients for CIFAR-10 and 125 clients for CIFAR-100.

Original: Similarly, for Shakespeare clients arealso uniformly assigned to a tier, resulting in 143 clients per tier.
Revised: Similarly, for Shakespeare clients are also uniformly assigned to a tier, resulting in 143 clients per tier.

Original: For SpeechCommands, we designeda more challenging setup and divide the clients into tiers as follows: 80% of clients are assigned tobe tier-1 devices, 17.5% are Tier-4 devices and the rest is evenly split into Tier-2 and Tier-3.
Revised: For SpeechCommands, we designed a more challenging setup and divide the clients into tiers as follows: 80% of clients are assigned to be tier-1 devices, 17.5% are Tier-4 devices and the rest is evenly split into Tier-2 and Tier-3.

Original: Thisdistribution better represents the the types of systems in the wild that perform keyword-spotting [67],where the majority of the commercially deployed systems run these applications on low end CPUs ormicrocontrollers due to their always-on nature.
Revised: This distribution better represents the the types of systems in the wild that perform keyword-spotting [79], where the majority of the commercially deployed systems run these applications on low end CPUs or microcontrollers due to their always-on nature.

Original: Across datasets, the client-to-tier assignments wasdone irrespective of the amount of data these contained or distribution over the labels. 
Revised: Across datasets, the client-to-tier assignments was done irrespective of the amount of data these contained or distribution over the labels. An alternative client to tier allocation is provided in Sec.

Original:  [Figure]
Revised: I. [Figure]

Original: D.5 Hyperparameters
Revised: D.5 Hyperparameters

Original: Here we present the hyperparameters used across all datasets and tasks to generate the resultspresented in Sec. 3.
Revised: Here we present the hyperparameters used across all datasets and tasks to generate the results presented in Sec. 4.

Original: In Tab. 7 we show the hyperparamters utilised for the first stage in the FedorASframework: federated training of the supernet.
Revised: In Tab. 7 we show the hyperparamters utilised for the first stage in the FedorAS framework: federated training of the supernet.

Original: After this stage, the best model for each tier isextracted from the supernet and then, they get finetuned in a per-tier aware manner (i
Revised: After this stage, the best model for each tier is extracted from the supernet and then, they get finetuned in a per-tier aware manner (i

Original: [Figure]
Revised: [Figure]

Original: e. clients intier T and above can finetune a model that belong to tier T ).
Revised: e. clients in tier T and above can finetune a model that belong to tier T ).

Original: The hyperparameters of these twoconsecutive stages are shown in Tab.8.
Revised: The hyperparameters of these two consecutive stages are shown in Tab.

Original: [Table]
Revised: [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: Regarding the End2End results (i
Revised: Regarding the rand-init results (i

Original: [Table]
Revised: [Table]

Original: e. models discovered by FedorAS but trained from scratch –discarding the weights given by the supernet)
Revised: e. models discovered by FedorAS but trained from scratch – discarding the weights given by the supernet)

Original: we present the hyperparameters in Tab. 9.
Revised: we present the hyperparameters in Tab. 9.

Original: Thesehyperparameters were the ones used to generate the results in Tab. 5 and 12 as well as Fig.
Revised: These hyperparameters were the ones used to generate the results in Tab. 5 and 12 as well as Fig.

Original: In
Revised: In

Original: Sec. 3.2, we present a CIFAR-100 result that largely outperforms existing federated baseline of [56].
Revised: Sec. 4.2, we present a CIFAR-100 result that largely outperforms existing federated baseline of [67].

Original: This was achieved by a model discovered by FedorAS but trained in End2End setting followingthe setup as in [56]: 10 clients per round for 4k rounds using batch 20, starting learning rate of 0.1decaying to 0.01 following a cosine scheduling.
Revised: This was achieved by a model discovered by FedorAS but trained in the rand-init setting following the setup as in [67]: 10 clients per round for 4k rounds using batch 20, starting learning rate of 0. decaying to 0.01 following a cosine scheduling.

Original: [Table]
Revised: [Table]

Original: In Table 2  we compare FedorAS against FedNAS in the cross-sile setting with 100 clients and 10clients randomly sampled on each round.
Revised: In Tab. 2 of Sec. 4.2 we compared FedorAS against FedNAS in the cross-sile setting with 100 clients and 10 clients randomly sampled on each round.

Original: Both methods follow a substantially different approachas far as NAS is concern and, as a result, FedNAS has a significantly higher memory peak than
Revised: Both methods follow a substantially different approach as far as NAS is concern and, as a result

Original:  FedorAS for the same batch size.
Revised: [Table] FedNAS has a significantly higher memory peak than FedorAS for the same batch size.

Original: For example, for a batch size of 64 images, FedorAS sees a memorypeak of 998MB whereas FedNAS requires 7674MB.
Revised: For example, for a batch size of 64 images, FedorAS sees a memory peak of 998MB whereas FedNAS requires 7674MB.

Original: Similarly, both methods translate in differentcompute footprints for each client.
Revised: Similarly, both methods translate in different compute footprints for each client.

Original: For example, FedNAS requires on average 716 GFLOPs per client(assuming each client), while clients in FedorAS need 280 GFLOPs for the same amount of localepochs and data in the client.
Revised: For example, FedNAS requires on average 716 GFLOPs per client (assuming each client), while clients in FedorAS need 280 GFLOPs for the same amount of local epochs and data in the client.

Original: Due to these differences, we aimed at normalising these aspects tomake the comparison fair.
Revised: Due to these differences, we aimed at normalising these aspects to make the comparison fair.

Original: This results  in the table below, which extends the content of Tab. [Table]
Revised: These results are shown in Tab. 10, which extends the content of Tab. 2.

Original: [Table]
Revised: [Table]

Original: D.6 Baselines
Revised: D.7 Baselines

Original: In this section we faithfully describe what each baseline represents in the experiments of the mainpaper and the appendix to clarify with what we are comparing in each section.
Revised: In this section we faithfully describe what each baseline represents in the experiments of the main paper and the appendix to clarify with what we are comparing in each section.

Original: Tier-unaware (Fig.2): This baseline represents model architectures that have been trained end-toend in a federated manner without any awareness of client eligibility.
Revised: Tier-unaware (Fig.3): This baseline represents model architectures that have been trained end-toend in a federated manner without any awareness of client eligibility.

Original: We are using FedAvg withhyperparameters similar to those presented in Tab. 9.
Revised: We are using FedAvg with hyperparameters similar to those presented in Tab. 9.

Original: Tier-aware (Fig.2): This baseline adds client eligibility awareness to the previous baseline.
Revised: Tier-aware (Fig.3): This baseline adds client eligibility awareness to the previous baseline.

Original: Thismeans that models of certain footprint can be trained only on clients of the eligible cluster and above.
Revised: This means that models of certain footprint can be trained only on clients of the eligible cluster and above.

Original: FjORD [28] (Fig.2, Tab. 4): FjORD is a baseline that is tackling system heterogeneity by means of
Revised: FjORD [33] (Fig.3, Tab. 4): FjORD is a baseline that is tackling system heterogeneity by means of

Original: Ordered Dropout.
Revised: Ordered Dropout.

Original: This particular variant is assuming the experimetnal setup of the original paperwhich shards the CIFAR-10 dataset per client without LDA.
Revised: This particular variant is assuming the experimental setup of the original paper which shards the CIFAR-10 dataset per client without LDA.

Original: The Shakespeare setup remains the sameas ours.
Revised: The Shakespeare setup remains the same as ours.

Original: FjORD-LDA [28] (Fig.2, Tab. 4): For this baseline, we took FjORD and ran it on the same numberof clusters as FedorAS , with LDA for CIFAR-10.

Original: ZeroFL [52] (Tab. 4):
Revised: FjORD-LDA [33] (Fig.3, Tab. 4):

Original: For this baseline, we borrow the results of respective paper for CIFAR-10 with LDA= { 1000 , 1 } , which assumes the same setup as ours. We present the results for sparsity level of90% and annotate its footprint as the original model FLOPS and the number of non-zero parameters.
Revised: For this baseline, we implemented FjORD and ran it on the same number of clusters as FedorAS , with LDA for CIFAR-10.

Original: FedNAS [25] (Tab. 2, 3): FedNAS was one of the first papers attempting to perform NAS in afederated setting.
Revised: FedNAS [30] (Tab. 2, 3): FedNAS was one of the first papers attempting to perform NAS in a federated setting.

Original: Its search is DARTS-based and it was primarily aimed for the cross-silo setting.
Revised: Its search is DARTS-based and it was primarily aimed for the cross-silo setting.

Original: Cross-device FL NAS of limited scale (20 total clients with 4 clients sample per round) was alsoshowcased in the original paper.
Revised: Cross-device FL NAS of limited scale (20 total clients with 4 clients sample per round) was also showcased in the original paper.

Original: We adjusted the code of FedNAS to perform cross-silo and crossdevice FL NAS with setups of varying training footprints by adjusting the amount of local epochsand batch size.
Revised: We adjusted the code of FedNAS to perform cross-silo and crossdevice FL NAS with setups of varying training footprints by adjusting the amount of local epochs and batch size.

Original: We have kept the original search space.
Revised: We have kept the original search space.

Original: SPIDER [49] (Tab. 3): SPIDER is another paper performing personalised Federated NAS in thecross-silo setting.
Revised: SPIDER [59] (Tab. 3): SPIDER is another paper performing personalised Federated NAS in the cross-silo setting.

Original: As there is no publicly available codebasse, we assume their setting with FedorASand compare on CIFAR-10. 
Revised: As there is no publicly available codebasse, we assume their setting with FedorAS and compare on CIFAR-10. ZeroFL [63] (Tab. 4):

Original: End2End (Tab. 5, 12, 14):
Revised: For this baseline, we borrow the results of respective paper for CIFAR-10 α = { 1 , 1000 } , which assumes the same setup as ours. We present the results for sparsity level of 90% and annotate its footprint as the original model FLOPS and the number of non-zero parameters.

Original: This baseline refers to the concept of running the search, coming up withan architecture and subsequently re-initialising the weights and running a conventional federatedtraining setup.
Revised: Oort [43] (Tab. 4): The Oort framework proposes a participation sampling criterion by which clients are sampled based on their utility (i.e. how much their data can contribute to the global model) while also taking the device capabilities into consideration. Over the course of FL training, the sampling of clients with high data and system utility is prioritised to reduce the time needed to convergence. For SpeechCommands (see Tab. 4), Oort made use of a ResNet-34 model.

Revised: PyramidFL [48](Tab. 4): At a high-level, PyramidFL proposes a framework similar to that in Oort.

Revised: The core difference between these two methods is that PyramidFL leverages more fine-grained statistics when assessing the contribution potential (i.e. utility) of the selected clients.

Revised: It also uses

Revised: ResNet-34 for SpeechCommands.

Revised: rand-init (Tab. 5, 12, 14): This baseline refers to the concept of running the search, coming up with an architecture and subsequently re-initialising the weights and running a conventional federated training setup.

Original: It shows vanilla scaling that would be obtained if we simply trained identified modelsin a standard way, using random initialization and full FL training using standard practices.
Revised: It shows vanilla scaling that would be obtained if we simply trained identified models in a standard way, using random initialization and full FL training using standard practices.

Original: Modelarchitectures are kept the same between FedorAS and “End2End” experiments.
Revised: Model architectures are kept the same between FedorAS and “ rand-init ” experiments.

Original: Models belonging tohigher tiers, as a whole, are never trained on data belonging to devices from lower tiers.
Revised: Models belonging to higher tiers, as a whole, are never trained on data belonging to devices from lower tiers.

Original: E Comparison against random search
Revised: E Comparison against random search

Original: In the main paper we compare our FedorAS to the state-of-the-art models from the literature (Sec. 3.2)and also investigate the impact of initializing models with weights from a supernet (Sec. 3.3).
Revised: In the main paper we compare our FedorAS to the state-of-the-art methods from the literature (Sec. 4.2) and also investigate the impact of initializing models with weights from a supernet (Sec. 4.3).

Original: Herewe present one more important baseline for any NAS algorithm - comparison to a random search.
Revised: Here we present one more important baseline for any NAS algorithm - comparison to a random search.

Original: Specifically, while we have already shown that models found by FedorAS in most cases benefit fromsupernet training, this does constitute a conclusive argument justifying the necessity of the entireprocess in a broader context.
Revised: Specifically, while we have already shown that models found by FedorAS in most cases benefit from supernet training, this does constitute a conclusive argument justifying the necessity of the entire process in a broader context.

Original: Perhaps there exist models that do not need to be initialized from asupernet but, since the focus of our work is on supernet training, we missed them due to biasedconditioning?
Revised: Perhaps there exist models that do not need to be initialized from a supernet but, since the focus of our work is on supernet training, we missed them due to biased conditioning?

Original: [Table]
Revised: [Table]

Original: In order to check this hypothesis we run a simple random search algorithm
Revised: 81.01 ± 0.46 81.53 ± 0.29 80.64 ± 0.66 80.85 ± 0. - 66.75 ± 0.74 70.85 ± 1.53 69.00 ± 0.18 66.08 ± 2. - CIFAR45.25 ± 0.13 45.84 ± 0.18 45.42 ± 0.39 45.07 ± 0. - 34.41 ± 0.90 37.64 ± 0.46 36.98 ± 0.64 35.45 ± 0.

Revised: - SpeechCommands 80.19 ± 1.78 80.47 ± 1.69 81.00 ± 1.58 80.56 ± 0.

Original: [Table]
Revised: - 82.84 ± 1.08 81.17 ± 0.26 81.10 ± 0.74 80.61 ± 0. - Shakespeare 3.43 ± 0. 3.39 ± 0. 3.38 ± 0.

Original: to get an estimate of the
Revised: 3.40 ± 0.01 3.42 ± 0.01 3.86 ± 0. 3.54 ± 0. 3.45 ± 0.

Original: [Table]
Revised: 3.80 ± 0.30 3.60 ± 0.

Original:  expected best case performance of models from our search spaces when trained following standard
Revised: In order to check this hypothesis we run a simple random search algorithm, to get an estimate of the expected best case performance of models from our search spaces when trained following standard

Original: FL procedure.
Revised: FL procedure.

Original: This simple baseline was allowed to train random models until the total training costexceeded the cost of running FedorAS – this turned out to be equivalent to roughly 40 fully-trainedmodels.
Revised: This simple baseline was allowed to train random models until the total training cost exceeded the cost of running FedorAS – this turned out to be equivalent to roughly 40 fully-trained models.

Original: After we run out of the training budget, we simply get the best model for each tier as oursolution to NAS.
Revised: After we run out of the training budget, we simply get the best model for each tier as our solution to NAS.

Original: We repeated the entire process 3 times for each search space and report average andstandard deviation compared to the average performance of models found by FedorAS in Tab. 11.
Revised: We repeated the entire process 3 times for each search space and report average and standard deviation compared to the average performance of models found by FedorAS in Tab. 11.

Original: Additionally, we also plot detailed performance of each model trained during this process in Fig. 5for the sake of completeness.[Figure]
Revised: Additionally, we also plot detailed performance of each model trained during this process in Fig.for the sake of completeness.

Original: The proposed OPA aggregation scheme converges faster than an alternative aggregation method thatdoes FedAvg of the updated supernets. We show this in Fig. When supernets from each clientparticipating in the round get aggregated with OPA, the resulting supernet is consistently better interms of validation accuracy than when aggregation is done with FedAvg.
Revised: Noticeably, FedorAS achieves significantly better results in almost all cases, with SpeechCommands being the only exception.

Original: This difference becomesmore evident as B Φ comm is reduced, i.e., as smaller portions of the supernet are sent to  the clients. In Fig. 6 (d) we asses the feasibility of reducing the number of rounds and end the federated supernettraining stage when validation accuracy reaches approximately 70%.
Revised: We suspect this is due to problem with fine-tuning rather than architectures themselves – specifically, we witness accuracy of the discovered models can vary significantly as we repeat the fine-tuning process (this is also visible in the case of full training, although the extend is smaller).

Original: Those points correspond to250 rounds and 150 rounds for the setting when 50% and 25% of the supernet is sent to the clients,respectively. The results  thatthese settings yield show a significant loss (compared to their respectivesettings but over 500 rounds) and we therefore also run the same settings but when allowing for 50 additional runs. We observe a significant jump in per-tier model performance.
Revised: Consequently, although on average FedorAS performs slightly worse, in many cases the best results surpass that of random search – because of that we suspect that fine-tuning for longer would improve FedorAS ’s performance.

Original: We leave asfuture work investigating alternative metrics to more accurately (but without incurring into heavycomputational costs) measure the quality of the supernet at any given training iteration and leveragethis to better inform an early stopping mechanism to further reduce communication costs [Figure] [Figure] [Figure]
Revised: We leave this for future work, considering that a single shortcoming like that does not seem significant in the light of the rest of our results.

Original: [Figure]

Original: [Figure] [Figure] [Figure] G Impact of weight-sharing on different task training
Revised: F Impact of weight-sharing on different task training

Original: In this section, we expand on the analysis of Sec. 3.3.
Revised: In this section, we expand on the analysis of Sec. 4.3.

Original: Specifically for CIFAR-100 and SpeechCommands, we also create a second scenario, where not all clients train in the same domain of labels.
Revised: Specifically for CIFAR-100 and SpeechCommands, we also create a second scenario, where not all clients train in the same domain of labels.

Original: For
Revised: For

Original: CIFAR-100, there are 20 superclasses that are coarse-grained categories of the 100 standard labels.
Revised: CIFAR-100, there are 20 superclasses that are coarse-grained categories of the 100 standard labels.

Original: In contrast, for SpeechCommands, there are 12 and 35-class label sets, with the latter annotatingthe “other” class more specifically.
Revised: In contrast, for SpeechCommands, there are 12 and 35-class label sets, with the latter annotating the “other” class more specifically.

Original: In both cases, we assume a non-uniform distribution of clientsto clusters, assuming that most data (75-80%) reside on lower-tier devices, whereas the two highertier-devices (holding 20-25% of data) can train on the fine-grained label set.
Revised: In both cases, we assume a non-uniform distribution of clients to clusters, assuming that most data (75-80%) reside on lower-tier devices, whereas the two higher tier-devices (holding 20-25% of data) can train on the fine-grained label set.

Original: Our aim is to test whetherfew data allocation on the high tier devices can benefit from the knowledge and feature extractionlearned from the coarse-grained label set.
Revised: Our aim is to test whether few data allocation on the high tier devices can benefit from the knowledge and feature extraction learned from the coarse-grained label set.

Original: We adopt two different setups.
Revised: We adopt two different setups.

Original: For CIFAR-100, we trainboth tasks simultaneously, having essentially two distinct linear layers across tiers (1,2) and (3,4).
Revised: For CIFAR-100, we train both tasks simultaneously, having essentially two distinct linear layers across tiers (1,2) and (3,4).

Original: Wecall this setup multi-task.
Revised: We call this setup multi-task.

Original: On the other hand, for SpeechCommands, we train all clients on the samecoarse domain and subsequently transfer learn a fine-grained 35 classes linear layer for the client oftiers (3,4).
Revised: On the other hand, for SpeechCommands, we train all clients on the same coarse domain and subsequently transfer learn a fine-grained 35 classes linear layer for the client of tiers (3,4).

Original: We fine-tune the linear layer keeping the rest of the network frozen for 25 epochs and thenallow for finetuning of the whole network for another 100.
Revised: We fine-tune the linear layer keeping the rest of the network frozen for 25 epochs and then allow for finetuning of the whole network for another 100.

Original: We present the results in Tab. 12.
Revised: We present the results in Tab. 12.

Original: [Table]
Revised: [Table]

Original: In both cases FedorAS learns better models and transfer knowledge from the low-tiers to high-tiersand vice-versa through weight sharing of the supernet.
Revised: In both cases FedorAS learns better models and transfer knowledge from the low-tiers to high-tiers and vice-versa through weight sharing of the supernet.

Original: Indicatively, for CIFAR-100 we are ableto achieve +14 .{++14.
+}
Revised: Indicatively, for CIFAR-100 we are able to achieve {++14.
+}

Original: 91 p.p (percentage points) of accuracy compared to training the same architecturesend-to-end on eligible devices.
Revised: pp (percentage points) of accuracy compared to training the same architectures end-to-end on eligible devices.

Original: Similarly, we achieve +1 . 6 p.p higher accuracy for SpeechCommands.
Revised: Similarly, we achieve +1.6 pp higher accuracy for SpeechCommands [Table]

Original: H Sensitivity to size of validation set
Revised: G Convergence and Impact of Supernet Sampling

Original: In order to more meaningfully examine the impact of the validation set size, we run experimentson all datasets with 20% and 50% of the size of the initial global validation set. This means thatduring Stage-II ofFedorAS , architectures sampled from the supernet are scored using a fraction ofthe global validation set.
Revised: The proposed OPA aggregation scheme converges faster than an alternative aggregation method that does FedAvg of the updated supernets.

Original: These validation subsets are extracted uniformly.
Revised: We show this in Fig. 8.

Original: After obtaining the bestperforming models for each tier, these are fine-tuned in a federated fashion (Stage-III in FedorAS )for 100 rounds just like it was done for Tab. 5. We maintain the same hyperparameters as those usedto generate Tab. 5. Results are depicted in Tab 13 for all datasets.
Revised: When supernets from each client participating in the round get aggregated with OPA, the resulting supernet is consistently better in terms of validation accuracy than when aggregation is done with FedAvg. This difference becomes more evident as B Φ comm is reduced, i.e., as smaller portions of the supernet are sent to the clients.

Original: [Table]
Revised: In Fig.

Original: I Different allocation of clients to tiers
Revised: 8 (d) we asses the feasibility of reducing the number of rounds and end the federated supernet training stage when validation accuracy reaches approximately 70%. Those points correspond to 250 rounds and 150 rounds for the setting when 50% and 25% of the supernet is sent to the clients, respectively.

Original: In this experiment, contrary to what was described in Appendix D.4, we adopt the device capabilitiestrace from Oort [38] for our device to cluster allocation and performed training on CIFAR-10 and CIFAR-100.
Revised: The results that these settings yield show a significant loss (compared to their respective settings but over 500 rounds) and we therefore also run the same settings but when allowing for 50 additional runs.

Original: Results are depicted on Tab. 14.
Revised: We observe a significant jump in per-tier model performance.

Original: It can be witnessed that FedorAS is able to operateeven under harsher heterogeneity conditions and still output state-of-the-art models in the federatedsetting.
Revised: We leave as future work investigating alternative metrics to more accurately (but without incurring into heavy computational costs) measure the quality of the supernet at any given training iteration and leverage this to better inform an early stopping mechanism to further reduce communication costs.

Original: [Table]
Revised: [Figure]

Original: J Fairness
Revised: H Fairness

Original: Up to this point [Table] we have reported on the global test accuracy, which is a union over the test sets ofthe clients for all datasets but SpeechCommands [Table]
Revised: Up to this point, we have reported on the global test accuracy, which is a union over the test sets of the clients for all datasets but SpeechCommands, which defines a separate set of clients mutually exclusive to the ones participating in federated training.

Original: which defines a separate set of clients mutuallyexclusive to the ones participating in federated training.

Original: In this section we want to measure howthe ability of our system to aggregate knowledge across tiers during supernet training affects thefairness in participation.
Revised: In this section we want to measure how the ability of our system to aggregate knowledge across tiers during supernet training affects the fairness in participation.

Original: We quantify this through variance and worst-case statistical measures ofclient performance on their respective test set.
Revised: We quantify this through variance and worst-case statistical measures of client performance on their respective test set.

Original: These test sets are considered to be following the samedistribution (LDA or pre-split) of their train and validation sets.
Revised: These test sets are considered to be following the same distribution (LDA or pre-split) of their train and validation sets.

Original: [Table] Results are shown on Tab. 15 for the different datasets offering per-client test data splits for all clients.
Revised: Results are shown on Tab. 13 for the different datasets offering per-client test data splits for all clients.

Original: It can be seen that FedorAS leads consistently to better performance compared to end-to-end FLtrained networks of the same architectures, where only eligible clients can train models.
Revised: It can be seen that FedorAS leads consistently to better performance compared to end-to-end FL trained networks of the same architectures, where only eligible clients can train models.

Original: With respectto the standard deviation of per client performance, we witness FedorAS offering lower variance,except for the case of CIFAR-100.
Revised: With respect to the standard deviation of per client performance, we witness FedorAS offering lower variance, except for the case of CIFAR-100.

Original: We consider this to be a consequence of our significantly higheraccuracy.
Revised: We consider this to be a consequence of our significantly higher accuracy.

Original: Similar behaviour, is also witnessed when we measure variance per tier of devices.
Revised: Similar behaviour, is also witnessed when we measure variance per tier of devices.

Original: Last,we also showcase the worst-case result of a clients performance in the last column of the table.
Revised: Last, we also showcase the worst-case result of a clients performance in the last column of the table.

Original: An in-depth view of how each client behaves is depicted in Fig.
Revised: An in-depth view of how each client behaves is depicted in Fig.

Original: 7 for CIFAR-10, where we showperformance of Fedoras vs. end-to-end FL-trained models per client. 
Revised: 9 for CIFAR-10, where we show performance of Fedoras vs. end-to-end FL-trained models per client. [Table] [Figure] I Different allocation of clients to tiers

Revised: In this experiment

Original: [Figure]
Revised: [Table]

Revised: contrary to what was described in Appendix D.

Revised: [Table]

Revised: we adopt the device capabilities trace from Oort [43] for our device to cluster allocation and performed training on CIFAR-10 and

Revised: CIFAR-100.

Revised: Results are depicted on Tab. 14.

Revised: It can be witnessed that FedorAS is able to operate even under harsher heterogeneity conditions and still output competitive models in the federated setting.

Revised: [Table]

Revised: J Evaluation of the Search Phase J.1 Sensitivity to size of validation set

Revised: In order to more meaningfully examine the impact of the validation set size

Revised: [Table]

Revised: we run experiments on all datasets with 20% and 50% of the size of the initial global validation set.

Revised: This means that during Stage-II of FedorAS , architectures sampled from the supernet are scored using a fraction of the global validation set.

Revised: These validation subsets are extracted uniformly.

Revised: After obtaining the best performing models for each tier, these are fine-tuned in a federated fashion (Stage-III in FedorAS ) for 100 rounds just like it was done for Tab. 5.

Revised: We maintain the same hyperparameters as those used to generate Tab. 5.

Revised: Results are depicted in Tab. 15 for all datasets.

Revised: [Table]

Revised: J.2 Federated Search

Revised: In this Section, we provide additional details and results concerning our federated variant of NSGA-II, discussed in Sec.

Revised: 4.5 of the main paper.

Revised: Below, we provide some context about how the algorithm works, how we setup the federated experiments and commentary of the results on CIFAR10 { 1000 , 1 , 0 . 1 } and CIFAR-100.

Revised: Algorithm details.

Revised: Federation is achieved by delegating evaluation of models to individual clients.

Revised: This means that evaluation is done stochastically on clients’ local datasets.

Revised: This setting is similar to Federated Evaluation in [ ? ].

Revised: In order to avoid sending a large number of models to clients, thus saving communication cost, we leverage the fact that NSGA-II operates in “batches" of models – in each iteration, a number of models (i.e. sample size ) from the population is selected to produce new models that replace the ones not selected.

Revised: Since models from a single “batch” are all selected at the same time, we do not need to evaluate them sequentially.

Revised: Instead, considering that weights between architectures are shared, we can construct a minimal supernet that encapsulates all selected models and send it to relevant clients.

Revised: This way, we can achieve parallel evaluation of all selected models and optimal 4 communication cost for a given “batch”.

Revised: Experimental setting.

Revised: The experimental setting is following exactly what was used in other experiments.

Revised: Specifically, population and sample sizes were the same as the ones used in centralised

Revised: NSGA-II (128 and 64, respectively), and federated evaluation (FE) rounds were analogous to rounds during federated training of the supernet from which models to evaluate were extracted.

Revised: Concretely, number of clients, allocation of clients to tiers, allocation of data to clients, number of available clients per round, and client selection mechanism were all exactly the same for FE as the ones used during the relevant Stage 1 of FedorAS .

Revised: However, to present more meaningful results, we assumed that as we fluctuate the number of clients used to evaluate models (number of evaluation rounds) additional clients are always unique.

Revised: In other words, reshuffling and "forgetting" of clients is happening only between federated NSGA-II iterations.

Revised: Consequently, it is possible that: i) if all clients are used to evaluate a model (e.g. 10 evaluation rounds for CIFAR-10), FE is equivalent to centralised evaluation; ii) with a relatively small number of FE rounds, it is possible that one set of models produced by NSGA-II is evaluated on a completely disjoint set of data to another “batch”; iii) similarly, we do not enforce in any way that all clients have to be used (unless we use all clients in a single iteration of NSGA-II).

Revised: Finally, keeping consistent with the rest of the paper, we performed evaluation in a tier-aware manner, meaning that models belonging to higher tiers could only access a subset of all validation data based on client eligibility.

Revised: [Figure]

Revised: This is the reason why relevant curves finish at a different maximum number of FE rounds, that is a smaller number of FE rounds (i e. Federated NSGA-II iterations) is needed to exhaust the set of eligible clients

Revised: Results.

Revised: Fig.

Revised: 10 complements Fig.

Revised: 4 and presents results across three different experimental settings, with varying level of non-IID-ness ( α ∈ { 1 , 0 . 1 } ) and number of clients ( { 100 , 500 } ) .

Revised: As conjectured in the main paper, we can observe that the cost of achieving the same fidelity of FE increases with both the level of non-IID-ness and the number of clients.

Revised: Specifically, Kendall- τ of 0.8 is achieved at approximately three FE rounds for { α = 1 , clients = 100 } , and increases to four and six for { α = 0 . 1 , clients = 100 } and { α = 0 .

Revised: 1 , clients = 500 } , respectively.

Revised: At the same time, we can see that regardless of the setting NSGA-II tends to produce smaller supernets as the search progresses, suggesting that searching for longer does not have to incur proportional increase in the communication cost.