Original: Cross-device Federated Architecture Search
Revised: Cross-device Federated Architecture Search

Original: Abstract
Revised: Abstract

Original: Federated learning (FL) has recently gained considerable attention due to its ability to learn on decentralised data while preserving client privacy.
Revised: Federated learning (FL) has recently gained considerable attention due to its ability to learn on decentralised data while preserving client privacy.

Original: However, it also poses additional challenges related to the heterogeneity of the participating devices, both in terms of their computational capabilities and contributed data.
Revised: However, it also poses additional challenges related to the heterogeneity of the participating devices, both in terms of their computational capabilities and contributed data.

Original: Meanwhile,
Revised: Meanwhile,

Original: Neural Architecture Search (NAS) has been successfully used with centralised datasets, producing state-of-the-art results in constrained or unconstrained settings.
Revised: Neural Architecture Search (NAS) has been successfully used with centralised datasets, producing state-of-the-art results in constrained or unconstrained settings.

Original: However, such centralised datasets may not be always available for training.
Revised: However, such centralised datasets may not be always available for training.

Original: Most recent work at the intersection of NAS and FL attempts to alleviate this issue in a cross-silo federated setting, which assumes homogeneous compute environments with datacenter-grade hardware.
Revised: Most recent work at the intersection of NAS and FL attempts to alleviate this issue in a cross-silo federated setting, which assumes homogeneous compute environments with datacenter-grade hardware.

Original: In this paper we explore the question of whether we can design architectures of different footprints in a cross-device federated setting, where the device landscape, availability and scale are very different.
Revised: In this paper we explore the question of whether we can design architectures of different footprints in a cross-device federated setting, where the device landscape, availability and scale are very different.

Original: To this end, we design our system, FedorAS , to discover and train promising architectures in a resource-aware manner when dealing with devices of varying capabilities holding non-IID distributed data.
Revised: To this end, we design our system, FedorAS , to discover and train promising architectures in a resource-aware manner when dealing with devices of varying capabilities holding non-IID distributed data.

Original: We present empirical evidence of its effectiveness across different settings, spanning across three different modalities (vision, speech, text), and showcase its better performance compared to state-of-the-art federated solutions, while maintaining resource efficiency.
Revised: We present empirical evidence of its effectiveness across different settings, spanning across three different modalities (vision, speech, text), and showcase its better performance compared to state-of-the-art federated solutions, while maintaining resource efficiency.

Original:  Introduction
Revised:  Introduction

Original: As smart devices become omnipresent where we live, work and socialise the ML-powered services that these provide grow in sophistication.
Revised: As smart devices become omnipresent where we live, work and socialise the ML-powered services that these provide grow in sophistication.

Original: This ambient intelligence has undoubtedly been sustained by recent advances in Deep Learning (DL) across a multitude of tasks and modalities.
Revised: This ambient intelligence has undoubtedly been sustained by recent advances in Deep Learning (DL) across a multitude of tasks and modalities.

Original: Parallel to this race for state-of-the-art performance in various in DL benchmarks, mobile and embedded devices also became more capable to accommodate new Deep Neural Network (DNN) designs [36], some even integrating specialised accelerators to their System-On-Chips (SoC) (e g. NPUs) to efficiently run DL workloads [3].
Revised: Parallel to this race for state-of-the-art performance in various in DL benchmarks, mobile and embedded devices also became more capable to accommodate new Deep Neural Network (DNN) designs [36], some even integrating specialised accelerators to their System-On-Chips (SoC) (e g. NPUs) to efficiently run DL workloads [3].

Original: These often come in various configurations in terms of their compute/memory capabilities and power envelopes [4] and co-exist in the wild as a rich multi-generational ecosystem ( system heterogeneity ) [75].
Revised: These often come in various configurations in terms of their compute/memory capabilities and power envelopes [4] and co-exist in the wild as a rich multi-generational ecosystem ( system heterogeneity ) [76].

Original: These devices bring intelligence through users’ interactions, also innately heterogeneous amongst them, leading to non-independent or identically distributed (non-IID) data in the wild ( data heterogeneity ).
Revised: These devices bring intelligence through users’ interactions, also innately heterogeneous amongst them, leading to non-independent or identically distributed (non-IID) data in the wild ( data heterogeneity ).

Original: Powered by the recent advances in SoCs’ capabilities and motivated by privacy concerns [71] over the custody of data, Federated Learning (FL) [57] has emerged as a way of training on-device on user data without it ever directly leaving the device premises.
Revised: Powered by the recent advances in SoCs’ capabilities and motivated by privacy concerns [72] over the custody of data, Federated Learning (FL) [57] has emerged as a way of training on-device on user data without it ever directly leaving the device premises.

Original: However, FL training has largely been focused on the weights of a static global model architecture, assumed to be runnable by every participating client [39].
Revised: However, FL training has largely been focused on the weights of a static global model architecture, assumed to be runnable by every participating client [39].

Original: Not only may this not be the case, but it can also lead to subpar performance of the overall training process in the presence of stragglers or biases in the case of consistently dropping certain low-powered devices.
Revised: Not only may this not be the case, but it can also lead to subpar performance of the overall training process in the presence of stragglers or biases in the case of consistently dropping certain low-powered devices.

Original: On the opposite end, more capable devices might not fully take advantage of their data if the deployed model is of reduced capacity to ensure all devices can participate [51].
Revised: On the opposite end, more capable devices might not fully take advantage of their data if the deployed model is of reduced capacity to ensure all devices can participate [51].

Original: Parallel to these trends, Neural Architecture Search (NAS) has become the de facto mechanism to automate the design of DNNs that can meet the requirements (e.g. latency, model size) for these to run on resource-constrained devices.
Revised: Parallel to these trends, Neural Architecture Search (NAS) has become the de facto mechanism to automate the design of DNNs that can meet the requirements (e.g. latency, model size) for these to run on resource-constrained devices.

Original: The success of NAS can be partly attributed to the fact that these frameworks are commonly run in datacenters, where high-performing hardware and/or large curated datasets [42, 24, 21, 38, 60, 60] are available.
Revised: The success of NAS can be partly attributed to the fact that these frameworks are commonly run in datacenters, where high-performing hardware and/or large curated datasets [42, 23, 20, 38, 60, 60] are available.

Original: However, this also imposes two major limitations on current NAS approaches: i) privacy , i.e. these methods were often not designed to work in situations when user’s data must remain on-device; and, consequently, ii) tail data non-discoverability , i.e. they might never be exposed to infrequent or time/user-specific data that exist in the wild but not necessarily in centralized datasets.
Revised: However, this also imposes two major limitations on current NAS approaches: i) privacy , i.e. these methods were often not designed to work in situations when user’s data must remain on-device; and, consequently, ii) tail data non-discoverability , i.e. they might never be exposed to infrequent or time/user-specific data that exist in the wild but not necessarily in centralized datasets.

Original: On top of these, the whole cost is born by the provider and separate on-device modelling/profiling needs to be done in the case of hardware-aware NAS [26, 70, 44], which has mainly focused on inference performance hitherto.
Revised: On top of these, the whole cost is born by the provider and separate on-device modelling/profiling needs to be done in the case of hardware-aware NAS [25, 71, 44], which has mainly focused on inference performance hitherto.

Original: Motivated by the aforementioned phenomena and limitations of the existing NAS methods, we propose FedorAS , a system that performs NAS over heterogeneous devices holding heterogeneous data in a resource-aware and federated manner.
Revised: Motivated by the aforementioned phenomena and limitations of the existing NAS methods, we propose FedorAS , a system that performs NAS over heterogeneous devices holding heterogeneous data in a resource-aware and federated manner.

Original: To this direction, we cluster clients into tiers based on their capabilities and design supernet comprising operations covering the whole spectrum of compute complexities.
Revised: To this direction, we cluster clients into tiers based on their capabilities and design supernet comprising operations covering the whole spectrum of compute complexities.

Original: This supernet acts both as search space and a weight-sharing backbone.
Revised: This supernet acts both as search space and a weight-sharing backbone.

Original: Upon federation, it is only partially and stochastically shared to clients, respecting their computational and bandwidth capabilities.
Revised: Upon federation, it is only partially and stochastically shared to clients, respecting their computational and bandwidth capabilities.

Original: In turn, we leverage resource-aware one-shot path sampling [28] and adapt it to facilitate lightweight on-device NAS.
Revised: In turn, we leverage resource-aware one-shot path sampling [27] and adapt it to facilitate lightweight on-device NAS.

Original: In this way, networks in a given search space are not only deployed in a resource-aware manner, but also trained as such, by tuning the downstream communication (i.e. the subspace explored by each client) and computation (i.e. FLOPs of sampled paths) to meet the device’s training budget.
Revised: In this way, networks in a given search space are not only deployed in a resource-aware manner, but also trained as such, by tuning the downstream communication (i.e. the subspace explored by each client) and computation (i.e. FLOPs of sampled paths) to meet the device’s training budget.

Original: Once federated training of the supernetwork has completed, usable pretrained networks can be extracted even before performing fine-tuning or personalising per device, thus minimising the number of extra on-device training rounds to achieve competitive performance.
Revised: Once federated training of the supernetwork has completed, usable pretrained networks can be extracted even before performing fine-tuning or personalising per device, thus minimising the number of extra on-device training rounds to achieve competitive performance.

Original: In summary, in this work we make the following contributions:
Revised: In summary, in this work we make the following contributions:

Original: • We propose a system for resource efficient federated NAS that can be applied in cross-device settings, where partial device participation, device and data heterogeneity are innate characteristics.
Revised: • We propose a system for resource efficient federated NAS that can be applied in cross-device settings, where partial device participation, device and data heterogeneity are innate characteristics.

Original: • We implement a system called FedorAS (Federated nAS) that leverages a server-resident supernet enabling weight sharing for efficient knowledge exchange between clients, without directly sharing common model architectures with one another.
Revised: • We implement a system called FedorAS (Federated nAS) that leverages a server-resident supernet enabling weight sharing for efficient knowledge exchange between clients, without directly sharing common model architectures with one another.

Original: • We propose a novel aggregation method named OPA (OPerator Aggregation) for weighing updates from multiple “single-path one-shot” client updates in a frequency-aware manner.
Revised: • We propose a novel aggregation method named OPA (OPerator Aggregation) for weighing updates from multiple “single-path one-shot” client updates in a frequency-aware manner.

Original: • We empirically evaluate the performance and convergence of our system under IID and non-IID settings across different datasets, tasks and modalities, spanning different device distributions and compare our system’s behaviour against state-of-the-art FL techniques.
Revised: • We empirically evaluate the performance and convergence of our system under IID and non-IID settings across different datasets, tasks and modalities, spanning different device distributions and compare our system’s behaviour against state-of-the-art FL techniques.

Original:  Background & Motivation
Revised:  Background & Motivation

Original: Federated Learning.
Revised: Federated Learning.

Original: A typical FL pipeline is comprised of three distinct stages: given a global model initialised on a central server, ω ( t =0)
Revised: A typical FL pipeline is comprised of three distinct stages: given a global model initialised on a central server, ω ( t =0)

Original: g , i) the server randomly samples k clients out of the available K ( k ≪ K for cross-device ; [11] k = K for cross-silo setting) and sends them the current state of the global model; ii) those k clients perform training on-device using their own data partition,
Revised: g , i) the server randomly samples k clients out of the available K ( k ≪ K for cross-device ; [10] k = K for cross-silo setting) and sends them the current state of the global model; ii) those k clients perform training on-device using their own data partition,

Original: D i , for a number of epochs and send the updated models, ω ( t ) i , back to the server after local training is completed; finally, iii) the server aggregates these models and a new global model, ω ( t +1) g , is obtained.
Revised: D i , for a number of epochs and send the updated models, ω ( t ) i , back to the server after local training is completed; finally, iii) the server aggregates these models and a new global model, ω ( t +1) g , is obtained.

Original: This aggregation can be implemented in different ways [57, 67, 52].
Revised: This aggregation can be implemented in different ways [57, 68, 52].

Original: For example, in FedAvg [57] each update is weighted by the relative amount of data on each client: ω ( t +1)
Revised: For example, in FedAvg [57] each update is weighted by the relative amount of data on each client: ω ( t +1)

Original: g = (cid:80) ki =
Revised: g = (cid:80) ki =

Original: | D i |
Revised: | D i |

Original: [Table]
Revised: [Table]

Original: Stages ii) and iii) repeat until convergence.
Revised: Stages ii) and iii) repeat until convergence.

Original: The quality of the global model ω g can be assessed: on the global test set; by evaluating the fit of the ω g to each participating client’s data ( D i ) and derive fairness metrics [53]; or, by evaluating the adaptability of the ω g to each client’s data or new data these might generate over time, this is commonly referred to as personalised FL [27, 50].
Revised: The quality of the global model ω g can be assessed: on the global test set; by evaluating the fit of the ω g to each participating client’s data ( D i ) and derive fairness metrics [53]; or, by evaluating the adaptability of the ω g to each client’s data or new data these might generate over time, this is commonly referred to as personalised FL [26, 50].

Original: Contrary to traditional distributed learning, cross-device FL performs the bulk of the compute on a highly heterogeneous [39] set of devices in terms of their compute capabilities, availability and data
Revised: Contrary to traditional distributed learning, cross-device FL performs the bulk of the compute on a highly heterogeneous [39] set of devices in terms of their compute capabilities, availability and data

Original: [Figure]
Revised: [Figure]

Original: distribution.
Revised: distribution.

Original: In such scenarios a trade-off between model capacity and client participation arises: larger architectures might result in more accurate models which may only be trained on a fraction of the available devices; on the other hand, deploying smaller footprint networks could target more devices – and thus more data – for training, but these might be of inferior quality (gap in Fig. 1)
Revised: In such scenarios a trade-off between model capacity and client participation arises: larger architectures might result in more accurate models which may only be trained on a fraction of the available devices; on the other hand, deploying smaller footprint networks could target more devices – and thus more data – for training, but these might be of inferior quality (gap in Fig. 1)

Original: Neural Architecture Search.
Revised: Neural Architecture Search.

Original: NAS is usually defined as a bi-level optimisation problem:
Revised: NAS is usually defined as a bi-level optimisation problem:

Original: [Equation]
Revised: [Equation]

Original: where A is a finite (discrete) set of architectures to search from (a search space ), L is a loss function, ω a are weights of a model with architecture a , and D { v,t } are validation and training datasets, respectively.
Revised: where A is a finite (discrete) set of architectures to search from (a search space ), L is a loss function, ω a are weights of a model with architecture a , and D { v,t } are validation and training datasets, respectively.

Original: The main challenge of NAS comes directly from the fact that in order to assess quality of different architectures ( outer optimisation ), we have to obtain their optimal weights which entitles conducting full training ( inner optimisation ).
Revised: The main challenge of NAS comes directly from the fact that in order to assess quality of different architectures ( outer optimisation ), we have to obtain their optimal weights which entitles conducting full training ( inner optimisation ).

Original: There exist multiple approaches to speed up NAS [62, 55,  25, 17, 26, 80, 1, 56, 68, 28, 58].
Revised: There exist multiple approaches to speed up NAS [63, 55, 24, 16, 25, 81, 1, 56, 69, 27, 58].

Original: More relevant to our work are those utilising the concept of a supernet [16, 8]; where a single model that encapsulates all possible architectures from the search space is created and trained.
Revised: More relevant to our work are those utilising the concept of a supernet [15, 7]; where a single model that encapsulates all possible architectures from the search space is created and trained.

Original: Specifically, a supernet is constructed by defining an operator that incorporates all candidate operations (the set of which we denote by O l ) for each searchable layer l .
Revised: Specifically, a supernet is constructed by defining an operator that incorporates all candidate operations (the set of which we denote by O l ) for each searchable layer l .

Original: A common choice is to define it as a weighted sum of candidates’ individual outputs y l
Revised: A common choice is to define it as a weighted sum of candidates’ individual outputs y l

Original: = (cid:80) o ∈
Revised: = (cid:80) o ∈

Original: O l α ( o ) l ∗ o ( y l − 1 ) , where factors { α ( o ) l } o ∈ O l of each layer l can be defined differently for different methods (e.g, continuous parameters [55, 17, 25] or random one-hot vectors [28]).
Revised: O l α ( o ) l ∗ o ( y l − 1 ) , where factors { α ( o ) l } o ∈ O l of each layer l can be defined differently for different methods (e.g, continuous parameters [55, 16, 24] or random one-hot vectors [27]).

Original: Importantly for us, methods that use sparse weighting factors can avoid executing operations associated with zero weights, saving both memory and compute [17, 28].
Revised: Importantly for us, methods that use sparse weighting factors can avoid executing operations associated with zero weights, saving both memory and compute [16, 27].

Original: After a supernet has been constructed and trained, searching for a ⋆ is usually performed by either investigating architectural parameters [55, 25, 17], or using zero-th order optimisation methods to directly solve the outer loop of Eq.
Revised: After a supernet has been constructed and trained, searching for a ⋆ is usually performed by either investigating architectural parameters [55, 24, 16], or using zero-th order optimisation methods to directly solve the outer loop of Eq.

Original: 1 while approximating ω ∗ a with weights taken from the supernet (thus avoiding the costly inner optimisation)
Revised: 1 while approximating ω ∗ a with weights taken from the supernet (thus avoiding the costly inner optimisation)

Original: The final model can then be retrained in isolation using either random initialisation or weights from the supernet as a starting point.
Revised: The final model can then be retrained in isolation using either random initialisation or weights from the supernet as a starting point.

Original: Challenges of Federated NAS.
Revised: Challenges of Federated NAS.

Original: As highlighted before, devices in the wild exhibit different compute capabilities and can hold non-IID distributed data, resulting in system and data heterogeneity.
Revised: As highlighted before, devices in the wild exhibit different compute capabilities and can hold non-IID distributed data, resulting in system and data heterogeneity.

Original: In the context of NAS, system heterogeneity has a particularly significant effects, as we might no longer be able to guarantee that any model from our search space can be efficiently trained on all devices.
Revised: In the context of NAS, system heterogeneity has a particularly significant effects, as we might no longer be able to guarantee that any model from our search space can be efficiently trained on all devices.

Original: This inability can be attributed to insufficient compute power, limited network bandwidth or unavailability of the client at hand.
Revised: This inability can be attributed to insufficient compute power, limited network bandwidth or unavailability of the client at hand.

Original: Consequently, some of the models might be deemed worse than others not because of their worse ability to generalise, but because they might not be exposed to the same subsets of data as others – as shown in Fig.
Revised: Consequently, some of the models might be deemed worse than others not because of their worse ability to generalise, but because they might not be exposed to the same subsets of data as others – as shown in Fig.

Original: 1, where we show models of different footprint trained across client of varying capabilities under constrained ( tier-aware ) and full ( tier-unaware ) participation.
Revised: 1, where we show models of different footprint trained across client of varying capabilities under constrained ( tier-aware ) and full ( tier-unaware ) participation.

Original:  The FedorAS Framework
Revised:  The FedorAS Framework

Original: FedorAS is a resource-aware Federated NAS framework that combines the best of both worlds: learning from clients across all tiers and yielding models tailored to each tier that benefit from this collective knowledge.
Revised: FedorAS is a resource-aware Federated NAS framework that combines the best of both worlds: learning from clients across all tiers and yielding models tailored to each tier that benefit from this collective knowledge.

Original: [Figure]
Revised: [Figure]

Original: Workflow.
Revised: Workflow.

Original: FedorAS ’ workflow consists of three stages (Fig. 2): i) supernet training , ii) model search and validation and iii) model fine-tuning .
Revised: FedorAS ’ workflow consists of three stages (Fig. 2): i) supernet training , ii) model search and validation and iii) model fine-tuning .

Original: First, we train the supernet in a resource-aware and federated manner ( Stage 1 , Sec. 3.1).
Revised: First, we train the supernet in a resource-aware and federated manner ( Stage 1 , Sec. 3.1).

Original: We then search for models from the supernet with the goal of finding the best architecture per tier ( Stage 2 , Sec. 3.2).
Revised: We then search for models from the supernet with the goal of finding the best architecture per tier ( Stage 2 , Sec. 3.2).

Original: Models are effectively sampled, validated on a global validation set and ranked per tier.
Revised: Models are effectively sampled, validated on a global validation set and ranked per tier.

Original: These architectures and their associated weights act as initialisation to the next phase, where each model is fine-tuned in a per-tier manner ( Stage 3 .
Revised: These architectures and their associated weights act as initialisation to the next phase, where each model is fine-tuned in a per-tier manner ( Stage 3 .

Original: Sec. 3.3).
Revised: Sec. 3.3).

Original: The end goal of our system is to have the best possible model per each cluster of devices.
Revised: The end goal of our system is to have the best possible model per each cluster of devices.

Original: Design rationale.
Revised: Design rationale.

Original: We build our system around the concept of a supernet to facilitate weight-sharing between architectures of various footprints.
Revised: We build our system around the concept of a supernet to facilitate weight-sharing between architectures of various footprints.

Original: Operations in the supernet are samplable from paths (i.e. models) of different footprint.
Revised: Operations in the supernet are samplable from paths (i.e. models) of different footprint.

Original: As such, while normally large models would not be directly trained on data of low-tier clients, our design allows for indirectly sharing this knowledge through the association of the same operation to different paths.
Revised: As such, while normally large models would not be directly trained on data of low-tier clients, our design allows for indirectly sharing this knowledge through the association of the same operation to different paths.

Original: To ensure efficient training of a supernet, we chose to base our approach on SPOS [28] and adapt it (see Eq. 4) since its training procedure is lightweight, introducing very little overhead in terms of memory consumption and/or required floating-point operations, especially compared to [55].
Revised: To ensure efficient training of a supernet, we chose to base our approach on SPOS [27] and adapt it (see Eq. 4) since its training procedure is lightweight, introducing very little overhead in terms of memory consumption and/or required floating-point operations, especially compared to [55].

Original: Last, we opted for clustering devices into “tiers” based on their computational capabilities and search for an architecture for each tier as a balance between having one model to fit all needs [30] and a different architecture per client [59].
Revised: Last, we opted for clustering devices into “tiers” based on their computational capabilities and search for an architecture for each tier as a balance between having one model to fit all needs [29] and a different architecture per client [59].

Original: 3.1 SuperNet Training
Revised: 3.1 SuperNet Training

Original: 0 Search space & models.
Revised: 0 Search space & models.

Original: First, we define the search space in terms of a range of operators options per layer in the network that are valid choices to form a network.
Revised: First, we define the search space in terms of a range of operators options per layer in the network that are valid choices to form a network.

Original: This search space resides as a whole on the server and is only partially communicated to participating clients of a training round to keep communication overheads minimal.
Revised: This search space resides as a whole on the server and is only partially communicated to participating clients of a training round to keep communication overheads minimal.

Original: Specific models (i.e. paths) consist of selections of one operator option per layer, sampled in a single-path-one-shot manner on-device per local iteration.
Revised: Specific models (i.e. paths) consist of selections of one operator option per layer, sampled in a single-path-one-shot manner on-device per local iteration.

Original: 1 Subspace sampling.
Revised: 1 Subspace sampling.

Original: It is obvious that communicating the whole space of operators along with the associated weights to each device becomes quickly intractable, especially bearing in mind that communication is usually a primary bottleneck in Federated Learning [39, 51].
Revised: It is obvious that communicating the whole space of operators along with the associated weights to each device becomes quickly intractable, especially bearing in mind that communication is usually a primary bottleneck in Federated Learning [39, 51].

Original: To this direction,
Revised: To this direction,

Original: FedorAS adopts a uniform parameter size budget, B Φ comm , and samples 2 the search space for operators until this limit is hit (Eq. 2).
Revised: FedorAS adopts a uniform parameter size budget, B Φ comm , and samples 2 the search space for operators until this limit is hit (Eq. 2).

Original: Setting the limit to half the size of a typical network deployed for a task worked sufficiently well in our experiments and in fact accelerated convergence (Sec. 4.4).
Revised: Setting the limit to half the size of a typical network deployed for a task worked sufficiently well in our experiments and in fact accelerated convergence (Sec. 4.4).

Original: [Equation]
Revised: [Equation]

Original: where L is the number of layer in the supernet, O l the candidate and ˆ O l the selected operations in layer l , 1 a unit vector of valid operations and Φ comm a measure of DNN size (e.g. #parameters).
Revised: where L is the number of layer in the supernet, O l the candidate and ˆ O l the selected operations in layer l , 1 a unit vector of valid operations and Φ comm a measure of DNN size (e.g. #parameters).

Original: In terms of sampling strategies, we experimented with uniform operator sampling, which we found to work sufficiently well and provide uniform coverage over the search space.
Revised: In terms of sampling strategies, we experimented with uniform operator sampling, which we found to work sufficiently well and provide uniform coverage over the search space.

Original: It is also worth noting that a different subspace (not necessarily mutex) could be selected for each participating client in a round.
Revised: It is also worth noting that a different subspace (not necessarily mutex) could be selected for each participating client in a round.

Original: 2 Client-side sampling & local training.
Revised: 2 Client-side sampling & local training.

Original: Participating clients receive the subspace sampled on the server, { ˆ O l } Ll =0 , from which they sample a single operator on every layer.
Revised: Participating clients receive the subspace sampled on the server, { ˆ O l } Ll =0 , from which they sample a single operator on every layer.

Original: This constitutes a path along the supernet ( p L ) representing a single model.
Revised: This constitutes a path along the supernet ( p L ) representing a single model.

Original: For every batch, clients sample paths that do not surpass the assigned training budget B Tier Φ train .
Revised: For every batch, clients sample paths that do not surpass the assigned training budget B Tier Φ train .

Original: Throughout this work, we consider Φ train ( · ) to be a cost function that counts the FLOPs of a given operator.
Revised: Throughout this work, we consider Φ train ( · ) to be a cost function that counts the FLOPs of a given operator.

Original: This FLOPs limit is defined per tier so that a network does not exceed the capabilities of the target device.
Revised: This FLOPs limit is defined per tier so that a network does not exceed the capabilities of the target device.

Original: Our goal is to sample valid paths uniformly, to ensure systematic coverage of the entire (sub) search space during training:
Revised: Our goal is to sample valid paths uniformly, to ensure systematic coverage of the entire (sub) search space during training:

Original: [Equation]
Revised: [Equation]

Original: However, realising Eq.
Revised: However, realising Eq.

Original: 3 efficiently is not a trivial task.
Revised: 3 efficiently is not a trivial task.

Original: Originally [28], the authors considered a naive approach of repeatedly sampling a path until it fits the given budget, which results in non-negligible overhead if the probability of finding a model under the threshold is low.
Revised: Originally [27], the authors considered a naive approach of repeatedly sampling a path until it fits the given budget, which results in non-negligible overhead if the probability of finding a model under the threshold is low.

Original: Were we to employ such a method, the most restricted devices, for which the set of eligible models is the smallest, would be the ones burdened with the largest overhead.
Revised: Were we to employ such a method, the most restricted devices, for which the set of eligible models is the smallest, would be the ones burdened with the largest overhead.

Original: Therefore, we propose a greedy approximation in which operations are selected sequentially.
Revised: Therefore, we propose a greedy approximation in which operations are selected sequentially.

Original: Specifically, in order to obtain a path p
Revised: Specifically, in order to obtain a path p

Original: L = { o i } Li =

Revised: L = { o i } Li =

Original: we sample operations layer-by-layer, according to a random permutation σ , in such a way that the i -th operation is chosen from the candidates for layer σ ( i ) whose total overhead would not violate the constraint:
Revised: we sample operations layer-by-layer, according to a random permutation σ , in such a way that the i -th operation is chosen from the candidates for layer σ ( i ) whose total overhead would not violate the constraint:

Original: [Equation]
Revised: [Equation]

Original: We can ensure that Eq.
Revised: We can ensure that Eq.

Original: 4 can always obtain a valid architecture without resampling if layers have
Revised: 4 can always obtain a valid architecture without resampling if layers have

Original: Identity among their candidate operations and prioritising the selection of those which do not.
Revised: Identity among their candidate operations and prioritising the selection of those which do not.

Original: After having sampled the path, a model is instantiated and a single optimization step using a single batch of data is performed.
Revised: After having sampled the path, a model is instantiated and a single optimization step using a single batch of data is performed.

Original: The number of samples passing through each operator are kept and communicated back to the server, along with the updates, for properly weighting updated parameters upon aggregation, as we will see next.
Revised: The number of samples passing through each operator are kept and communicated back to the server, along with the updates, for properly weighting updated parameters upon aggregation, as we will see next.

Original: 3 Aggregation with OPA.
Revised: 3 Aggregation with OPA.

Original: An operator gets stochastically exposed to clients data.
Revised: An operator gets stochastically exposed to clients data.

Original: This stems from subspace sampling and client-side path sampling .
Revised: This stems from subspace sampling and client-side path sampling .

Original: As such, naively aggregating updates in an equiweighed manner (Plain-Avg) or total samples residing on a client (FedAvg [57]) would not reflect the training experience during a round.
Revised: As such, naively aggregating updates in an equiweighed manner (Plain-Avg) or total samples residing on a client (FedAvg [57]) would not reflect the training experience during a round.

Original: For this reason, we propose OPA, OPerator Aggregation, an aggregation method that weights updates based on the relative experience of an operator across all clients that have updated that operator.
Revised: For this reason, we propose OPA, OPerator Aggregation, an aggregation method that weights updates based on the relative experience of an operator across all clients that have updated that operator.

Original: Concretely, our method is generalisation of FedAvg where normalisation is performed independently for each layer, rather than collectively for full models.
Revised: Concretely, our method is generalisation of FedAvg where normalisation is performed independently for each layer, rather than collectively for full models.

Original: In order to enable that, we keep track of how many examples were used to update each searchable operation o l , independently on each client, and later use this information to weight updates.
Revised: In order to enable that, we keep track of how many examples were used to update each searchable operation o l , independently on each client, and later use this information to weight updates.

Original: Formally:
Revised: Formally:

Original: [Equation]
Revised: [Equation]

Original: where ω ( t ) g ( · ) are global weights, ω ( t ) i ( · ) are local weights of client
Revised: where ω ( t ) g ( · ) are global weights, ω ( t ) i ( · ) are local weights of client

Original: i at global step t , | D ( t ) i,o l | is the number of samples having backpropagated through an operator
Revised: i at global step t , | D ( t ) i,o l | is the number of samples having backpropagated through an operator

Original: o l for client
Revised: o l for client

Original: i in round t , and C ( t )
Revised: i in round t , and C ( t )

Original: o l is the set of clients i s.t.
Revised: o l is the set of clients i s.t.

Original: | D ( t ) i,o l | > 0 .
Revised: | D ( t ) i,o l | > 0 .

Original: Updates to ω ( t ) g happen only if | C ( t )
Revised: Updates to ω ( t ) g happen only if | C ( t )

Original: o l | > 1 in order to protect the privacy of single clients.
Revised: o l | > 1 in order to protect the privacy of single clients.

Original: Finally, if an operation is always selected, then Eq.
Revised: Finally, if an operation is always selected, then Eq.

Original: 5 recovers FedAvg, which means we can effectively use OPA throughout the model and not only for searchable layers.
Revised: 5 recovers FedAvg, which means we can effectively use OPA throughout the model and not only for searchable layers.

Original: 3.2 Model Search & Validation
Revised: 3.2 Model Search & Validation

Original: After training the supernet, a search phase is implemented to discover the best trained architecture per device tier.
Revised: After training the supernet, a search phase is implemented to discover the best trained architecture per device tier.

Original: Models are sampled with NSGA-II [23] and evaluated on a global validation set.
Revised: Models are sampled with NSGA-II [22] and evaluated on a global validation set.

Original: The rationale behind selecting the NSGA-II algorithm for our search is the fact that it is multi-objective and allows us for efficient space exploration with the goal of optimising for model size in a specific tier and accuracy.
Revised: The rationale behind selecting the NSGA-II algorithm for our search is the fact that it is multi-objective and allows us for efficient space exploration with the goal of optimising for model size in a specific tier and accuracy.

Original: Other search algorithms can be used in place of NSGA-II, based on the objective at hand.
Revised: Other search algorithms can be used in place of NSGA-II, based on the objective at hand.

Original: At the end of this stage, we have a model architecture per tier, already encompassing knowledge across devices of different tiers, which serves as initialisation for further training.
Revised: At the end of this stage, we have a model architecture per tier, already encompassing knowledge across devices of different tiers, which serves as initialisation for further training.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: 3.3 Fine-tuning Phase
Revised: 3.3 Fine-tuning Phase

Original: Subsequently, we move to train each of the previously produced models in a federated manner across all eligible clients.
Revised: Subsequently, we move to train each of the previously produced models in a federated manner across all eligible clients.

Original: This means that architectures falling in a specific tier, in terms of footprint, can be trained across clients belonging to that tier and above.
Revised: This means that architectures falling in a specific tier, in terms of footprint, can be trained across clients belonging to that tier and above.

Original: Our hypothesis here is that compared to regular
Revised: Our hypothesis here is that compared to regular

Original: FL where one needs to trade off capacity of participation, FedorAS allows for better generalisation due to knowledge sharing through its supernet structure.
Revised: FL where one needs to trade off capacity of participation, FedorAS allows for better generalisation due to knowledge sharing through its supernet structure.

Original: Here we use conventional FedAvg and partial client participation per round, again with the goal of training a single network per tier
Revised: Here we use conventional FedAvg and partial client participation per round, again with the goal of training a single network per tier

Original:  Evaluation
Revised:  Evaluation

Original: This section provides a thorough evaluation of FedorAS across different tasks to show its performance and generality.
Revised: This section provides a thorough evaluation of FedorAS across different tasks to show its performance and generality.

Original: First, we compare FedorAS to existing approaches in the context of federated NAS in the cross-device and cross silo setting.
Revised: First, we compare FedorAS to existing approaches in the context of federated NAS in the cross-device and cross silo setting.

Original: Next, we we draw from the broader FL literature and showcase our technique’s performance gains compared to homogeneous and heterogeneous federated solutions (Sec. 4.2).
Revised: Next, we we draw from the broader FL literature and showcase our technique’s performance gains compared to homogeneous and heterogeneous federated solutions (Sec. 4.2).

Original: This can be traced back to the benefits of supernet weight sharing ; as such we, subsequently, quantify its contribution by comparing it to FedorAS with randomly initialised networks trained on eligible clients in a federated way (Sec. 4.3) without weight sharing.
Revised: This can be traced back to the benefits of supernet weight sharing ; as such we, subsequently, quantify its contribution by comparing it to FedorAS with randomly initialised networks trained on eligible clients in a federated way (Sec. 4.3) without weight sharing.

Original: Last, we showcase the contribution of specific components of our system through an ablation study (Sec. 4.4) and also the performance and behaviour of alternative search methods (Sec. 4.5).
Revised: Last, we showcase the contribution of specific components of our system through an ablation study (Sec. 4.4) and also the performance and behaviour of alternative search methods (Sec. 4.5).

Original: 4.1 Experimental Setup
Revised: 4.1 Experimental Setup

Original: Datasets & Baselines.
Revised: Datasets & Baselines.

Original: Datasets are summarised in Tab. 1.
Revised: Datasets are summarised in Tab. 1.

Original: More information in Appendix D.
Revised: More information in Appendix D.

Original: Search space.
Revised: Search space.

Original: The adopted search spaces are specific to the dataset and task at hand, both in terms of size and type of operators.
Revised: The adopted search spaces are specific to the dataset and task at hand, both in terms of size and type of operators.

Original: In general, we assume our highest tier of model sizes to coincide with a typical network footprint used for that task in the FL literature (e.g. ResNet-18 for CIFAR-10).
Revised: In general, we assume our highest tier of model sizes to coincide with a typical network footprint used for that task in the FL literature (e.g. ResNet-18 for CIFAR-10).

Original: Nevertheless, there may be operators in the space that cannot be sampled in some tiers due to their footprint.
Revised: Nevertheless, there may be operators in the space that cannot be sampled in some tiers due to their footprint.

Original: FedorAS sets the communication budget B Φ comm to be half the size of the supernet.
Revised: FedorAS sets the communication budget B Φ comm to be half the size of the supernet.

Original: The full set of available operators per task is provided in the Appendix D.3.
Revised: The full set of available operators per task is provided in the Appendix D.3.

Original: Clusters definition.
Revised: Clusters definition.

Original: In our experiments, we cluster clients based on the amount of operations they can sustain per round of training (#FLOPs), for simplicity.
Revised: In our experiments, we cluster clients based on the amount of operations they can sustain per round of training (#FLOPs), for simplicity.

Original: Other resources (e.g. #parameters, energy consumption) can be used in place or in conjunction with FLOPs.
Revised: Other resources (e.g. #parameters, energy consumption) can be used in place or in conjunction with FLOPs.

Original: More details in Appendix D.4.
Revised: More details in Appendix D.4.

Original: 4.2 Performance evaluation
Revised: 4.2 Performance evaluation

Original: Federated NAS.
Revised: Federated NAS.

Original: We start the evaluation by comparing our system with existing works in the federated
Revised: We start the evaluation by comparing our system with existing works in the federated

Original: NAS domain.
Revised: NAS domain.

Original: Specifically, we find two systems that perform federated NAS, namely FedNAS [30] and SPIDER [59] and compare in the cross-device and cross-silo settings.
Revised: Specifically, we find two systems that perform federated NAS, namely FedNAS [29] and SPIDER [59] and compare in the cross-device and cross-silo settings.

Original: In the former setting, we adapt FedNAS to support partial participation and compare their technique to FedorAS under the same cross-device settings in CIFAR-10.
Revised: In the former setting, we adapt FedNAS to support partial participation and compare their technique to FedorAS under the same cross-device settings in CIFAR-10.

Original: Results are shown in Tab. 2, showing FedorAS performing 1.04% better than FedNAS on CIFAR-10 α =1 and 48.7% on CIFAR-10 α =0 .
Revised: Results are shown in Tab. 2, showing FedorAS performing 1.04% better than FedNAS on CIFAR-10 α =1 and 48.7% on CIFAR-10 α =0 .

Original: 1 , for the same training memory footprint.
Revised: 1 , for the same training memory footprint.

Original: Further details can be found in Appendix D.6.
Revised: Further details can be found in Appendix D.6.

Original: At the same time, while running in cross-silo setting is not the main focus of FedorAS , we have adapted our experimental setting to match that of FedNAS and SPIDER.
Revised: At the same time, while running in cross-silo setting is not the main focus of FedorAS , we have adapted our experimental setting to match that of FedNAS and SPIDER.

Original: Results are shown in Tab. 3 with FedorAS performing +11.6%
Revised: Results are shown in Tab. 3 with FedorAS performing +11.6%

Original: and -1.3% than the baselines, respectively, on the test set of their selected settings.
Revised: and -1.3% than the baselines, respectively, on the test set of their selected settings.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Comparison with Random Search.
Revised: Comparison with Random Search.

Original: We further compare our search with a random search baseline – which is a naive way of running FL NAS in our experimental setting (the same search space, etc.) – and find that FedorAS performs significantly better at an average of +5.11pp, +3.24pp, +12.84pp, +9.27pp, +0.25p across tiers for CIFAR-

Revised: We further compare our search with a random search baseline – which is a naive way of running FL NAS in our experimental setting (the same search space, etc.) – and find that FedorAS performs significantly better at an average of +5.11pp, +3.24pp, +12.84pp, +9.27pp, +0.25p across tiers for CIFAR-

Original: α = { 1000 , 1 , 0 . 1 } , CIFAR-100 and Shakespeare, respectively.
Revised: α = { 1000 , 1 , 0 . 1 } , CIFAR-100 and Shakespeare, respectively.

Original: Only in the case of SpeechCommands did our search result in -0.87pp of accuracy on average.
Revised: Only in the case of SpeechCommands did our search result in -0.87pp of accuracy on average.

Original: We suspect this is an artifact of insufficiently-long fine-tuning which means models might not have converged (suggested by a high variance).
Revised: We suspect this is an artifact of insufficiently-long fine-tuning which means models might not have converged (suggested by a high variance).

Original: Detailed results are provided in Appendix E.
Revised: Detailed results are provided in Appendix E.

Original: Federated Learning baselines.
Revised: Federated Learning baselines.

Original: Next, we compare the performance of FedorAS with different federated baselines, including homogeneous [67] and system heterogeneous frameworks [33, 43, 63].
Revised: Next, we compare the performance of FedorAS with different federated baselines, including homogeneous [68] and system heterogeneous frameworks [33, 43, 64].

Original: In the former setting, we compare with results from [67] on CIFAR100 α =0 .
Revised: In the former setting, we compare with results from [68] on CIFAR100 α =0 .

Original: FedorAS performs 1 percentage point (pp) better than the FedAvg baseline, at 45.84%, but at a fraction of the cost 3 .
Revised: FedorAS performs 1 percentage point (pp) better than the FedAvg baseline, at 45.84%, but at a fraction of the cost 3 .

Original: Simultaneously, retraining the discovered model from scratch using random initialisation under the same training setting as the baseline results in 11.43 pp higher accuracy than the best FedAdam (63.94% vs 52.50%), showcasing the quality of FedorAS -produced bespoke architectures.
Revised: Simultaneously, retraining the discovered model from scratch using random initialisation under the same training setting as the baseline results in 11.43 pp higher accuracy than the best FedAdam (63.94% vs 52.50%), showcasing the quality of FedorAS -produced bespoke architectures.

Original: With respect to heterogeneous baselines (ZeroFL [63], FjORD[33]), we see that FedorAS consistently leads to models with higher accuracy across tiers that are in the respective clients’ computational budget (Tab. 4).
Revised: With respect to heterogeneous baselines (ZeroFL [64], FjORD[33]), we see that FedorAS consistently leads to models with higher accuracy across tiers that are in the respective clients’ computational budget (Tab. 4).

Original: At the same time, we depict how FedorAS performs compared to FjORD and randomly selected architectures trained naively in a resource-aware manner in Fig.
Revised: At the same time, we depict how FedorAS performs compared to FjORD and randomly selected architectures trained naively in a resource-aware manner in Fig.

Original: One can clearly see that the degrees of architectural freedom that our solution offers leads to significantly better accuracy per resource tier.
Revised: One can clearly see that the degrees of architectural freedom that our solution offers leads to significantly better accuracy per resource tier.

Original: Notably, we perform 15.20% and 12.58% better on average than FjORD on CIFAR-10 and Shakespeare, respectively, while still respecting client eligibility.
Revised: Notably, we perform 15.20% and 12.58% better on average than FjORD on CIFAR-10 and Shakespeare, respectively, while still respecting client eligibility.

Original: While model accuracy may not seem to scale proportionally to their size, we attribute this to the limited participation eligibility of clients, an innate trait of system heterogeneous landscape.
Revised: While model accuracy may not seem to scale proportionally to their size, we attribute this to the limited participation eligibility of clients, an innate trait of system heterogeneous landscape.

Original: Normally, this can cause performance gaps due to limited exposure to federated data (Fig. 1).
Revised: Normally, this can cause performance gaps due to limited exposure to federated data (Fig. 1).

Original: We argue that
Revised: We argue that

Original: FedorAS is able to bridge this gap (+0.03 points (p) avg), by means of weight sharing and OPA, +1.72 p more effectively than FjORD (avg Tier 4 vs Tier 1 gap across datasets).
Revised: FedorAS is able to bridge this gap (+0.03 points (p) avg), by means of weight sharing and OPA, +1.72 p more effectively than FjORD (avg Tier 4 vs Tier 1 gap across datasets).

Original: Additional results.
Revised: Additional results.

Original: Additional results on the convergence and fairness of FedorAS as well as alternative client allocation [43] to clusters are provided in Appendix G, H and I, respectively.
Revised: Additional results on the convergence and fairness of FedorAS as well as alternative client allocation [43] to clusters are provided in Appendix G, H and I, respectively.

Original: [Table]
Revised: [Table]

Original: 4.3 Supernet weight-sharing with FedorAS
Revised: 4.3 Supernet weight-sharing with FedorAS

Original: Here
Revised: Here

Original: [Table]
Revised: [Table]

Original: we evaluate the impact of weight sharing through FedorAS ’ supernet.
Revised: we evaluate the impact of weight sharing through FedorAS ’ supernet.

Original: We compare the performance of FedorAS ’ models to the same architectures models
Revised: We compare the performance of FedorAS ’ models to the same architectures models

Original: [Table]
Revised: [Table]

Original: but trained end-to-end in a federated manner, across all four datasets.
Revised: but trained end-to-end in a federated manner, across all four datasets.

Original: With this comparison
Revised: With this comparison

Original: [Table]
Revised: [Table]

Original: we aim to showcase that FedorAS
Revised: we aim to showcase that FedorAS

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: not only comes up with better architectures, but it also effectively transfers knowledge between tiers through its supernet structure and OPA aggregation scheme.
Revised: not only comes up with better architectures, but it also effectively transfers knowledge between tiers through its supernet structure and OPA aggregation scheme.

Original: In order to accomplish that, we first train with FedorAS across the three stages described in Sec. 3 ( supernet-init ).
Revised: In order to accomplish that, we first train with FedorAS across the three stages described in Sec. 3 ( supernet-init ).

Original: Subsequently, we take the output architectures from our system, randomly initialise them and train end-to-end in a federated manner, where each model trains across all eligible clients ( rand-init ).
Revised: Subsequently, we take the output architectures from our system, randomly initialise them and train end-to-end in a federated manner, where each model trains across all eligible clients ( rand-init ).

Original: Results are presented in Tab. 5.
Revised: Results are presented in Tab. 5.

Original: Indeed, models benefit from being initialised from a supernet across cases; this means that weightsharing mitigates both the limited model capacity of the lower-tier and the limited data exposure of large models.
Revised: Indeed, models benefit from being initialised from a supernet across cases; this means that weightsharing mitigates both the limited model capacity of the lower-tier and the limited data exposure of large models.

Original: The accuracy improvement is further magnified as non-IID-ness increases, leading up to +15 pp over rand-init .
Revised: The accuracy improvement is further magnified as non-IID-ness increases, leading up to +15 pp over rand-init .

Original: Results on different tasks of the same dataset presented in Appendix F.
Revised: Results on different tasks of the same dataset presented in Appendix F.

Original: 4.4 Ablation study
Revised: 4.4 Ablation study

Original: Next, we measure the contribution of specific components of FedorAS to the quality and performance of these models.
Revised: Next, we measure the contribution of specific components of FedorAS to the quality and performance of these models.

Original: To this direction, we firstly compare the performance of our system’s aggregation compared to naive averaging.
Revised: To this direction, we firstly compare the performance of our system’s aggregation compared to naive averaging.

Original: Subsequently, we measure the impact of sending smaller subspaces to clients for supernet training.
Revised: Subsequently, we measure the impact of sending smaller subspaces to clients for supernet training.

Original: We provide indicative results on CIFAR-10 α =0 .
Revised: We provide indicative results on CIFAR-10 α =0 .

Original: OPA vs. naive averaging.
Revised: OPA vs. naive averaging.

Original: OPA compared to FedAvg leads to increased accuracy by +1.2, +0.9, +1.

Revised: OPA compared to FedAvg leads to increased accuracy by +1.2, +0.9, +1.

Original: and +1.78 pp for tiers 1-4, respectively.
Revised: and +1.78 pp for tiers 1-4, respectively.

Original: More details in Appendix G.
Revised: More details in Appendix G.

Original: Subspace sampling size.
Revised: Subspace sampling size.

Original: FedorAS samples the supernet before communicating it to a client.
Revised: FedorAS samples the supernet before communicating it to a client.

Original: For
Revised: For

Original: CIFAR-10 ( α = 1 . 0 ) and when B Φ comm is set to 22.3M parameters (i.e. allowing sending whole supernet), FedorAS yields 85.48%, 86.73%, 86.50% average accuracies across tiers for full, 1 / 2 and 1 / 4 of the size of the search space, respectively.
Revised: CIFAR-10 ( α = 1 . 0 ) and when B Φ comm is set to 22.3M parameters (i.e. allowing sending whole supernet), FedorAS yields 85.48%, 86.73%, 86.50% average accuracies across tiers for full, 1 / 2 and 1 / 4 of the size of the search space, respectively.

Original: Thus, not only is the overhead of communication reduced with FedorAS , but convergence to the same level of accuracy is faster.
Revised: Thus, not only is the overhead of communication reduced with FedorAS , but convergence to the same level of accuracy is faster.

Original: We hypothesise that this is due a better balance in the exploration vs. exploitation trade-off.
Revised: We hypothesise that this is due a better balance in the exploration vs. exploitation trade-off.

Original: More details in Appendix G.
Revised: More details in Appendix G.

Original: 4.5 Evaluation of the Search Phase
Revised: 4.5 Evaluation of the Search Phase

Original: In order to assess the quality of models during search (Stage 2), so far we have needed a proxy dataset to evaluate different paths and rank them.
Revised: In order to assess the quality of models during search (Stage 2), so far we have needed a proxy dataset to evaluate different paths and rank them.

Original: We consider this as a set of examples that each application provider has centrally to measure the quality of their deployed models in production settings.
Revised: We consider this as a set of examples that each application provider has centrally to measure the quality of their deployed models in production settings.

Original: Validation set size.
Revised: Validation set size.

Original: Nevertheless, the size and representativeness of the centralised dataset might affect the quality of the search.
Revised: Nevertheless, the size and representativeness of the centralised dataset might affect the quality of the search.

Original: To gauge the sensitivity of the end models to the underlying distribution of the validation set, we sample down the validation set, as a held-out portion of the clients datasets, to 20% and 50% of the original size.
Revised: To gauge the sensitivity of the end models to the underlying distribution of the validation set, we sample down the validation set, as a held-out portion of the clients datasets, to 20% and 50% of the original size.

Original: We find no noticeable impact on the quality of the final models.
Revised: We find no noticeable impact on the quality of the final models.

Original: Detailed results are presented in Appendix J.1.
Revised: Detailed results are presented in Appendix J.1.

Original: Federated search.
Revised: Federated search.

Original: There may be also cases where no such dataset can be centralised.
Revised: There may be also cases where no such dataset can be centralised.

Original: To this direction, we test whether our search can operate under a federated setting with partial participation in order to faithfully rank the quality of models stemming from the supernet.
Revised: To this direction, we test whether our search can operate under a federated setting with partial participation in order to faithfully rank the quality of models stemming from the supernet.

Original: In this setting, we have implemented a federated version of NSGA-II.
Revised: In this setting, we have implemented a federated version of NSGA-II.

Original: Instead of candidate models being evaluated on the same validation set, they are stochastically evaluated on sampled federated client datasets [ ? ].
Revised: Instead of candidate models being evaluated on the same validation set, they are stochastically evaluated on sampled federated client datasets [62].

Original: We hypothesise it is possible to maintain faithful ranking of models compared to centralised evaluation if enough clients are leveraged to evaluate models, at the cost of increased communication cost.
Revised: We hypothesise it is possible to maintain faithful ranking of models compared to centralised evaluation if enough clients are leveraged to evaluate models, at the cost of increased communication cost.

Original: Furthermore, we expect the overall cost of achieving robust evaluation to increase as non-IID-ness and #clients increase, and the instantaneous cost of sending models to decrease over time, as NSGA-II converges to well-performing models (i.e. decreased diversity of models).
Revised: Furthermore, we expect the overall cost of achieving robust evaluation to increase as non-IID-ness and #clients increase, and the instantaneous cost of sending models to decrease over time, as NSGA-II converges to well-performing models (i.e. decreased diversity of models).

Original: Fig.
Revised: Fig.

Original: 4 shows results for
Revised: 4 shows results for

Original: [Figure]
Revised: [Figure]

Original: CIFAR

Revised: CIFAR

Original: α =0 .
Revised: α =0 .

Original: 1 , with extra results and details presented in Appendix J.2.
Revised: 1 , with extra results and details presented in Appendix J.2.

Original: Our experiments support the aforementioned conjectures.
Revised: Our experiments support the aforementioned conjectures.

Original: Noticeably, even under highly non-IID settings, we can attain faithful
Revised: Noticeably, even under highly non-IID settings, we can attain faithful

Original: FE at a reasonable cost (4 rounds to Kendall- τ > 0 .
Revised: FE at a reasonable cost (4 rounds to Kendall- τ > 0 .

Original: 8 for the results presented in Fig. 4)
Revised: 8 for the results presented in Fig. 4)

Original:  Related Work
Revised:  Related Work

Original: Data and system heterogeneity.
Revised: Data and system heterogeneity.

Original: Traditionally, works on heterogeneous FL have focused on tackling the statistical data heterogeneity [69, 49, 34, 27, 53] or minimising the upstream communication cost [45, 41, 73, 29, 5] of sending updates from client to servers, as the primary bottleneck in the federated training process.
Revised: Traditionally, works on heterogeneous FL have focused on tackling the statistical data heterogeneity [70, 49, 34, 26, 53] or minimising the upstream communication cost [45, 41, 74, 28, 5] of sending updates from client to servers, as the primary bottleneck in the federated training process.

Original: However, it has become apparent that computational disparity becomes an equally important barrier for contributing knowledge in FL.
Revised: However, it has become apparent that computational disparity becomes an equally important barrier for contributing knowledge in FL.

Original: As such, lately, there has been a line of work focusing on this very problem, where the discrepancy between the dynamics of different clients affects the convergence rate or fairness of the deployed system.
Revised: As such, lately, there has been a line of work focusing on this very problem, where the discrepancy between the dynamics of different clients affects the convergence rate or fairness of the deployed system.

Original: Specifically, such solutions draw from efficient ML and attempt to dynamically alter the footprint of local models my means of structured (PruneFL [37]), unstructured (Adaptive Federated Dropout [13], LotteryFL [46]) or importance-based pruning (FjORD [33]), quantisation (AQFL [2]), low-rank factorisation (FedHM [76]), sparsityinducing training (ZeroFL [64]) or distillation (GKT [31]).
Revised: Specifically, such solutions draw from efficient ML and attempt to dynamically alter the footprint of local models my means of structured (PruneFL [37]), unstructured (Adaptive Federated Dropout [12], LotteryFL [46]) or importance-based pruning (FjORD [33]), quantisation (AQFL [2]), low-rank factorisation (FedHM [77]), sparsityinducing training (ZeroFL [65]) or distillation (GKT [30]).

Original: However, each approach has limitations, either because they involve extra training overhead [31] and residence of multiple DNN copies in memory [2], or because they require specialised hardware for performance gains ([64, 13]).
Revised: However, each approach has limitations, either because they involve extra training overhead [30] and residence of multiple DNN copies in memory [2], or because they require specialised hardware for performance gains ([65, 12]).

Original: Last, some of the architectural changes proposed may not offer the degrees of freedom that NAS exposes [33, 76, 46] or may have a different optimisation objective altogether (i.e. personalisation [46]).
Revised: Last, some of the architectural changes proposed may not offer the degrees of freedom that NAS exposes [33, 77, 46] or may have a different optimisation objective altogether (i.e. personalisation [46]).

Original: Federated NAS.
Revised: Federated NAS.

Original: The concept of performing NAS in a federated setting has been visited before in the literature [30, 59,77, 78,  54].
Revised: The concept of performing NAS in a federated setting has been visited before in the literature [29, 59, 78, 79, 54].

Original: However, current solutions vary greatly in their approaches and goals, with most of them being applied and applicable to the cross-silo setting, where full participation and small number of participants are expected.
Revised: However, current solutions vary greatly in their approaches and goals, with most of them being applied and applicable to the cross-silo setting, where full participation and small number of participants are expected.

Original: Specifically, one of the first works in the area was FedNAS [30], which adopts a DARTS-based approach and aims to find a globally good model for all clients, to be personalised at a later stage.
Revised: Specifically, one of the first works in the area was FedNAS [29], which adopts a DARTS-based approach and aims to find a globally good model for all clients, to be personalised at a later stage.

Original: This approach requires the whole supernet to be transmitted and kept in memory for architectural updates, which leads to excessive requirements (Fig. 2) that make it largely inapplicable for cross-device setups spanning many clients of limited capabilities partially participating in training.
Revised: This approach requires the whole supernet to be transmitted and kept in memory for architectural updates, which leads to excessive requirements (Fig. 2) that make it largely inapplicable for cross-device setups spanning many clients of limited capabilities partially participating in training.

Original: To mitigate this requirement, [77] proposes an RL-based approach for cross-silo FL-based NAS.
Revised: To mitigate this requirement, [78] proposes an RL-based approach for cross-silo FL-based NAS.

Original: Despite the intention, it still incurs significant overheads due to RL-based model sampling convergence and single model training per client.
Revised: Despite the intention, it still incurs significant overheads due to RL-based model sampling convergence and single model training per client.

Original: A somewhat different approach is adopted by HAFL [54], which leverages graph hypernetworks as a means to generate outputs of a model.
Revised: A somewhat different approach is adopted by HAFL [54], which leverages graph hypernetworks as a means to generate outputs of a model.

Original: While interesting, performance and scalability are not on par with current state-of-the-art.
Revised: While interesting, performance and scalability are not on par with current state-of-the-art.

Original: On the front of personalisation, FedPNAS [6] searches for architectures with a shared based component across clients and a personalised component.
Revised: On the front of personalisation, FedPNAS [31] searches for architectures with a shared based component across clients and a personalised component.

Original: However, this work is only aimed at IID vision tasks and involves a meta-step for personalisation, which increases training overheads significantly.
Revised: However, this work is only aimed at IID vision tasks and involves a meta-step for personalisation, which increases training overheads significantly.

Original: At the other extreme for cross-silo personalised FL, SPIDER [59] aims at finding personalised architectures per client.
Revised: At the other extreme for cross-silo personalised FL, SPIDER [59] aims at finding personalised architectures per client.

Original: It requires the whole space on-device and federation is accomplished through a second static on-device network.
Revised: It requires the whole space on-device and federation is accomplished through a second static on-device network.

Original: These non-negligible overheads make porting this approach to the cross-device setting non-trivial.
Revised: These non-negligible overheads make porting this approach to the cross-device setting non-trivial.

Original: In contrast, FedorAS brings Federated NAS to the cross-device setting and is designed with resource efficiency and system heterogeneity in mind.
Revised: In contrast, FedorAS brings Federated NAS to the cross-device setting and is designed with resource efficiency and system heterogeneity in mind.

Original: This way, communication and computation cost is kept within the defined limits, respecting the runtime on constrained devices.
Revised: This way, communication and computation cost is kept within the defined limits, respecting the runtime on constrained devices.

Original: Crucially, our system does not assume full participation of clients and is able to efficiently exchange knowledge through the supernet weight sharing and generalise across different modalities and granularity of tasks.
Revised: Crucially, our system does not assume full participation of clients and is able to efficiently exchange knowledge through the supernet weight sharing and generalise across different modalities and granularity of tasks.

Original:  Conclusion
Revised:  Conclusion

Original: In this work we have presented FedorAS , a system that performs resource-efficient federated NAS in the cross-device setting.
Revised: In this work we have presented FedorAS , a system that performs resource-efficient federated NAS in the cross-device setting.

Original: Our method offers significantly lower overhead compared to existing federated NAS techniques, while achieving state-of-the-art performance compared to heterogeneous
Revised: Our method offers significantly lower overhead compared to existing federated NAS techniques, while achieving state-of-the-art performance compared to heterogeneous

Original: FL solutions, by means of effective weight sharing and flexible resource-aware training
Revised: FL solutions, by means of effective weight sharing and flexible resource-aware training

Original: A Introduction
Revised: A Introduction

Original: B
Revised: B

Original: D Detailed Experimental Setup
Revised: D Detailed Experimental Setup

Original: D.1 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: D.1 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: D.2 Datasets . .
Revised: D.2 Datasets . .

Original: D.3 Search Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: D.3 Search Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: D. client assignment .
Revised: D. client assignment .

Original: D.

Revised: D.

Original: Hyperparameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Hyperparameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: D. device Federated evaluation . . . . . . . . . . . . . . . . . . . . . .
Revised: D. device Federated evaluation . . . . . . . . . . . . . . . . . . . . . .

Original: D.7 Baselines . .
Revised: D.7 Baselines . .

Original: Sampling
Revised: Sampling

Original: Fairness the
Revised: Fairness the

Original: Phase
Revised: Phase

Original: J.
Revised: J.

Original: Sensitivity to size of validation set . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Sensitivity to size of validation set . . . . . . . . . . . . . . . . . . . . . . . . .

Original: Federated Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Federated Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A Introduction
Revised: A Introduction

Original: This Appendix extends the content of the main paper by providing support material to the study presented in Sec. 3, 4 and, additional insights about FedorAS .
Revised: This Appendix extends the content of the main paper by providing support material to the study presented in Sec. 3, 4 and, additional insights about FedorAS .

Original: Concretely, the Appendix is divided into three main blocks of extra content:
Revised: Concretely, the Appendix is divided into three main blocks of extra content:

Original: • Impact and Limitations.
Revised: • Impact and Limitations.

Original: We concisely present the broad impact and limitations of our work as well as future research directions in Sec.
Revised: We concisely present the broad impact and limitations of our work as well as future research directions in Sec.

Original: B, C. • Experimental Setup.
Revised: B, C. • Experimental Setup.

Original: Sec.
Revised: Sec.

Original: D provides details on the libraries used to build FedorAS , the datasets considered for experiments as well as the hyperparameters used to obtain the results presented in Section 4.
Revised: D provides details on the libraries used to build FedorAS , the datasets considered for experiments as well as the hyperparameters used to obtain the results presented in Section 4.

Original: Crucially, we provide a detailed description of the search spaces utilised in for each dataset domain, how tiers are defined and, a concise description of each baseline method included in this work.
Revised: Crucially, we provide a detailed description of the search spaces utilised in for each dataset domain, how tiers are defined and, a concise description of each baseline method included in this work.

Original: • Additional Experiments.
Revised: • Additional Experiments.

Original: Sec.
Revised: Sec.

Original: E through H study different aspects of FedorAS such as: i) learning multiple tasks using the supernet in Sec.
Revised: E through H study different aspects of FedorAS such as: i) learning multiple tasks using the supernet in Sec.

Original: F; or ii) the convergence behaviour in Sec.
Revised: F; or ii) the convergence behaviour in Sec.

Original: G; iii) the fairness aspect of FedorAS ; or under new scenarios altogether, such as: iv) an alternative procedure to assign clients to tiers in Section I; or iv) alternative search methods in Sec.
Revised: G; iii) the fairness aspect of FedorAS ; or under new scenarios altogether, such as: iv) an alternative procedure to assign clients to tiers in Section I; or iv) alternative search methods in Sec.

Original: J.
Revised: J.

Original: Overall, the following content substantially extends what is already presented in the main text.
Revised: Overall, the following content substantially extends what is already presented in the main text.

Original: B Broader Impact
Revised: B Broader Impact

Original: Our system, FedorAS , performs federated and resource-aware NAS in the cross-device setting.
Revised: Our system, FedorAS , performs federated and resource-aware NAS in the cross-device setting.

Original: Despite the benefits illustrated compared to centralised NAS and cross-silo FL solutions, running
Revised: Despite the benefits illustrated compared to centralised NAS and cross-silo FL solutions, running

Original: Neural Architecture Search is still a resource-demanding process, in terms of compute, memory and network bandwidth.
Revised: Neural Architecture Search is still a resource-demanding process, in terms of compute, memory and network bandwidth.

Original: While FedorAS ’s target devices can be of significantly lower TDP (i.e. smartphones and IoT devices vs. server-grade GPUs) – with consequences on the overall energy consumption of training [64] – they are resources not directly owned by the operator.
Revised: While FedorAS ’s target devices can be of significantly lower TDP (i.e. smartphones and IoT devices vs. server-grade GPUs) – with consequences on the overall energy consumption of training [65] – they are resources not directly owned by the operator.

Original: As such, special care needs to be taken with respect to how many device resources are leveraged at any point in, so as not hinder the usability of the device or invoke monetary costs to the user [40].
Revised: As such, special care needs to be taken with respect to how many device resources are leveraged at any point in, so as not hinder the usability of the device or invoke monetary costs to the user [40].

Original: C Limitations & Future Work
Revised: C Limitations & Future Work

Original: Despite the challenges addressed by FedorAS , our prototype has certain limitations.
Revised: Despite the challenges addressed by FedorAS , our prototype has certain limitations.

Original: First and foremost, we have opted to cluster devices (i.e. in tiers) based on their FLOPS.
Revised: First and foremost, we have opted to cluster devices (i.e. in tiers) based on their FLOPS.

Original: While it is perfectly normal to divide clusters based on other criteria (e.g. memory, latency, energy) or in a multi-objective manner, we have kept it simple.
Revised: While it is perfectly normal to divide clusters based on other criteria (e.g. memory, latency, energy) or in a multi-objective manner, we have kept it simple.

Original: Moreover, one can opt for biased sampling of i) clients participating in a round, ii) the subspace they get communicated and iii) the paths sampled from that subspace, we opted for uniform sampling for all of the above for simplicity, uniform coverage of the search space and fairness in participation.
Revised: Moreover, one can opt for biased sampling of i) clients participating in a round, ii) the subspace they get communicated and iii) the paths sampled from that subspace, we opted for uniform sampling for all of the above for simplicity, uniform coverage of the search space and fairness in participation.

Original: We leave the exploration of such strategies as future work.
Revised: We leave the exploration of such strategies as future work.

Original: Last but not least, we have considered privacy-enhancing techniques, such as Differential Privacy [15] or Secure
Revised: Last but not least, we have considered privacy-enhancing techniques, such as Differential Privacy [14] or Secure

Original: Aggregation [12, 7], as orthogonal to our scheme.
Revised: Aggregation [11, 6], as orthogonal to our scheme.

Original: Combining Federated NAS with such strategies can expose interesting trade-offs of exploration-exploitation-privacy budgets that could be explored in the future.
Revised: Combining Federated NAS with such strategies can expose interesting trade-offs of exploration-exploitation-privacy budgets that could be explored in the future.

Original: D Detailed Experimental Setup D.1 Implementation
Revised: D Detailed Experimental Setup D.1 Implementation

Original: FedorAS was implemented on top of the Flower (v0.18) [10] framework and PyTorch (v1.11.0) [61].
Revised: FedorAS was implemented on top of the Flower (v0.18) [9] framework and PyTorch (v1.11.0) [61].

Original: We run all our experiments on a private cloud cluster in a simulated manner, across four iterations each and report averages and standard deviations.
Revised: We run all our experiments on a private cloud cluster in a simulated manner, across four iterations each and report averages and standard deviations.

Original: D.2 Datasets
Revised: D.2 Datasets

Original: We partition CIFAR-10/100 following the Latent Dirichlet Allocation (LDA) partitioning [35], with each client receiving approximately equisized training sets.
Revised: We partition CIFAR-10/100 following the Latent Dirichlet Allocation (LDA) partitioning [35], with each client receiving approximately equisized training sets.

Original: For CIFAR-10, we consider α ∈ { 1000 , 1 . 0 , 0 .
Revised: For CIFAR-10, we consider α ∈ { 1000 , 1 . 0 , 0 .

Original: 1 } configurations, while for CIFAR-100, we adopt α = 0 .
Revised: 1 } configurations, while for CIFAR-100, we adopt α = 0 .

Original: 1 as in [67].
Revised: 1 as in [68].

Original: The remaining datasets come naturally partitioned.
Revised: The remaining datasets come naturally partitioned.

Original: CIFAR-10/100.
Revised: CIFAR-10/100.

Original: The CIFAR-10 datasets contains 10k and 50k 32x32 RGB images in its test and training sets respectively comprising ten classes.
Revised: The CIFAR-10 datasets contains 10k and 50k 32x32 RGB images in its test and training sets respectively comprising ten classes.

Original: The goal is to classify these images correctly.
Revised: The goal is to classify these images correctly.

Original: Similarly, CIFAR-100 follows an identical partitioning but, this time, across 100 classes ( fine lebels) or 20 superclasses ( coarse labels).
Revised: Similarly, CIFAR-100 follows an identical partitioning but, this time, across 100 classes ( fine lebels) or 20 superclasses ( coarse labels).

Original: Both CIFAR datasets have a uniform coverage across their classes.
Revised: Both CIFAR datasets have a uniform coverage across their classes.

Original: SpeechCommands.
Revised: SpeechCommands.

Original: The Speech Commands datasets [74] is comprised of 105,829, 16KHz 1-second long audio clips of a spoken word (e.g. "yes", "up", "stop") and the task is to classify these in aor 35 classes setting.
Revised: The Speech Commands datasets [75] is comprised of 105,829, 16KHz 1-second long audio clips of a spoken word (e.g. "yes", "up", "stop") and the task is to classify these in aor 35 classes setting.

Original: The datasets comes pre-partitioned into 35 classes and in order to obtain the 12-classes version, the standard approach [9, 20, 72] is to keep 10 classes of interest (i.e. "yes", "no", "up", "down", "left", "right", "on", "off", "stop", "go"), place the remaining 25 under the "unknown" class and, introduce a new class "silence" where no spoken word appear is the audio clip.
Revised: The datasets comes pre-partitioned into 35 classes and in order to obtain the 12-classes version, the standard approach [8, 19, 73] is to keep 10 classes of interest (i.e. "yes", "no", "up", "down", "left", "right", "on", "off", "stop", "go"), place the remaining 25 under the "unknown" class and, introduce a new class "silence" where no spoken word appear is the audio clip.

Original: In this work we consider SpeechCommandsV2, the most recent version of this dataset.
Revised: In this work we consider SpeechCommandsV2, the most recent version of this dataset.

Original: The dataset spans three disjoint set of speakers: 2112 form the training set, 256 for validation and, 250 for testing.
Revised: The dataset spans three disjoint set of speakers: 2112 form the training set, 256 for validation and, 250 for testing.

Original: In FedorAS , the supernet training phase makes uses of the 2112 clients in the training partition only.
Revised: In FedorAS , the supernet training phase makes uses of the 2112 clients in the training partition only.

Original: The results are obtained by measuring the performance of the discovered models on the data of the 250 clients comprising the test set.
Revised: The results are obtained by measuring the performance of the discovered models on the data of the 250 clients comprising the test set.

Original: Data is not uniformly distributed and some clients have more data than others.
Revised: Data is not uniformly distributed and some clients have more data than others.

Original: This dataset is processed by first extracting MFCC [22] features from each audio clip [79, 9].
Revised: This dataset is processed by first extracting MFCC [21] features from each audio clip [80, 8].

Original: Across our experiments we extract 40 MFCC features from a MelSpectrogram where each audio signal is first sub-sampled down to 8KHz and then sampled using 40ms wide time windows strided 20ms appart.
Revised: Across our experiments we extract 40 MFCC features from a MelSpectrogram where each audio signal is first sub-sampled down to 8KHz and then sampled using 40ms wide time windows strided 20ms appart.

Original: This results in 1-second audio clip being transformed into a 1 × 40 ×input that can be passed to a CNN.
Revised: This results in 1-second audio clip being transformed into a 1 × 40 ×input that can be passed to a CNN.

Original: Shakespeare.
Revised: Shakespeare.

Original: This dataset is built [18, 57] from The Complete Works of William Shakespeare and partitioned in such a way data that from each role in the play is assigned to a client.
Revised: This dataset is built [17, 57] from The Complete Works of William Shakespeare and partitioned in such a way data that from each role in the play is assigned to a client.

Original: This results in a total of 1,129 partitions where the average number of samples per device is 3.7K and the standard deviation 6.2K samples.
Revised: This results in a total of 1,129 partitions where the average number of samples per device is 3.7K and the standard deviation 6.2K samples.

Original: This makes Shakespeare a relatively imbalanced dataset.
Revised: This makes Shakespeare a relatively imbalanced dataset.

Original: The task is to correctly predict the next character in a dialog given the previously seen characters in the sentence.
Revised: The task is to correctly predict the next character in a dialog given the previously seen characters in the sentence.

Original: The vocabulary considered has 86 English characters as well as four special tokens: start and end of sentence, padding and out-of-vocabulary tokens.
Revised: The vocabulary considered has 86 English characters as well as four special tokens: start and end of sentence, padding and out-of-vocabulary tokens.

Original: D.3 Search Spaces
Revised: D.3 Search Spaces

Original: We assume that the largest model in the search space is the maximal size that any of the clients can handle.
Revised: We assume that the largest model in the search space is the maximal size that any of the clients can handle.

Original: The minimum cost is set to the fixed cost of the network (i.e. cost of non-searchable components).
Revised: The minimum cost is set to the fixed cost of the network (i.e. cost of non-searchable components).

Original: Given this range, and number of clusters C , we define the first and last clusters at prct l , prct r percentiles of a randomly sampled set of models from the search space and linearly scale the C − 2 clusters in-between.
Revised: Given this range, and number of clusters C , we define the first and last clusters at prct l , prct r percentiles of a randomly sampled set of models from the search space and linearly scale the C − 2 clusters in-between.

Original: All supernet search spaces are depicted in Fig.
Revised: All supernet search spaces are depicted in Fig.

Original: 5 and described below.
Revised: 5 and described below.

Original: C on v ( 3 )
Revised: C on v ( 3 )

Original: O pe r a t i on O pe r a t i on
Revised: O pe r a t i on O pe r a t i on

Original: O pe r a t i on O pe r a t i on
Revised: O pe r a t i on O pe r a t i on

Original: D S C on v ( 3 )
Revised: D S C on v ( 3 )

Original: C on v ( 2 )
Revised: C on v ( 2 )

Original: G l oba l poo li ng
Revised: G l oba l poo li ng

Original: L i nea r repeated 4 times
Revised: L i nea r repeated 4 times

Original: O pe r a t i on
Revised: O pe r a t i on

Original: S k i p S k i p
Revised: S k i p S k i p

Original: O pe r a t i on
Revised: O pe r a t i on

Original: E m bedd
Revised: E m bedd

Original: i ng
Revised: i ng

Original: L i nea r
Revised: L i nea r

Original: [Figure]
Revised: [Figure]

Original: c=same c=prev*1.5s=
Revised: = c =same c=prev*1.5s=

Original: C on v ( 3 )
Revised: C on v ( 3 )

Original: O pe r a t i on O pe r a t i on
Revised: O pe r a t i on O pe r a t i on

Original: G l oba l poo li ng
Revised: G l oba l poo li ng

Original: L i nea r
Revised: L i nea r

Original: [Figure]
Revised: [Figure]

Original: c
Revised: c

Original: =same
Revised: =same

Original: C on v ( 1 )
Revised: C on v ( 1 )

Original: A v g .
Revised: A v g .

Original: P oo l ( 3 ) c=prev*1.5s=
Revised: P oo l ( 3 ) c=prev*1.5s=

Original: O pe r a t i on c=same
Revised: O pe r a t i on c=same

Original: C on v ( 1 )
Revised: C on v ( 1 )

Original: A v g .
Revised: A v g .

Original: P oo l ( 3 ) c=prev*1.5s= repeated 2 times d= d=
Revised: P oo l ( 3 ) c=prev*1.5s= repeated 2 times d= d=

Original: CIFAR-10 and CIFAR-100.
Revised: CIFAR-10 and CIFAR-100.

Original: We use a ResNet-like search space similar to the one used in, for example, Cai et al.
Revised: We use a ResNet-like search space similar to the one used in, for example, Cai et al.

Original: Specifically, our model is a feedforward model with a fixed ( i.e. nonsearchable) stem layer followed by a sequence of 3 searchable blocks, each followed by a fixed reduction block.
Revised: Specifically, our model is a feedforward model with a fixed ( i.e. nonsearchable) stem layer followed by a sequence of 3 searchable blocks, each followed by a fixed reduction block.

Original: A standard block consists of 4 searchable layers organized again in a simple feedforward manner.
Revised: A standard block consists of 4 searchable layers organized again in a simple feedforward manner.

Original: Operations within a standard block preserve shape of their inputs, but are allowed to change dimensions of intermediate results.
Revised: Operations within a standard block preserve shape of their inputs, but are allowed to change dimensions of intermediate results.

Original: On the other hand, the goal of reduction blocks is to reduce spatial resolution ( 2 × in each dimension) and increase number of channels ( 1 . 5 × ).
Revised: On the other hand, the goal of reduction blocks is to reduce spatial resolution ( 2 × in each dimension) and increase number of channels ( 1 . 5 × ).

Original: Reduction blocks are fixed throughout and consists of a depthwise separable convolution 3 × 3 , with the first part performing spatial reduction and the second increasing the number of channels, and a standard 2 × 2 convolution applied to a residual link.
Revised: Reduction blocks are fixed throughout and consists of a depthwise separable convolution 3 × 3 , with the first part performing spatial reduction and the second increasing the number of channels, and a standard 2 × 2 convolution applied to a residual link.

Original: The sequence of blocks is finished with a global average pooling per-channel and a classification layer outputting either 10 or 20/100 logits, which is the only difference between the two models.
Revised: The sequence of blocks is finished with a global average pooling per-channel and a classification layer outputting either 10 or 20/100 logits, which is the only difference between the two models.

Original: Each convolution operation is followed by a batch normalization (BN) and ReLU.
Revised: Each convolution operation is followed by a batch normalization (BN) and ReLU.

Original: In our experiments we considered the following candidate operations:
Revised: In our experiments we considered the following candidate operations:

Original: • standard convolution 1 × 1 with BN and ReLU • depthwise separable convolution 3 × 3 with expansion ratio (controlling the number of intermediate channels) set to { 0 . 5 , 1 , 2 } , with each expansion ratio being an independent choice in our search space • MobileNet-like block consisting of a convolution with kernel size k and expansion ratio e , followed by a Squeeze-and-Excitation layer, followed by a 1 × 1 convolution reverting the channels expansion, we considered { ( k = 1 , e = 2) , ( k = 3 , e = 0 . 5) , ( k = 3 , e = 1) , ( k = 3 , e = 2) } • identity operation
Revised: • standard convolution 1 × 1 with BN and ReLU • depthwise separable convolution 3 × 3 with expansion ratio (controlling the number of intermediate channels) set to { 0 . 5 , 1 , 2 } , with each expansion ratio being an independent choice in our search space • MobileNet-like block consisting of a convolution with kernel size k and expansion ratio e , followed by a Squeeze-and-Excitation layer, followed by a 1 × 1 convolution reverting the channels expansion, we considered { ( k = 1 , e = 2) , ( k = 3 , e = 0 . 5) , ( k = 3 , e = 1) , ( k = 3 , e = 2) } • identity operation

Original: The stem layer was set to a 3 × 3 convolution outputting 64 channels.
Revised: The stem layer was set to a 3 × 3 convolution outputting 64 channels.

Original: Speech Commands.
Revised: Speech Commands.

Original: The model follows the one used for the CIFAR datasets but is made more lightweight – to roughly match what can be found in the literature – by reducing stem channels toand including only 1 (resp. 2) searchable operations in the first (resp.
Revised: The model follows the one used for the CIFAR datasets but is made more lightweight – to roughly match what can be found in the literature – by reducing stem channels toand including only 1 (resp. 2) searchable operations in the first (resp.

Original: last) two blocks.
Revised: last) two blocks.

Original: Additionally, reduction block only includes a single 1 × 1 convolution, that changes the number of channels, followed by a 3 × 3 average pooling that reduces spatial dimensions.
Revised: Additionally, reduction block only includes a single 1 × 1 convolution, that changes the number of channels, followed by a 3 × 3 average pooling that reduces spatial dimensions.

Original: We also include additional candidate operations:
Revised: We also include additional candidate operations:

Original: • standard convolution 3 ×• depthwise separable convolution with kernel size 1 and expansion ratio
Revised: • standard convolution 3 ×• depthwise separable convolution with kernel size 1 and expansion ratio

Original: All other operations from the CIFAR model are also included.
Revised: All other operations from the CIFAR model are also included.

Original: Shakespeare.
Revised: Shakespeare.

Original: We base our design on the model used by FjORD.
Revised: We base our design on the model used by FjORD.

Original: Specifically, the model is a recurrent network that begins with a fixed embedding layer outputting 8-dimensional feature vector per each input character.
Revised: Specifically, the model is a recurrent network that begins with a fixed embedding layer outputting 8-dimensional feature vector per each input character.

Original: This is then followed by a searchable operation #1 that increases dimensionality from 8 to 128; in parallel, we have a searchable skip connection #1 operation whose output is added to the output of the operation
Revised: This is then followed by a searchable operation #1 that increases dimensionality from 8 to 128; in parallel, we have a searchable skip connection #1 operation whose output is added to the output of the operation

Original: Later there is another searchable operation #2 with its own skip connection #2, both keeping the hidden dimension at 128.
Revised: Later there is another searchable operation #2 with its own skip connection #2, both keeping the hidden dimension at 128.

Original: Again, their outputs are added and passed to the final classification layer.
Revised: Again, their outputs are added and passed to the final classification layer.

Original: Candidate operations for each of the four searchable layers are mainly the same, with minor adjustments made to make sure a valid model is always constructed.
Revised: Candidate operations for each of the four searchable layers are mainly the same, with minor adjustments made to make sure a valid model is always constructed.

Original: These include:
Revised: These include:

Original: • an LSTM layer [32], • a GRU layer [19], • a LiGRU layer with tanh activation [65], • a QuasiRNN layer [14], • a simple Linear layer (no recurrent connection) followed by a sigmoid activation, • a 1D convolution with kernel 5 spanning time dimension (looking at the current character and 4 previous ones), followed by a sigmoid activation (no normalisation), • identity operation (only included in later operation, after feature dimension has been increased to 128), • zero operation, outputting zeros of the correct shape (only included in skip connection layers).
Revised: • an LSTM layer [32], • a GRU layer [18], • a LiGRU layer with tanh activation [66], • a QuasiRNN layer [13], • a simple Linear layer (no recurrent connection) followed by a sigmoid activation, • a 1D convolution with kernel 5 spanning time dimension (looking at the current character and 4 previous ones), followed by a sigmoid activation (no normalisation), • identity operation (only included in later operation, after feature dimension has been increased to 128), • zero operation, outputting zeros of the correct shape (only included in skip connection layers).

Original: For LiGRU and QuasiRNN we used implementations provided by the Speechbrain project [66], after minor modifications.
Revised: For LiGRU and QuasiRNN we used implementations provided by the Speechbrain project [67], after minor modifications.

Original: For others, we used standard operations from PyTorch.
Revised: For others, we used standard operations from PyTorch.

Original: All operations were used as unidirectional.
Revised: All operations were used as unidirectional.

Original: We did not use any normalisation throughout the model.
Revised: We did not use any normalisation throughout the model.

Original: D.4 Tiers: definition and client assignment
Revised: D.4 Tiers: definition and client assignment

Original: With FedorAS the discovery and training of architectures happens in a tier-aware fashion as a federated process.
Revised: With FedorAS the discovery and training of architectures happens in a tier-aware fashion as a federated process.

Original: In this work we considered splitting each search space along the FLOPs dimensions (but other splits are possible, e.g: energy, peak-memory usage, etc – or a combination of these).
Revised: In this work we considered splitting each search space along the FLOPs dimensions (but other splits are possible, e.g: energy, peak-memory usage, etc – or a combination of these).

Original: Fig.illustrates the span in terms of model parameters and FLOPs of each search space presented in D. and the split along the FLOPs dimensions for each of them.
Revised: Fig.illustrates the span in terms of model parameters and FLOPs of each search space presented in D. and the split along the FLOPs dimensions for each of them.

Original: These search spaces vary considerably in terms of size and span, which motivated us to use different number of tiers or client-to-tier assignment strategies.
Revised: These search spaces vary considerably in terms of size and span, which motivated us to use different number of tiers or client-to-tier assignment strategies.

Original: Tab. 6 shows the the FLOPs ranges considered for each tier, the number of models in those sub-spaces exposed to FedorAS as well as the ratio of models in the entire search space that fall onto each tier.
Revised: Tab. 6 shows the the FLOPs ranges considered for each tier, the number of models in those sub-spaces exposed to FedorAS as well as the ratio of models in the entire search space that fall onto each tier.

Original: [Table]
Revised: [Table]

Original: Identifying FLOPs ranges for each tier.
Revised: Identifying FLOPs ranges for each tier.

Original: Regardless of the dataset, we follow a common approach to divide the FLOPs dimension that each dataset spans.
Revised: Regardless of the dataset, we follow a common approach to divide the FLOPs dimension that each dataset spans.

Original: The main aim is to finely control how many models fall onto the smallest/largest tier, given that these are considerably sparse regions of the entire search space.
Revised: The main aim is to finely control how many models fall onto the smallest/largest tier, given that these are considerably sparse regions of the entire search space.

Original: If we were to evenly split the FLOPs dimension, almost no architecture would fall onto the largest tier.
Revised: If we were to evenly split the FLOPs dimension, almost no architecture would fall onto the largest tier.

Original: To simplify the process of defining where each tier’s boundaries lie, we follow these steps which require two hyperparameters: (1) we construct a long array of FLOPs from models sampled from the search space and this array is then sorted from lowest to highest; then, (2) we normalise this array of FLOPs and compute the cumulative sum of such normalised array; finally, (3) we identify the lower limit of the highest tier as maximum FLOPs value found in the first ρN, ρ ∈ [0 , 1] , elements of the array where N is the number of samples taken (we found 100K to work well).
Revised: To simplify the process of defining where each tier’s boundaries lie, we follow these steps which require two hyperparameters: (1) we construct a long array of FLOPs from models sampled from the search space and this array is then sorted from lowest to highest; then, (2) we normalise this array of FLOPs and compute the cumulative sum of such normalised array; finally, (3) we identify the lower limit of the highest tier as maximum FLOPs value found in the first ρN, ρ ∈ [0 , 1] , elements of the array where N is the number of samples taken (we found 100K to work well).

Original: Essentially, we find the FLOPs for which the approximated PDF over the FLOPS of a given search space doesn’t surpass ρ .
Revised: Essentially, we find the FLOPs for which the approximated PDF over the FLOPS of a given search space doesn’t surpass ρ .

Original: Once that FLOPs value is identified, the FLOPs range is evenly split for the tiers below and the higher limit for the largest tier becomes the maximum FLOPs a model in the search space can have.
Revised: Once that FLOPs value is identified, the FLOPs range is evenly split for the tiers below and the higher limit for the largest tier becomes the maximum FLOPs a model in the search space can have.

Original: This is the approach for CIFAR-10 and CIFAR-100 as illustrated in
Revised: This is the approach for CIFAR-10 and CIFAR-100 as illustrated in

Original: Fig.
Revised: Fig.

Original: 6 (a) and (b).
Revised: 6 (a) and (b).

Original: For SpeechCommands and Shakespeare we introduce a similar approach to give a wider range to the smallest tier.
Revised: For SpeechCommands and Shakespeare we introduce a similar approach to give a wider range to the smallest tier.

Original: We refer to ρ L and ρ H as the PDF ratios to identify the boundaries splitting Tier1&2 and Tier3&4 respectively.
Revised: We refer to ρ L and ρ H as the PDF ratios to identify the boundaries splitting Tier1&2 and Tier3&4 respectively.

Original: The concrete values for each dataset as well as the
Revised: The concrete values for each dataset as well as the

Original: FLOPs values for each tier boundary are shown in Tab. 6.
Revised: FLOPs values for each tier boundary are shown in Tab. 6.

Original: Clients to tiers assignment.
Revised: Clients to tiers assignment.

Original: For CIFAR-10/100 clients are uniformly assigned to a cluster of devices or tier.
Revised: For CIFAR-10/100 clients are uniformly assigned to a cluster of devices or tier.

Original: For these datasets we consider four tiers, so each ends up containing 25% the clients resulting in 25 clients for CIFAR-10 and 125 clients for CIFAR-100.
Revised: For these datasets we consider four tiers, so each ends up containing 25% the clients resulting in 25 clients for CIFAR-10 and 125 clients for CIFAR-100.

Original: Similarly, for Shakespeare clients are also uniformly assigned to a tier, resulting in 143 clients per tier.
Revised: Similarly, for Shakespeare clients are also uniformly assigned to a tier, resulting in 143 clients per tier.

Original: For SpeechCommands, we designed a more challenging setup and divide the clients into tiers as follows: 80% of clients are assigned to be tier-1 devices, 17.5% are Tier-4 devices and the rest is evenly split into Tier-2 and Tier-3.
Revised: For SpeechCommands, we designed a more challenging setup and divide the clients into tiers as follows: 80% of clients are assigned to be tier-1 devices, 17.5% are Tier-4 devices and the rest is evenly split into Tier-2 and Tier-3.

Original: This distribution better represents the the types of systems in the wild that perform keyword-spotting [79], where the majority of the commercially deployed systems run these applications on low end CPUs or microcontrollers due to their always-on nature.
Revised: This distribution better represents the the types of systems in the wild that perform keyword-spotting [80], where the majority of the commercially deployed systems run these applications on low end CPUs or microcontrollers due to their always-on nature.

Original: Across datasets, the client-to-tier assignments was done irrespective of the amount of data these contained or distribution over the labels.
Revised: Across datasets, the client-to-tier assignments was done irrespective of the amount of data these contained or distribution over the labels.

Original: An alternative client to tier allocation is provided in Sec.
Revised: An alternative client to tier allocation is provided in Sec.

Original: I.
Revised: I.

Original: [Figure]
Revised: [Figure]

Original: D.5 Hyperparameters
Revised: D.5 Hyperparameters

Original: Here we present the hyperparameters used across all datasets and tasks to generate the results presented in Sec. 4.
Revised: Here we present the hyperparameters used across all datasets and tasks to generate the results presented in Sec. 4.

Original: In Tab.
Revised: In Tab.

Original: 7 we show the hyperparamters utilised for the first stage in the FedorAS framework: federated training of the supernet.
Revised: 7 we show the hyperparamters utilised for the first stage in the FedorAS framework: federated training of the supernet.

Original: After this stage, the best model for each tier is extracted from the supernet and then, they get finetuned in a per-tier aware manner (i
Revised: After this stage, the best model for each tier is extracted from the supernet and then, they get finetuned in a per-tier aware manner (i

Original: [Figure]
Revised: [Figure]

Original: e. clients in tier T and above can finetune a model that belong to tier T ).
Revised: e. clients in tier T and above can finetune a model that belong to tier T ).

Original: The hyperparameters of these two consecutive stages are shown in Tab.
Revised: The hyperparameters of these two consecutive stages are shown in Tab.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Regarding the rand-init results (i
Revised: Regarding the rand-init results (i

Original: [Table]
Revised: [Table]

Original: e. models discovered by FedorAS but trained from scratch – discarding the weights given by the supernet)
Revised: e. models discovered by FedorAS but trained from scratch – discarding the weights given by the supernet)

Original: we present the hyperparameters in Tab. 9.
Revised: we present the hyperparameters in Tab. 9.

Original: These hyperparameters were the ones used to generate the results in Tab. 5 and 12 as well as Fig.
Revised: These hyperparameters were the ones used to generate the results in Tab. 5 and 12 as well as Fig.

Original: In
Revised: In

Original: Sec.
Revised: Sec.

Original: 4.2, we present a CIFAR-100 result that largely outperforms existing federated baseline of [67].
Revised: 4.2, we present a CIFAR-100 result that largely outperforms existing federated baseline of [68].

Original: This was achieved by a model discovered by FedorAS but trained in the rand-init setting following the setup as in [67]: 10 clients per round for 4k rounds using batch 20, starting learning rate of 0.
Revised: This was achieved by a model discovered by FedorAS but trained in the rand-init setting following the setup as in [68]: 10 clients per round for 4k rounds using batch 20, starting learning rate of 0.

Original: decaying to 0.01 following a cosine scheduling.
Revised: decaying to 0.01 following a cosine scheduling.

Original: [Table]
Revised: [Table]

Original: D.6 Cross-device Federated NAS evaluation
Revised: D.6 Cross-device Federated NAS evaluation

Original: In Tab. 2 of Sec.
Revised: In Tab. 2 of Sec.

Original: 4.2 we compared FedorAS against FedNAS in the cross-sile setting with 100 clients and 10 clients randomly sampled on each round.
Revised: 4.2 we compared FedorAS against FedNAS in the cross-sile setting with 100 clients and 10 clients randomly sampled on each round.

Original: Both methods follow a substantially different approach as far as NAS is concern and, as a result
Revised: Both methods follow a substantially different approach as far as NAS is concern and, as a result

Original: [Table]
Revised: [Table]

Original: FedNAS has a significantly higher memory peak than FedorAS for the same batch size.
Revised: FedNAS has a significantly higher memory peak than FedorAS for the same batch size.

Original: For example, for a batch size of 64 images, FedorAS sees a memory peak of 998MB whereas FedNAS requires 7674MB.
Revised: For example, for a batch size of 64 images, FedorAS sees a memory peak of 998MB whereas FedNAS requires 7674MB.

Original: Similarly, both methods translate in different compute footprints for each client.
Revised: Similarly, both methods translate in different compute footprints for each client.

Original: For example, FedNAS requires on average 716 GFLOPs per client (assuming each client), while clients in FedorAS need 280 GFLOPs for the same amount of local epochs and data in the client.
Revised: For example, FedNAS requires on average 716 GFLOPs per client (assuming each client), while clients in FedorAS need 280 GFLOPs for the same amount of local epochs and data in the client.

Original: Due to these differences, we aimed at normalising these aspects to make the comparison fair.
Revised: Due to these differences, we aimed at normalising these aspects to make the comparison fair.

Original: These results are shown in Tab. 10, which extends the content of Tab. 2.
Revised: These results are shown in Tab. 10, which extends the content of Tab. 2.

Original: [Table]
Revised: [Table]

Original: D.7 Baselines
Revised: D.7 Baselines

Original: In this section we faithfully describe what each baseline represents in the experiments of the main paper and the appendix to clarify with what we are comparing in each section.
Revised: In this section we faithfully describe what each baseline represents in the experiments of the main paper and the appendix to clarify with what we are comparing in each section.

Original: Tier-unaware (Fig.3): This baseline represents model architectures that have been trained end-toend in a federated manner without any awareness of client eligibility.
Revised: Tier-unaware (Fig.3): This baseline represents model architectures that have been trained end-toend in a federated manner without any awareness of client eligibility.

Original: We are using FedAvg with hyperparameters similar to those presented in Tab. 9.
Revised: We are using FedAvg with hyperparameters similar to those presented in Tab. 9.

Original: Tier-aware (Fig.3): This baseline adds client eligibility awareness to the previous baseline.
Revised: Tier-aware (Fig.3): This baseline adds client eligibility awareness to the previous baseline.

Original: This means that models of certain footprint can be trained only on clients of the eligible cluster and above.
Revised: This means that models of certain footprint can be trained only on clients of the eligible cluster and above.

Original: FjORD [33] (Fig.3, Tab. 4): FjORD is a baseline that is tackling system heterogeneity by means of
Revised: FjORD [33] (Fig.3, Tab. 4): FjORD is a baseline that is tackling system heterogeneity by means of

Original: Ordered Dropout.
Revised: Ordered Dropout.

Original: This particular variant is assuming the experimental setup of the original paper which shards the CIFAR-10 dataset per client without LDA.
Revised: This particular variant is assuming the experimental setup of the original paper which shards the CIFAR-10 dataset per client without LDA.

Original: The Shakespeare setup remains the same as ours.
Revised: The Shakespeare setup remains the same as ours.

Original: FjORD-LDA [33] (Fig.3, Tab. 4):
Revised: FjORD-LDA [33] (Fig.3, Tab. 4):

Original: For this baseline, we implemented FjORD and ran it on the same number of clusters as FedorAS , with LDA for CIFAR-10.
Revised: For this baseline, we implemented FjORD and ran it on the same number of clusters as FedorAS , with LDA for CIFAR-10.

Original: FedNAS [30] (Tab. 2, 3): FedNAS was one of the first papers attempting to perform NAS in a federated setting.
Revised: FedNAS [29] (Tab. 2, 3): FedNAS was one of the first papers attempting to perform NAS in a federated setting.

Original: Its search is DARTS-based and it was primarily aimed for the cross-silo setting.
Revised: Its search is DARTS-based and it was primarily aimed for the cross-silo setting.

Original: Cross-device FL NAS of limited scale (20 total clients with 4 clients sample per round) was also showcased in the original paper.
Revised: Cross-device FL NAS of limited scale (20 total clients with 4 clients sample per round) was also showcased in the original paper.

Original: We adjusted the code of FedNAS to perform cross-silo and crossdevice FL NAS with setups of varying training footprints by adjusting the amount of local epochs and batch size.
Revised: We adjusted the code of FedNAS to perform cross-silo and crossdevice FL NAS with setups of varying training footprints by adjusting the amount of local epochs and batch size.

Original: We have kept the original search space.
Revised: We have kept the original search space.

Original: SPIDER [59] (Tab. 3): SPIDER is another paper performing personalised Federated NAS in the cross-silo setting.
Revised: SPIDER [59] (Tab. 3): SPIDER is another paper performing personalised Federated NAS in the cross-silo setting.

Original: As there is no publicly available codebasse, we assume their setting with FedorAS and compare on CIFAR-10.
Revised: As there is no publicly available codebasse, we assume their setting with FedorAS and compare on CIFAR-10.

Original: ZeroFL [63] (Tab. 4):
Revised: ZeroFL [64] (Tab. 4):

Original: For this baseline, we borrow the results of respective paper for
Revised: For this baseline, we borrow the results of respective paper for

Original: CIFAR-10 α
Revised: CIFAR-10 α

Original: = { 1 , 1000 } , which assumes the same setup as ours.
Revised: = { 1 , 1000 } , which assumes the same setup as ours.

Original: We present the results for sparsity level of 90% and annotate its footprint as the original model FLOPS and the number of non-zero parameters.
Revised: We present the results for sparsity level of 90% and annotate its footprint as the original model FLOPS and the number of non-zero parameters.

Original: Oort [43] (Tab. 4): The Oort framework proposes a participation sampling criterion by which clients are sampled based on their utility (i.e. how much their data can contribute to the global model) while also taking the device capabilities into consideration.
Revised: Oort [43] (Tab. 4): The Oort framework proposes a participation sampling criterion by which clients are sampled based on their utility (i.e. how much their data can contribute to the global model) while also taking the device capabilities into consideration.

Original: Over the course of FL training, the sampling of clients with high data and system utility is prioritised to reduce the time needed to convergence.
Revised: Over the course of FL training, the sampling of clients with high data and system utility is prioritised to reduce the time needed to convergence.

Original: For
Revised: For

Original: SpeechCommands (see Tab. 4), Oort made use of a ResNet-34 model.
Revised: SpeechCommands (see Tab. 4), Oort made use of a ResNet-34 model.

Original: PyramidFL [48](Tab. 4): At a high-level, PyramidFL proposes a framework similar to that in Oort.
Revised: PyramidFL [48](Tab. 4): At a high-level, PyramidFL proposes a framework similar to that in Oort.

Original: The core difference between these two methods is that PyramidFL leverages more fine-grained statistics when assessing the contribution potential (i.e. utility) of the selected clients.
Revised: The core difference between these two methods is that PyramidFL leverages more fine-grained statistics when assessing the contribution potential (i.e. utility) of the selected clients.

Original: It also uses
Revised: It also uses

Original: ResNet-34 for SpeechCommands.
Revised: ResNet-34 for SpeechCommands.

Original: rand-init (Tab. 5, 12, 14): This baseline refers to the concept of running the search, coming up with an architecture and subsequently re-initialising the weights and running a conventional federated training setup.
Revised: rand-init (Tab. 5, 12, 14): This baseline refers to the concept of running the search, coming up with an architecture and subsequently re-initialising the weights and running a conventional federated training setup.

Original: It shows vanilla scaling that would be obtained if we simply trained identified models in a standard way, using random initialization and full FL training using standard practices.
Revised: It shows vanilla scaling that would be obtained if we simply trained identified models in a standard way, using random initialization and full FL training using standard practices.

Original: Model architectures are kept the same between FedorAS and “ rand-init ” experiments.
Revised: Model architectures are kept the same between FedorAS and “ rand-init ” experiments.

Original: Models belonging to higher tiers, as a whole, are never trained on data belonging to devices from lower tiers.
Revised: Models belonging to higher tiers, as a whole, are never trained on data belonging to devices from lower tiers.

Original: E Comparison against random search
Revised: E Comparison against random search

Original: In the main paper we compare our FedorAS to the state-of-the-art methods from the literature (Sec. 4.2) and also investigate the impact of initializing models with weights from a supernet (Sec. 4.3).
Revised: In the main paper we compare our FedorAS to the state-of-the-art methods from the literature (Sec. 4.2) and also investigate the impact of initializing models with weights from a supernet (Sec. 4.3).

Original: Here we present one more important baseline for any NAS algorithm - comparison to a random search.
Revised: Here we present one more important baseline for any NAS algorithm - comparison to a random search.

Original: Specifically, while we have already shown that models found by FedorAS in most cases benefit from supernet training, this does constitute a conclusive argument justifying the necessity of the entire process in a broader context.
Revised: Specifically, while we have already shown that models found by FedorAS in most cases benefit from supernet training, this does constitute a conclusive argument justifying the necessity of the entire process in a broader context.

Original: Perhaps there exist models that do not need to be initialized from a supernet but, since the focus of our work is on supernet training, we missed them due to biased conditioning?
Revised: Perhaps there exist models that do not need to be initialized from a supernet but, since the focus of our work is on supernet training, we missed them due to biased conditioning?

Original: [Table]
Revised: [Table]

Original: 81.01 ± 0.46 81.53 ± 0.29 80.64 ± 0.66 80.85 ± 0.
Revised: 81.01 ± 0.46 81.53 ± 0.29 80.64 ± 0.66 80.85 ± 0.

Original: - 66.75 ± 0.74 70.85 ± 1.53 69.00 ± 0.18 66.08 ± 2.
Revised: - 66.75 ± 0.74 70.85 ± 1.53 69.00 ± 0.18 66.08 ± 2.

Original: - CIFAR45.25 ± 0.13 45.84 ± 0.18 45.42 ± 0.39 45.07 ± 0.
Revised: - CIFAR45.25 ± 0.13 45.84 ± 0.18 45.42 ± 0.39 45.07 ± 0.

Original: - 34.41 ± 0.90 37.64 ± 0.46 36.98 ± 0.64 35.45 ± 0.
Revised: - 34.41 ± 0.90 37.64 ± 0.46 36.98 ± 0.64 35.45 ± 0.

Original: - SpeechCommands 80.19 ± 1.78 80.47 ± 1.69 81.00 ± 1.58 80.56 ± 0.
Revised: - SpeechCommands 80.19 ± 1.78 80.47 ± 1.69 81.00 ± 1.58 80.56 ± 0.

Original: - 82.84 ± 1.08 81.17 ± 0.26 81.10 ± 0.74 80.61 ± 0.
Revised: - 82.84 ± 1.08 81.17 ± 0.26 81.10 ± 0.74 80.61 ± 0.

Original: - Shakespeare 3.43 ± 0.
Revised: - Shakespeare 3.43 ± 0.

Original: 3.39 ± 0.
Revised: 3.39 ± 0.

Original: 3.38 ± 0.
Revised: 3.38 ± 0.

Original: 3.40 ± 0.01 3.42 ± 0.01 3.86 ± 0.
Revised: 3.40 ± 0.01 3.42 ± 0.01 3.86 ± 0.

Original: 3.54 ± 0.
Revised: 3.54 ± 0.

Original: 3.45 ± 0.
Revised: 3.45 ± 0.

Original: 3.80 ± 0.30 3.60 ± 0.
Revised: 3.80 ± 0.30 3.60 ± 0.

Original: In order to check this hypothesis we run a simple random search algorithm, to get an estimate of the expected best case performance of models from our search spaces when trained following standard
Revised: In order to check this hypothesis we run a simple random search algorithm, to get an estimate of the expected best case performance of models from our search spaces when trained following standard

Original: FL procedure.
Revised: FL procedure.

Original: This simple baseline was allowed to train random models until the total training cost exceeded the cost of running FedorAS – this turned out to be equivalent to roughly 40 fully-trained models.
Revised: This simple baseline was allowed to train random models until the total training cost exceeded the cost of running FedorAS – this turned out to be equivalent to roughly 40 fully-trained models.

Original: After we run out of the training budget, we simply get the best model for each tier as our solution to NAS.
Revised: After we run out of the training budget, we simply get the best model for each tier as our solution to NAS.

Original: We repeated the entire process 3 times for each search space and report average and standard deviation compared to the average performance of models found by FedorAS in Tab. 11.
Revised: We repeated the entire process 3 times for each search space and report average and standard deviation compared to the average performance of models found by FedorAS in Tab. 11.

Original: Additionally, we also plot detailed performance of each model trained during this process in Fig.for the sake of completeness.
Revised: Additionally, we also plot detailed performance of each model trained during this process in Fig.for the sake of completeness.

Original: Noticeably, FedorAS achieves significantly better results in almost all cases, with SpeechCommands being the only exception.
Revised: Noticeably, FedorAS achieves significantly better results in almost all cases, with SpeechCommands being the only exception.

Original: We suspect this is due to problem with fine-tuning rather than architectures themselves – specifically, we witness accuracy of the discovered models can vary significantly as we repeat the fine-tuning process (this is also visible in the case of full training, although the extend is smaller).
Revised: We suspect this is due to problem with fine-tuning rather than architectures themselves – specifically, we witness accuracy of the discovered models can vary significantly as we repeat the fine-tuning process (this is also visible in the case of full training, although the extend is smaller).

Original: Consequently, although on average FedorAS performs slightly worse, in many cases the best results surpass that of random search – because of that we suspect that fine-tuning for longer would improve FedorAS ’s performance.
Revised: Consequently, although on average FedorAS performs slightly worse, in many cases the best results surpass that of random search – because of that we suspect that fine-tuning for longer would improve FedorAS ’s performance.

Original: We leave this for future work, considering that a single shortcoming like that does not seem significant in the light of the rest of our results.
Revised: We leave this for future work, considering that a single shortcoming like that does not seem significant in the light of the rest of our results.

Original: F Impact of weight-sharing on different task training
Revised: F Impact of weight-sharing on different task training

Original: In this section, we expand on the analysis of Sec. 4.3.
Revised: In this section, we expand on the analysis of Sec. 4.3.

Original: Specifically for CIFAR-100 and SpeechCommands, we also create a second scenario, where not all clients train in the same domain of labels.
Revised: Specifically for CIFAR-100 and SpeechCommands, we also create a second scenario, where not all clients train in the same domain of labels.

Original: For
Revised: For

Original: CIFAR-100, there are 20 superclasses that are coarse-grained categories of the 100 standard labels.
Revised: CIFAR-100, there are 20 superclasses that are coarse-grained categories of the 100 standard labels.

Original: In contrast, for SpeechCommands, there are 12 and 35-class label sets, with the latter annotating the “other” class more specifically.
Revised: In contrast, for SpeechCommands, there are 12 and 35-class label sets, with the latter annotating the “other” class more specifically.

Original: In both cases, we assume a non-uniform distribution of clients to clusters, assuming that most data (75-80%) reside on lower-tier devices, whereas the two higher tier-devices (holding 20-25% of data) can train on the fine-grained label set.
Revised: In both cases, we assume a non-uniform distribution of clients to clusters, assuming that most data (75-80%) reside on lower-tier devices, whereas the two higher tier-devices (holding 20-25% of data) can train on the fine-grained label set.

Original: Our aim is to test whether few data allocation on the high tier devices can benefit from the knowledge and feature extraction learned from the coarse-grained label set.
Revised: Our aim is to test whether few data allocation on the high tier devices can benefit from the knowledge and feature extraction learned from the coarse-grained label set.

Original: We adopt two different setups.
Revised: We adopt two different setups.

Original: For CIFAR-100, we train both tasks simultaneously, having essentially two distinct linear layers across tiers (1,2) and (3,4).
Revised: For CIFAR-100, we train both tasks simultaneously, having essentially two distinct linear layers across tiers (1,2) and (3,4).

Original: We call this setup multi-task.
Revised: We call this setup multi-task.

Original: On the other hand, for SpeechCommands, we train all clients on the same coarse domain and subsequently transfer learn a fine-grained 35 classes linear layer for the client of tiers (3,4).
Revised: On the other hand, for SpeechCommands, we train all clients on the same coarse domain and subsequently transfer learn a fine-grained 35 classes linear layer for the client of tiers (3,4).

Original: We fine-tune the linear layer keeping the rest of the network frozen for 25 epochs and then allow for finetuning of the whole network for another 100.
Revised: We fine-tune the linear layer keeping the rest of the network frozen for 25 epochs and then allow for finetuning of the whole network for another 100.

Original: We present the results in Tab. 12.
Revised: We present the results in Tab. 12.

Original: [Table]
Revised: [Table]

Original: In both cases FedorAS learns better models and transfer knowledge from the low-tiers to high-tiers and vice-versa through weight sharing of the supernet.
Revised: In both cases FedorAS learns better models and transfer knowledge from the low-tiers to high-tiers and vice-versa through weight sharing of the supernet.

Original: Indicatively, for CIFAR-100 we are able to achieve +14.

Revised: Indicatively, for CIFAR-100 we are able to achieve +14.

Original: pp (percentage points) of accuracy compared to training the same architectures end-to-end on eligible devices.
Revised: pp (percentage points) of accuracy compared to training the same architectures end-to-end on eligible devices.

Original: Similarly, we achieve +1.6 pp higher accuracy for SpeechCommands
Revised: Similarly, we achieve +1.6 pp higher accuracy for SpeechCommands

Original: [Table]
Revised: [Table]

Original: G Convergence and Impact of Supernet Sampling
Revised: G Convergence and Impact of Supernet Sampling

Original: The proposed OPA aggregation scheme converges faster than an alternative aggregation method that does FedAvg of the updated supernets.
Revised: The proposed OPA aggregation scheme converges faster than an alternative aggregation method that does FedAvg of the updated supernets.

Original: We show this in Fig. 8.
Revised: We show this in Fig. 8.

Original: When supernets from each client participating in the round get aggregated with OPA, the resulting supernet is consistently better in terms of validation accuracy than when aggregation is done with FedAvg.
Revised: When supernets from each client participating in the round get aggregated with OPA, the resulting supernet is consistently better in terms of validation accuracy than when aggregation is done with FedAvg.

Original: This difference becomes more evident as B Φ comm is reduced, i.e., as smaller portions of the supernet are sent to the clients.
Revised: This difference becomes more evident as B Φ comm is reduced, i.e., as smaller portions of the supernet are sent to the clients.

Original: In
Revised: In

Original: Fig.
Revised: Fig.

Original: 8 (d) we asses the feasibility of reducing the number of rounds and end the federated supernet training stage when validation accuracy reaches approximately 70%.
Revised: 8 (d) we asses the feasibility of reducing the number of rounds and end the federated supernet training stage when validation accuracy reaches approximately 70%.

Original: Those points correspond to 250 rounds and 150 rounds for the setting when 50% and 25% of the supernet is sent to the clients, respectively.
Revised: Those points correspond to 250 rounds and 150 rounds for the setting when 50% and 25% of the supernet is sent to the clients, respectively.

Original: The results that these settings yield show a significant loss (compared to their respective settings but over 500 rounds) and we therefore also run the same settings but when allowing for 50 additional runs.
Revised: The results that these settings yield show a significant loss (compared to their respective settings but over 500 rounds) and we therefore also run the same settings but when allowing for 50 additional runs.

Original: We observe a significant jump in per-tier model performance.
Revised: We observe a significant jump in per-tier model performance.

Original: We leave as future work investigating alternative metrics to more accurately (but without incurring into heavy computational costs) measure the quality of the supernet at any given training iteration and leverage this to better inform an early stopping mechanism to further reduce communication costs.
Revised: We leave as future work investigating alternative metrics to more accurately (but without incurring into heavy computational costs) measure the quality of the supernet at any given training iteration and leverage this to better inform an early stopping mechanism to further reduce communication costs.

Original: [Figure]
Revised: [Figure]

Original: H Fairness
Revised: H Fairness

Original: Up to this point, we have reported on the global test accuracy, which is a union over the test sets of the clients for all datasets but SpeechCommands, which defines a separate set of clients mutually exclusive to the ones participating in federated training.
Revised: Up to this point, we have reported on the global test accuracy, which is a union over the test sets of the clients for all datasets but SpeechCommands, which defines a separate set of clients mutually exclusive to the ones participating in federated training.

Original: In this section we want to measure how the ability of our system to aggregate knowledge across tiers during supernet training affects the fairness in participation.
Revised: In this section we want to measure how the ability of our system to aggregate knowledge across tiers during supernet training affects the fairness in participation.

Original: We quantify this through variance and worst-case statistical measures of client performance on their respective test set.
Revised: We quantify this through variance and worst-case statistical measures of client performance on their respective test set.

Original: These test sets are considered to be following the same distribution (LDA or pre-split) of their train and validation sets.
Revised: These test sets are considered to be following the same distribution (LDA or pre-split) of their train and validation sets.

Original: Results are shown on Tab. 13 for the different datasets offering per-client test data splits for all clients.
Revised: Results are shown on Tab. 13 for the different datasets offering per-client test data splits for all clients.

Original: It can be seen that FedorAS leads consistently to better performance compared to end-to-end FL trained networks of the same architectures, where only eligible clients can train models.
Revised: It can be seen that FedorAS leads consistently to better performance compared to end-to-end FL trained networks of the same architectures, where only eligible clients can train models.

Original: With respect to the standard deviation of per client performance, we witness FedorAS offering lower variance, except for the case of CIFAR-100.
Revised: With respect to the standard deviation of per client performance, we witness FedorAS offering lower variance, except for the case of CIFAR-100.

Original: We consider this to be a consequence of our significantly higher accuracy.
Revised: We consider this to be a consequence of our significantly higher accuracy.

Original: Similar behaviour, is also witnessed when we measure variance per tier of devices.
Revised: Similar behaviour, is also witnessed when we measure variance per tier of devices.

Original: Last, we also showcase the worst-case result of a clients performance in the last column of the table.
Revised: Last, we also showcase the worst-case result of a clients performance in the last column of the table.

Original: An in-depth view of how each client behaves is depicted in Fig.
Revised: An in-depth view of how each client behaves is depicted in Fig.

Original: 9 for CIFAR-10, where we show performance of Fedoras vs. end-to-end FL-trained models per client.
Revised: 9 for CIFAR-10, where we show performance of Fedoras vs. end-to-end FL-trained models per client.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: I Different allocation of clients to tiers
Revised: I Different allocation of clients to tiers

Original: In this experiment
Revised: In this experiment

Original: [Table]
Revised: [Table]

Original: contrary to what was described in Appendix D.
Revised: contrary to what was described in Appendix D.

Original: [Table]
Revised: [Table]

Original: we adopt the device capabilities trace from Oort [43] for our device to cluster allocation and performed training on CIFAR-10 and
Revised: we adopt the device capabilities trace from Oort [43] for our device to cluster allocation and performed training on CIFAR-10 and

Original: CIFAR-100.
Revised: CIFAR-100.

Original: Results are depicted on Tab. 14.
Revised: Results are depicted on Tab. 14.

Original: It can be witnessed that FedorAS is able to operate even under harsher heterogeneity conditions and still output competitive models in the federated setting.
Revised: It can be witnessed that FedorAS is able to operate even under harsher heterogeneity conditions and still output competitive models in the federated setting.

Original: [Table]
Revised: [Table]

Original: J Evaluation of the Search Phase J.1 Sensitivity to size of validation set
Revised: J Evaluation of the Search Phase J.1 Sensitivity to size of validation set

Original: In order to more meaningfully examine the impact of the validation set size
Revised: In order to more meaningfully examine the impact of the validation set size

Original: [Table]
Revised: [Table]

Original: we run experiments on all datasets with 20% and 50% of the size of the initial global validation set.
Revised: we run experiments on all datasets with 20% and 50% of the size of the initial global validation set.

Original: This means that during Stage-II of FedorAS , architectures sampled from the supernet are scored using a fraction of the global validation set.
Revised: This means that during Stage-II of FedorAS , architectures sampled from the supernet are scored using a fraction of the global validation set.

Original: These validation subsets are extracted uniformly.
Revised: These validation subsets are extracted uniformly.

Original: After obtaining the best performing models for each tier, these are fine-tuned in a federated fashion (Stage-III in FedorAS ) for 100 rounds just like it was done for Tab. 5.
Revised: After obtaining the best performing models for each tier, these are fine-tuned in a federated fashion (Stage-III in FedorAS ) for 100 rounds just like it was done for Tab. 5.

Original: We maintain the same hyperparameters as those used to generate Tab. 5.
Revised: We maintain the same hyperparameters as those used to generate Tab. 5.

Original: Results are depicted in Tab. 15 for all datasets.
Revised: Results are depicted in Tab. 15 for all datasets.

Original: [Table]
Revised: [Table]

Original: J.2 Federated Search
Revised: J.2 Federated Search

Original: In this Section, we provide additional details and results concerning our federated variant of NSGA-II, discussed in Sec.
Revised: In this Section, we provide additional details and results concerning our federated variant of NSGA-II, discussed in Sec.

Original: 4.5 of the main paper.
Revised: 4.5 of the main paper.

Original: Below, we provide some context about how the algorithm works, how we setup the federated experiments and commentary of the results on CIFAR10 { 1000 , 1 , 0 . 1 } and CIFAR-100.
Revised: Below, we provide some context about how the algorithm works, how we setup the federated experiments and commentary of the results on CIFAR10 { 1000 , 1 , 0 . 1 } and CIFAR-100.

Original: Algorithm details.
Revised: Algorithm details.

Original: Federation is achieved by delegating evaluation of models to individual clients.
Revised: Federation is achieved by delegating evaluation of models to individual clients.

Original: This means that evaluation is done stochastically on clients’ local datasets.
Revised: This means that evaluation is done stochastically on clients’ local datasets.

Original: This setting is similar to Federated Evaluation in [ ? ].
Revised: This setting is similar to Federated Evaluation in [62].

Original: In order to avoid sending a large number of models to clients, thus saving communication cost, we leverage the fact that NSGA-II operates in “batches" of models – in each iteration, a number of models (i.e. sample size ) from the population is selected to produce new models that replace the ones not selected.
Revised: In order to avoid sending a large number of models to clients, thus saving communication cost, we leverage the fact that NSGA-II operates in “batches" of models – in each iteration, a number of models (i.e. sample size ) from the population is selected to produce new models that replace the ones not selected.

Original: Since models from a single “batch” are all selected at the same time, we do not need to evaluate them sequentially.
Revised: Since models from a single “batch” are all selected at the same time, we do not need to evaluate them sequentially.

Original: Instead, considering that weights between architectures are shared, we can construct a minimal supernet that encapsulates all selected models and send it to relevant clients.
Revised: Instead, considering that weights between architectures are shared, we can construct a minimal supernet that encapsulates all selected models and send it to relevant clients.

Original: This way, we can achieve parallel evaluation of all selected models and optimal 4 communication cost for a given “batch”.
Revised: This way, we can achieve parallel evaluation of all selected models and optimal 4 communication cost for a given “batch”.

Original: Experimental setting.
Revised: Experimental setting.

Original: The experimental setting is following exactly what was used in other experiments.
Revised: The experimental setting is following exactly what was used in other experiments.

Original: Specifically, population and sample sizes were the same as the ones used in centralised
Revised: Specifically, population and sample sizes were the same as the ones used in centralised

Original: NSGA-II (128 and 64, respectively), and federated evaluation (FE) rounds were analogous to rounds during federated training of the supernet from which models to evaluate were extracted.
Revised: NSGA-II (128 and 64, respectively), and federated evaluation (FE) rounds were analogous to rounds during federated training of the supernet from which models to evaluate were extracted.

Original: Concretely, number of clients, allocation of clients to tiers, allocation of data to clients, number of available clients per round, and client selection mechanism were all exactly the same for FE as the ones used during the relevant Stage 1 of FedorAS .
Revised: Concretely, number of clients, allocation of clients to tiers, allocation of data to clients, number of available clients per round, and client selection mechanism were all exactly the same for FE as the ones used during the relevant Stage 1 of FedorAS .

Original: However, to present more meaningful results, we assumed that as we fluctuate the number of clients used to evaluate models (number of evaluation rounds) additional clients are always unique.
Revised: However, to present more meaningful results, we assumed that as we fluctuate the number of clients used to evaluate models (number of evaluation rounds) additional clients are always unique.

Original: In other words, reshuffling and "forgetting" of clients is happening only between federated NSGA-II iterations.
Revised: In other words, reshuffling and "forgetting" of clients is happening only between federated NSGA-II iterations.

Original: Consequently, it is possible that: i) if all clients are used to evaluate a model (e.g. 10 evaluation rounds for CIFAR-10), FE is equivalent to centralised evaluation; ii) with a relatively small number of FE rounds, it is possible that one set of models produced by NSGA-II is evaluated on a completely disjoint set of data to another “batch”; iii) similarly, we do not enforce in any way that all clients have to be used (unless we use all clients in a single iteration of NSGA-II).
Revised: Consequently, it is possible that: i) if all clients are used to evaluate a model (e.g. 10 evaluation rounds for CIFAR-10), FE is equivalent to centralised evaluation; ii) with a relatively small number of FE rounds, it is possible that one set of models produced by NSGA-II is evaluated on a completely disjoint set of data to another “batch”; iii) similarly, we do not enforce in any way that all clients have to be used (unless we use all clients in a single iteration of NSGA-II).

Original: Finally, keeping consistent with the rest of the paper, we performed evaluation in a tier-aware manner, meaning that models belonging to higher tiers could only access a subset of all validation data based on client eligibility.
Revised: Finally, keeping consistent with the rest of the paper, we performed evaluation in a tier-aware manner, meaning that models belonging to higher tiers could only access a subset of all validation data based on client eligibility.

Original: [Figure]
Revised: [Figure]

Original: This is the reason why relevant curves finish at a different maximum number of FE rounds, that is a smaller number of FE rounds (i e. Federated NSGA-II iterations) is needed to exhaust the set of eligible clients
Revised: This is the reason why relevant curves finish at a different maximum number of FE rounds, that is a smaller number of FE rounds (i e. Federated NSGA-II iterations) is needed to exhaust the set of eligible clients

Original: Results.
Revised: Results.

Original: Fig.
Revised: Fig.

Original: 10 complements Fig.
Revised: 10 complements Fig.

Original: 4 and presents results across three different experimental settings, with varying level of non-IID-ness ( α ∈ { 1 , 0 . 1 } ) and number of clients ( { 100 , 500 } ) .
Revised: 4 and presents results across three different experimental settings, with varying level of non-IID-ness ( α ∈ { 1 , 0 . 1 } ) and number of clients ( { 100 , 500 } ) .

Original: As conjectured in the main paper, we can observe that the cost of achieving the same fidelity of FE increases with both the level of non-IID-ness and the number of clients.
Revised: As conjectured in the main paper, we can observe that the cost of achieving the same fidelity of FE increases with both the level of non-IID-ness and the number of clients.

Original: Specifically, Kendall- τ of 0.8 is achieved at approximately three FE rounds for { α = 1 , clients = 100 } , and increases to four and six for { α = 0 . 1 , clients = 100 } and { α = 0 .
Revised: Specifically, Kendall- τ of 0.8 is achieved at approximately three FE rounds for { α = 1 , clients = 100 } , and increases to four and six for { α = 0 . 1 , clients = 100 } and { α = 0 .

Original: 1 , clients = 500 } , respectively.
Revised: 1 , clients = 500 } , respectively.

Original: At the same time, we can see that regardless of the setting NSGA-II tends to produce smaller supernets as the search progresses, suggesting that searching for longer does not have to incur proportional increase in the communication cost.
Revised: At the same time, we can see that regardless of the setting NSGA-II tends to produce smaller supernets as the search progresses, suggesting that searching for longer does not have to incur proportional increase in the communication cost.