Original: S UPER F ED : W EIGHT S HARED F EDERATED L EARNING
Revised: S UPER F ED : W EIGHT S HARED F EDERATED L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Federated Learning (FL) is a well-established technique for privacy preserving distributed training.
Revised: Federated Learning (FL) is a well-established technique for privacy preserving distributed training.

Original: Much attention has been given to various aspects of FL training.
Revised: Much attention has been given to various aspects of FL training.

Original: A growing number of applications that consume FL-trained models, however, increasingly operate under dynamically and unpredictably variable conditions, rendering a single model insufficient.
Revised: A growing number of applications that consume FL-trained models, however, increasingly operate under dynamically and unpredictably variable conditions, rendering a single model insufficient.

Original: We argue for training a global “family of models” cost efficiently in a federated fashion.
Revised: We argue for training a global “family of models” cost efficiently in a federated fashion.

Original: Training them independently for different tradeoff points incurs ≈ O ( k ) cost for any k architectures of interest, however.
Revised: Training them independently for different tradeoff points incurs ≈ O ( k ) cost for any k architectures of interest, however.

Original: Straightforward applications of FL techniques to recent weight-shared training approaches is either infeasible or prohibitively expensive.
Revised: Straightforward applications of FL techniques to recent weight-shared training approaches is either infeasible or prohibitively expensive.

Original: We propose SuperFed— an architectural framework that incurs O (1) cost to co-train a large family of models in a federated fashion by leveraging weight-shared learning.
Revised: We propose SuperFed— an architectural framework that incurs O (1) cost to co-train a large family of models in a federated fashion by leveraging weight-shared learning.

Original: We achieve an order of magnitude cost savings on both communication and computation by proposing two novel training mechanisms: (a) distribution of weight-shared models to federated clients, (b) central aggregation of arbitrarily overlapping weight-shared model parameters.
Revised: We achieve an order of magnitude cost savings on both communication and computation by proposing two novel training mechanisms: (a) distribution of weight-shared models to federated clients, (b) central aggregation of arbitrarily overlapping weight-shared model parameters.

Original: The combination of these mechanisms is shown to reach an order of magnitude ( 9 . 43 x) reduction in computation and communication cost for training a 5 ∗ 10 8 -sized family of models, compared to independently training as few as k = 9 DNNs without any accuracy loss.
Revised: The combination of these mechanisms is shown to reach an order of magnitude ( 9 . 43 x) reduction in computation and communication cost for training a 5 ∗ 10 8 -sized family of models, compared to independently training as few as k = 9 DNNs without any accuracy loss.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: With the increase in the computational power of smartphones, the use of on-device inference in mobile applications is on the rise, ranging from image recognition (google vision; azure vision) , virtual assistant (Alexa) , voice recognition (google ASR) to recommendation systems (Bin et al., 2019).
Revised: With the increase in the computational power of smartphones, the use of on-device inference in mobile applications is on the rise, ranging from image recognition (google vision; azure vision) , virtual assistant (Alexa) , voice recognition (google ASR) to recommendation systems (Bin et al., 2019).

Original: Indeed, on-device inference is pervasive, especially with recent advances in software (Chen et al., 2018; torch mobile), accelerators (samsung exynos; apple neural engine), and neural architecture optimizations (Howard et al., 2019; Sun et al., 2020; Wu et al., 2019a).
Revised: Indeed, on-device inference is pervasive, especially with recent advances in software (Chen et al., 2018; torch mobile), accelerators (samsung exynos; apple neural engine), and neural architecture optimizations (Howard et al., 2019; Sun et al., 2020; Wu et al., 2019a).

Original: The surge in its use cases (Cai et al., 2017; Han et al., 2019; Kang et al., 2017; Lane et al., 2016; Reddi et al., 2021; Wu et al., 2019b) has led to a growing interest in providing support not only for on-device inference, but also for on-device training of these models (Dhar et al., 2021).
Revised: The surge in its use cases (Cai et al., 2017; Han et al., 2019; Kang et al., 2017; Lane et al., 2016; Reddi et al., 2021; Wu et al., 2019b) has led to a growing interest in providing support not only for on-device inference, but also for on-device training of these models (Dhar et al., 2021).

Original: Federated Learning (FL) is an emerging distributed training technique that allows smartphones with different data sources to collaboratively train an ML model (McMahan et al., 2017; Chen & Chao, 2020; Wang et al., 2020;
Revised: Federated Learning (FL) is an emerging distributed training technique that allows smartphones with different data sources to collaboratively train an ML model (McMahan et al., 2017; Chen & Chao, 2020; Wang et al., 2020;

Original: Karimireddy et al., 2021; Koneˇcn`y et al., 2016).
Revised: Karimireddy et al., 2021; Koneˇcn`y et al., 2016).

Original: FL enjoys three key properties, it — a) has smaller communication cost, b) is massively parallel, and c) involves no data-sharing.
Revised: FL enjoys three key properties, it — a) has smaller communication cost, b) is massively parallel, and c) involves no data-sharing.

Original: As a result, numerous applications such as GBoard (Hard et al., 2018), Apple’s Siri (sir, 2019), pharmaceutical discovery (CORDIS., 2019), medical imaging
Revised: As a result, numerous applications such as GBoard (Hard et al., 2018), Apple’s Siri (sir, 2019), pharmaceutical discovery (CORDIS., 2019), medical imaging

Original: (Silva et al., 2019), health record mining (Huang & Liu, 2019), and recommendation systems (Ammad-ud-din et al., 2019) are readily adopting federated learning.
Revised: (Silva et al., 2019), health record mining (Huang & Liu, 2019), and recommendation systems (Ammad-ud-din et al., 2019) are readily adopting federated learning.

Original: However, adoption of FL in smartphone applications is non-trivial.
Revised: However, adoption of FL in smartphone applications is non-trivial.

Original: As a result, recent works pay attention to the emerging challenges that occur in training, such as data heterogeneity (Karimireddy et al., 2021;
Revised: As a result, recent works pay attention to the emerging challenges that occur in training, such as data heterogeneity (Karimireddy et al., 2021;

Original: Li et al., 2020; Acar et al., 2021), heterogeneous resources (Alistarh et al., 2017;
Revised: Li et al., 2020; Acar et al., 2021), heterogeneous resources (Alistarh et al., 2017;

Original: Ivkin et al., 2019; Li et al., 2020; Koneˇcn`y et al., 2016), and privacy (Truex et al., 2019; Mo et al., 2021;
Revised: Ivkin et al., 2019; Li et al., 2020; Koneˇcn`y et al., 2016), and privacy (Truex et al., 2019; Mo et al., 2021;

Original: Gong et al., 2021).
Revised: Gong et al., 2021).

Original: These helped FL adoption, particularly in challenging training conditions.
Revised: These helped FL adoption, particularly in challenging training conditions.

Original: However, the success of FL adoption depends not only on tackling challenges that occur in training but also post-training (deployment).
Revised: However, the success of FL adoption depends not only on tackling challenges that occur in training but also post-training (deployment).

Original: Indeed, deploying ML models for on-device inference is exceedingly challenging (Wu et al., 2019b; Reddi et al., 2021).
Revised: Indeed, deploying ML models for on-device inference is exceedingly challenging (Wu et al., 2019b; Reddi et al., 2021).

Original: Yet, most of the existing training techniques in FL do not take these deployment challenges into consideration.
Revised: Yet, most of the existing training techniques in FL do not take these deployment challenges into consideration.

Original: In this paper, we focus on developing FL training algorithms specifically designed to address deployment challenges related to post-training inference.
Revised: In this paper, we focus on developing FL training algorithms specifically designed to address deployment challenges related to post-training inference.

Original: It is well-established that any single model statically chosen for on-device inference is sub-optimal.
Revised: It is well-established that any single model statically chosen for on-device inference is sub-optimal.

Original: This is because the deployment conditions may continuously change on a multi-task system like smartphones (Xu et al., 2019) due to dynamic resource availability (Fang et al., 2018).
Revised: This is because the deployment conditions may continuously change on a multi-task system like smartphones (Xu et al., 2019) due to dynamic resource availability (Fang et al., 2018).

Original: For instance, the computational budget may vary due to excessive consumption by background apps; the energy budget may vary if the smartphone is in low power or power-saver mode (Yu et al., 2019).
Revised: For instance, the computational budget may vary due to excessive consumption by background apps; the energy budget may vary if the smartphone is in low power or power-saver mode (Yu et al., 2019).

Original: Furthermore, an increasing number of applications require flexibility with respect to resource-accuracy trade-offs in order to efficiently utilize the dynamic resources in deployment (Fang et al., 2018).
Revised: Furthermore, an increasing number of applications require flexibility with respect to resource-accuracy trade-offs in order to efficiently utilize the dynamic resources in deployment (Fang et al., 2018).

Original: In all of these deployment scenarios, a single model neither satisfies variable constraints nor offers the flexibility to make trade-offs.
Revised: In all of these deployment scenarios, a single model neither satisfies variable constraints nor offers the flexibility to make trade-offs.

Original: In contrast to existing FL approaches that produce a single model, we need to produce multiple model variants (varying in size/latency) for efficient on-device inference.
Revised: In contrast to existing FL approaches that produce a single model, we need to produce multiple model variants (varying in size/latency) for efficient on-device inference.

Original: However, training these model variants independently is computationally prohibitive (Cai et al., 2020).
Revised: However, training these model variants independently is computationally prohibitive (Cai et al., 2020).

Original: This is particularly true for
Revised: This is particularly true for

Original: FL, where training these variants independently will cumulatively inflate the communication cost as well as the computation cost.
Revised: FL, where training these variants independently will cumulatively inflate the communication cost as well as the computation cost.

Original: Thus, it is imperative to develop techniques for training multiple models in a federated fashion cost efficiently without any accuracy loss—achieving asymptotic cost improvements relative to independently training them.
Revised: Thus, it is imperative to develop techniques for training multiple models in a federated fashion cost efficiently without any accuracy loss—achieving asymptotic cost improvements relative to independently training them.

Original: [Figure]
Revised: [Figure]

Original: u r
Revised: u r

Original: [Figure]
Revised: [Figure]

Original: To achieve this goal, we propose SuperFed- a novel federated framework that targets the problem of efficient ondevice inference on smartphones with better training algorithms.
Revised: To achieve this goal, we propose SuperFed- a novel federated framework that targets the problem of efficient ondevice inference on smartphones with better training algorithms.

Original: SuperFed co-trains a family of model variants in a federated fashion simultaneously by leveraging weightsharing (Cai et al., 2020; Yu et al., 2020).
Revised: SuperFed co-trains a family of model variants in a federated fashion simultaneously by leveraging weightsharing (Cai et al., 2020; Yu et al., 2020).

Original: After federated training, the clients perform local neural architecture search to find the appropriate model variants for their deployment scenarios.
Revised: After federated training, the clients perform local neural architecture search to find the appropriate model variants for their deployment scenarios.

Original: In weight-sharing, all model variants are subnetworks of a supernetwork (Cai et al., 2020) and share their parameters partially.
Revised: In weight-sharing, all model variants are subnetworks of a supernetwork (Cai et al., 2020) and share their parameters partially.

Original: The largest subnetwork’s (or supernetwork’s) parameters contain other subnetworks’ parameters within it as proper subgraphs.
Revised: The largest subnetwork’s (or supernetwork’s) parameters contain other subnetworks’ parameters within it as proper subgraphs.

Original: There are two key benefits that weight-sharing brings in FL, it a) significantly reduces the communication and computation cost for training k model variants, and b) requires no re-training after the federated training of the supernetwork is complete.
Revised: There are two key benefits that weight-sharing brings in FL, it a) significantly reduces the communication and computation cost for training k model variants, and b) requires no re-training after the federated training of the supernetwork is complete.

Original: Hence, SuperFed decouples training from neural architecture search which allows local clients to dynamically select subnetworks of their choice from the globally-trained supernetwork without any re-training.
Revised: Hence, SuperFed decouples training from neural architecture search which allows local clients to dynamically select subnetworks of their choice from the globally-trained supernetwork without any re-training.

Original: However, applying existing weight-shared training techniques to federated learning is challenging.
Revised: However, applying existing weight-shared training techniques to federated learning is challenging.

Original: First, weightshared training techniques like Progressive Shrinking (PS) (Cai et al., 2020) work on centralized i.i.d data, whereas the data is decentralized and typically non-i.i.d in FL.
Revised: First, weightshared training techniques like Progressive Shrinking (PS) (Cai et al., 2020) work on centralized i.i.d data, whereas the data is decentralized and typically non-i.i.d in FL.

Original: Second, PS uses a pre-trained largest subnetwork during the weight-shared training.
Revised: Second, PS uses a pre-trained largest subnetwork during the weight-shared training.

Original: This requirement becomes impractical in the context of FL as it - a) is hard to obtain a globally pre-trained FL model, or b) may significantly increase the overall communication cost to train it first.
Revised: This requirement becomes impractical in the context of FL as it - a) is hard to obtain a globally pre-trained FL model, or b) may significantly increase the overall communication cost to train it first.

Original: Third, weight-sharing training techniques need to minimize interference (Cai et al., 2020; Xu et al., 2022).
Revised: Third, weight-sharing training techniques need to minimize interference (Cai et al., 2020; Xu et al., 2022).

Original: Interference occurs when smaller subnetworks interfere with the training of larger subnetworks (Fig 2a in Yu et al.
Revised: Interference occurs when smaller subnetworks interfere with the training of larger subnetworks (Fig 2a in Yu et al.

Original: To mitigate interference,
Revised: To mitigate interference,

Original: PS adopts multi-phase training approach that prioritizes the training of larger subnetworks before training smaller subnetworks.
Revised: PS adopts multi-phase training approach that prioritizes the training of larger subnetworks before training smaller subnetworks.

Original: Such multi-phase training may lead to significant communication cost in federated learning.
Revised: Such multi-phase training may lead to significant communication cost in federated learning.

Original: Instead, we argue that the weight-shared training technique in FL must be be one-shot (single phase) to mitigate interference.
Revised: Instead, we argue that the weight-shared training technique in FL must be be one-shot (single phase) to mitigate interference.

Original: As a part of SuperFed framework, we propose MaxNet — a weight-shared training technique for FL.
Revised: As a part of SuperFed framework, we propose MaxNet — a weight-shared training technique for FL.

Original: Figure 1 provides a high level overview of our proposed approach.
Revised: Figure 1 provides a high level overview of our proposed approach.

Original: MaxNet hosts the supernetwork in the server and assumes no prior pre-trained model before the federated training.
Revised: MaxNet hosts the supernetwork in the server and assumes no prior pre-trained model before the federated training.

Original: MaxNet decides which individual subnetworks are distributed for training on participating clients and when (subnetwork distribution).
Revised: MaxNet decides which individual subnetworks are distributed for training on participating clients and when (subnetwork distribution).

Original: MaxNet ’s subnetwork distribution optimizes both lower bound (smallest subnet) and upper bound (largest subnet) to increase the accuracy of every subnet in between.
Revised: MaxNet ’s subnetwork distribution optimizes both lower bound (smallest subnet) and upper bound (largest subnet) to increase the accuracy of every subnet in between.

Original: Since the subnetworks partially share their parameters, MaxNet also introduces a novel shared model weight aggregation mechanism that mitigates interference in weight-shared FL training.
Revised: Since the subnetworks partially share their parameters, MaxNet also introduces a novel shared model weight aggregation mechanism that mitigates interference in weight-shared FL training.

Original: To summarize, our contributions are as follows:
Revised: To summarize, our contributions are as follows:

Original: • SuperFed: A weight shared training framework for FL that trains a family of model variants (DNN models), cost-efficiently in a federated fashion.
Revised: • SuperFed: A weight shared training framework for FL that trains a family of model variants (DNN models), cost-efficiently in a federated fashion.

Original: • MaxNet’s subnetwork distribution: a heuristic that trains the upper and lower bounds in the model family by optimizing both bounds and load balancing their distribution over time.
Revised: • MaxNet’s subnetwork distribution: a heuristic that trains the upper and lower bounds in the model family by optimizing both bounds and load balancing their distribution over time.

Original: • MaxNet’s Aggregation: A mechanism for subnetwork weight aggregation with variable overlaps mitigating intra-network interference.
Revised: • MaxNet’s Aggregation: A mechanism for subnetwork weight aggregation with variable overlaps mitigating intra-network interference.

Original: We perform rigorous evaluation of SuperFed ’s weight-shared training technique MaxNet against non-weight shared baselines, where each model variant is trained independently in a federated fashion over CIFAR10/100 and CINIC-10 (downsampled ImageNet images) datasets.
Revised: We perform rigorous evaluation of SuperFed ’s weight-shared training technique MaxNet against non-weight shared baselines, where each model variant is trained independently in a federated fashion over CIFAR10/100 and CINIC-10 (downsampled ImageNet images) datasets.

Original: MaxNet trains a family of ≈ 5 ∗ 10 8 subnets showing 9.43x lower computational and 10.94x lower communication cost compared to training as few as 9 subnetworks separately with FedAvg.
Revised: MaxNet trains a family of ≈ 5 ∗ 10 8 subnets showing 9.43x lower computational and 10.94x lower communication cost compared to training as few as 9 subnetworks separately with FedAvg.

Original: This order of magnitude reduction in training cost is achieved with no accuracy loss compared to independently trained subnetworks (without weight-sharing) with FedAvg.
Revised: This order of magnitude reduction in training cost is achieved with no accuracy loss compared to independently trained subnetworks (without weight-sharing) with FedAvg.

Original: 1. R ELATED W ORK
Revised: 1. R ELATED W ORK

Original: In this section, we describe FL approaches that are closely related to SuperFed.
Revised: In this section, we describe FL approaches that are closely related to SuperFed.

Original: We provide a background of weight shared training in non-federated environments in App. A.
Revised: We provide a background of weight shared training in non-federated environments in App. A.

Original: FL-training challenges.
Revised: FL-training challenges.

Original: Our proposed work targets a challenge of efficient deployment of FL-models for on-device inference which is fundamentally different from challenges that occur during training.
Revised: Our proposed work targets a challenge of efficient deployment of FL-models for on-device inference which is fundamentally different from challenges that occur during training.

Original: Specifically, SuperFed aims to reduce the cost of training k global model variants in FL whereas existing works train a single model in FL-training.
Revised: Specifically, SuperFed aims to reduce the cost of training k global model variants in FL whereas existing works train a single model in FL-training.

Original: FedAvg (McMahan et al., 2017) and local SGD (Lin et al., 2020b) train a single model by averaging model updates received from the participating clients.
Revised: FedAvg (McMahan et al., 2017) and local SGD (Lin et al., 2020b) train a single model by averaging model updates received from the participating clients.

Original: We show that FedAvg is a special case of SuperFed in App.
Revised: We show that FedAvg is a special case of SuperFed in App.

Original: C. Recent modifications to
Revised: C. Recent modifications to

Original: FedAvg like FedDyn (Acar et al., 2021), FedProx (Li et al., 2020), Scaffold (Karimireddy et al., 2021) modify client local training to minimize communication cost for non-i.i.d client datasets.
Revised: FedAvg like FedDyn (Acar et al., 2021), FedProx (Li et al., 2020), Scaffold (Karimireddy et al., 2021) modify client local training to minimize communication cost for non-i.i.d client datasets.

Original: These techniques are complimentary to the proposed work and incorporating them in SuperFed framework is left as future work.
Revised: These techniques are complimentary to the proposed work and incorporating them in SuperFed framework is left as future work.

Original: Other techniques like Diao et al.
Revised: Other techniques like Diao et al.

Original: Lin et al.
Revised: Lin et al.

Original: (2020a) enable FL clients to train neural-nets (NNs) that differ in architecture enabling system heterogeneity in FL.
Revised: (2020a) enable FL clients to train neural-nets (NNs) that differ in architecture enabling system heterogeneity in FL.

Original: SuperFed also allows clients to train architecturally different NNs during training.
Revised: SuperFed also allows clients to train architecturally different NNs during training.

Original: However, the goal of SuperFed is not to enable system-heterogeneity but satisfy varying deployment scenarios of clients after training.
Revised: However, the goal of SuperFed is not to enable system-heterogeneity but satisfy varying deployment scenarios of clients after training.

Original: Hence, SuperFed provides k global models to clients for their varying deployment constraints, whereas, Diao et al.
Revised: Hence, SuperFed provides k global models to clients during on-device inference, whereas, Diao et al.

Original:  Lin et al.
Revised: (2020); Lin et al.

Original: (2020a) provide a single global model to every client.
Revised: (2020a) provide a single global model to every client during inference (post-training).

Original: Developing system-heterogeneity aware weight-shared FL training is future work.
Revised: Developing system-heterogeneity aware weight-shared FL training is future work.

Original: AutoML in FL. AutoML for federated learning is an emerging field.
Revised: AutoML in FL. AutoML for federated learning is an emerging field.

Original: SuperFed automates the neural architecture selection for varied client deployment scenarios.
Revised: SuperFed automates the neural architecture selection for varied client deployment scenarios.

Original: A known AutoML technique FedNAS (He et al., 2020) automatically searches for NN architecture that maximizes global accuracy in FL.
Revised: A known AutoML technique FedNAS (He et al., 2020) automatically searches for NN architecture that maximizes global accuracy in FL.

Original: We argue that the goal of SuperFed fundamentally differs from that of FedNAS.
Revised: We argue that the goal of SuperFed fundamentally differs from that of FedNAS.

Original: FedNAS doesn’t target varying deployment scenarios of FL clients whereas SuperFed does.
Revised: FedNAS doesn’t target varying deployment scenarios of FL clients whereas SuperFed does.

Original: Hence, SuperFed trains k global model variants while FedNAS trains a single global model.
Revised: Hence, SuperFed trains k global model variants while FedNAS trains a single global model.

Original: SuperFed decouples training from searching of NN architectures whereas FedNAS doesn’t.
Revised: SuperFed decouples training from searching of NN architectures whereas FedNAS doesn’t.

Original: This decoupling is important to allow clients to run local NAS after training in SuperFed.
Revised: This decoupling is important to allow clients to run local NAS after training in SuperFed.

Original: Another line of work FedEx (Khodak et al., 2021) automates hyper-parameter search in FL.
Revised: Another line of work FedEx (Khodak et al., 2021) automates hyper-parameter search in FL.

Original: Similar to SuperFed, it uses weight-sharing to reduce the communication cost.
Revised: Similar to SuperFed, it uses weight-sharing to reduce the communication cost.

Original: However, FedEx uses weight-sharing in a different context than that of
Revised: However, FedEx uses weight-sharing in a different context than that of

Original: SuperFed.
Revised: SuperFed.

Original: Specifically, FedEx shares parameter of an NN across different hyper-parameters like learning rate schedules, momentum and weight-decay.
Revised: Specifically, FedEx shares parameter of an NN across different hyper-parameters like learning rate schedules, momentum and weight-decay.

Original: Whereas, SuperFed shares parameters of an
Revised: Whereas, SuperFed shares parameters of an

Original: NN across different NNs that differ in shape and sizes (similar to Cai et al.
Revised: NN across different NNs that differ in shape and sizes (similar to Cai et al.

Original: Moreover, FedEx produces a single global model at the end of hyper-parameter optimization in FL.
Revised: Moreover, FedEx produces a single global model at the end of hyper-parameter optimization in FL.

Original: Whereas, SuperFed produces k global models in FL.
Revised: Whereas, SuperFed produces k global models in FL.

Original: Multi-Model FL.
Revised: Multi-Model FL.

Original: Another line of FL-research trains multiple models simultaneously but on a fairly different FL scenario than SuperFed.
Revised: Another line of FL-research trains multiple models simultaneously but on a fairly different FL scenario than SuperFed.

Original: Multi-Model FL (Bhuyan & Moharir, 2022; da Silva et al., 2022) trains k global models (simultaneously) that make predictions on k different tasks (for e.g. one model may predict birds’ species while another predicts dogs’ species).
Revised: Multi-Model FL (Bhuyan & Moharir, 2022; da Silva et al., 2022) trains k global models (simultaneously) that make predictions on k different tasks (for e.g. one model may predict birds’ species while another predicts dogs’ species).

Original: Whereas, SuperFed trains k
Revised: Whereas, SuperFed trains k

Original: [Table]
Revised: [Table]

Original: such
Revised: such

Original: [Table]
Revised: [Table]

Original: different global models on a single task (every model in SuperFed makes prediction for the same classes) for varying client deployment scenarios.
Revised: different global models on a single task (every model in SuperFed makes prediction for the same classes) for varying client deployment scenarios.

Original: Multi-Model FL assumes availability of k different datasets on a client device whereas SuperFed trains every model on a single (and same) dataset per client.
Revised: Multi-Model FL assumes availability of k different datasets on a client device whereas SuperFed trains every model on a single (and same) dataset per client.

Original:  S UPER F ED : W EIGHT S HARED FL F RAMEWORK
Revised:  S UPER F ED : W EIGHT S HARED FL F RAMEWORK

Original: We start by describing our weight shared FL framework in this section.
Revised: We start by describing our weight shared FL framework in this section.

Original: It provides a pluggable algorithmic and implementation base for subnetwork distribution and aggregation heuristics between participating clients—a novel challenge introduced by weight shared FL.
Revised: It provides a pluggable algorithmic and implementation base for subnetwork distribution and aggregation heuristics between participating clients—a novel challenge introduced by weight shared FL.

Original: We also propose one such
Revised: We also propose one such

Original: SuperFed’s heuristic MaxNet which instantiates a specific approach to subnetwork distribution and aggregation.
Revised: SuperFed’s heuristic MaxNet which instantiates a specific approach to subnetwork distribution and aggregation.

Original: Overall, the goal of weight-shared FL training is to train all subnetworks contained inside supernetwork on all the data partitions.
Revised: Overall, the goal of weight-shared FL training is to train all subnetworks contained inside supernetwork on all the data partitions.

Original: The problem formulation of weight-shared FL is discussed in detail in App.
Revised: The problem formulation of weight-shared FL is discussed in detail in App.

Original: B. Tab. 1 lists some notations used to describe SuperFed.
Revised: B. Tab. 1 lists some notations used to describe SuperFed.

Original: 2. F EDERATED S ETUP
Revised: 2. F EDERATED S ETUP

Original: Fundamentally, to train a supernetwork in a federated fashion, a server must decide how to distribute its sub-architectures between participating clients.
Revised: Fundamentally, to train a supernetwork in a federated fashion, a server must decide how to distribute its sub-architectures between participating clients.

Original: This happens at the beginning of each FL round (line 4 of algorithm 1).
Revised: This happens at the beginning of each FL round (line 4 of algorithm 1).

Original: The server extracts the subnetworks from the hosted supernet and sends them to the clients (algorithm 1 line 6).
Revised: The server extracts the subnetworks from the hosted supernet and sends them to the clients (algorithm 1 line 6).

Original: Each participating client trains the received subnetwork for a fixed number of local epochs and sends the updated subnetwork parameters back to the server (algorithmline 7).
Revised: Each participating client trains the received subnetwork for a fixed number of local epochs and sends the updated subnetwork parameters back to the server (algorithmline 7).

Original: Upon receipt of individual updated subnetwork parameters, the server is then responsible for merging them and assimilating the updates into the shared supernetwork.
Revised: Upon receipt of individual updated subnetwork parameters, the server is then responsible for merging them and assimilating the updates into the shared supernetwork.

Original: Critically, it must take into consideration any overlaps or contributions to individual parameters from more than one client (algorithm 1 line 11-15).
Revised: Critically, it must take into consideration any overlaps or contributions to individual parameters from more than one client (algorithm 1 line 11-15).

Original: Note that FedAvg (McMahan et al., 2017) is a special case of algorithm(App.
Revised: Note that FedAvg (McMahan et al., 2017) is a special case of algorithm(App.

Original: C) where this overlap is constant and equals to the number of clients K .
Revised: C) where this overlap is constant and equals to the number of clients K .

Original: However, the overlap cardinality may vary ( ∈ [1 , K ] ) in weight shared FL.
Revised: However, the overlap cardinality may vary ( ∈ [1 , K ] ) in weight shared FL.

Original: This is handled on lines 10-16 of algorithm 1.
Revised: This is handled on lines 10-16 of algorithm 1.

Original: This shared-parameter averaging is one of the key contributions and further investigation opportunity.
Revised: This shared-parameter averaging is one of the key contributions and further investigation opportunity.

Original: Now, we describe these two principal framework components in more detail.
Revised: Now, we describe these two principal framework components in more detail.

Original: 2. S UBNETWORK D ISTRIBUTION
Revised: 2. S UBNETWORK D ISTRIBUTION

Original: We start by taxonomizing different subnetwork distribution heuristics on the server side.
Revised: We start by taxonomizing different subnetwork distribution heuristics on the server side.

Original: Fundamentally, the design space for subnetwork distribution heuristics can be divided into two high-level categories: spatial and temporal .
Revised: Fundamentally, the design space for subnetwork distribution heuristics can be divided into two high-level categories: spatial and temporal .

Original: Spatial refers to the distribution of subnetworks among participating clients within a given FL round.
Revised: Spatial refers to the distribution of subnetworks among participating clients within a given FL round.

Original: Intuitively, this class of heuristics concerns itself with which subnets go where .
Revised: Intuitively, this class of heuristics concerns itself with which subnets go where .

Original: Temporal refers to the distribution of subnetworks for an individual client across rounds.
Revised: Temporal refers to the distribution of subnetworks for an individual client across rounds.

Original: Intuitively, this class of heuristics concerns itself with when a given subnet is forwarded to a given client.
Revised: Intuitively, this class of heuristics concerns itself with when a given subnet is forwarded to a given client.

Original: Both spatial (which) and temporal (when) aspects should be taken into consideration for subnetwork distribution for best performance.
Revised: Both spatial (which) and temporal (when) aspects should be taken into consideration for subnetwork distribution for best performance.

Original: We propose one concrete spatio-temporal heuristic in this paper (§2.4).
Revised: We propose one concrete spatio-temporal heuristic in this paper (§2.4).

Original: Indeed, spatial distribution makes sure that the upper and lower bound of the model family is optimized, while temporal heuristics ensure exposure to all data partitions.
Revised: Indeed, spatial distribution makes sure that the upper and lower bound of the model family is optimized, while temporal heuristics ensure exposure to all data partitions.

Original: The latter can be thought of as a form of temporal load balancing of individual subnetworks across partitioned datasets.
Revised: The latter can be thought of as a form of temporal load balancing of individual subnetworks across partitioned datasets.

Original: To further validate the importance of spatio-temporal heuristics, we perform an ablation study, comparing it with random subnetwork distribution (§3.4).
Revised: To further validate the importance of spatio-temporal heuristics, we perform an ablation study, comparing it with random subnetwork distribution (§3.4).

Original: 1: Initialize W 2: for round t = 1,2, ...
Revised: 1: Initialize W 2: for round t = 1,2, ...

Original: T do 3:
Revised: T do 3:

Original: S t ← random set of max ( C · K, 1) clients ( S t ⊆ S ) 4: h t = H ( t ) //
Revised: S t ← random set of max ( C · K, 1) clients ( S t ⊆ S ) 4: h t = H ( t ) //

Original: t th spatial subnet distr.
Revised: t th spatial subnet distr.

Original: 5: for client k ∈ S t do 6: arch kt ← h t ( k ) , w kt ← G ( W t , arch kt ) 7: w kt +1 ← ClientUpdate( k , w kt ) 8: end for 9:
Revised: 5: for client k ∈ S t do 6: arch kt ← h t ( k ) , w kt ← G ( W t , arch kt ) 7: w kt +1 ← ClientUpdate( k , w kt ) 8: end for 9:

Original: W 0 ← zeros( W t ) 10: // shared-param avg by overlap cardinality 11:
Revised: W 0 ← zeros( W t ) 10: // shared-param avg by overlap cardinality 11:

Original: W sum ← (cid:80) k ∈ S t n k
Revised: W sum ← (cid:80) k ∈ S t n k

Original: [Equation]
Revised: [Equation]

Original: M ( W 0 , arch kt , w kt +1 ) 12:
Revised: M ( W 0 , arch kt , w kt +1 ) 12:

Original: W sum ← replace_zeros( W sum , W t ) 13:
Revised: W sum ← replace_zeros( W sum , W t ) 13:

Original: W cnt ← (cid:80) k ∈ S t M ( W 0 , arch kt , n k ∗ ones ( w kt +1 ))
Revised: W cnt ← (cid:80) k ∈ S t M ( W 0 , arch kt , n k ∗ ones ( w kt +1 ))

Original: W cnt ← replace_zeros( W cnt , 1 ) 15:
Revised: W cnt ← replace_zeros( W cnt , 1 ) 15:

Original: W t +1 ← W sum W cnt 16: end for
Revised: W t +1 ← W sum W cnt 16: end for

Original: 1: Input: Client j ( j ∈ S t ) which has been assigned the largest subnetwork at t th round ( H ( t )( j ) = arch M ),
Revised: 1: Input: Client j ( j ∈ S t ) which has been assigned the largest subnetwork at t th round ( H ( t )( j ) = arch M ),

Original: S t is set of randomly sampled clients t th round 2: β t = decay ( β 0 , t ) 3: W 0 ← zeros( W t ) 4: W sum ← (cid:80) k ∈ S t \ j (1 − β t )
Revised: S t is set of randomly sampled clients t th round 2: β t = decay ( β 0 , t ) 3: W 0 ← zeros( W t ) 4: W sum ← (cid:80) k ∈ S t \ j (1 − β t )

Original: | S t |−
Revised: | S t |−

Original: [Equation]
Revised: [Equation]

Original: n k
Revised: n k

Original: [Equation]
Revised: [Equation]

Original: M ( W 0 , arch kt , w kt +1 ) 5: W sum += β t ∗ n j
Revised: M ( W 0 , arch kt , w kt +1 ) 5: W sum += β t ∗ n j

Original: [Equation]
Revised: [Equation]

Original: M ( W 0 , arch M , w jt +1 ) //
Revised: M ( W 0 , arch M , w jt +1 ) //

Original: largest subnet 6: W sum ← replace_zeros
Revised: largest subnet 6: W sum ← replace_zeros

Original: ( W sum , W t ) //
Revised: ( W sum , W t ) //

Original: no overlaps 7: W cnt ← (cid:80) k ∈ S t \ j M
Revised: no overlaps 7: W cnt ← (cid:80) k ∈ S t \ j M

Original: ( W 0 , arch kt , (1 − β t ) | S t |− 1 ∗ n k ∗ ones ( w kt +1 ))
Revised: ( W 0 , arch kt , (1 − β t ) | S t |− 1 ∗ n k ∗ ones ( w kt +1 ))

Original: 8: W cnt += β t ∗ M
Revised: 8: W cnt += β t ∗ M

Original: ( W 0 , arch M , n j ∗ ones ( w jt +1 ))
Revised: ( W 0 , arch M , n j ∗ ones ( w jt +1 ))

Original: 9: W cnt ← replace_zeros
Revised: 9: W cnt ← replace_zeros

Original: ( W cnt , 1 ) //
Revised: ( W cnt , 1 ) //

Original: no overlaps 10: W t +1 ← W sum W cnt
Revised: no overlaps 10: W t +1 ← W sum W cnt

Original: 2. S HARED -P ARAMETER A VERAGING
Revised: 2. S HARED -P ARAMETER A VERAGING

Original: Shared parameter averaging is the second principal component of the weight shared FL framework.
Revised: Shared parameter averaging is the second principal component of the weight shared FL framework.

Original: Fundamentally, it provides a scaffolding for implementing heuristics that aggregate parameter updates from participating clients.
Revised: Fundamentally, it provides a scaffolding for implementing heuristics that aggregate parameter updates from participating clients.

Original: The server performs shared-parameter averaging of spatio-temporally distributed subnetworks at the end of each FL round.
Revised: The server performs shared-parameter averaging of spatio-temporally distributed subnetworks at the end of each FL round.

Original: One naive way of shared-parameter averaging is to keep track of the cardinality of the overlap—the number of overlaps of shared parameters—and average the parameters based on this cardinality (Algorithm 1 line 10-16).
Revised: One naive way of shared-parameter averaging is to keep track of the cardinality of the overlap—the number of overlaps of shared parameters—and average the parameters based on this cardinality (Algorithm 1 line 10-16).

Original: We call this simplest
Revised: We call this simplest

Original: FedAvg extension to weight shared supernets as averaging by overlap cardinality and analyze it in §3.4.
Revised: FedAvg extension to weight shared supernets as averaging by overlap cardinality and analyze it in §3.4.

Original: While the framework is designed to accommodate arbitrary spatio-temporally aware aggregation mechanisms, we propose a specific aggregation mechanism in §2.4.
Revised: While the framework is designed to accommodate arbitrary spatio-temporally aware aggregation mechanisms, we propose a specific aggregation mechanism in §2.4.

Original: Limitation.
Revised: Limitation.

Original: In order to fully replace previous round’s supernetwork parameters in the current round t , the largest subnetwork must be trained by at least one client.
Revised: In order to fully replace previous round’s supernetwork parameters in the current round t , the largest subnetwork must be trained by at least one client.

Original: Note that the weights with zero overlap are kept same as the previous round’s weights (algorithm 1 Line 12 and 14).
Revised: Note that the weights with zero overlap are kept same as the previous round’s weights (algorithm 1 Line 12 and 14).

Original: Since largest subnetwork weights arch M are same as supernetwork’s parameters W , the largest subnetwork should be included in each round to ensure the cardinality of overlap ≥ 1 .
Revised: Since largest subnetwork weights arch M are same as supernetwork’s parameters W , the largest subnetwork should be included in each round to ensure the cardinality of overlap ≥ 1 .

Original: Hence, our framework makes a simplifying assumption (justified in App.
Revised: Hence, our framework makes a simplifying assumption (justified in App.

Original: I) that a spatio-temporal heuristic will sample the largest network each round for at least one client.
Revised: J) that a spatio-temporal heuristic will sample the largest network each round for at least one client.

Original: 2. S UPER F ED ’ S W EIGHT S HARED FL T RAINING
Revised: 2. S UPER F ED ’ S W EIGHT S HARED FL T RAINING

Original: We propose a novel spatio-temporal subnet distribution heuristic and shared-parameter averaging in this section.
Revised: We propose a novel spatio-temporal subnet distribution heuristic and shared-parameter averaging in this section.

Original: We collectively refer to this combination as MaxNet.
Revised: We collectively refer to this combination as MaxNet.

Original: MaxNet’s subnet distribution.
Revised: MaxNet’s subnet distribution.

Original: In order to optimize the weight-shared FL objective (App.
Revised: In order to optimize the weight-shared FL objective (App.

Original: B equation 1), we first use the sandwich rule (Yu & Huang, 2019b).
Revised: B equation 1), we first use the sandwich rule (Yu & Huang, 2019b).

Original: With the sandwich rule, optimizing the lower and upper bound can implicitly optimize all the architectures in ϕ .
Revised: With the sandwich rule, optimizing the lower and upper bound can implicitly optimize all the architectures in ϕ .

Original: Hence, if | S t | is the number of clients participating in the t
Revised: Hence, if | S t | is the number of clients participating in the t

Original: th FL round, then MaxNet heuristic spatially samples one smallest, one largest and | S t | − 2 random subnets.
Revised: th FL round, then MaxNet heuristic spatially samples one smallest, one largest and | S t | − 2 random subnets.

Original: However, not every client participates in even single communication round.
Revised: However, not every client participates in even single communication round.

Original: The heuristic should train the lower and upper bound on as many clients as possible to train muliple model variants.
Revised: The heuristic should train the lower and upper bound on as many clients as possible to train muliple model variants.

Original: Therefore, temporally , MaxNet keeps track of the number of times the smallest and largest subnetworks have been assigned to each client.
Revised: Therefore, temporally , MaxNet keeps track of the number of times the smallest and largest subnetworks have been assigned to each client.

Original: Hence, in each communication round, the heuristic assigns the smallest and largest subnetworks to the clients which have seen them the least .
Revised: Hence, in each communication round, the heuristic assigns the smallest and largest subnetworks to the clients which have seen them the least .

Original: Random subnetworks are assigned to the rest of the clients.
Revised: Random subnetworks are assigned to the rest of the clients.

Original: We call this subnet distribution as Tracking-Sandwich .
Revised: We call this subnet distribution as Tracking-Sandwich .

Original: MaxNet’s shared-parameter averaging.
Revised: MaxNet’s shared-parameter averaging.

Original: The shared-parameter averaging of MaxNet is derived from averaging by overlap cardinality described in §2.3.
Revised: The shared-parameter averaging of MaxNet is derived from averaging by overlap cardinality described in §2.3.

Original: We find that naive way of shared-parameter averaging leads to interference Cai et al.
Revised: We find that naive way of shared-parameter averaging leads to interference Cai et al.

Original: We report our findings in §3.4.
Revised: We report our findings in §3.4.

Original: In order to reduce interference, we propose performing a weighted average among subnetworks (algorithm 2).
Revised: In order to reduce interference, we propose performing a weighted average among subnetworks (algorithm 2).

Original: While averaging the parameters of | S t | subnetworks, we assign β t ( β t ∈ R, 0 < β t < 1 ) weight to the largest subnetwork’s parameters and (1 − β t ) | S t |− 1 weight to the parameters of rest of the subnetworks (algorithm 2 line 4-5).
Revised: While averaging the parameters of | S t | subnetworks, we assign β t ( β t ∈ R, 0 < β t < 1 ) weight to the largest subnetwork’s parameters and (1 − β t ) | S t |− 1 weight to the parameters of rest of the subnetworks (algorithm 2 line 4-5).

Original: The training starts with an initial value β t = β
Revised: The training starts with an initial value β t = β

Original: o (say β o = 0 . 9 ), and β t value is decayed over some communication rounds to reach β T (say β T =
Revised: o (say β o = 0 . 9 ), and β t value is decayed over some communication rounds to reach β T (say β T =

Original: (1 / | S T | ) )
Revised: (1 / | S T | ) )

Original: (algorithmline 2).
Revised: (algorithmline 2).

Original: Our proposed heuristic has three hyperparameters — 1) initial weight given to maximum subnetwork in averaging β 0 2) decay function of β t 3) decay period of β t w.r.t percentage of total communication rounds.
Revised: Our proposed heuristic has three hyperparameters — 1) initial weight given to maximum subnetwork in averaging β 0 2) decay function of β t 3) decay period of β t w.r.t percentage of total communication rounds.

Original: We perform a detailed ablation on these hyperparameters in §3.4.
Revised: We perform a detailed ablation on these hyperparameters in §3.4.

Original: We use β 0 = 0 .
Revised: We use β 0 = 0 .

Original: 9 and decay it for 80% of the total communication rounds using cosine decay function as standard hyperparameters for the majority of experiments.
Revised: 9 and decay it for 80% of the total communication rounds using cosine decay function as standard hyperparameters for the majority of experiments.

Original: 2.4. R EDUCING I NTERFERENCE
Revised: 2.4. R EDUCING I NTERFERENCE

Original: Interference is a known phenomenon in centralized weight shared training (Cai et al., 2020) in which smaller subnetworks interfere with the training of the larger subnetworks.
Revised: Interference is a known phenomenon in centralized weight shared training (Cai et al., 2020) in which smaller subnetworks interfere with the training of the larger subnetworks.

Original: We demonstrate interference in weight shared federated learning (§3.4) and that weight-shared training in FL context is non-trivial .
Revised: We demonstrate interference in weight shared federated learning (§3.4) and that weight-shared training in FL context is non-trivial .

Original: We argue that in order to reduce interference, it is paramount to facilitate/priortize the training of the largest subnetwork first in the global model family .
Revised: We argue that in order to reduce interference, it is paramount to facilitate/priortize the training of the largest subnetwork first in the global model family .

Original: Hence, MaxNet weighs the importance of largest subnetwork’s parameters and reduces it’s influence over time.
Revised: Hence, MaxNet weighs the importance of largest subnetwork’s parameters and reduces it’s influence over time.

Original: Specifically, weighted averaging with β t prioritizes the contributions of the largest subnetwork in the earlier phase of training.
Revised: Specifically, weighted averaging with β t prioritizes the contributions of the largest subnetwork in the earlier phase of training.

Original: We later demonstrate that MaxNet’s shared-parameter averaging reduces interference among subnetworks (figure 7).
Revised: We later demonstrate that MaxNet’s shared-parameter averaging reduces interference among subnetworks (figure 7).

Original: In §3.4, we perform a detailed ablation on MaxNet heuristic and its derivations to show attribution of benefit (Tab. 2).
Revised: In §3.4, we perform a detailed ablation on MaxNet heuristic and its derivations to show attribution of benefit (Tab. 2).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We show generality of our proposed algorithm on various federated learning scenarios through comprehensive evaluation.
Revised: We show generality of our proposed algorithm on various federated learning scenarios through comprehensive evaluation.

Original: We compare SuperFed’s MaxNet and independent training (using FedAvg) on the following dimensions : (1) real-world datasets (2) non-i.i.d degree α (3) client participation ratio C .
Revised: We compare SuperFed’s MaxNet and independent training (using FedAvg) on the following dimensions : (1) real-world datasets (2) non-i.i.d degree α (3) client participation ratio C .

Original: Our goal is to match FedAvg across diverse FL scenarios and achieve an order-of-magnitude cost reduction w.r.t training a global model family.
Revised: Our goal is to match FedAvg across diverse FL scenarios and achieve an order-of-magnitude cost reduction w.r.t training a global model family.

Original: Furthermore, we also provide a cost analysis for global model family training.
Revised: Furthermore, we also provide a cost analysis for global model family training.

Original: Later, we perform a detailed ablation study to highlight key factors which contribute to our results.
Revised: Later, we perform a detailed ablation study to highlight key factors which contribute to our results.

Original: We run every experiment for three different seeds and report mean and standard deviation.
Revised: We run every experiment for three different seeds and report mean and standard deviation.

Original: 3. S ETUP
Revised: 3. S ETUP

Original: Baseline.
Revised: Baseline.

Original: To our knowledge, there does not exist an FL algorithm which trains a global family of models using weight-sharing.
Revised: To our knowledge, there does not exist an FL algorithm which trains a global family of models using weight-sharing.

Original: Hence, our baseline is independent SOTA training of DNN subnetworks (without weight sharing) using FedAvg.
Revised: Hence, our baseline is independent SOTA training of DNN subnetworks (without weight sharing) using FedAvg.

Original: We call this iFedAvg.
Revised: We call this iFedAvg.

Original: While SuperFed trains ≈ 5 ∗ 10subnetworks jointly, it is computationally infeasible to train such a large number of subnetworks with FedAvg independently.
Revised: While SuperFed trains upto 5 ∗ 10subnetworks jointly, it is computationally infeasible to train such a large number of subnetworks with FedAvg independently.

Original: We train a family of four DNN modelsranging from 0.46-3.41 GFLOPs using FedAvg (independently) containing both the smallest (lower bound) and the largest (upper bound) subnetworks of the supernetwork (App. F Tab. 4).
Revised: We train a family of four DNN models using FedAvg (independently) containing both the smallest (lower bound) and the largest (upper bound) subnetworks of the supernetwork (App. F Tab. 4).

Original: In all the graphs, we compare test accuracy of these subnetworks trained using the two approaches and add  5 more subnetworks for SuperFed to illustrate that SuperFed trains an order of magnitude more subnetworks which is computationally infeasible for iFedAvg.
Revised: We compare test accuracy/perplexity of these subnetworks trained using the two approaches and add upto 5 more subnetworks for SuperFed to illustrate that SuperFed trains an order of magnitude more subnetworks which is computationally infeasible for iFedAvg.

Original: Dataset and Models.
Revised: Dataset and Models.

Original: We compare our algorithm with the baseline on three datasets CIFAR10/ (Krizhevsky et al., 2009) and CINIC-10 (Darlow et al., 2018).
Revised: We compare our algorithm with the baseline on four datasets CIFAR10/ (Krizhevsky et al., 2009), CINIC-10 (Darlow et al., 2018) and PennTreeBank (PTB) (Marcus et al., 1993).

Original:  The base supernetwork is a ResNet (He et al., 2016) based architecture and contains ResNet-10/26 as the smallest and largest subnetworks respectively. 
Revised: For image datasets, The base supernetwork is a ResNet (He et al., 2016) based architecture and contains ResNet-10/26 as the smallest and largest subnetworks respectively. For text dataset, the base supernetwork is a TCN (Bai et al., 2018) based architecture (defined in App. H.1).

Original: In SuperFed, we train subnetworks varying in the depth and number of filters in convolution layers (App. F).
Revised: In SuperFed, we train subnetworks varying in the depth and number of filters in convolution layers (App. F).

Original: We keep the elastic dimensions same as described in OFA (Cai et al., 2020).

Original: Heterogeneity in Client Data Similar to (Chen & Chao, 2020; Lin et al., 2020a), we use the Dirichlet distribution for disjoint non-iid training data.
Revised: Heterogeneity in Client Data Similar to (Chen & Chao, 2020; Lin et al., 2020a), we use the Dirichlet distribution for disjoint non-iid training data.

Original: The α parameter controls the degree of heterogeneity — α = 100 is a close to uniform distribution of classes.
Revised: The α parameter controls the degree of heterogeneity — α = 100 is a close to uniform distribution of classes.

Original: Lower α values increase per-class differences between data partitions (App. J).
Revised: Lower α values increase per-class differences between data partitions (App. K).

Original: Local Training Hyper-params.
Revised: Local Training Hyper-params.

Original: For fair comparison, we keep client’s local training hyperparameters 
Revised: We keep client’s local training hyperparameters same for SuperFed and iFedAvg. For image datasets, our training setting is similar to (Lin et al., 2020a) using local

Original: [Figure]
Revised: [Figure]

Original: same for SuperFed and iFedAvg.

Original: Our training setting is similar to (Lin et al., 2020a) using local

Original: SGD (Lin et al., 2020b) with no weight decay and a constant learning rate (no decay).
Revised: SGD (Lin et al., 2020b) with no weight decay and a constant learning rate (no decay).

Original: We perform the grid search for learning rate between {0.03, 0.1} on training largest subnetwork using FedAvg (App. J).
Revised: We perform the grid search for learning rate between {0.03, 0.1} on training largest subnetwork using FedAvg (App. K).

Original: We find the optimal learning rate to be 0.1 and train the model for 5 local epochs for each communication round.
Revised: We find the optimal learning rate to be 0.1 and train the model for 5 local epochs for each communication round 1 .

Original: We adopt SOTA approach to batchnorm which is known to be problematic in FL non-iid settings (Hsieh et al., 2020).
Revised: We defer local training hyper-params for text dataset to App. H.2.

Original: 3. E VALUATION
Revised: 3. E VALUATION

Original: We compare our weight shared framework SuperFed with iFedAvg across various federated learning settings in this section.
Revised: We compare our weight shared framework SuperFed with iFedAvg across various federated learning settings in this section.

Original: Accuracy on Real-world Datasets.
Revised: Accuracy on Real-world Datasets.

Original: figure 2 compares MaxNet and iFedAvg on three datasets.
Revised: figure 2 compares MaxNet and iFedAvg on four datasets.

Original: For this experiment, we keep C = 0 .
Revised: For this experiment, we keep C = 0 .

Original: 4 and α = 100 .
Revised: 4 and α = 100 .

Original: CINIC10/ CIFAR10/100 experiments are run for R = 1000 / 1500 / 2000  communication rounds with total clients as 100 / 20 / 20 respectively.
Revised: CINIC10/CIFAR10/100 and PTB experiments are run for R = 1000 / 1500 / 2000 / 100 communication rounds with total clients as 100 / 20 / 20 /respectively.

Original: Takeaway .
Revised: Takeaway .

Original: SuperFed’s MaxNet achieves at par (or better) test accuracy (for all subnetworks) than iFedAvg.
Revised: SuperFed’s MaxNet achieves at par (or better) test accuracy/perplexity (for all subnetworks) than iFedAvg.

Original: MaxNet gets closer to iFedAvg as the dataset becomes harder (CIFAR100 is a harder dataset than CIFAR10 Krizhevsky et al.
Revised: MaxNet gets closer to iFedAvg as the dataset becomes harder (CIFAR100 is a harder dataset than CIFAR10 Krizhevsky et al.

Original: Effect of Non-i.i.d Degree α .
Revised: Effect of Non-i.i.d Degree α .

Original: figure 3 evaluates MaxNet as the degree of heterogeneity increases.
Revised: figure 3 evaluates MaxNet as the degree of heterogeneity increases.

Original: In this experiment, we use CIFAR10 dataset with C = 0 .
Revised: In this experiment, we use CIFAR10 dataset with C = 0 .

Original: 4 and run different
Revised: 4 and run different

Original: α = { 0 . 1 , 1 , 100 } settings for R = 1500 / 1500 / 2500 communications rounds respectively.
Revised: α = { 0 . 1 , 1 , 100 } settings for R = 1500 / 1500 / 2500 communications rounds respectively.

Original: Takeaway.
Revised: Takeaway.

Original: SuperFed’s MaxNet is robust to non-i.i.d-ness and experiences no loss of accuracy relative to iFedAvg across all the α settings.
Revised: SuperFed’s MaxNet is robust to non-i.i.d-ness and experiences no loss of accuracy relative to iFedAvg across all the α settings.

Original: As the degree of non-i.i.d-ness increases, the difference of the test accuracy between both the approaches decreases.
Revised: As the degree of non-i.i.d-ness increases, the difference of the test accuracy between both the approaches decreases.

Original: Effect of Client Participation Ratio C .
Revised: Effect of Client Participation Ratio C .

Original: figure 4 compares SuperFed’s MaxNet and iFedAvg by varying percentage of clients participating in each communication round.
Revised: figure 4 compares SuperFed’s MaxNet and iFedAvg by varying percentage of clients participating in each communication round.

Original: The experiment is run using CIFAR10 dataset, α = 100 and 20 clients.
Revised: The experiment is run using CIFAR10 dataset, α = 100 and 20 clients.

Original: Different C = { 0 . 2 , 0 . 4 , 0 .
Revised: Different C = { 0 . 2 , 0 . 4 , 0 .

Original: 8 } settings are run for
Revised: 8 } settings are run for

Original: R = 1500 / 1500 / 1000 communication rounds respectively.
Revised: R = 1500 / 1500 / 1000 communication rounds respectively.

Original: Takeaway.
Revised: Takeaway.

Original: MaxNet experiences no loss of accuracy relative to iFedAvg across all the C settings as well.
Revised: MaxNet experiences no loss of accuracy relative to iFedAvg across all the C settings as well.

Original: MaxNet’s tracking makes sure that every client contributes to the global supernet eventually.
Revised: MaxNet’s tracking makes sure that every client contributes to the global supernet eventually.

Original: Hence, MaxNet is resilient to client participation ratios.
Revised: Hence, MaxNet is resilient to client participation ratios.

Original: [Figure]
Revised: [Figure]

Revised: s

Original: [Figure]
Revised: [Figure]

Revised: A

Original: [Figure]
Revised: [Figure]

Revised: u r a

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: 3. C OST C OMPARISON
Revised: 3. C OST C OMPARISON

Original: We now compare SuperFed and iFedAvg w.r.t computational and communication costs for training a family of models.
Revised: We now compare SuperFed and iFedAvg w.r.t computational and communication costs for training a family of models.

Original: We defer cost calculation details to App.
Revised: We defer cost calculation details to App.

Original: G.
Revised: G.

Original: Takeaway.
Revised: Takeaway.

Original: figure 6 shows computational and communication costs of iFedAvg as the model family size increases.
Revised: figure 6 shows computational and communication costs of iFedAvg as the model family size increases.

Original: The dotted line represents cost for training ≈ 5 ∗ 10 8 subnetworks using SuperFed’s
Revised: The dotted line represents cost for training ≈ 5 ∗ 10 8 subnetworks using SuperFed’s

Original: MaxNet.
Revised: MaxNet.

Original: Clearly, as the size of model family increases the cost of training model family with iFedAvg approach increases.
Revised: Clearly, as the size of model family increases the cost of training model family with iFedAvg approach increases.

Original: In fact, training a family of 9 models with iFedAvg is 9.43x costlier computationally and 10.94x costlier in communication than training ≈ 5 ∗ 10 8 subnetworks using
Revised: In fact, training a family of 9 models with iFedAvg is 9.43x costlier computationally and 10.94x costlier in communication than training ≈ 5 ∗ 10 8 subnetworks using

Original: SuperFed’s MaxNet.
Revised: SuperFed’s MaxNet.

Original: Hence, iFedAvg is infeasible if used for training subnetworks of the same order of magnitude as SuperFed.
Revised: Hence, iFedAvg is infeasible if used for training subnetworks of the same order of magnitude as SuperFed.

Original: Furthermore, we emphasize that SuperFed incurs smaller cost than independently training the largest network with iFedAvg (model family size as 1, App. E), making it the preferred method of federated training for any family of models, however small.
Revised: Furthermore, we emphasize that SuperFed incurs smaller cost than independently training the largest network with iFedAvg (model family size as 1, App. E), making it the preferred method of federated training for any family of models, however small.

Original: 3. A BLATION S TUDY
Revised: 3. A BLATION S TUDY

Original: We now study attribution of benefits by performing an ablation on 1)
Revised: We now study attribution of benefits by performing an ablation on 1)

Original: Spatio Temporal Heuristics 2)
Revised: Spatio Temporal Heuristics 2)

Original: Hyperparameters of MaxNet heuristic.
Revised: Hyperparameters of MaxNet heuristic.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Spatio Temporal Heuristics.
Revised: Spatio Temporal Heuristics.

Original: Tab.
Revised: Tab.

Original: 2 lists the spatio-temporal heuristics tried as a part of this ablation study.
Revised: 2 lists the spatio-temporal heuristics tried as a part of this ablation study.

Original: For fair comparison, all the training hyperparameters are kept constant — CIFAR10 dataset,
Revised: For fair comparison, all the training hyperparameters are kept constant — CIFAR10 dataset,

Original: C = 0 .
Revised: C = 0 .

Original: 4 and α = 100 .
Revised: 4 and α = 100 .

Original: figure 7 compares the test accuracy of all the spatio-temporal heuristics.
Revised: figure 7 compares the test accuracy of all the spatio-temporal heuristics.

Original: Takeaway.
Revised: Takeaway.

Original: First, MaxNet has the best accuracy for all the subnetworks among all other heuristics.
Revised: First, MaxNet has the best accuracy for all the subnetworks among all other heuristics.

Original: Second, notice that the simple overlap cardinality averaging (overlap + R) with random subnetwork distribution underperforms for all subnetworks compared to independently training them.
Revised: Second, notice that the simple overlap cardinality averaging (overlap + R) with random subnetwork distribution underperforms for all subnetworks compared to independently training them.

Original: While, both sandwich (overlap + S) and tracking-sandwich (overlap + TS) subnetwork distribution improve smallest subnetwork’s accuracy as compared to random subnetwork distribution (overlap + R) and match the accuracy of independently training it.
Revised: While, both sandwich (overlap + S) and tracking-sandwich (overlap + TS) subnetwork distribution improve smallest subnetwork’s accuracy as compared to random subnetwork distribution (overlap + R) and match the accuracy of independently training it.

Original: Tracking-sandwich (overlap + TS) performs slightly better than sandwich (overlap + S).
Revised: Tracking-sandwich (overlap + TS) performs slightly better than sandwich (overlap + S).

Original: However, the largest subnetwork’s accuracy is still sub-par compared to iFedAvg.
Revised: However, the largest subnetwork’s accuracy is still sub-par compared to iFedAvg.

Original: We attribute the reason for sub-par accuracy of the largest subnetwork to interference (for overlap + (R,S,TS) in figure 7), i.e. smaller subnetworks interfere with the training of the larger subnetworks Cai et al.
Revised: We attribute the reason for sub-par accuracy of the largest subnetwork to interference (for overlap + (R,S,TS) in figure 7), i.e. smaller subnetworks interfere with the training of the larger subnetworks Cai et al.

Original: MaxNet prioritizes contributions of the largest subnetwork in the earlier phase of training and decays its importance over time.
Revised: MaxNet prioritizes contributions of the largest subnetwork in the earlier phase of training and decays its importance over time.

Original: This reduces interference and achieves no loss of accuracy compared to independently trained subnetworks.
Revised: This reduces interference and achieves no loss of accuracy compared to independently trained subnetworks.

Original: Hyperparameters of MaxNet.
Revised: Hyperparameters of MaxNet.

Original: MaxNet’s hyperparameters are described in §2.4.
Revised: MaxNet’s hyperparameters are described in §2.4.

Original: We perform a grid search on the following hyperparameters : β 0 : { 0 . 9 , 0 . 5 } , decay function {linear, cosine, constant}, decay period {50%, 80%}.
Revised: We perform a grid search on the following hyperparameters : β 0 : { 0 . 9 , 0 . 5 } , decay function {linear, cosine, constant}, decay period {50%, 80%}.

Original: figure 5 compares test accuracy of the subnetworks for different hyperparameters.
Revised: figure 5 compares test accuracy of the subnetworks for different hyperparameters.

Original: Takeaway.
Revised: Takeaway.

Original: Hyperparameter β 0 has a major effect on test accuracy of subnetworks (figure 5a).
Revised: Hyperparameter β 0 has a major effect on test accuracy of subnetworks (figure 5a).

Original: Starting the training with larger β 0 value helps reduce interference.
Revised: Starting the training with larger β 0 value helps reduce interference.

Original: Hence, training of the largest subnetwork is preferred more in the earlier phase of training, which also benefits the smaller networks.
Revised: Hence, training of the largest subnetwork is preferred more in the earlier phase of training, which also benefits the smaller networks.

Original: Also, keeping β t = 0 .
Revised: Also, keeping β t = 0 .

Original: 9 = const only improves largest subnetwork’s accuracy.
Revised: 9 = const only improves largest subnetwork’s accuracy.

Original: Decaying β t over communication rounds increases accuracy for smaller subnetworks without reducing the largest subnetwork accuracy(figure 5b).
Revised: Decaying β t over communication rounds increases accuracy for smaller subnetworks without reducing the largest subnetwork accuracy(figure 5b).

Original: Similar empirical evidence of the benefit of gradually reducing larger subnetwork’s importance in training is also seen in OFA (Cai et al., 2020).
Revised: Similar empirical evidence of the benefit of gradually reducing larger subnetwork’s importance in training is also seen in OFA (Cai et al., 2020).

Original: Moreover, cosine decay performs better than linear decay.
Revised: Moreover, cosine decay performs better than linear decay.

Original: Longer decay period ( 80% of the total comm.
Revised: Longer decay period ( 80% of the total comm.

Original: rounds) further helps in achieving better test accuracy (figure 5c).
Revised: rounds) further helps in achieving better test accuracy (figure 5c).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: SuperFed is the new framework for weight shared FL.
Revised: SuperFed is the new framework for weight shared FL.

Original: Its pluggable architecture is used to evaluate a newly proposed set of distributed training algorithms for spatio-temporal subnetwork distribution and model parameter aggregation, collectively called MaxNet.
Revised: Its pluggable architecture is used to evaluate a newly proposed set of distributed training algorithms for spatio-temporal subnetwork distribution and model parameter aggregation, collectively called MaxNet.

Original: MaxNet is able to train multiple
Revised: MaxNet is able to train multiple

Original: DNNs jointly with weight sharing in a federated fashion, matching state-of-the-art FedAvg w.r.t accuracy.
Revised: DNNs jointly with weight sharing in a federated fashion, matching state-of-the-art FedAvg w.r.t accuracy.

Original: SuperFed amortizes the cost of training over an arbitrary selection of subnetworks, while conventional FL techniques (e.g., FedAvg) incur O ( k ) training cost for any selection of k models from the “family of models”.
Revised: SuperFed amortizes the cost of training over an arbitrary selection of subnetworks, while conventional FL techniques (e.g., FedAvg) incur O ( k ) training cost for any selection of k models from the “family of models”.

Original: SuperFed opens new directions for future work: adjusting MaxNet for client resource constraints (e.g., compute, bandwidth, memory capacity) and developing theoretical insights on training enabled by MaxNet.
Revised: SuperFed opens new directions for future work: adjusting MaxNet for client resource constraints (e.g., compute, bandwidth, memory capacity) and developing theoretical insights on training enabled by MaxNet.

Original: We believe that SuperFed takes an important first step towards co-training model families in a federated fashion cost-efficiently, achieving an order of magnitude communication and computation cost reduction (even with as few as k = 9 models) while maintaining accuracy through interference mitigation.  
Revised: We believe that SuperFed takes an important first step towards co-training model families in a federated fashion cost-efficiently, achieving an order of magnitude communication and computation cost reduction (even with as few as k = 9 models) while maintaining accuracy through interference mitigation.  REPRODUCIBILITY STATEMENT

Revised: We provide anonymized source code https://anonymous.4open.science/r/

Revised: SuperFed/ for the experimental results included in the main text.

Revised: The code includes full instructions to perform weight-shared federated learning on the datasets and settings considered in the paper.

Original: A B ACKGROUND ON W EIGHT - SHARED T RAINING
Revised: A B ACKGROUND ON W EIGHT - SHARED T RAINING

Original: OFA (Cai et al., 2020) proposes a progressive shrinking method for weight-shared training of 10subnetworks jointly by using a neural supernetwork.
Revised: OFA (Cai et al., 2020) proposes a progressive shrinking method for weight-shared training of 10subnetworks jointly by using a neural supernetwork.

Original: The training is facilitated by a pre-trained teacher network which is used for distillation throughout the training.
Revised: The training is facilitated by a pre-trained teacher network which is used for distillation throughout the training.

Original: Training a supernetwork naively causes “interference” (Cai et al., 2020).
Revised: Training a supernetwork naively causes “interference” (Cai et al., 2020).

Original: Hence, OFA takes a phased approach to training, by training larger subnetworks first and progressively shrinking the dimensions of a subnetwork being trained.
Revised: Hence, OFA takes a phased approach to training, by training larger subnetworks first and progressively shrinking the dimensions of a subnetwork being trained.

Original: BigNAS (Yu et al., 2020) proposes a one-shot approach (doesn’t use any pretrained network) for training a supernetwork using inplace distillation (Yu & Huang, 2019a).
Revised: BigNAS (Yu et al., 2020) proposes a one-shot approach (doesn’t use any pretrained network) for training a supernetwork using inplace distillation (Yu & Huang, 2019a).

Original: It also proposes several training hyperparameters which optimize the supernetwork training.
Revised: It also proposes several training hyperparameters which optimize the supernetwork training.

Original: However, these weight-shared training techniques are proposed for centralised setting with i.i.d data.
Revised: However, these weight-shared training techniques are proposed for centralised setting with i.i.d data.

Original: The proposed SuperFed is a one-shot weight shared training framework for non-i.i.d decentralised data.
Revised: The proposed SuperFed is a one-shot weight shared training framework for non-i.i.d decentralised data.

Original: We empirically show that naive application of FL techniques for globally training weight shared supernetwork structures causes interference among subnetworks in non-i.i.d setting (§3.4).
Revised: We empirically show that naive application of FL techniques for globally training weight shared supernetwork structures causes interference among subnetworks in non-i.i.d setting (§3.4).

Original: B P ROBLEM F ORMULATION
Revised: B P ROBLEM F ORMULATION

Original: Now, we formulate the objective for our weight shared federated learning (notations listed in App.
Revised: Now, we formulate the objective for our weight shared federated learning (notations listed in App.

Original: D
Revised: D

Original: Tab. 3).
Revised: Tab. 3).

Original: Later, we discuss our proposed heuristic to minimize the objective.
Revised: Later, we discuss our proposed heuristic to minimize the objective.

Original: In FedAvg (McMahan et al., 2017), a single global model with parameters w is trained by minimizing the following objective function - min w (cid:80)
Revised: In FedAvg (McMahan et al., 2017), a single global model with parameters w is trained by minimizing the following objective function - min w (cid:80)

Original: Kk =1 n k n ∗ F k ( w ) .
Revised: Kk =1 n k n ∗ F k ( w ) .

Original: where there are K clients with their own data partition P k such that the size of the partition is denoted by n k ( where n = (cid:80)
Revised: where there are K clients with their own data partition P k such that the size of the partition is denoted by n k ( where n = (cid:80)

Original: Kk =1 n k is the total number of data points).
Revised: Kk =1 n k is the total number of data points).

Original: F k ( W ) = (cid:80) i ∈
Revised: F k ( W ) = (cid:80) i ∈

Original: P k f i ( w ) denotes the loss for data points of partition P k
Revised: P k f i ( w ) denotes the loss for data points of partition P k

Original: Meanwhile, in weight shared training, OFA’s goal is to train every possible architecture in the supernetwork.
Revised: Meanwhile, in weight shared training, OFA’s goal is to train every possible architecture in the supernetwork.

Original: Assume ϕ : { arch 1 , arch 2 , ... arch M } as a DNN model family with arch M being the largest subnetwork.
Revised: Assume ϕ : { arch 1 , arch 2 , ... arch M } as a DNN model family with arch M being the largest subnetwork.

Original: The shared weights of these architectures are denoted by W (supernet’s parameters).
Revised: The shared weights of these architectures are denoted by W (supernet’s parameters).

Original: Hence the objective of OFA (Cai et al., 2020) can be formalized as follows:
Revised: Hence the objective of OFA (Cai et al., 2020) can be formalized as follows:

Original: min W (cid:80) arch i ∈
Revised: min W (cid:80) arch i ∈

Original: ϕ L ( G ( W, arch i ))) .
Revised: ϕ L ( G ( W, arch i ))) .

Original: where w = G ( W, arch i ) denotes a selection (subset) of arch i parameters from shared weights W .
Revised: where w = G ( W, arch i ) denotes a selection (subset) of arch i parameters from shared weights W .

Original: L ( w ) denotes the loss of subnetwork’s parameters w ( w ∈ W ) on a single central dataset.
Revised: L ( w ) denotes the loss of subnetwork’s parameters w ( w ∈ W ) on a single central dataset.

Original: As stated in §1, the goal of this work is to train a global model family of size | ϕ | .
Revised: As stated in §1, the goal of this work is to train a global model family of size | ϕ | .

Original: Thus, the objective function of such training takes the following form:
Revised: Thus, the objective function of such training takes the following form:

Original: [Equation]
Revised: [Equation]

Original: Indeed, the goal of weight shared FL training is to train all the architectures in the family ϕ on all the data partitions { P 1 , P 2 , ..., P K } .
Revised: Indeed, the goal of weight shared FL training is to train all the architectures in the family ϕ on all the data partitions { P 1 , P 2 , ..., P K } .

Original: Weight shared FL aims at training a family of models ϕ at an order of magnitude lesser training cost (both communication and computational) relative to independently training these networks in FL while matching or even surpassing accuracy.
Revised: Weight shared FL aims at training a family of models ϕ at an order of magnitude lesser training cost (both communication and computational) relative to independently training these networks in FL while matching or even surpassing accuracy.

Original: We compare weight shared FL trained subnetworks with independently FL trained subnetworks in §3.2.
Revised: We compare weight shared FL trained subnetworks with independently FL trained subnetworks in §3.2.

Original: C F ED A VG : A S PECIAL C ASE OF S UPER F ED
Revised: C F ED A VG : A S PECIAL C ASE OF S UPER F ED

Original: FedAvg (McMahan et al., 2017) is a special case of SuperFed’s algorithm (algorithm 1).
Revised: FedAvg (McMahan et al., 2017) is a special case of SuperFed’s algorithm (algorithm 1).

Original: When the family of models contains only a single subnetwork i.e. ϕ = { arch } , then algorithm 1 only trains the weights of that subnetwork.
Revised: When the family of models contains only a single subnetwork i.e. ϕ = { arch } , then algorithm 1 only trains the weights of that subnetwork.

Original: The spatio-temporal heuristic H becomes H ( t )( k ) =
Revised: The spatio-temporal heuristic H becomes H ( t )( k ) =

Original: arch ∀ t ∈ { 1 , 2 , ..T } , k ∈ S .
Revised: arch ∀ t ∈ { 1 , 2 , ..T } , k ∈ S .

Original: Since the same subnetwork is sent to all the clients participating in a round, the number of overlaps for weights of subnetwork arch equals the number of participating clients | S t | .
Revised: Since the same subnetwork is sent to all the clients participating in a round, the number of overlaps for weights of subnetwork arch equals the number of participating clients | S t | .

Original: Hence the averaging method reduces to FedAvg’s averaging mechanism for subnetwork arch .
Revised: Hence the averaging method reduces to FedAvg’s averaging mechanism for subnetwork arch .

Original: D D ESCRIPTION OF N OTATIONS
Revised: D D ESCRIPTION OF N OTATIONS

Original: [Table]
Revised: [Table]

Original: DNN architecture (subnetwork) such that arch ∈ ϕ
Revised: DNN architecture (subnetwork) such that arch ∈ ϕ

Original: G ( W, arch )
Revised: G ( W, arch )

Original: Selection of weights for DNN arch from supernet weights W arch M largest subnetwork in ϕ such that G ( W, arch M ) =
Revised: Selection of weights for DNN arch from supernet weights W arch M largest subnetwork in ϕ such that G ( W, arch M ) =

Original: w M = W h
Revised: w M = W h

Original: Spatial heuristic for subnetwork distribution such that h : S → ϕ
Revised: Spatial heuristic for subnetwork distribution such that h : S → ϕ

Original: H ( t )
Revised: H ( t )

Original: Spatio-temporal heuristic for subnetwork distribution such that H : { 1 , 2 , ...T } → h
Revised: Spatio-temporal heuristic for subnetwork distribution such that H : { 1 , 2 , ...T } → h

Original: M ( W, arch, w )
Revised: M ( W, arch, w )

Original: Weight w of subnetwork arch partially replacing weight W of supernetwork (super-imposition)
Revised: Weight w of subnetwork arch partially replacing weight W of supernetwork (super-imposition)

Original: E C OST C OMPARISON F OR T RAINING T HE L ARGEST S UBNETWORK
Revised: E C OST C OMPARISON F OR T RAINING T HE L ARGEST S UBNETWORK

Original: Since supernetwork’s weights are same as the largest subnetwork’s weights, we compare SuperFed and FedAvg w.r.t communication and computational costs of training the same weights for total communication rounds T .
Revised: Since supernetwork’s weights are same as the largest subnetwork’s weights, we compare SuperFed and FedAvg w.r.t communication and computational costs of training the same weights for total communication rounds T .

Original: In SuperFed, other subnetworks (i.e., subsets of the largest subnetwork parameters) are also trained along with the largest subnetwork while FedAvg only trains the largest subnetwork.
Revised: In SuperFed, other subnetworks (i.e., subsets of the largest subnetwork parameters) are also trained along with the largest subnetwork while FedAvg only trains the largest subnetwork.

Original: We assume that the largest subnetwork is assigned to atleast one client for every round in SuperFed (assumption in §2.3).
Revised: We assume that the largest subnetwork is assigned to atleast one client for every round in SuperFed (assumption in §2.3).

Original: Note that the overall time spent by a client in training the neural network(s) send by the server over multiple rounds would be less in SuperFed than FedAvg.
Revised: Note that the overall time spent by a client in training the neural network(s) send by the server over multiple rounds would be less in SuperFed than FedAvg.

Original: Moreover, the amount of bytes exchanged between the server and clients would be less in SuperFed than FedAvg in every round.
Revised: Moreover, the amount of bytes exchanged between the server and clients would be less in SuperFed than FedAvg in every round.

Original: This is because in SuperFed, smaller subnetworks are also sent to some clients other than just sending the largest subnetwork to every client (FedAvg).
Revised: This is because in SuperFed, smaller subnetworks are also sent to some clients other than just sending the largest subnetwork to every client (FedAvg).

Original: These smaller subnetworks not only take less training time (computational) but also have less parameters (communication).
Revised: These smaller subnetworks not only take less training time (computational) but also have less parameters (communication).

Original: Hence, our proposed weight shared federated learning framework trains a family of models with lesser computational and communication costs than training the largest subnetwork using FedAvg.
Revised: Hence, our proposed weight shared federated learning framework trains a family of models with lesser computational and communication costs than training the largest subnetwork using FedAvg.

Original: F DNN A RCHITECTURE S PACE AND S ELECTION 
Revised: F DNN A RCHITECTURE S PACE AND S ELECTION FOR I MAGE D ATASETS

Original: We modified OFAResNets — a supernetwork proposed in OFA (Cai et al., 2020) to have smaller size (10-26 layers) models and make it amenable to 32 x 32 image size.
Revised: We modified OFAResNets — a supernetwork proposed in OFA (Cai et al., 2020) to have smaller size (10-26 layers) models and make it amenable to 32 x 32 image size.

Original: A typical ResNet architecture
Revised: A typical ResNet architecture

Original: He et al.
Revised: He et al.

Original: (2016) has multiple stages and each stage contains a list of residual blocks.
Revised: (2016) has multiple stages and each stage contains a list of residual blocks.

Original: We implement the supernetwork with 4 such stages.
Revised: We implement the supernetwork with 4 such stages.

Original: Next, we describe the two elastic dimensions we use to vary subnetworks in size.
Revised: Next, we describe the two elastic dimensions we use to vary subnetworks in size.

Original: Depth.
Revised: Depth.

Original: It decides the no. of residual blocks per stage.
Revised: It decides the no. of residual blocks per stage.

Original: Hence, this dimension is usually specified using a list of size 4 with each value denoting the depth of that stage.
Revised: Hence, this dimension is usually specified using a list of size 4 with each value denoting the depth of that stage.

Original: Our smallest subnetwork keeps a depth of two per stage.
Revised: Our smallest subnetwork keeps a depth of two per stage.

Original: Hence these two per-stage residual blocks are shared among every subnetwork.
Revised: Hence these two per-stage residual blocks are shared among every subnetwork.

Original: Depth elastic dimension can also be specified using a single integer say d which denotes d depth for every stage in the supernetwork.
Revised: Depth elastic dimension can also be specified using a single integer say d which denotes d depth for every stage in the supernetwork.

Original: Elastic ratio.
Revised: Elastic ratio.

Original: This dimension changes number of filters and channels of convolution layers in a residual block.
Revised: This dimension changes number of filters and channels of convolution layers in a residual block.

Original: The block has two convolutional layers with input, middle and output channels.
Revised: The block has two convolutional layers with input, middle and output channels.

Original: Elastic ratio is a ratio of output to middle channel only affecting the number middle channels in the block.
Revised: Elastic ratio is a ratio of output to middle channel only affecting the number middle channels in the block.

Original: The elastic ratio is specified via list of size 12 expressing number of middle channels for each residual block.
Revised: The elastic ratio is specified via list of size 12 expressing number of middle channels for each residual block.

Original: It can also be specified using a single integer say e which denotes e elastic ratio for
Revised: It can also be specified using a single integer say e which denotes e elastic ratio for

Original: Tab. 4 lists the subnetworks selected for comparing SuperFed with iFedAvg.
Revised: Tab. 4 lists the subnetworks selected for comparing SuperFed with iFedAvg.

Original: Out of the 9 subnetworks, 4 of them are also trained using FedAvg independently.
Revised: Out of the 9 subnetworks, 4 of them are also trained using FedAvg independently.

Original: SuperFed ≈ 5 ∗ 10 8 subnetworks including the nine subnetworks listed in Tab. 4.
Revised: SuperFed ≈ 5 ∗ 10 8 subnetworks including the nine subnetworks listed in Tab. 4.

Original: [Table]
Revised: [Table]

Original: G C OST CALCULATION G. C OMPUTATIONAL C OST
Revised: G C OST CALCULATION G. C OMPUTATIONAL C OST

Original: We define computational cost as the average time spent/computations done by the client in FL training.
Revised: We define computational cost as the average time spent/computations done by the client in FL training.

Original: Hence, we estimate average computational cost in a single FL communication round.
Revised: Hence, we estimate average computational cost in a single FL communication round.

Original: Note that the cost is proportional to the GFLOPs of the model (subnetwork) that it is training.
Revised: Note that the cost is proportional to the GFLOPs of the model (subnetwork) that it is training.

Original: Now, we describe cost calculation for the two approaches compared in §3.
Revised: Now, we describe cost calculation for the two approaches compared in §3.

Original: SuperFed.
Revised: SuperFed.

Original: Sum of GFLOPs of all the subnetworks that each clients sees over the course of training procedure is done.
Revised: Sum of GFLOPs of all the subnetworks that each clients sees over the course of training procedure is done.

Original: Then, an average is calculated over the total number of clients and total number of communication rounds.
Revised: Then, an average is calculated over the total number of clients and total number of communication rounds.

Original: iFedAvg.
Revised: iFedAvg.

Original: Here all the subnetworks of a model family are trained independently.
Revised: Here all the subnetworks of a model family are trained independently.

Original: The GFLOPs of a subnetwork is proportional to the time client spends per round in FedAvg training .
Revised: The GFLOPs of a subnetwork is proportional to the time client spends per round in FedAvg training .

Original: Hence, the sum of GFLOPs of all subnetworks in the family is the average computational cost per round in iFedAvg.
Revised: Hence, the sum of GFLOPs of all subnetworks in the family is the average computational cost per round in iFedAvg.

Original: G. C OMMUNICATION C OST
Revised: G. C OMMUNICATION C OST

Original: Communication cost is defined by the average bytes transferred between the clients and the server in
Revised: Communication cost is defined by the average bytes transferred between the clients and the server in

Original: FL training.
Revised: FL training.

Original: The total communication cost is the sum of communication cost per round.
Revised: The total communication cost is the sum of communication cost per round.

Original: Hence, we first calculate average communication cost per FL round.
Revised: Hence, we first calculate average communication cost per FL round.

Original: The average includes both the download and upload of the model.
Revised: The average includes both the download and upload of the model.

Original: The bytes transferred to the clients and vice-versa depend on the size of the subnetwork.
Revised: The bytes transferred to the clients and vice-versa depend on the size of the subnetwork.

Original: Now we describe communication cost calculationSuperFed.
Revised: Now we describe communication cost calculationSuperFed.

Original: The sum of size (in GB) of subnetworks distributed to each client at each communication round is calculated.
Revised: The sum of size (in GB) of subnetworks distributed to each client at each communication round is calculated.

Original: Then average is taken over total communication round.
Revised: Then average is taken over total communication round.

Original: iFedAvg.
Revised: iFedAvg.

Original: We first calculate the average communicaton cost per round for training a single subnetwork (say arch i ).
Revised: We first calculate the average communicaton cost per round for training a single subnetwork (say arch i ).

Original: This is typically given by - 2 ∗| S t |∗ sizeOf ( arch i ) ( | S t | denotes number of participating clients).
Revised: This is typically given by - 2 ∗| S t |∗ sizeOf ( arch i ) ( | S t | denotes number of participating clients).

Original: Since every subnetwork is trained independently the total communication cost per round is given by - 2 ∗ | S t | ∗ (cid:80) arch
Revised: Since every subnetwork is trained independently the total communication cost per round is given by - 2 ∗ | S t | ∗ (cid:80) arch

Original: i ∈
Revised: i ∈

Original: ϕ sizeOf ( arch i ) .
Revised: ϕ sizeOf ( arch i ) .

Original: In figure 6, we show costs for training a model family of size upto nine subnetworks with iFedAvg.
Revised: In figure 6, we show costs for training a model family of size upto nine subnetworks with iFedAvg.

Revised: H DNN A RCHITECTURE S PACE A ND C LIENT L OCAL T RAINING FOR T EXT D ATASET

Revised: In this section, we describe our TCN Bai et al.

Revised: (2018) supernetwork and local training hyperparameters associated with the text dataset experiment.

Original: H N EURAL A RCHITECTURE SEARCH
Revised: H. S UPERNET B ASED ON TCN S

Revised: A standard TCN architecture (Bai et al., 2018) for PTB dataset Marcus et al. (1993) consists of eight layer (or four temporal blocks) with 600 input and output channels.

Revised: We enable two elastic dimensions in the standard TCN architecture — depth and expand ratio.

Revised: Depth defines the the number of blocks to be activated whereas the expand ratio defines the number of output channels of the first layer of each block.

Revised: Overall, the depth choices considered are {0,1,2} with 0 denoting activating top two blocks only and expand ratio choices are {0.1, 0.2, 0.5, 1.0}.

Revised: H. L OCAL T RAINING H YPER - PARAMS

Revised: For client local training in text dataset, the initial learning rate is kept as 4 which is a standard learning rate to train TCN on PTB dataset.

Revised: We decay the learning rate every 50 local epochs by 0.1.

Revised: We use gradient norm clipping as 0 .

Revised: 35 and batch size 16.

Revised: We run five local epochs per client for every communication round

Revised: I N EURAL A RCHITECTURE S EARCH

Original:  The goal of weight shared FL is to offer a rich latency accuracy trade-off space so that clients can choose one or more models for her deployment needs.
Revised: [Table] The goal of weight shared FL is to offer a rich latency accuracy trade-off space so that clients can choose one or more models for her deployment needs.

Original: In this section, we demonstrate that once the supernetwork is trained, a neural architectural search (NAS) can be performed by the clients to extract Pareto-optimal subnetworks.
Revised: In this section, we demonstrate that once the supernetwork is trained, a neural architectural search (NAS) can be performed by the clients to extract Pareto-optimal subnetworks.

Original: Similar to Sahni et al.
Revised: Similar to Sahni et al.

Original: (2021), we use evolutionary algorithm to[Table] search for pareto-optimal subnetworks for a given FLOP constraint.
Revised: (2021), we use evolutionary algorithm to search for pareto-optimal subnetworks for a given FLOP constraint.

Original: Tab.
Revised: Tab.

Original: 5 lists different subnetworks found by the search algorithm for different GFLOPs constraints.
Revised: 5 lists different subnetworks found by the search algorithm for different GFLOPs constraints.

Original: The subnetworks are extracted from supernetwork trained with the following setting - C = 0 .
Revised: The subnetworks are extracted from supernetwork trained with the following setting - C = 0 .

Original: 4 , α = 100 , 20 clients and CIFARdataset.
Revised: 4 , α = 100 , 20 clients and CIFARdataset.

Original: The subnetworks are extracted without retraining .
Revised: The subnetworks are extracted without retraining .

Original: I J USTIFICATION OF S UPER F ED ’ S L IMITATION
Revised: J J USTIFICATION OF S UPER F ED ’ S L IMITATION

Original: We first highlight that goal of SuperFed is to target multiple deployment scenarios of clients rather than assume system-heterogeneity during training.
Revised: We first highlight that goal of SuperFed is to target multiple deployment scenarios of clients rather than assume system-heterogeneity during training.

Original: Furthermore, we argue that resources are extremely dynamic and scarce in deployment (discussed in §1) than during FL-training.
Revised: Furthermore, we argue that resources are extremely dynamic and scarce in deployment (discussed in §1) than during FL-training.

Original: One of the foremost requirement of FL is to make the model training invisible to the user i.e. it should not slow down the smartphone or drain its battery (Bonawitz et al., 2019).
Revised: One of the foremost requirement of FL is to make the model training invisible to the user i.e. it should not slow down the smartphone or drain its battery (Bonawitz et al., 2019).

Original: Hence, smartphones usually participate in
Revised: Hence, smartphones usually participate in

Original: FL-training when they are idle, above a certain battery level and connected to unmetered network (Kairouz et al., 2021).
Revised: FL-training when they are idle, above a certain battery level and connected to unmetered network (Kairouz et al., 2021).

Original: Since resources are fairly available at FL-training time, it is reasonable to assume that atleast one client takes the largest subnetwork during FL-training in a given comm.
Revised: Since resources are fairly available at FL-training time, it is reasonable to assume that atleast one client takes the largest subnetwork during FL-training in a given comm.

Original: round (a limitation discussed in §2.3).
Revised: round (a limitation discussed in §2.3).

Original: [Figure]
Revised: K L EARNING R ATE G RID S EARCH

Original: We perform a grid search on learning rate for the largest subnetwork in the model family using
Revised: We perform a grid search on learning rate for the largest subnetwork in the model family using

Original: FedAvg. figure 8 compares learning rates 0 .
Revised: FedAvg. figure 8 compares learning rates 0 .

Original: 03 and 0 .
Revised: 03 and 0 .

Original: This shows that learning rate 0 .
Revised: This shows that learning rate 0 .

Original: 1 achieves a better test accuracy for 1500 communication round.
Revised: 1 achieves a better test accuracy for 1500 communication round.

Original: To have less complexity, we keep the same learning rate for training each subnetwork using FedAvg.
Revised: To have less complexity, we keep the same learning rate for training each subnetwork using FedAvg.

Original: SuperFed also uses learning rate as 0 .
Revised: SuperFed also uses learning rate as 0 .

Original: 1 for fair comparison.
Revised: 1 for fair comparison.

Original: K C LIENT C LASS D ISTRIBUTION
Revised: [Figure] [Figure]

Original: figure 9 visualizes distribution of classes among 20 clients.
Revised: figure 9 visualizes distribution of classes among 20 clients.

Original: The size of dots symbolizes the amount of data points for that particular class in client’s dataset partition.
Revised: The size of dots symbolizes the amount of data points for that particular class in client’s dataset partition.

Original: Partition is done using dirichlet distribution which is parameterized by α .
Revised: Partition is done using dirichlet distribution which is parameterized by α .

Original: As seen in the figure, the lower the α value the higher the degree of non-i.i.d-ness.
Revised: As seen in the figure, the lower the α value the higher the degree of non-i.i.d-ness.

Original: Specifically for α = 0 .
Revised: Specifically for α = 0 .

Original: 1 , many clients don’t have even a single data point for some classes.
Revised: 1 , many clients don’t have even a single data point for some classes.

Original: [Figure]