Original: E FFICIENT S PLIT -M IX F EDERATED L EARNING FOR O ND EMAND AND I N -S ITU C USTOMIZATION
Revised: E FFICIENT S PLIT -M IX F EDERATED L EARNING FOR O ND EMAND AND I N -S ITU C USTOMIZATION

Original: A BSTRACT
Revised: A BSTRACT

Original: Federated learning (FL) provides a distributed learning framework for multiple participants to collaborate learning without sharing raw data.
Revised: Federated learning (FL) provides a distributed learning framework for multiple participants to collaborate learning without sharing raw data.

Original: In many practical FL scenarios, participants have heterogeneous resources due to disparities in hardware and inference dynamics that require quickly loading models of different sizes and levels of robustness.
Revised: In many practical FL scenarios, participants have heterogeneous resources due to disparities in hardware and inference dynamics that require quickly loading models of different sizes and levels of robustness.

Original: The heterogeneity and dynamics together impose significant challenges to existing FL approaches and thus greatly limit FL’s applicability.
Revised: The heterogeneity and dynamics together impose significant challenges to existing FL approaches and thus greatly limit FL’s applicability.

Original: In this paper, we propose a novel Split-Mix FL strategy for heterogeneous participants that, once training is done, provides in-situ customization of model sizes and robustness.
Revised: In this paper, we propose a novel Split-Mix FL strategy for heterogeneous participants that, once training is done, provides in-situ customization of model sizes and robustness.

Original: Specifically, we achieve customization by learning a set of base sub-networks of different sizes and robustness levels, which are later aggregated on-demand according to inference requirements.
Revised: Specifically, we achieve customization by learning a set of base sub-networks of different sizes and robustness levels, which are later aggregated on-demand according to inference requirements.

Original: This split-mix strategy achieves customization with high efficiency in communication, storage, and inference.
Revised: This split-mix strategy achieves customization with high efficiency in communication, storage, and inference.

Original: Extensive experiments demonstrate that our method provides better in-situ customization than the existing heterogeneous-architecture FL methods.
Revised: Extensive experiments demonstrate that our method provides better in-situ customization than the existing heterogeneous-architecture FL methods.

Original: Codes and pre-trained models are available: https://github.com/illidanlab/SplitMix .
Revised: Codes and pre-trained models are available: https://github.com/illidanlab/SplitMix .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Federated learning (FL) (Kone ˇ cný et al., 2015) is a distributed learning paradigm that leverages data from remote participants and aggregates their knowledge without requiring their raw data to be transferred to a central server, thereby largely reducing the concerns from data security and privacy.
Revised: Federated learning (FL) (Kone ˇ cný et al., 2015) is a distributed learning paradigm that leverages data from remote participants and aggregates their knowledge without requiring their raw data to be transferred to a central server, thereby largely reducing the concerns from data security and privacy.

Original: FedAvg (McMahan et al., 2017) is among the most popular federated instantiations, which aggregates knowledge by averaging models uploaded from different participants.
Revised: FedAvg (McMahan et al., 2017) is among the most popular federated instantiations, which aggregates knowledge by averaging models uploaded from different participants.

Original: When deploying federated learning, one challenge in real-world applications is the run-time (i.e., test-time) dynamics :
Revised: When deploying federated learning, one challenge in real-world applications is the run-time (i.e., test-time) dynamics :

Original: The requirements on model properties (e.g., inference efficiency, robustness, etc.) can be constantly changing during the run-time, depending on the status of the devices or the outside environment.
Revised: The requirements on model properties (e.g., inference efficiency, robustness, etc.) can be constantly changing during the run-time, depending on the status of the devices or the outside environment.

Original: One common and specific type of dynamics is resource dynamics :
Revised: One common and specific type of dynamics is resource dynamics :

Original: For each application, the allocated on-device resources (e.g., run-time memory, CPU bandwidth, etc.) may vary drastically during run-time, depending on how the resource allocation of the running programs are prioritized on a participant’s device (Xu et al., 2021).
Revised: For each application, the allocated on-device resources (e.g., run-time memory, CPU bandwidth, etc.) may vary drastically during run-time, depending on how the resource allocation of the running programs are prioritized on a participant’s device (Xu et al., 2021).

Original: Another type of dynamics is the robustness dynamics :
Revised: Another type of dynamics is the robustness dynamics :

Original: The constantly changing outside environment can make different requirements on the safety (or robustness) level of the model (Wang et al., 2020).
Revised: The constantly changing outside environment can make different requirements on the safety (or robustness) level of the model (Wang et al., 2020).

Original: For instance, the quality of real-time videos captured by autonomous cars can suddenly degrade, e.g., on entering a poor-lighted alley or tunnel from a well-lighted avenue, on entering a section of bumpy road which leads to a sudden burst of blurring in the videos, etc.
Revised: For instance, the quality of real-time videos captured by autonomous cars can suddenly degrade, e.g., on entering a poor-lighted alley or tunnel from a well-lighted avenue, on entering a section of bumpy road which leads to a sudden burst of blurring in the videos, etc.

Original: In such cases, a more robust model should be quickly switch in and replace the one used on benign conditions, in order to prevent catastrophic accidents caused by wrong recognition under poor visual conditions.
Revised: In such cases, a more robust model should be quickly switch in and replace the one used on benign conditions, in order to prevent catastrophic accidents caused by wrong recognition under poor visual conditions.

Original: Such dynamic run-time requirements demand the flexibility to customize the model.
Revised: Such dynamic run-time requirements demand the flexibility to customize the model.

Original: However, as illustrated in Fig.
Revised: However, as illustrated in Fig.

Original: 1a, we show that conventional static-model
Revised: 1a, we show that conventional static-model

Original: FL methods, represented FedAvg, cannot provide such customization.
Revised: FL methods, represented FedAvg, cannot provide such customization.

Original: A naive solution is to train multiple models with different desired properties and keep them all on device.
Revised: A naive solution is to train multiple models with different desired properties and keep them all on device.

Original: However, this leads to extra training and storage costs proportional to the number of models.
Revised: However, this leads to extra training and storage costs proportional to the number of models.

Original: Moreover, since it is not practical to keep all models simultaneously in run-time memory on a resource-limited device, it also introduces inference overhead to swap the models into and out of the run-time memory (Dosovitskiy & Djolonga, 2019).
Revised: Moreover, since it is not practical to keep all models simultaneously in run-time memory on a resource-limited device, it also introduces inference overhead to swap the models into and out of the run-time memory (Dosovitskiy & Djolonga, 2019).

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: To effectively and efficiently train models for on-demand an in-situ customization, new challenges will be raised by the ubiquitous heterogeneity of federated learning participants.
Revised: To effectively and efficiently train models for on-demand an in-situ customization, new challenges will be raised by the ubiquitous heterogeneity of federated learning participants.

Original: Fist, the participants can have resource heterogeneity : Different participants have different hardware resources available, such as memory, computing power, and network bandwidth (Ignatov et al., 2018).
Revised: Fist, the participants can have resource heterogeneity : Different participants have different hardware resources available, such as memory, computing power, and network bandwidth (Ignatov et al., 2018).

Original: For example, in a learning task for face recognition, clients may use different types of devices (e.g., computers, tablets or smartphones) to participate in learning.
Revised: For example, in a learning task for face recognition, clients may use different types of devices (e.g., computers, tablets or smartphones) to participate in learning.

Original: To accommodate different hardware, one can turn to more resource-flexible architectures trained by distillation from ensemble (Lin et al., 2020), partial model averaging (Diao et al., 2021), or directly combining predictions (Shi et al., 2021).
Revised: To accommodate different hardware, one can turn to more resource-flexible architectures trained by distillation from ensemble (Lin et al., 2020), partial model averaging (Diao et al., 2021), or directly combining predictions (Shi et al., 2021).

Original: Specifically, (Diao et al., 2021) is the first heterogeneous-width solution (HeteroFL) allowing insitu model-size switching.
Revised: Specifically, (Diao et al., 2021) is the first heterogeneous-width solution (HeteroFL) allowing insitu model-size switching.

Original: Nevertheless, it suffers from under-training in its large models due to local budget constraints as shown in Fig.
Revised: Nevertheless, it suffers from under-training in its large models due to local budget constraints as shown in Fig.

Original: 1a.
Revised: 1a.

Original: The degradation could be worsened as facing data heterogeneity : The training datasets from participants are not independent and identically distributed (non- i.i.d. )
Revised: The degradation could be worsened as facing data heterogeneity : The training datasets from participants are not independent and identically distributed (non- i.i.d. )

Original: (Li et al., 2020b; Fallah et al., 2020; Hong et al., 2021c; Zhu et al., 2021).
Revised: (Li et al., 2020b; Fallah et al., 2020; Hong et al., 2021c; Zhu et al., 2021).

Original: When one device with a unique data distribution cannot afford training a large model, the global large model may not transfer to the unseen distribution (Pan & Yang, 2010).
Revised: When one device with a unique data distribution cannot afford training a large model, the global large model may not transfer to the unseen distribution (Pan & Yang, 2010).

Original: Thus, HeteroFL may not provide effective customization such that more parameters brings in higher accuracy and how to train an effectively customizable model still remains unknown.
Revised: Thus, HeteroFL may not provide effective customization such that more parameters brings in higher accuracy and how to train an effectively customizable model still remains unknown.

Original: To address the aforementioned challenges from heterogeneity and dynamics, we study a novel SplitMix approach to enable FL on heterogeneous devices and achieve in-situ model customization for resource efficiency and robustness: The size and robustness of the resultant model can be efficiently customized at run-time.
Revised: To address the aforementioned challenges from heterogeneity and dynamics, we study a novel SplitMix approach to enable FL on heterogeneous devices and achieve in-situ model customization for resource efficiency and robustness: The size and robustness of the resultant model can be efficiently customized at run-time.

Original: Specifically, we first split the complete knowledge in a large model into several small base sub-networks (shards) according to model widths and robustness levels.
Revised: Specifically, we first split the complete knowledge in a large model into several small base sub-networks (shards) according to model widths and robustness levels.

Original: To complete the knowledge, we let the base models be fully trained on all clients.
Revised: To complete the knowledge, we let the base models be fully trained on all clients.

Original: To provide customized models, we mix selected base models to construct the desired model size and robustness.
Revised: To provide customized models, we mix selected base models to construct the desired model size and robustness.

Original: We illustrate the main idea in Fig.
Revised: We illustrate the main idea in Fig.

Original: 1b.
Revised: 1b.

Original: Overall, our contributions can be summarized in three folds:
Revised: Overall, our contributions can be summarized in three folds:

Original: • Within the domain of heterogeneous federated learning, we are the first to study training a model with the capability of in-situ customization with heterogeneous local computation budgets, which cannot be resolved by existing methods yet as shown in Fig.
Revised: • Within the domain of heterogeneous federated learning, we are the first to study training a model with the capability of in-situ customization with heterogeneous local computation budgets, which cannot be resolved by existing methods yet as shown in Fig.

Original: 1a.
Revised: 1a.

Original: • To address the challenge, we propose a novel Split-Mix framework that aggregates knowledge from heterogeneous clients into a width- and robustness-adjustable model structure.
Revised: • To address the challenge, we propose a novel Split-Mix framework that aggregates knowledge from heterogeneous clients into a width- and robustness-adjustable model structure.

Original: Remarkably, due to fewer parameters and modular nature, our framework is not only efficient in federated communication and flexibly adaptable to various client budgets during training , but also efficient and flexible in storage, model loading and execution during inference .
Revised: Remarkably, due to fewer parameters and modular nature, our framework is not only efficient in federated communication and flexibly adaptable to various client budgets during training , but also efficient and flexible in storage, model loading and execution during inference .

Original: • Empirically, we demonstrate that the performance of the proposed method is better than other
Revised: • Empirically, we demonstrate that the performance of the proposed method is better than other

Original: FL baselines under heterogeneous budget constraints.
Revised: FL baselines under heterogeneous budget constraints.

Original: Moreover, we show its effectiveness when facing the challenge of data heterogeneity.
Revised: Moreover, we show its effectiveness when facing the challenge of data heterogeneity.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Heterogeneous Federated Learning.
Revised: Heterogeneous Federated Learning.

Original: As increasing concerns have been gained on data privacy leakage in machine learning (Dwork, 2008; Weiss & Archick, 2016; Wu et al., 2020; Hong et al., 2021b), federated learning (FL) protects data privacy by training the model locally on users’ own devices without sharing data.
Revised: As increasing concerns have been gained on data privacy leakage in machine learning (Dwork, 2008; Weiss & Archick, 2016; Wu et al., 2020; Hong et al., 2021b), federated learning (FL) protects data privacy by training the model locally on users’ own devices without sharing data.

Original: In real-world applications, FL with budget-insufficient devices (e.g., mobile devices) has attracted a great amount of attention.
Revised: In real-world applications, FL with budget-insufficient devices (e.g., mobile devices) has attracted a great amount of attention.

Original: For example, FedDistill (Lin et al., 2020) used logit averages to distill a network of the same size or several prototypes, which will be communicated with users.
Revised: For example, FedDistill (Lin et al., 2020) used logit averages to distill a network of the same size or several prototypes, which will be communicated with users.

Original: FedDistill made an assumption that the central server has access to a public dataset of the same learning task, which is impractical for many applications.
Revised: FedDistill made an assumption that the central server has access to a public dataset of the same learning task, which is impractical for many applications.

Original: He et al.
Revised: He et al.

Original: (2020) introduced a distillation-based method after aggregating private representations from all participants.
Revised: (2020) introduced a distillation-based method after aggregating private representations from all participants.

Original: The method closely resembles centralized learning because all encoded samples are gathered, and however it is less efficient when local clients have large data dimensions or sample sizes.
Revised: The method closely resembles centralized learning because all encoded samples are gathered, and however it is less efficient when local clients have large data dimensions or sample sizes.

Original: Importantly, the method may not transfer adversarial robustness knowledge through the intermediate representations due to the decoupling of the input and prediction.
Revised: Importantly, the method may not transfer adversarial robustness knowledge through the intermediate representations due to the decoupling of the input and prediction.

Original: On the other hand,
Revised: On the other hand,

Original: HeteroFL (Diao et al., 2021) avoids distillation, allowing participants to train different prototypes and sharing parameters among prototypes to reduce redundant parameters.
Revised: HeteroFL (Diao et al., 2021) avoids distillation, allowing participants to train different prototypes and sharing parameters among prototypes to reduce redundant parameters.

Original: However, HeteroFL also reduces the samples available for training each prototype, which leads to degraded performance.
Revised: However, HeteroFL also reduces the samples available for training each prototype, which leads to degraded performance.

Original: Considering the high cost of training robust models, Hong et al.
Revised: Considering the high cost of training robust models, Hong et al.

Original: (2021a) proposed an efficient way to transfer model robustness from budget-sufficient devices to insufficient ones.
Revised: (2021a) proposed an efficient way to transfer model robustness from budget-sufficient devices to insufficient ones.

Original: FedEnsemble (Shi et al., 2021) is technically related to the proposed approach, which uses ensemble of diverse models to accommodate non- i.i.d.
Revised: FedEnsemble (Shi et al., 2021) is technically related to the proposed approach, which uses ensemble of diverse models to accommodate non- i.i.d.

Original: heterogeneity.
Revised: heterogeneity.

Original: The authors showed that combining multiple base models trained simultaneously in FL can outperform a single base model in testing.
Revised: The authors showed that combining multiple base models trained simultaneously in FL can outperform a single base model in testing.

Original: A critical difference between the proposed approach and FedEnsemble comes from the challenging problem setting of constrained heterogeneous computation budgets.
Revised: A critical difference between the proposed approach and FedEnsemble comes from the challenging problem setting of constrained heterogeneous computation budgets.

Original: For the first time, we show that base models can be trained adaptively under budget constraints and used to customize efficient inference networks that can outperform a single model of the same width but trained in a heterogeneous-architecture way.
Revised: For the first time, we show that base models can be trained adaptively under budget constraints and used to customize efficient inference networks that can outperform a single model of the same width but trained in a heterogeneous-architecture way.

Original: Customizable Models.
Revised: Customizable Models.

Original: To our best knowledge, this is the first paper discussing in-situ customization in federated learning and here we review similar concepts in central learning.
Revised: To our best knowledge, this is the first paper discussing in-situ customization in federated learning and here we review similar concepts in central learning.

Original: First, customization of robustness and accuracy was discussed by Wang et al.
Revised: First, customization of robustness and accuracy was discussed by Wang et al.

Original: (2020), where an adjustable-conditional layer together with decoupled batch-normalization were used.
Revised: (2020), where an adjustable-conditional layer together with decoupled batch-normalization were used.

Original: The conditional layer enables continuous trade-off but also brings in more parameters.
Revised: The conditional layer enables continuous trade-off but also brings in more parameters.

Original: In comparison, a simple weighted combination without additional parameters is used in our method and is very efficient in both communication and inference.
Revised: In comparison, a simple weighted combination without additional parameters is used in our method and is very efficient in both communication and inference.

Original: In terms of model complexity, a series of work on dynamic neural networks were proposed to provide instant, adaptive and efficient trade-off between accuracy and latency (mainly related to model complexity) of neural networks at the inference stage.
Revised: In terms of model complexity, a series of work on dynamic neural networks were proposed to provide instant, adaptive and efficient trade-off between accuracy and latency (mainly related to model complexity) of neural networks at the inference stage.

Original: Typically, sub-path networks of different complexity (Liu & Deng, 2018) or sub-depth networks (Huang et al., 2018; Wu et al., 2019; Wang et al., 2018; Zhang et al., 2021) are trained together.
Revised: Typically, sub-path networks of different complexity (Liu & Deng, 2018) or sub-depth networks (Huang et al., 2018; Wu et al., 2019; Wang et al., 2018; Zhang et al., 2021) are trained together.

Original: However, due to the large memory footprint brought by a constant number of channels per layer, the memory footprint at inference is barely reduced.
Revised: However, due to the large memory footprint brought by a constant number of channels per layer, the memory footprint at inference is barely reduced.

Original: To address the challenge, slimmable neural network (SNN) (Yu et al., 2018) was proposed to train networks with adaptive layer widths.
Revised: To address the challenge, slimmable neural network (SNN) (Yu et al., 2018) was proposed to train networks with adaptive layer widths.

Original: Distinct from SNN, we consider a more challenging scenario with distributed and non-sharable data and heterogeneous client capabilities.
Revised: Distinct from SNN, we consider a more challenging scenario with distributed and non-sharable data and heterogeneous client capabilities.

Original:  P ROBLEM S ETTING
Revised:  P ROBLEM S ETTING

Original: The goal of this work is to develop a heterogeneous federated learning (FL) strategy, that yields a global model, which can be efficiently customized for dynamic needs.
Revised: The goal of this work is to develop a heterogeneous federated learning (FL) strategy, that yields a global model, which can be efficiently customized for dynamic needs.

Original: Formally, FL minimizes the objective
Revised: Formally, FL minimizes the objective

Original: [Equation]
Revised: [Equation]

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: f
Revised: f

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: W ) , y ) , where L is loss function (e.g., cross-entropy loss), f is a model parameterized by W , and { D k } Kk =1 are the training datasets on K participants with the image-label pairs ( x, y ) .
Revised: W ) , y ) , where L is loss function (e.g., cross-entropy loss), f is a model parameterized by W , and { D k } Kk =1 are the training datasets on K participants with the image-label pairs ( x, y ) .

Original: Following the standard FL setting (McMahan et al., 2017), only model parameters can be shared to protect privacy.
Revised: Following the standard FL setting (McMahan et al., 2017), only model parameters can be shared to protect privacy.

Original: We require efficient run-time customization on model f for resource dynamics (Section 4.2) and robustness dynamics (Section 4.3).
Revised: We require efficient run-time customization on model f for resource dynamics (Section 4.2) and robustness dynamics (Section 4.3).

Original: When training customizable/adjustable models, significant challenges arise from heterogeneity among clients.
Revised: When training customizable/adjustable models, significant challenges arise from heterogeneity among clients.

Original: In this paper, we consider the following: 1) Heterogeneous computational budgets during training of clients constrain the maximal complexity of local models.
Revised: In this paper, we consider the following: 1) Heterogeneous computational budgets during training of clients constrain the maximal complexity of local models.

Original: The complexity of deep neural networks can be defined in multiple dimensions, like depths or widths (Zagoruyko & Komodakis, 2017; Tan & Le, 2019).
Revised: The complexity of deep neural networks can be defined in multiple dimensions, like depths or widths (Zagoruyko & Komodakis, 2017; Tan & Le, 2019).

Original: In this paper, we consider the width of hidden channels, because it can significantly reduce not only the number of model parameters but also the layer output cache.
Revised: In this paper, we consider the width of hidden channels, because it can significantly reduce not only the number of model parameters but also the layer output cache.

Original: Thus, we assume clients have confined width capabilities { R k ∈ (0 , 1] } Kk =1 , defined by width ratios w.r.t.
Revised: Thus, we assume clients have confined width capabilities { R k ∈ (0 , 1] } Kk =1 , defined by width ratios w.r.t.

Original: the original model, i.e., × R k net as presented in Fig.
Revised: the original model, i.e., × R k net as presented in Fig.

Original: 1b.
Revised: 1b.

Original: Similar to FedAvg and HeteroFL, the same architecture is used by clients and therefore the model width can be tailored according to local budgets.
Revised: Similar to FedAvg and HeteroFL, the same architecture is used by clients and therefore the model width can be tailored according to local budgets.

Original: In many applications, there are usually a significant number of devices with insufficient computational budgets (Enge, 2021).
Revised: In many applications, there are usually a significant number of devices with insufficient computational budgets (Enge, 2021).

Original: For example, we may assume exponentially distributed budgets in uniformly divided client groups: R k = (1 / 2) ⌈ 4 k/K ⌉ , i.e., the first group with 1 / 4 clients is capable for a 1 -width and the rest are for 0 . 5 , 0 .
Revised: For example, we may assume exponentially distributed budgets in uniformly divided client groups: R k = (1 / 2) ⌈ 4 k/K ⌉ , i.e., the first group with 1 / 4 clients is capable for a 1 -width and the rest are for 0 . 5 , 0 .

Original: 125 -widths respectively.
Revised: 125 -widths respectively.

Original: The budget distribution simulate a real scenario where most federated mobile phones prefer a low-energy solution with smaller models for training probably in the background and maintain more resources for major tasks.
Revised: The budget distribution simulate a real scenario where most federated mobile phones prefer a low-energy solution with smaller models for training probably in the background and maintain more resources for major tasks.

Original: Other budget distributions, e.g., log-normal distributions, are discussed in Appendix A.5.
Revised: Other budget distributions, e.g., log-normal distributions, are discussed in Appendix A.5.

Original: Heterogeneous data distributions , e.g., non- i.i.d.
Revised: Heterogeneous data distributions , e.g., non- i.i.d.

Original: features, D k ∼ D i for ordered domain i , induces additional challenges with a skewed budget distribution, since training a single model on multiple domains (Li et al., 2020b) or models in a single domain (due to budget constraints)
Revised: features, D k ∼ D i for ordered domain i , induces additional challenges with a skewed budget distribution, since training a single model on multiple domains (Li et al., 2020b) or models in a single domain (due to budget constraints)

Original: (Hong et al., 2021c; Dong et al., 2021) are known to be suboptimal on transfer performance.
Revised: (Hong et al., 2021c; Dong et al., 2021) are known to be suboptimal on transfer performance.

Original: In summary, training various model sizes has challenges from 1) resource heterogeneity, where disparity in computation budgets will significantly affect the training of large models because they can only be trained on scarce budget-sufficient clients; 2) data heterogeneity, where the under-trained large models may perform poorly on unseen in-domain or out-of-domain samples.
Revised: In summary, training various model sizes has challenges from 1) resource heterogeneity, where disparity in computation budgets will significantly affect the training of large models because they can only be trained on scarce budget-sufficient clients; 2) data heterogeneity, where the under-trained large models may perform poorly on unseen in-domain or out-of-domain samples.

Original:  M ETHOD
Revised:  M ETHOD

Original: To provide efficient in-situ customization, we introduce a simple yet powerful principle, Split-Mix: shatter complete knowledge into smaller pieces, and customize by flexible formations of pieces.
Revised: To provide efficient in-situ customization, we introduce a simple yet powerful principle, Split-Mix: shatter complete knowledge into smaller pieces, and customize by flexible formations of pieces.

Original: Based on the principle, we propose customizations of model size and robustness in the following.
Revised: Based on the principle, we propose customizations of model size and robustness in the following.

Original: 4. C ASE S TUDY : C USTOMIZABLE N ETWORKS FROM B UDGET -C ONSTRAINED FL
Revised: 4. C ASE S TUDY : C USTOMIZABLE N ETWORKS FROM B UDGET -C ONSTRAINED FL

Original: For motivation, we set up a standard non- i.i.d.
Revised: For motivation, we set up a standard non- i.i.d.

Original: FL benchmark (Li et al., 2020b) using DomainNet dataset (refer to experimental details in Section 5) and study how the budget constraint impedes effective training of customizable models.
Revised: FL benchmark (Li et al., 2020b) using DomainNet dataset (refer to experimental details in Section 5) and study how the budget constraint impedes effective training of customizable models.

Original: [Figure]
Revised: [Figure]

Original: Case 1: FL without budget constraint.
Revised: Case 1: FL without budget constraint.

Original: First, we individually train networks of different widths by FedAvg.
Revised: First, we individually train networks of different widths by FedAvg.

Original: We see that the slimmer networks converge slower and are less generalizable, even though they are trained on all clients (solid lines in Fig. 2).
Revised: We see that the slimmer networks converge slower and are less generalizable, even though they are trained on all clients (solid lines in Fig. 2).

Original: The results justified the motivation of training wider networks than slimmer ones.
Revised: The results justified the motivation of training wider networks than slimmer ones.

Original: Case 2: FL with budget constraint.
Revised: Case 2: FL with budget constraint.

Original: Following the above budget constraint, i.e., R k = (1 / 2) ⌈ 4 k/K ⌉ , we deploy HeteroFL to train budget-compatible prototype models locally.
Revised: Following the above budget constraint, i.e., R k = (1 / 2) ⌈ 4 k/K ⌉ , we deploy HeteroFL to train budget-compatible prototype models locally.

Original: Since
Revised: Since

Original: HeteroFL was not designed for model customization, data are not fully used: each model prototype is only trained by 1 /of clients, when clients of the R k = 1 budget can actually afford all slimmer models.
Revised: HeteroFL was not designed for model customization, data are not fully used: each model prototype is only trained by 1 /of clients, when clients of the R k = 1 budget can actually afford all slimmer models.

Original: Therefore, we extend HeteroFL to a slimmable version (SHeteroFL) by training all affordable prototypes locally.
Revised: Therefore, we extend HeteroFL to a slimmable version (SHeteroFL) by training all affordable prototypes locally.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2, wider models (e.g., × 1 net) converge to validation accuracy lower than not only the FedAvg counterparts, but also the slimmer × 0 .
Revised: 2, wider models (e.g., × 1 net) converge to validation accuracy lower than not only the FedAvg counterparts, but also the slimmer × 0 .

Original: 25 net, showing that the widest model may not be a good candidate model.
Revised: 25 net, showing that the widest model may not be a good candidate model.

Original: Therefore, switching models to wider configurations lowers efficiency but does not improve accuracy, which is not a valid customization.
Revised: Therefore, switching models to wider configurations lowers efficiency but does not improve accuracy, which is not a valid customization.

Original: From the perspective of data allocation, it is not surprising that SHeteroFL exhibits a non-monotonous relation between the model width and accuracy.
Revised: From the perspective of data allocation, it is not surprising that SHeteroFL exhibits a non-monotonous relation between the model width and accuracy.

Original: For × 1 nets, only 1 / 4 clients and data are accessible for training.
Revised: For × 1 nets, only 1 / 4 clients and data are accessible for training.

Original: In comparison, 3 / 4 data are accessible by × 0 .
Revised: In comparison, 3 / 4 data are accessible by × 0 .

Original: 25 nets and the more data empowers × 0 .
Revised: 25 nets and the more data empowers × 0 .

Original: 25 nets better generalization ability than × 1 nets.
Revised: 25 nets better generalization ability than × 1 nets.

Original: 4. C USTOMIZE M ODEL S IZE
Revised: 4. C USTOMIZE M ODEL S IZE

Original: Motivated by the above observations, we propose to increase accessible training data by splitting wide networks into universally-budget-compatible sub-networks and re-mix afterward.
Revised: Motivated by the above observations, we propose to increase accessible training data by splitting wide networks into universally-budget-compatible sub-networks and re-mix afterward.

Original: The overall
Revised: The overall

Original: FL algorithm is summarized in Algorithm 1.
Revised: FL algorithm is summarized in Algorithm 1.

Original: More accessible data by splitting wide networks.
Revised: More accessible data by splitting wide networks.

Original: Since a wide network cannot fit into budgetinsufficient clients, we split it into budget-compatible sub-networks by channels (width) while maintaining the the total width.
Revised: Since a wide network cannot fit into budgetinsufficient clients, we split it into budget-compatible sub-networks by channels (width) while maintaining the the total width.

Original: In terms of memory limitations, each sub-network can be painlessly and individually trained in all clients.
Revised: In terms of memory limitations, each sub-network can be painlessly and individually trained in all clients.

Original: However, sequentially training multiple slim base networks could be much slower than training a single integrated one and increases blocking time on communication.
Revised: However, sequentially training multiple slim base networks could be much slower than training a single integrated one and increases blocking time on communication.

Original: Noticing that all base models can be evaluated independently, we instead efficiently train ×
Revised: Noticing that all base models can be evaluated independently, we instead efficiently train ×

Original: r base models in parallel (see Algorithm 3).
Revised: r base models in parallel (see Algorithm 3).

Original: Despite the benefit, a client’s budget constraint ( R ) will limit the number of paralleled base models within ⌊ R/r ⌋ , as wider channels result in larger intermediate layer cache (activations), and excludes the rest base models from the client.
Revised: Despite the benefit, a client’s budget constraint ( R ) will limit the number of paralleled base models within ⌊ R/r ⌋ , as wider channels result in larger intermediate layer cache (activations), and excludes the rest base models from the client.

Original: Fortunately, we can select different sets of base models for a budget-limited client per round, which is inspired by FedEnsemble (Shi et al., 2021) and is presented in Algorithm 2.
Revised: Fortunately, we can select different sets of base models for a budget-limited client per round, which is inspired by FedEnsemble (Shi et al., 2021) and is presented in Algorithm 2.

Original: Hence, all base models can be ever trained on the client for multiple communication rounds, though not continuously every round.
Revised: Hence, all base models can be ever trained on the client for multiple communication rounds, though not continuously every round.

Original: Note that since the federated training processes of base models are independent without interference, the training could be as stable as FedAvg with partial participants.
Revised: Note that since the federated training processes of base models are independent without interference, the training could be as stable as FedAvg with partial participants.

Original: In addition, the combination of slimmer base models is flexible and can conform a variety of client budgets.
Revised: In addition, the combination of slimmer base models is flexible and can conform a variety of client budgets.

Original: Boost accuracy by mixture of subnet experts.
Revised: Boost accuracy by mixture of subnet experts.

Original: To craft a wide model, we combine the outputs of multiple × r base models until the size of the ensemble reaches the same as the number of channels, e.g., ⌊ R/r ⌋ bases for an × R net.
Revised: To craft a wide model, we combine the outputs of multiple × r base models until the size of the ensemble reaches the same as the number of channels, e.g., ⌊ R/r ⌋ bases for an × R net.

Original: We randomly initialize base models independently such that the diverse bases could extract different features from the same image (Allen-Zhu et al., 2020).
Revised: We randomly initialize base models independently such that the diverse bases could extract different features from the same image (Allen-Zhu et al., 2020).

Original: Therefore, the ensemble can predict based on a variety of features, resembling an integrated wide network.
Revised: Therefore, the ensemble can predict based on a variety of features, resembling an integrated wide network.

Original: We follow the common practice, Kaiming’s method (He et al., 2015), for initializing base networks with
Revised: We follow the common practice, Kaiming’s method (He et al., 2015), for initializing base networks with

Original: ReLU layers.
Revised: ReLU layers.

Original: As Kaiming’s method is width-dependent, we parameterize the initialization based on the width of × 1 net instead of the × r one, which leads to smaller initial convolutional weights.
Revised: As Kaiming’s method is width-dependent, we parameterize the initialization based on the width of × 1 net instead of the × r one, which leads to smaller initial convolutional weights.

Original: In
Revised: In

Original: Appendix A.3, intensive ablation studies show that the rescaled initialization is critical for improving test accuracy of wider networks.
Revised: Appendix A.3, intensive ablation studies show that the rescaled initialization is critical for improving test accuracy of wider networks.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: 4. E XTENSION TO A DVERSARIAL R OBUSTNESS C USTOMIZATION
Revised: 4. E XTENSION TO A DVERSARIAL R OBUSTNESS C USTOMIZATION

Original: In this section, we extend the customization from one dimension to two dimensions, by jointly customizing model size and model robustness under adversarial attacks (Goodfellow et al., 2014).
Revised: In this section, we extend the customization from one dimension to two dimensions, by jointly customizing model size and model robustness under adversarial attacks (Goodfellow et al., 2014).

Original: Model robustness has gained increasing interest (Hendrycks & Dietterich, 2018;
Revised: Model robustness has gained increasing interest (Hendrycks & Dietterich, 2018;

Original: Wang et al., 2021), especially in high-stakes federated learning applications (Hong et al., 2021a).
Revised: Wang et al., 2021), especially in high-stakes federated learning applications (Hong et al., 2021a).

Original: Adversarial training (AT) (Madry et al., 2018) is arguably the most popular and effective defense strategy against adversarial attacks.
Revised: Adversarial training (AT) (Madry et al., 2018) is arguably the most popular and effective defense strategy against adversarial attacks.

Original: Specifically, it uses on-the-fly adversarial samples as augmentation to improve robustness.
Revised: Specifically, it uses on-the-fly adversarial samples as augmentation to improve robustness.

Original: Formally, AT minimizes the following augmented loss:
Revised: Formally, AT minimizes the following augmented loss:

Original: [Equation]
Revised: [Equation]

Original: where δ is a subtle ϵ -constrained adversarial noise and transfers a clean sample x into an adversarial sample x + δ .
Revised: where δ is a subtle ϵ -constrained adversarial noise and transfers a clean sample x into an adversarial sample x + δ .

Original: In Eq.
Revised: In Eq.

Original: (1), L CE is the cross-entropy loss the hyper-parameter and λ n trades off accuracy (the 1st term) and robustness (the 2nd term).
Revised: (1), L CE is the cross-entropy loss the hyper-parameter and λ n trades off accuracy (the 1st term) and robustness (the 2nd term).

Original: When λ n = 0 or 1 , the optimization yields a standard-training (ST) model or an AT model, respectively.
Revised: When λ n = 0 or 1 , the optimization yields a standard-training (ST) model or an AT model, respectively.

Original: Since an AT model is commonly less accurate in predicting standard images (Tsipras et al., 2019), there is usually no such a sweet point of λ n simultaneously maximizing robustness and accuracy, and one typically needs to carefully gauge the trade-off according to the demand of robustness in specific application context.
Revised: Since an AT model is commonly less accurate in predicting standard images (Tsipras et al., 2019), there is usually no such a sweet point of λ n simultaneously maximizing robustness and accuracy, and one typically needs to carefully gauge the trade-off according to the demand of robustness in specific application context.

Original: [Figure]
Revised: [Figure]

Original: Splitting and sharing parameters.
Revised: Splitting and sharing parameters.

Original: Since standard performance and adversarial robustness are irreconcilable, we can directly use two separated ST and AT models to maximally capture the each property.
Revised: Since standard performance and adversarial robustness are irreconcilable, we can directly use two separated ST and AT models to maximally capture the each property.

Original: But do we really need two totally separated models?
Revised: But do we really need two totally separated models?

Original: Intuitively, the two models share some common knowledge, given that an adversarial image share a large part of common features with its original version.
Revised: Intuitively, the two models share some common knowledge, given that an adversarial image share a large part of common features with its original version.

Original: As introduced by (Xie & Yuille, 2019), sharing all parameters except the batch-normalization (BN) layers can maximize robustness and accuracy by expertised BNs, respectively.
Revised: As introduced by (Xie & Yuille, 2019), sharing all parameters except the batch-normalization (BN) layers can maximize robustness and accuracy by expertised BNs, respectively.

Original: Accordingly, we propose to split BN layers (instead of the whole model) into two sub-components: one for standard performance and the other for robustness.
Revised: Accordingly, we propose to split BN layers (instead of the whole model) into two sub-components: one for standard performance and the other for robustness.

Original: At training time, the first loss of Eq.
Revised: At training time, the first loss of Eq.

Original: (1) is computed with clean BN (BN c ) merely and the second adversarial loss is computed with noised BN (BN n ).
Revised: (1) is computed with clean BN (BN c ) merely and the second adversarial loss is computed with noised BN (BN n ).

Original: The FL local training is elaborated in Algorithm 4.
Revised: The FL local training is elaborated in Algorithm 4.

Original: As the two BNs are decoupled, there is no more trade-off in loss Eq.
Revised: As the two BNs are decoupled, there is no more trade-off in loss Eq.

Original: and thereby we choose λ n = 0 .
Revised: and thereby we choose λ n = 0 .

Original: 5 to balance their effects on parameter updates.
Revised: 5 to balance their effects on parameter updates.

Original: Customizable layer-wise mixing.
Revised: Customizable layer-wise mixing.

Original: After training, the problem is how to mix the two models (with different BNs) for prediction.
Revised: After training, the problem is how to mix the two models (with different BNs) for prediction.

Original: A straightforward solution is averaging their outputs.
Revised: A straightforward solution is averaging their outputs.

Original: However, forwarding memory footprint will be doubled in this way, as the two models have to be both executed separately.
Revised: However, forwarding memory footprint will be doubled in this way, as the two models have to be both executed separately.

Original: To avoid the doubled intermediate outputs, a gate function can be used to adaptively choose BNs like (Liu et al., 2020).
Revised: To avoid the doubled intermediate outputs, a gate function can be used to adaptively choose BNs like (Liu et al., 2020).

Original: Inspired by the method, we further propose a simple parameter-free method by weighted-averaging outputs of each BN layer (see Fig. 3):
Revised: Inspired by the method, we further propose a simple parameter-free method by weighted-averaging outputs of each BN layer (see Fig. 3):

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: given a BN-layer input x .
Revised: given a BN-layer input x .

Original: We note that the averaging strategy is entirely training-free and does not use extra parameters, and the customization weight λ is intuitive for trade-offs between SA and RA.
Revised: We note that the averaging strategy is entirely training-free and does not use extra parameters, and the customization weight λ is intuitive for trade-offs between SA and RA.

Original: Lastly, it is remarkable that the DBN structure is rather lightweight in terms of model complexity.
Revised: Lastly, it is remarkable that the DBN structure is rather lightweight in terms of model complexity.

Original: As investigated in (Yu et al., 2018)
Revised: As investigated in (Yu et al., 2018)

Original: (Table 2), the parameters of BN is no more than 1% in popular deep architectures, e.g., ResNet or MobileNet.
Revised: (Table 2), the parameters of BN is no more than 1% in popular deep architectures, e.g., ResNet or MobileNet.

Original: Therefore, we can plug DBN into base models in place of
Revised: Therefore, we can plug DBN into base models in place of

Original: BN, replace Algorithm 3 with Algorithm 4, and jointly customize robustness and model widths.
Revised: BN, replace Algorithm 3 with Algorithm 4, and jointly customize robustness and model widths.

Original:  E MPIRICAL S TUDIES
Revised:  E MPIRICAL S TUDIES

Original: We design experiments to compare the proposed method against FL classification benchmarks.
Revised: We design experiments to compare the proposed method against FL classification benchmarks.

Original: For class non-
Revised: For class non-

Original: i.i.d configuration, we use CIFAR10 dataset (Krizhevsky, 2009) with preactivated ResNet (PreResNet18)
Revised: i.i.d configuration, we use CIFAR10 dataset (Krizhevsky, 2009) with preactivated ResNet (PreResNet18)

Original: (He et al., 2016).
Revised: (He et al., 2016).

Original: CIFAR10 contains over 50 , 000 32 × 32 images of 10 classes.
Revised: CIFAR10 contains over 50 , 000 32 × 32 images of 10 classes.

Original: The
Revised: The

Original: CIFAR10 data are uniformly split into 100 clients and distribute 3 classes per client.
Revised: CIFAR10 data are uniformly split into 100 clients and distribute 3 classes per client.

Original: For (feature) non- i.i.d .
Revised: For (feature) non- i.i.d .

Original: configuration, we use Digits with a CNN defined and DomainNet datasets (Li et al., 2020b) with AlexNet extended with BN layers after each convolutional or linear layer (Li et al., 2020b).
Revised: configuration, we use Digits with a CNN defined and DomainNet datasets (Li et al., 2020b) with AlexNet extended with BN layers after each convolutional or linear layer (Li et al., 2020b).

Original: The first dataset is a subset (30%) of Digits, a benchmark for domain adaption (Peng et al., 2019b).
Revised: The first dataset is a subset (30%) of Digits, a benchmark for domain adaption (Peng et al., 2019b).

Original: Digits has 28 × 28 images and serves as a commonly used benchmark for FL (Caldas et al., 2019;
Revised: Digits has 28 × 28 images and serves as a commonly used benchmark for FL (Caldas et al., 2019;

Original: McMahan et al., 2017; Li et al., 2020a).
Revised: McMahan et al., 2017; Li et al., 2020a).

Original: The dataset includes 5 different domains: MNIST (Lecun et al., 1998), SVHN (Netzer et al., 2011), USPS (Hull, 1994), SynthDigits (Ganin & Lempitsky, 2015), and MNIST-M (Ganin & Lempitsky, 2015).
Revised: The dataset includes 5 different domains: MNIST (Lecun et al., 1998), SVHN (Netzer et al., 2011), USPS (Hull, 1994), SynthDigits (Ganin & Lempitsky, 2015), and MNIST-M (Ganin & Lempitsky, 2015).

Original: The second dataset is DomainNet (Peng et al., 2019a) processed by (Li et al., 2020b), which contains 6 distinct domains of large-size 256 ×real-world images: Clipart, Infograph, Painting, Quickdraw, Real, Sketch.
Revised: The second dataset is DomainNet (Peng et al., 2019a) processed by (Li et al., 2020b), which contains 6 distinct domains of large-size 256 ×real-world images: Clipart, Infograph, Painting, Quickdraw, Real, Sketch.

Original: Each domain of Digits (or
Revised: Each domain of Digits (or

Original: DomainNet) are split into 10 (or 5) clients, and therefore 50 (or 30) clients in total.
Revised: DomainNet) are split into 10 (or 5) clients, and therefore 50 (or 30) clients in total.

Original: We defer other details such as hyper-parameters to Appendix A, and focus on discussing the results.
Revised: We defer other details such as hyper-parameters to Appendix A, and focus on discussing the results.

Original: 5. C USTOMIZE MODEL SIZES
Revised: 5. C USTOMIZE MODEL SIZES

Original: In this section, we evaluate the proposed Split-Mix on tasks of customizing model sizes through adjusting model widths.
Revised: In this section, we evaluate the proposed Split-Mix on tasks of customizing model sizes through adjusting model widths.

Original: Recall that in Section 3, we assume a specific heterogeneous training budget to facilitate our discussion, such that one client can only train models within a maximal width, and the resource distribution is imbalance among clients:
Revised: Recall that in Section 3, we assume a specific heterogeneous training budget to facilitate our discussion, such that one client can only train models within a maximal width, and the resource distribution is imbalance among clients:

Original: R k = (1 / 2) ⌈ 4 k/K ⌉ .
Revised: R k = (1 / 2) ⌈ 4 k/K ⌉ .

Original: Following the common FL setting, we do not consider algorithms that use public data or representation sharing in our baselines.
Revised: Following the common FL setting, we do not consider algorithms that use public data or representation sharing in our baselines.

Original: Baselines .
Revised: Baselines .

Original: As an ideal upper bound but a memory-incompatible baseline, we (re-)train networks from scratch by FedAvg to obtain individual models with different widths.
Revised: As an ideal upper bound but a memory-incompatible baseline, we (re-)train networks from scratch by FedAvg to obtain individual models with different widths.

Original: The state-of-the-art heterogeneous-architecture FL method is HeteroFL (Diao et al., 2021), which trains different slim models in different clients.
Revised: The state-of-the-art heterogeneous-architecture FL method is HeteroFL (Diao et al., 2021), which trains different slim models in different clients.

Original: For a fair comparison, we extend HeteroFL with bounded-slimmable training in clients who can afford the larger models, named SHeteroFL .
Revised: For a fair comparison, we extend HeteroFL with bounded-slimmable training in clients who can afford the larger models, named SHeteroFL .

Original: For example, if a client in
Revised: For example, if a client in

Original: HeteroFL can afford × 0 .
Revised: HeteroFL can afford × 0 .

Original: 5 net, then the client meanwhile trains × 0 .
Revised: 5 net, then the client meanwhile trains × 0 .

Original: 25 net and other smaller subnets in slimmable training manner (Yu et al., 2018) by SHeteroFL.
Revised: 25 net and other smaller subnets in slimmable training manner (Yu et al., 2018) by SHeteroFL.

Original: [Figure]
Revised: [Figure]

Original: Convergence.
Revised: Convergence.

Original: In Fig.
Revised: In Fig.

Original: 4, we compare the convergence curves of full-width models on three datasets.
Revised: 4, we compare the convergence curves of full-width models on three datasets.

Original: For FedAvg, the budget-compatible width is × 0 .
Revised: For FedAvg, the budget-compatible width is × 0 .

Original: Both for SHeteroFL and Split-Mix, the fullwidth is × 1 .
Revised: Both for SHeteroFL and Split-Mix, the fullwidth is × 1 .

Original: Compared to baselines, the proposed Split-Mix converges faster, largely due to the splitting strategy.
Revised: Compared to baselines, the proposed Split-Mix converges faster, largely due to the splitting strategy.

Original: Specifically, because all base models are independently trained, the convergence of each base model mainly depends on how frequently they are trained.
Revised: Specifically, because all base models are independently trained, the convergence of each base model mainly depends on how frequently they are trained.

Original: When enough clients participate
Revised: When enough clients participate

Original: FL, the training frequency of the × 1 net is more than one, as all bases will be selected at least once in a communication round.
Revised: FL, the training frequency of the × 1 net is more than one, as all bases will be selected at least once in a communication round.

Original: Performance.
Revised: Performance.

Original: In Table 1, we compare the test accuracy of different model widths.
Revised: In Table 1, we compare the test accuracy of different model widths.

Original: We measure the latency in terms of MACs (number of multiplication-and-addition operations) and model size in parameter numbers.
Revised: We measure the latency in terms of MACs (number of multiplication-and-addition operations) and model size in parameter numbers.

Original: With the same width, our method outperforms SHeteroFL, using much fewer parameters and thus conducts inference in much lower latency.
Revised: With the same width, our method outperforms SHeteroFL, using much fewer parameters and thus conducts inference in much lower latency.

Original: Remarkably, compared to the best model by SHeteroFL, e.g., 81 . 8% × 1 net in CIFAR10, Split-Mix uses only 1.8% parameters and
Revised: Remarkably, compared to the best model by SHeteroFL, e.g., 81 . 8% × 1 net in CIFAR10, Split-Mix uses only 1.8% parameters and

Original: [Table]
Revised: [Table]

Original: 1.6% MACs ( × 0 .
Revised: 1.6% MACs ( × 0 .

Original: 125 net) to reach a similar level of test accuracy.
Revised: 125 net) to reach a similar level of test accuracy.

Original: We notice that SHeteroFL has a much lower test accuracy with × 0 .
Revised: We notice that SHeteroFL has a much lower test accuracy with × 0 .

Original: 125 net on the CIFAR10 dataset.
Revised: 125 net on the CIFAR10 dataset.

Original: By investigating the loss curves, we find that the inference between parameter-shared different prototypes results in unstable convergence of the × 0 .
Revised: By investigating the loss curves, we find that the inference between parameter-shared different prototypes results in unstable convergence of the × 0 .

Original: 125 net.
Revised: 125 net.

Original: Attributed to the independent splitting, our method is more stable on convergence with different widths.
Revised: Attributed to the independent splitting, our method is more stable on convergence with different widths.

Original: Remarkably, Split-Mix only requires 12.7% parameters and 19.8% MACs (by × 1 ensemble) to achieve the comparable accuracy as the × 1 individual network on Digits.
Revised: Remarkably, Split-Mix only requires 12.7% parameters and 19.8% MACs (by × 1 ensemble) to achieve the comparable accuracy as the × 1 individual network on Digits.

Original: Results on CIFAR10 and DomainNet show a potential limitation of our method.
Revised: Results on CIFAR10 and DomainNet show a potential limitation of our method.

Original: The accuracy of Split-Mix is not comparable with the wider individual models due to the locally limited complexity.
Revised: The accuracy of Split-Mix is not comparable with the wider individual models due to the locally limited complexity.

Original: However, the limitation is more from the problem setting itself, and will not undermine our advantage in budget-limited FL, since wide individual models cannot be trained in this case.
Revised: However, the limitation is more from the problem setting itself, and will not undermine our advantage in budget-limited FL, since wide individual models cannot be trained in this case.

Original: [Figure]
Revised: [Figure]

Original: Client-wise evaluation.
Revised: Client-wise evaluation.

Original: In addition to comparisons of inference on average, we also demonstrate the statistics of test accuracy and the training and communication efficiency in Fig. 5.
Revised: In addition to comparisons of inference on average, we also demonstrate the statistics of test accuracy and the training and communication efficiency in Fig. 5.

Original: Conforming the budget constraints, our method outputs more accurate full-width models, transfer fewer parameters per round and execute fewer multiplication-and-add operations for gradient descent than SHeteroFL, either in terms of average or variance.
Revised: Conforming the budget constraints, our method outputs more accurate full-width models, transfer fewer parameters per round and execute fewer multiplication-and-add operations for gradient descent than SHeteroFL, either in terms of average or variance.

Original: Because only the individual × 0 .
Revised: Because only the individual × 0 .

Original: 125 net can fit into the budget constraint, FedAvg requires the least MACs and parameters, which however significantly sacrifices the final accuracy.
Revised: 125 net can fit into the budget constraint, FedAvg requires the least MACs and parameters, which however significantly sacrifices the final accuracy.

Original: Domain-wise evaluation.
Revised: Domain-wise evaluation.

Original: To understand why Split-Mix outperforms SHeteroFL, we investigate the total percentage of parameters that can be trained in each domain, in Fig.
Revised: To understand why Split-Mix outperforms SHeteroFL, we investigate the total percentage of parameters that can be trained in each domain, in Fig.

Original: 6 (Left).
Revised: 6 (Left).

Original: We count the total parameters that were ever trained in clients of a domain during the learning.
Revised: We count the total parameters that were ever trained in clients of a domain during the learning.

Original: Thanks to the base-model sampling strategy (i.e., Algorithm 2), Split-Mix allows all base models, rather than a subset, to be trained on all clients.
Revised: Thanks to the base-model sampling strategy (i.e., Algorithm 2), Split-Mix allows all base models, rather than a subset, to be trained on all clients.

Original: On the other hand, varying client budgets greatly impacted SHeteroFL by limiting the width of models trained in budget-insufficient clients, e.g., in the clipart, infograph and painting domains.
Revised: On the other hand, varying client budgets greatly impacted SHeteroFL by limiting the width of models trained in budget-insufficient clients, e.g., in the clipart, infograph and painting domains.

Original: Hence, SHeteroFL leaves a large amount of parameters under-trained and suffers
Revised: Hence, SHeteroFL leaves a large amount of parameters under-trained and suffers

Original: [Figure]
Revised: [Figure]

Original: 5. C USTOMIZE ROBUSTNESS
Revised: 5. C USTOMIZE ROBUSTNESS

Original: Training and evaluation.
Revised: Training and evaluation.

Original: For local AT, we use an n -step projected gradient descent (PGD) attack (Madry et al., 2018) with a constant noise magnitude ϵ .
Revised: For local AT, we use an n -step projected gradient descent (PGD) attack (Madry et al., 2018) with a constant noise magnitude ϵ .

Original: Following (Madry et al., 2018), we set ( ϵ, n ) =
Revised: Following (Madry et al., 2018), we set ( ϵ, n ) =

Original: (8 / 255 , 7) , and attack inner-loop step size 2 / 255 , for training, validation, and test.
Revised: (8 / 255 , 7) , and attack inner-loop step size 2 / 255 , for training, validation, and test.

Original: For simplicity, we temporarily relax the budget constraint R k and let the base model be × 1 net.
Revised: For simplicity, we temporarily relax the budget constraint R k and let the base model be × 1 net.

Original: For comparison, we extend FedAvg with AT which yields individual models optimized with different trade-off variables, i.e., λ n ∈ { 0 , 0 . 1 , 0 . 2 , 0 . 3 , 0 . 5 , 1 } .
Revised: For comparison, we extend FedAvg with AT which yields individual models optimized with different trade-off variables, i.e., λ n ∈ { 0 , 0 . 1 , 0 . 2 , 0 . 3 , 0 . 5 , 1 } .

Original: Also, we extend FedAvg with state-of-the-art in-situ trade-off method, OAT (Wang et al., 2020), as a baseline.
Revised: Also, we extend FedAvg with state-of-the-art in-situ trade-off method, OAT (Wang et al., 2020), as a baseline.

Original: Split-Mix+DAT is an extension of
Revised: Split-Mix+DAT is an extension of

Original: Split-Mix by the proposed DBN-based AT in Algorithm 4.
Revised: Split-Mix by the proposed DBN-based AT in Algorithm 4.

Original: We evaluate and contrast models in two metrics: standard accuracy (SA) on the clean test samples and robust accuracy (RA) on adversarial images generated from the clean test set by the PGD attack.
Revised: We evaluate and contrast models in two metrics: standard accuracy (SA) on the clean test samples and robust accuracy (RA) on adversarial images generated from the clean test set by the PGD attack.

Original: Both metric values are averaged by users.
Revised: Both metric values are averaged by users.

Original: We evaluate the trade-off effectiveness by comparing the RA-SA curves, which is better approaching the right-upper corner.
Revised: We evaluate the trade-off effectiveness by comparing the RA-SA curves, which is better approaching the right-upper corner.

Original: To plot the curves for FedAvg+OAT and Split-Mix+DAT, we vary their condition variable λ in { 0 , 0 . 2 , 0 . 5 , 0 . 8 , 1 } .
Revised: To plot the curves for FedAvg+OAT and Split-Mix+DAT, we vary their condition variable λ in { 0 , 0 . 2 , 0 . 5 , 0 . 8 , 1 } .

Original: [Figure]
Revised: [Figure]

Original: Trade-off curves are presented in Fig.
Revised: Trade-off curves are presented in Fig.

Original: 7 (a) and (b).
Revised: 7 (a) and (b).

Original: Since the naive extension of OAT with FedAvg adopts heterogeneous objectives and over-parameterization of conditional layers which suffers from convergence issues, its adversarial training does not converge and RA is incredibly poor in some cases.
Revised: Since the naive extension of OAT with FedAvg adopts heterogeneous objectives and over-parameterization of conditional layers which suffers from convergence issues, its adversarial training does not converge and RA is incredibly poor in some cases.

Original: As a result, the trade-off curve is not smooth.
Revised: As a result, the trade-off curve is not smooth.

Original: Instead, the proposed Split-Mix+DAT method has a smoother trade-off curve without heavy conditional training or over-parameterization.
Revised: Instead, the proposed Split-Mix+DAT method has a smoother trade-off curve without heavy conditional training or over-parameterization.

Original: By training in one pass, Split-Mix+DAT even outperforms and is more efficient than the FedAvg+AT baselines.
Revised: By training in one pass, Split-Mix+DAT even outperforms and is more efficient than the FedAvg+AT baselines.

Original: Joint customization of width and robustness under budget constraints.
Revised: Joint customization of width and robustness under budget constraints.

Original: Now we consider the width customization and the training budgets as Section 5.1.
Revised: Now we consider the width customization and the training budgets as Section 5.1.

Original: Due to the constraint, FedAvg can only train × 0 .
Revised: Due to the constraint, FedAvg can only train × 0 .

Original: 125 net.
Revised: 125 net.

Original: We omit OAT for its unstable convergence and use SplitMixDAT as a short name of Split-Mix+DAT.
Revised: We omit OAT for its unstable convergence and use SplitMixDAT as a short name of Split-Mix+DAT.

Original: In Fig.
Revised: In Fig.

Original: 7 (c) and (d), the trade-off curves with different widths are depicted.
Revised: 7 (c) and (d), the trade-off curves with different widths are depicted.

Original: As the width increases, both RA and SA of SplitMixDAT are improved, when they are smoothly traded off.
Revised: As the width increases, both RA and SA of SplitMixDAT are improved, when they are smoothly traded off.

Original: The results demonstrate the flexibility and the modular nature of our method.
Revised: The results demonstrate the flexibility and the modular nature of our method.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we proposed a novel federated learning approach for in-situ and on-demand customization to address challenges arise from resource heterogeneity and inference dyanmics.
Revised: In this paper, we proposed a novel federated learning approach for in-situ and on-demand customization to address challenges arise from resource heterogeneity and inference dyanmics.

Original: We proposed a
Revised: We proposed a

Original: Split-Mix strategy that efficiently transfer clients’ knowledge to collaboratively learn a customizable model.
Revised: Split-Mix strategy that efficiently transfer clients’ knowledge to collaboratively learn a customizable model.

Original: Extensive experiments demonstrate the effectiveness of the principle in adjusting model widths and robustness when much fewer parameters are used compared to baselines.
Revised: Extensive experiments demonstrate the effectiveness of the principle in adjusting model widths and robustness when much fewer parameters are used compared to baselines.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: This material is based in part upon work supported by the National Institute of Aging 1RF1AG072449,
Revised: This material is based in part upon work supported by the National Institute of Aging 1RF1AG072449,

Original: Office of Naval Research N00014-20-1-2382, National Science Foundation under Grant IIS-1749940.
Revised: Office of Naval Research N00014-20-1-2382, National Science Foundation under Grant IIS-1749940.

Original: Z.W. is supported by the U
Revised: Z.W. is supported by the U

Original: S. Army Research Laboratory Cooperative Research Agreement
Revised: S. Army Research Laboratory Cooperative Research Agreement

Original: W911NF17-2-0196 (IOBT REIGN).
Revised: W911NF17-2-0196 (IOBT REIGN).

Original: A E XPERIMENTS
Revised: A E XPERIMENTS

Original: In this section, we provide more details about our experiments and additional evaluation results.
Revised: In this section, we provide more details about our experiments and additional evaluation results.

Original: A. P ARALLEL IMPLEMENTATION OF S PLIT -M IX AND CONVERGENCE
Revised: A. P ARALLEL IMPLEMENTATION OF S PLIT -M IX AND CONVERGENCE

Original: In this section, we elaborate on the implementation and efficiency of Split-Mix.
Revised: In this section, we elaborate on the implementation and efficiency of Split-Mix.

Original: In Fig.
Revised: In Fig.

Original: 8, we conceptually compare the training by three methods, when two clients capable of training × 1 and × 0 .
Revised: 8, we conceptually compare the training by three methods, when two clients capable of training × 1 and × 0 .

Original: 5 are considered.
Revised: 5 are considered.

Original: FedAvg can train the × 1 net on the × 1 -capable client but not on the × 0 .
Revised: FedAvg can train the × 1 net on the × 1 -capable client but not on the × 0 .

Original: 5capable client.
Revised: 5capable client.

Original: Through parameter sharing among different model widths, SHeteroFL can train multiple widths in a sequential manner and can fit into × 0 .
Revised: Through parameter sharing among different model widths, SHeteroFL can train multiple widths in a sequential manner and can fit into × 0 .

Original: 5 -capable clients.
Revised: 5 -capable clients.

Original: Unlike SHeteroFL,
Revised: Unlike SHeteroFL,

Original: Split-Mix trains four base models in one parallel pass and therefore is the most efficient and flexible method.
Revised: Split-Mix trains four base models in one parallel pass and therefore is the most efficient and flexible method.

Original: Remarkably, in the worst case, training × 1 net of Split-Mix is as efficient as FedAvg and more efficient than SHeteroFL, if all the base weights, w l 0 ,r , . . .
Revised: Remarkably, in the worst case, training × 1 net of Split-Mix is as efficient as FedAvg and more efficient than SHeteroFL, if all the base weights, w l 0 ,r , . . .

Original: , w l 3 ,r ∈ R r × r (where r = 0 .here), are embedded into a 4 r × 4 r weight matrix 2 .
Revised: , w l 3 ,r ∈ R r × r (where r = 0 .here), are embedded into a 4 r × 4 r weight matrix 2 .

Original: When base models are embedded into a full net, non-trainable parameters can be masked out (grey areas) to avoid interference between base models.
Revised: When base models are embedded into a full net, non-trainable parameters can be masked out (grey areas) to avoid interference between base models.

Original: [Figure]
Revised: [Figure]

Original: With the aforementioned implementation, we compare the convergence versus the wall-clock time in Fig.
Revised: With the aforementioned implementation, we compare the convergence versus the wall-clock time in Fig.

Original: We implement all algorithms in PyTorch 1.4.1 run on a single NVIDIA RTX
Revised: We implement all algorithms in PyTorch 1.4.1 run on a single NVIDIA RTX

Original: A
Revised: A

Original: GPU and a 104-thread CPU.
Revised: GPU and a 104-thread CPU.

Original: Fig.
Revised: Fig.

Original: 9 shows the elapsed computation time from the initialization to a maximal number of iterations.
Revised: 9 shows the elapsed computation time from the initialization to a maximal number of iterations.

Original: The maximal number of iterations is set to be the same for all methods, such that it is easier to compare the stopping time.
Revised: The maximal number of iterations is set to be the same for all methods, such that it is easier to compare the stopping time.

Original: In Fig.
Revised: In Fig.

Original: 9, Split-Mix is much more efficient than the
Revised: 9, Split-Mix is much more efficient than the

Original: SheteroFL.
Revised: SheteroFL.

Original: Note that FedAvg can only train the × 0 .
Revised: Note that FedAvg can only train the × 0 .

Original: 125 net which includes much fewer parameters,
Revised: 125 net which includes much fewer parameters,

Original: For this reason, Split-Mix is slightly slower than FedAvg, but the degradation of efficiency trades in better accuracy than FedAvg.
Revised: For this reason, Split-Mix is slightly slower than FedAvg, but the degradation of efficiency trades in better accuracy than FedAvg.

Original: [Figure]
Revised: [Figure]

Original: A. E XPERIMENTAL CONFIGURATIONS
Revised: A. E XPERIMENTAL CONFIGURATIONS

Original: In Algorithm 4, we elaborate the local training of DBN.
Revised: In Algorithm 4, we elaborate the local training of DBN.

Original: Specifically, each BN is trained independent with different input samples.
Revised: Specifically, each BN is trained independent with different input samples.

Original: The maximization problem in Algorithm 4 can be solved by an n -step projected gradient descent, and is commonly known as PGD attack (Madry et al., 2018).
Revised: The maximization problem in Algorithm 4 can be solved by an n -step projected gradient descent, and is commonly known as PGD attack (Madry et al., 2018).

Original: Algorithm
Revised: [Table] [Table] [Table] [Table]

Original: , D
Revised: [Table]

Original: , E,
Revised: [Table] [Table] [Table]

Original: with DBN and
Revised: [Table] [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: (˜ x,y ) ∈ ˜ B

Original: [ L CE ( f (˜ x ; ˆ w i,r ) , y )] (cid:111) 13: ˆ w i,r ← ˆ w i,r − η ∂L∂ ˆ w i,r 14: Return ˆ W k

Original: Data .
Revised: Data .

Original: Both CIFAR10 and Digits are 10-way classification tasks.
Revised: Both CIFAR10 and Digits are 10-way classification tasks.

Original: We follow the non- i.i.d benchmark of Li et al.
Revised: We follow the non- i.i.d benchmark of Li et al.

Original: (2020b) to extract 10 classes from DomainNet, which is publicly available in FedBN codes.
Revised: (2020b) to extract 10 classes from DomainNet, which is publicly available in FedBN codes.

Original: To illustrate the multiple-domain datasets, we sample several images from Digits and DomainNet datasets in Fig.
Revised: To illustrate the multiple-domain datasets, we sample several images from Digits and DomainNet datasets in Fig.

Original: mnist svhn usps syn mnistm clipart infograph painting quickdraw real sketch
Revised: [Figure]

Original: Hyper-parameters.
Revised: Hyper-parameters.

Original: In general, for local optimization we use stochastic gradient descent (SGD) with 0.9 momentum and 5 × 10 − 4 weight decay.
Revised: In general, for local optimization we use stochastic gradient descent (SGD) with 0.9 momentum and 5 × 10 − 4 weight decay.

Original: Dataset specific settings are stated as follows.
Revised: Dataset specific settings are stated as follows.

Original: CIFAR10:
Revised: CIFAR10:

Original: Following HeteroFL
Revised: Following HeteroFL

Original: (Diao et al., 2021), we train with 5 local epochs and 400 global
Revised: (Diao et al., 2021), we train with 5 local epochs and 400 global

Original: [Table]
Revised: [Table]

Original: communication rounds.
Revised: communication rounds.

Original: Globally, we initialize the learning rate as 0 . 01 and adjust the learning rate at 150, 250 communication rounds with a scale rate of 0 . 1 .
Revised: Globally, we initialize the learning rate as 0 . 01 and adjust the learning rate at 150, 250 communication rounds with a scale rate of 0 . 1 .

Original: Locally, we use a larger batch size of 128, to speed up the training in simulation.
Revised: Locally, we use a larger batch size of 128, to speed up the training in simulation.

Original: Digits: We use a cosine annealing learning rate decaying from 0 .
Revised: Digits: We use a cosine annealing learning rate decaying from 0 .

Original: 1 to 0 across 400 global communication rounds.
Revised: 1 to 0 across 400 global communication rounds.

Original: SGD is executed with one epoch for each local client.
Revised: SGD is executed with one epoch for each local client.

Original: DomainNet: We use a constant learning rate 0 .
Revised: DomainNet: We use a constant learning rate 0 .

Original: 01 and run 400 communication rounds in total.
Revised: 01 and run 400 communication rounds in total.

Original: Similar to Digits, SGD is executed with one epoch for each local client
Revised: Similar to Digits, SGD is executed with one epoch for each local client

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Network architectures .
Revised: Network architectures .

Original: Architectures of modified AlexNet (for DomaiNet) and CNN (for Digits) can be found in (Li et al., 2020b) and public codes 3 .
Revised: Architectures of modified AlexNet (for DomaiNet) and CNN (for Digits) can be found in (Li et al., 2020b) and public codes 3 .

Original: For reader’s reference, we provide the layer details in in Tables 2 and 3.
Revised: For reader’s reference, we provide the layer details in in Tables 2 and 3.

Original: For the convolutional layer (Conv2D or Conv1D), the first argument is the number channel.
Revised: For the convolutional layer (Conv2D or Conv1D), the first argument is the number channel.

Original: For a fully connected layer (FC), we list the number of hidden units as the first argument.
Revised: For a fully connected layer (FC), we list the number of hidden units as the first argument.

Original: The implementation of the preactivated ResNet can be found in the repository of HeteroFL (Diao et al., 2021).
Revised: The implementation of the preactivated ResNet can be found in the repository of HeteroFL (Diao et al., 2021).

Original: Batch-normalization for customizable model sizes .
Revised: Batch-normalization for customizable model sizes .

Original: As pointed out by Diao et al.
Revised: As pointed out by Diao et al.

Original: (2021), HeteroFL relies on mini-batch batch-normalization to stabilize the training with multiple model widths and compute the statistics afterwards.
Revised: (2021), HeteroFL relies on mini-batch batch-normalization to stabilize the training with multiple model widths and compute the statistics afterwards.

Original: Another advantage of such strategy is on federation of multi-domain clients.
Revised: Another advantage of such strategy is on federation of multi-domain clients.

Original: Li et al.
Revised: Li et al.

Original: (2020b) showed that using local batch-normalization helps model personalization for non- i.i.d local features.
Revised: (2020b) showed that using local batch-normalization helps model personalization for non- i.i.d local features.

Original: As min-batch BN statistics simulate the local BN idea, it should enjoy the similar benefit.
Revised: As min-batch BN statistics simulate the local BN idea, it should enjoy the similar benefit.

Original: To eliminate the potential biases in personalization or convergence caused by different estimation methods of BN statistic, we let all compared algorithms using the same mini-batch strategy.
Revised: To eliminate the potential biases in personalization or convergence caused by different estimation methods of BN statistic, we let all compared algorithms using the same mini-batch strategy.

Original: To reveal the effect of different BN statistic solutions for Split-Mix, we provide a detailed ablation study regarding the estimation of BN statistics in Appendix A.3.
Revised: To reveal the effect of different BN statistic solutions for Split-Mix, we provide a detailed ablation study regarding the estimation of BN statistics in Appendix A.3.

Original: Loss function for class non-
Revised: Loss function for class non-

Original: i.i.d
Revised: i.i.d

Original: FL .
Revised: FL .

Original: As some classes are missing locally in non-
Revised: As some classes are missing locally in non-

Original: i.i.d setting, classspecific parameters in the classifier head may be updated without proper supervision and results in random updates.
Revised: i.i.d setting, classspecific parameters in the classifier head may be updated without proper supervision and results in random updates.

Original: To mitigate the effect of missing classes locally, we use the same masked cross-entropy loss as introduced by HeteroFL, where absent classes are masked out.
Revised: To mitigate the effect of missing classes locally, we use the same masked cross-entropy loss as introduced by HeteroFL, where absent classes are masked out.

Original: A. A BLATION STUDY OF NETWORK SCALING AND BN STATISTICS
Revised: A. A BLATION STUDY OF NETWORK SCALING AND BN STATISTICS

Original: In this section, we evaluate how the network rescaling and BN statistics affect the performance.
Revised: In this section, we evaluate how the network rescaling and BN statistics affect the performance.

Original: For rescaling, we consider the parameter initialization ( rescale init ) and layer outputs ( rescale layer ).
Revised: For rescaling, we consider the parameter initialization ( rescale init ) and layer outputs ( rescale layer ).

Original: For
Revised: For

Original: BN statistics, we consider four options.
Revised: BN statistics, we consider four options.

Original: The batch average one will estimate the statistics by one batch of data.
Revised: The batch average one will estimate the statistics by one batch of data.

Original: The post average one will use the batch average strategy during training but re-estimate the statistics using client data afterward, which was adopted by HeteroFL.
Revised: The post average one will use the batch average strategy during training but re-estimate the statistics using client data afterward, which was adopted by HeteroFL.

Original: To gain better BN statistics, we run the model on a training set for 20 epochs.
Revised: To gain better BN statistics, we run the model on a training set for 20 epochs.

Original: The tracked one will track statistics during training.
Revised: The tracked one will track statistics during training.

Original: The locally tracked one will also track statistics but the statistics will not be shared with the server for averaging, which can benefit clients’ privacy and personalization (Li et al., 2020b).
Revised: The locally tracked one will also track statistics but the statistics will not be shared with the server for averaging, which can benefit clients’ privacy and personalization (Li et al., 2020b).

Original: We report full ablation results in Table 4. 1)
Revised: We report full ablation results in Table 4. 1)

Original: First we compare the use of BNs without in-training tracking.
Revised: First we compare the use of BNs without in-training tracking.

Original: The post-average BN performs best compared to other BN choices.
Revised: The post-average BN performs best compared to other BN choices.

Original: With similar performance, the batch average BN does not need multiple rounds of evaluation of BN statistics, which is more efficient for inference.
Revised: With similar performance, the batch average BN does not need multiple rounds of evaluation of BN statistics, which is more efficient for inference.

Original: Then we compare the use of BNs tracked during training.
Revised: Then we compare the use of BNs tracked during training.

Original: Consistent with the prior study (Li et al., 2020b), locally tracked BN performs better than the globally averaged one. 3)
Revised: Consistent with the prior study (Li et al., 2020b), locally tracked BN performs better than the globally averaged one. 3)

Original: Rescaling layer outputs barely affect the accuracy, but it could be poisonous for tracked BN statistics.
Revised: Rescaling layer outputs barely affect the accuracy, but it could be poisonous for tracked BN statistics.

Original: The initialization rescaling greatly improves the performance regardless of the choice of
Revised: The initialization rescaling greatly improves the performance regardless of the choice of

Original: BN statistics.
Revised: BN statistics.

Original: In conclusion
Revised: In conclusion

Original: [Table]
Revised: [Table]

Original: either locally tracked or batch averaged BN statistics can yield both efficient and accurate performance.
Revised: either locally tracked or batch averaged BN statistics can yield both efficient and accurate performance.

Original: Post-averaged BN statistics may be preferred if post averaging is bearable for efficiency, especially for large-scale datasets.
Revised: Post-averaged BN statistics may be preferred if post averaging is bearable for efficiency, especially for large-scale datasets.

Original: Rescaled initialization is an essential ingredient for Split-Max to perform well.
Revised: Rescaled initialization is an essential ingredient for Split-Max to perform well.

Original: Layer rescale is not recommended if batch average or tracked BN statistics are utilized.
Revised: Layer rescale is not recommended if batch average or tracked BN statistics are utilized.

Original: [Table]
Revised: [Table]

Original: A. E XPERIMENTS WITH I . I . D FL
Revised: A. E XPERIMENTS WITH I . I . D FL

Original: In addition to non- i.i.d FL settings
Revised: In addition to non- i.i.d FL settings

Original: [Table]
Revised: [Table]

Original: we experiment with i.i.d
Revised: we experiment with i.i.d

Original: FL where each client will own data of 10 classes from the CIFAR10 dataset.
Revised: FL where each client will own data of 10 classes from the CIFAR10 dataset.

Original: Results using 100% and 50% training data are included in
Revised: Results using 100% and 50% training data are included in

Original: Table 5.
Revised: Table 5.

Original: We observe a great increase in the accuracy compared to the non- i.i.d experiments, which is a common phenomenon that non-iid FL will perform worse globally.
Revised: We observe a great increase in the accuracy compared to the non- i.i.d experiments, which is a common phenomenon that non-iid FL will perform worse globally.

Original: The similar performance degradation was observed in (Diao et al., 2021), as well.
Revised: The similar performance degradation was observed in (Diao et al., 2021), as well.

Original: In Table 5, our method performs better in larger widths with fewer training data, whose performance approaches that of unconstrained individual FedAvg.
Revised: In Table 5, our method performs better in larger widths with fewer training data, whose performance approaches that of unconstrained individual FedAvg.

Original: Our method provide a monotonous relation between model size and accuracy (larger models are more accurate) and uses fewer parameters and MACs even compared to wider baseline networks, though performs worse in smaller widths because the slimmest networks are updated less frequently in budget-insufficient clients compared to the SHeteroFL or FedAvg.
Revised: Our method provide a monotonous relation between model size and accuracy (larger models are more accurate) and uses fewer parameters and MACs even compared to wider baseline networks, though performs worse in smaller widths because the slimmest networks are updated less frequently in budget-insufficient clients compared to the SHeteroFL or FedAvg.

Original: Worth to mention, our method uses much fewer parameters and operation counts for the same accuracy.
Revised: Worth to mention, our method uses much fewer parameters and operation counts for the same accuracy.

Original: For example, Split-Mix requires 7.2M MACs and 1.4M parameters for 81.1% accuracy while SHeteroFL needs twice of the complexity, given the 50% CIFAR10 i.i.d configuration.
Revised: For example, Split-Mix requires 7.2M MACs and 1.4M parameters for 81.1% accuracy while SHeteroFL needs twice of the complexity, given the 50% CIFAR10 i.i.d configuration.

Original: [Table]
Revised: [Table]

Original: A. M ORE BUDGET DISTRIBUTIONS
Revised: A. M ORE BUDGET DISTRIBUTIONS

Original: In our experiments
Revised: In our experiments

Original: [Table]
Revised: [Table]

Original: we generally use an exponential budget distribution: R k = (1 / 2) ⌈ 4 k/K ⌉ .
Revised: we generally use an exponential budget distribution: R k = (1 / 2) ⌈ 4 k/K ⌉ .

Original: Though the distribution represents the imbalance between the budget-sufficient and budget-insufficient clients, real-world applications may encounter a wider variety of budget distributions.
Revised: Though the distribution represents the imbalance between the budget-sufficient and budget-insufficient clients, real-world applications may encounter a wider variety of budget distributions.

Original: Thus
Revised: Thus

Original: [Table]
Revised: [Table]

Original: we extend our problem assumption to budget distributions with more budget-sufficient clients where we let more groups to have × 1 or × 0 .
Revised: we extend our problem assumption to budget distributions with more budget-sufficient clients where we let more groups to have × 1 or × 0 .

Original: 5 net training capability and with step-increase budgets where we increase budgets by a fixed step (e.g., × 0 . 25 ).
Revised: 5 net training capability and with step-increase budgets where we increase budgets by a fixed step (e.g., × 0 . 25 ).

Original: In addition, we consider a log normal distribution which concentrate around 0.45 budget with few wider or extremely budget-insufficient clients.
Revised: In addition, we consider a log normal distribution which concentrate around 0.45 budget with few wider or extremely budget-insufficient clients.

Original: We partition the budget distribution into 0.125-width bins and each client will only train the maximal compatible width varying from 0.125 to 1, which greatly increases the number of slimmable subnetworks (8 now compared to previous 4).
Revised: We partition the budget distribution into 0.125-width bins and each client will only train the maximal compatible width varying from 0.125 to 1, which greatly increases the number of slimmable subnetworks (8 now compared to previous 4).

Original: To reduce the overhead of slimmable training, we use HeteroFL instead of
Revised: To reduce the overhead of slimmable training, we use HeteroFL instead of

Original: SHeteroFL.
Revised: SHeteroFL.

Original: Fig.
Revised: Fig.

Original: For SHeteroFL, we only report the evaluated performance on trained widths.
Revised: For SHeteroFL, we only report the evaluated performance on trained widths.

Original: In other words, if the maximal width is × 0 . 5 , we will not report results of the × 1 net.
Revised: In other words, if the maximal width is × 0 . 5 , we will not report results of the × 1 net.

Original: For Individually-trained FedAvg (Ind. FedAvg), models individually trained for each width are reported, which ignores the width constraints by users and therefore only serves as reference upper bounds.
Revised: For Individually-trained FedAvg (Ind. FedAvg), models individually trained for each width are reported, which ignores the width constraints by users and therefore only serves as reference upper bounds.

Original: Regardless of the budget distributions, for instance, more budget-sufficient clients (Fig. 11a) or non-exponential distributions (Fig. 11b), Split-Mix outperforms SHeteroFL with larger widths.
Revised: Regardless of the budget distributions, for instance, more budget-sufficient clients (Fig. 11a) or non-exponential distributions (Fig. 11b), Split-Mix outperforms SHeteroFL with larger widths.

Original: A. E FFECT OF LOWER CONTACT RATES
Revised: A. E FFECT OF LOWER CONTACT RATES

Original: Because of varying communication conditions in deployment, the times that clients actively and successfully upload their models could be fewer than expected.
Revised: Because of varying communication conditions in deployment, the times that clients actively and successfully upload their models could be fewer than expected.

Original: To evaluate the robustness of customization federated algorithms, we conduct federated experiments with a varying number of active clients per round in Fig.
Revised: To evaluate the robustness of customization federated algorithms, we conduct federated experiments with a varying number of active clients per round in Fig.

Original: Because of the limited number of communication rounds (within
Revised: Because of the limited number of communication rounds (within

Original: 300 rounds), the test accuracy decreases by fewer contact clients.
Revised: 300 rounds), the test accuracy decreases by fewer contact clients.

Original: This is a common phenomenon because lower contact rates requires more communication rounds to reach the same performance as the full-contact competitors do.
Revised: This is a common phenomenon because lower contact rates requires more communication rounds to reach the same performance as the full-contact competitors do.

Original: Though global performance generally decreases, we find that the wider networks are more accurate if trained by SplitMix.
Revised: Though global performance generally decreases, we find that the wider networks are more accurate if trained by SplitMix.

Original: One source for the advantage is the modular base models in SplitMix, which can be easily distributed into different rounds for training.
Revised: One source for the advantage is the modular base models in SplitMix, which can be easily distributed into different rounds for training.

Original: Rather, the wider integrated networks in SHeteroFL lower their chance to be aggregated globally and therefore their global performance declines
Revised: Rather, the wider integrated networks in SHeteroFL lower their chance to be aggregated globally and therefore their global performance declines

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure][Figure] [Figure] [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: A. N EGATIVE IMPACT OF CONSTRAINED BUDGETS ON WIDER NETWORKS
Revised: A. N EGATIVE IMPACT OF CONSTRAINED BUDGETS ON WIDER NETWORKS

Original: In Fig.
Revised: In Fig.

Original: [Figure]
Revised: [Figure]

Original: we show how the SHeteroFL fails to train wider networks with budget-constrained clients.
Revised: we show how the SHeteroFL fails to train wider networks with budget-constrained clients.

Original: To show the generality of such a problem, we extend the experiments to Digits and CIFAR10 datasets in Fig. 13.
Revised: To show the generality of such a problem, we extend the experiments to Digits and CIFAR10 datasets in Fig. 13.

Original: Though wider networks converges faster at the beginning, they meanwhile overfit limited data in a few clients and therefore their validation accuracy no longer improves.
Revised: Though wider networks converges faster at the beginning, they meanwhile overfit limited data in a few clients and therefore their validation accuracy no longer improves.