Original: FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-model Extraction
Revised: FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-model Extraction

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Federated learning (FL) is a collaborative machine learning paradigm to trainmodels from decentralized private data.
Revised: Federated learning (FL) is a collaborative machine learning paradigm to trainmodels from decentralized private data.

Original: Most FL research focuses on modelhomogeneous setting where models deployed across all the participating clientsand the server are required to be identical.
Revised: Most FL research focuses on modelhomogeneous setting where models deployed across all the participating clientsand the server are required to be identical.

Original: However, in real-world scenarios,such a requirement acts as a constraint that restricts the outreach to clients withheterogeneous device resources and unfairly excludes users with low-end deviceswho would otherwise benefit from FL.
Revised: However, in real-world scenarios,such a requirement acts as a constraint that restricts the outreach to clients withheterogeneous device resources and unfairly excludes users with low-end deviceswho would otherwise benefit from FL.

Original: In this work, we propose a simple yeteffective model-heterogeneous FL method named FedRolex to tackle this constraint.
Revised: In this work, we propose a simple yeteffective model-heterogeneous FL method named FedRolex to tackle this constraint.

Original: Unlike the model-homogeneous scenario, the fundamental challenge ofmodel heterogeneity in FL is that different parameters of the global model aretrained on heterogeneous data distributions.
Revised: Unlike the model-homogeneous scenario, the fundamental challenge ofmodel heterogeneity in FL is that different parameters of the global model aretrained on heterogeneous data distributions.

Original: Our method addresses this challengeby rolling the sub-model in each federated iteration so that the parameters of theglobal model are evenly trained on the global data distribution across all devices,making it more akin to model-homogeneous training.
Revised: Our method addresses this challengeby rolling the sub-model in each federated iteration so that the parameters of theglobal model are evenly trained on the global data distribution across all devices,making it more akin to model-homogeneous training.

Original: Our experiments show that
Revised: Our experiments show that

Original: FedRolex outperforms other state-of-the-art model-heterogeneous FL methods,especially under high data-heterogeneity scenarios.
Revised: FedRolex outperforms other state-of-the-art model-heterogeneous FL methods,especially under high data-heterogeneity scenarios.

Original: We have conducted ablationstudies to show that submodel rolling is an effective technique to reduce the gapbetween model-heterogeneous and standard model-homogeneous settings.
Revised: We have conducted ablationstudies to show that submodel rolling is an effective technique to reduce the gapbetween model-heterogeneous and standard model-homogeneous settings.

Original: Lastly,we consider the distribution of client capabilities that is similar to real-world income distribution instead of the uniform distribution used in existing works.
Revised: Lastly,we consider the distribution of client capabilities that is similar to real-world income distribution instead of the uniform distribution used in existing works.

Original: Ourresults show a consistent improvement in the accuracies on low-end devices, whichenhances the inclusiveness of federated learning.
Revised: Ourresults show a consistent improvement in the accuracies on low-end devices, whichenhances the inclusiveness of federated learning.

Original:  Introduction
Revised:  Introduction

Original: Cross-device federated learning (FL) was originally proposed by McMahan et al.
Revised: Cross-device federated learning (FL) was originally proposed by McMahan et al.

Original: (2017) as a privacypreserving machine learning paradigm to train a machine learning model on a federation of resourceconstrained edge devices without accessing their data.
Revised: (2017) as a privacypreserving machine learning paradigm to train a machine learning model on a federation of resourceconstrained edge devices without accessing their data.

Original: This idea has far-reaching applications, whichis made urgent by modern demands for privacy.
Revised: This idea has far-reaching applications, whichis made urgent by modern demands for privacy.

Original: Indeed, the goal of strong data privacy is a central motivation for FL.
Revised: Indeed, the goal of strong data privacy is a central motivation for FL.

Original: By storing data locally, instead of replicating them on a remote server, the attacksurface of the system is decreased (Jere et al., 2020), and by using focused ephemeral updates andearly aggregation, the communication cost is also minimized (Nishio and Yonetani, 2019).
Revised: By storing data locally, instead of replicating them on a remote server, the attacksurface of the system is decreased (Jere et al., 2020), and by using focused ephemeral updates andearly aggregation, the communication cost is also minimized (Nishio and Yonetani, 2019).

Original: The majority of existing cross-device FL studies focus on model-homogeneous setting, where models deployed across all the participating clients and the server are required to be identical.
Revised: The majority of existing cross-device FL studies focus on model-homogeneous setting, where models deployed across all the participating clients and the server are required to be identical.

Original: In practice, device heterogeneity is a more realistic constraint to consider for deploying in real-world environments
Revised: In practice, device heterogeneity is a more realistic constraint to consider for deploying in real-world environments

Original: Different edge devices have different on-device resources and, hence, are only capable of training models with capacities that match their on-device resources.
Revised: Different edge devices have different on-device resources and, hence, are only capable of training models with capacities that match their on-device resources.

Original: Having the same model on all the devices forces a limitation that restricts the outreach to clients with heterogeneous device resources and unfairly excludes users with low-end devices who would otherwise benefit from FL
Revised: Having the same model on all the devices forces a limitation that restricts the outreach to clients with heterogeneous device resources and unfairly excludes users with low-end devices who would otherwise benefit from FL

Original: [Table]
Revised: [Table]

Original: To relax this fundamental constraint in standard FL
Revised: To relax this fundamental constraint in standard FL

Original: [Table]
Revised: [Table]

Original: recent works proposed model-heterogeneous
Revised: recent works proposed model-heterogeneous

Original: FL
Revised: FL

Original: [Table]
Revised: [Table]

Original: where models with different capacities are trained on edge devices with heterogeneous resourcesduring the federated training process.
Revised: where models with different capacities are trained on edge devices with heterogeneous resourcesduring the federated training process.

Original: One fundamental challenge in model-heterogeneous FL isthe aggregation of heterogeneous client models.
Revised: One fundamental challenge in model-heterogeneous FL isthe aggregation of heterogeneous client models.

Original: Lopes et al.
Revised: Lopes et al.

Original: ; He et al.
Revised: ; He et al.

Original: ; Cho et al.(2022) proposed to use knowledge distillation (KD) (Hinton et al., 2015)
Revised: ; Cho et al.(2022) proposed to use knowledge distillation (KD) (Hinton et al., 2015)

Original: [Table]
Revised: [Table]

Original: where the client modelsserve as teachers and the server ensembles the knowledge distilled from the individual client models.
Revised: where the client modelsserve as teachers and the server ensembles the knowledge distilled from the individual client models.

Original: However, KD-based methods generally require access to public data to achieve competitive modelaccuracy, where the accessibility of high-quality public data is not always practical.
Revised: However, KD-based methods generally require access to public data to achieve competitive modelaccuracy, where the accessibility of high-quality public data is not always practical.

Original: Moreover, KDbecomes computationally expensive when the cohort size in each FL round scales up.
Revised: Moreover, KDbecomes computationally expensive when the cohort size in each FL round scales up.

Original: To remove thedependency on public data, another set of studies focuses on designing partial training (PT) basedtechniques such as federated dropout (Caldas et al., 2018), ordered dropout (Horvath et al., 2021),and static sub-model extraction (Diao et al., 2020), where each client trains a smaller sub-model ofthe larger global model on the server.
Revised: To remove thedependency on public data, another set of studies focuses on designing partial training (PT) basedtechniques such as federated dropout (Caldas et al., 2018), ordered dropout (Horvath et al., 2021),and static sub-model extraction (Diao et al., 2020), where each client trains a smaller sub-model ofthe larger global model on the server.

Original: However, the fundamental issue of existing PT-based methodsis that the sub-models are extracted in ways such that the parameters of the global model obtained byaggregating the extracted sub-models are not evenly trained, making the global model vulnerable to aphenomenon called client drift (Wang et al., 2021) where it is used to describe the error in gradientestimation due to the differences in data distributions between clients.
Revised: However, the fundamental issue of existing PT-based methodsis that the sub-models are extracted in ways such that the parameters of the global model obtained byaggregating the extracted sub-models are not evenly trained, making the global model vulnerable to aphenomenon called client drift (Wang et al., 2021) where it is used to describe the error in gradientestimation due to the differences in data distributions between clients.

Original: However, this estimation errorcan originate from any discrepancy across the federation.
Revised: However, this estimation errorcan originate from any discrepancy across the federation.

Original: For instance, in model heterogeneousfederated learning, client drift will also occur due to the inconsistency between the clients’ individualmodel architectures in addition to the difference in private data distributions.
Revised: For instance, in model heterogeneousfederated learning, client drift will also occur due to the inconsistency between the clients’ individualmodel architectures in addition to the difference in private data distributions.

Original: This also becomes moreprominent when the cohort size in each communication round is small or data heterogeneity acrossclients becomes more severe, resulting in lower global model accuracy.
Revised: This also becomes moreprominent when the cohort size in each communication round is small or data heterogeneity acrossclients becomes more severe, resulting in lower global model accuracy.

Original: In this work, we propose a simple yet effective PT-based model-heterogeneous FL method named
Revised: In this work, we propose a simple yet effective PT-based model-heterogeneous FL method named

Original: FedRolex that tackles the fundamental issue of existing PT-based methods.
Revised: FedRolex that tackles the fundamental issue of existing PT-based methods.

Original: Compared to existing
Revised: Compared to existing

Original: PT-based methods
Revised: PT-based methods

Original: , FedRolex proposes a rolling sub-model extraction scheme, where the sub-modelis extracted from the global model using a rolling window that advances in each communicationround.
Revised: , FedRolex proposes a rolling sub-model extraction scheme, where the sub-modelis extracted from the global model using a rolling window that advances in each communicationround.

Original: As such, sub-models are extracted from different parts of the global model in differentrounds.
Revised: As such, sub-models are extracted from different parts of the global model in differentrounds.

Original: Since the extraction window is rolling in each round, parameters of the global model areevenly trained to minimize the client drift.
Revised: Since the extraction window is rolling in each round, parameters of the global model areevenly trained to minimize the client drift.

Original: Throughout communication rounds, all the parametersof the global model are updated over the entire data distribution.
Revised: Throughout communication rounds, all the parametersof the global model are updated over the entire data distribution.

Original: As we show in §4.2, such arolling sub-model extraction strategy significantly reduces the gap between model-heterogeneous andmodel-homogeneous settings.
Revised: As we show in §4.2, such arolling sub-model extraction strategy significantly reduces the gap between model-heterogeneous andmodel-homogeneous settings.

Original: We evaluate the performance of FedRolex on two datasets – CIFAR-10 and CIFAR-100  – andcompare it against state-of-the-art KD-based model-heterogeneous FL methods FedDF (Lin et al.,2020), DS-FL (Itahara et al., 2020) and Fed-ET (Cho et al., 2021) as well as PT-based modelheterogeneous FL methods HeteroFL (Diao et al., 2020) and Federated Dropout (Caldas et al., 2018) 1 .
Revised: We evaluate the performance of FedRolex on three datasets – CIFAR-10, CIFAR-100 (Krizhevsky,2009) and StackOverflow (tff) – and compare it against state-of-the-art KD-based modelheterogeneous FL methods FedDF (Lin et al., 2020), DS-FL (Itahara et al., 2020) and Fed-ET (Choet al., 2021) as well as PT-based model-heterogeneous FL methods HeteroFL (Diao et al., 2020) and Federated Dropout (Caldas et al., 2018) 1 .

Original: Our results show that FedRolex consistently outperforms state-of-the-art PT-based methods in bothhigh and low data heterogeneity scenarios.
Revised: Our results show that FedRolex consistently outperformsstate-of-the-art PT-based methods in both high and low data heterogeneity scenarios.

Original: FedRolex also outperforms state-of-the-art KD-basedmethods in low data heterogeneity scenarios as well as on the more challenging CIFAR-100 datasetin high data heterogeneity scenarios without public data.
Revised: FedRolex alsooutperforms state-of-the-art KD-based methods in low data heterogeneity scenarios as well as on themore challenging CIFAR-100 dataset in high data heterogeneity scenarios without public data.

Original: Furthermore, we show that our proposedrolling sub-model extraction scheme is flexible to train a global model that can be much larger thanthe largest client model.
Revised: Furthermore, we show that our proposed rolling sub-model extraction scheme is flexible to train a globalmodel that can be much larger than the largest client model.

Original: Lastly, we conducted an experiment that uses real-world household income distribution to emulate real-world device distribution.
Revised: Lastly, we conducted an experiment that uses real-world household income distribution to emulate real-world device distribution.

Original: Our results show that FedRolex effectivelyboosts the model accuracy of low-end devices, which enhances the inclusiveness of federated learning.
Revised: Our resultsshow that FedRolex effectively boosts the model accuracy of low-end devices, which enhances theinclusiveness of federated learning.

Original: The code of FedRolex will be publicly available on Github.
Revised: The code of FedRolex will be publicly available on Github.

Original:  Related Work
Revised:  Related Work

Original: Our work focuses on model-heterogeneous FL.
Revised: Our work focuses on model-heterogeneous FL.

Original: Existing methods can be generally grouped into twocategories: knowledge distillation (KD) based and partial training (PT) based methods.
Revised: Existing methods can be generally grouped into twocategories: knowledge distillation (KD) based and partial training (PT) based methods.

Original: Knowledge Distillation (KD) based Model-Heterogeneous FL.
Revised: Knowledge Distillation (KD) based Model-Heterogeneous FL.

Original: The key idea of knowledge distillation (Hinton et al., 2015; Mirzadeh et al., 2020; Liu et al., 2019) is to compress a large pre-trainedmodel by teaching a smaller network, step by step, exactly what to do using the larger pre-trainednetwork.
Revised: The key idea of knowledge distillation (Hinton et al., 2015; Mirzadeh et al., 2020; Liu et al., 2019) is to compress a large pre-trainedmodel by teaching a smaller network, step by step, exactly what to do using this larger pre-trainednetwork.

Original: This mechanism has been adopted in model-heterogeneous FL to train the server modelfrom a federation of client models with different architectures.
Revised: This mechanism has been adopted in model-heterogeneous FL to train the server modelfrom a federation of client models with different architectures.

Original: For example, in FedDF, (Lin et al.,2020) uses KD from a set of classifiers pooled from a federation of client devices with their ownprivate data.
Revised: For example, FedDF (Lin et al., 2020)distills knowledge from a set of classifiers trained with private data from a federation of client devices.

Original: The logit outputs of each of the classifiers against a public datasetinput are then usedto train a student model at the server and thereby distill knowledge.
Revised: The logit outputs of each classifier against a public dataset are then used to train a student modelat the server with KD.

Original: In DS-FL (Itahara et al., 2020),an unlabeled public dataset is similarly used. The logit output for this dataset is aggregated andaveraged at the server and is broadcast back to the clients.
Revised: Similarly, DS-FL (Itahara et al., 2020) keeps an unlabeled public datasetat the server, aggregates, and averages the logit outputs, and broadcasts the updated model backto the clients.

Original: Local models now train on the additionalpseudo-labeled data, and performance is enhanced owing to the data augmentation effect.
Revised: Local models now train on the additional pseudo-labeled data, and performance isenhanced owing to the data augmentation effect.

Original: Fed-ET(Cho et al., 2022) considers a weighted consensus distillation approach with diversity regularizationthat enables training of a large server model with smaller models at the clients.
Revised: Fed-ET (Cho et al., 2022) considers a weightedconsensus distillation approach with diversity regularization that enables training of a large servermodel with smaller models at the clients.

Original: These models haveseveral drawbacks.
Revised: These models have several drawbacks.

Original: KD is generally a computationally expensive task and so the costs blow upas the system scales up for larger models and more clients.
Revised: KD is generally acomputationally expensive task and so the costs blow up as the system scales up for larger modelsand more clients.

Original: These are also typically restricted toclassification tasks as they depend on logits for implementing KD.
Revised: These are also typically restricted to classification tasks as they depend on logitsfor implementing KD.

Original: The methods are also likely toface deployment issues, especially for tasks with a large number of possible output classes.
Revised: The methods are also likely to face deployment issues, especially for taskswith a large number of possible output classes.

Original: Partial Training (PT) based Model-Heterogeneous FL.
Revised: Partial Training (PT) based Model-Heterogeneous FL.

Original: There have been several studies employingdifferent methods to train models of different sizes.
Revised: There have been several studies employingdifferent methods to train models of different sizes.

Original: These generally follow PT.
Revised: These generally follow PT.

Original: HeteroFL (Diaoet al., 2020) uses this concept to aggregate models of different sizes.
Revised: HeteroFL (Diaoet al., 2020) uses this concept to aggregate models of different sizes.

Original: Fjord (Horvath et al., 2021) usesordered dropout, which similarly samples submodels from a global model, trains them, and then doesa weighted aggregation.
Revised: Fjord (Horvath et al., 2021) usesordered dropout, which similarly samples submodels from a global model, trains them, and then doesa weighted aggregation.

Original: PruneFL (Jiang et al., 2022) uses model pruning to extract submodels.
Revised: PruneFL (Jiang et al., 2022) uses model pruning to extract submodels.

Original: Thesealgorithms are easily scalable and have minimum overheads on the server.
Revised: Thesealgorithms are easily scalable and have minimum overheads on the server.

Original: However, these methodshave a significant drawback in the sense that different parts of the model are restricted to seeingupdates from a fixed pool of client data.
Revised: However, these methodshave a significant drawback in the sense that different parts of the model are restricted to seeingupdates from a fixed pool of client data.

Original: This limits the performance as the global knowledge isconcentrated on a smaller portion of the model.
Revised: This limits the performance as the global knowledge isconcentrated on a smaller portion of the model.

Original: Furthermore, the server model capability is restrictedto the same capability as the largest client model.
Revised: Furthermore, the server model capability is restrictedto the same capability as the largest client model.

Original: Servers are usually much more computationallycapable and so should be able to use a larger model.
Revised: Servers are usually much more computationallycapable and so should be able to use a larger model.

Original: In comparison, Federated Dropout (Caldaset al., 2018; Guliani et al., 2022; Ro et al., 2022) can train on larger global models and use its entirecapability, especially when the dataset and the number of clients are large.
Revised: In comparison, Federated Dropout (Caldaset al., 2018; Guliani et al., 2022; Ro et al., 2022) can train on larger global models and use its entirecapability, especially when the dataset and the number of clients are large.

Original: Here, a portion of theparameters in each layer of the global model is dropped, leaving a smaller submodel.
Revised: Here, a portion of theparameters in each layer of the global model is dropped, leaving a smaller submodel.

Original: Submodels aretrained on the client devices according to their capacities and the updates are aggregated via weightedaveraging similar to traditional dropout.
Revised: Submodels aretrained on the client devices according to their capacities and the updates are aggregated via weightedaveraging similar to traditional dropout.

Original: However, as reported by Bouacida et al.
Revised: However, as reported by Bouacida et al.

Original: (2020), the approachworks well only with less heterogeneous data and large client pools.
Revised: (2020), the approachworks well only with less heterogeneous data and large client pools.

Original: This is because of high variancein heterogeneous data when only a small subset of the client pool is selected.
Revised: This is because of high variancein heterogeneous data when only a small subset of the client pool is selected.

Original:  Our Method
Revised:  Our Method

Original: 3.1 Problem Formulation of Model-Heterogeneous FL
Revised: 3.1 Problem Formulation of Model-Heterogeneous FL

Original: Consider N different client devices, N , with different compute capacities { C 1 , C 2 , ..., C N } and private data { D 1 , D 2 , ..., D N } , each with different distributions.
Revised: Consider N different client devices, N , with different compute capacities { C 1 , C 2 , ..., C N } and private data { D 1 , D 2 , ..., D N } , each with different distributions.

Original: The objective is to train a large globalmodel characterized by the set of trainable model parameters θ .
Revised: The objective is to train a large globalmodel characterized by the set of trainable model parameters θ .

Original: Therefore, model-heterogeneous FLcan be formulated as the following distributed optimization problem:
Revised: Therefore, model-heterogeneous FLcan be formulated as the following distributed optimization problem:

Original: [Equation]
Revised: [Equation]

Original: where F ( θ )
Revised: where F ( θ )

Original: =
Revised: =

Original: E ξ ∼ D l ( θ ; ξ ) , F n ( · ) is the local objective function on the client dataset D n ={ d n, 1 , d n, 2 , d n, 3 ...
Revised: E ξ ∼ D l ( θ ; ξ ) , F n ( · ) is the local objective function on the client dataset D n ={ d n, 1 , d n, 2 , d n, 3 ...

Original: d n,m n } with respect to the user defined loss function l ( · ; · ) and p n is the corre- sponding weight of the n th client such that p n ≥ 0 and (cid:80) Nn
Revised: d n,m n } with respect to the user defined loss function l ( · ; · ) and p n is the corre- sponding weight of the n th client such that p n ≥ 0 and (cid:80) Nn

Original: =1 p n = 1 .
Revised: =1 p n = 1 .

Original: The n th device contains m ndata points and has model parameters θ n (this can be changed from one iteration to another).
Revised: The n th device contains m ndata points and has model parameters θ n (this can be changed from one iteration to another).

Original: Thesize of θ n depends on the client’s model capacity C n and are extracted from the global model i.e.,θ n ⊂ θ .
Revised: Thesize of θ n depends on the client’s model capacity C n and are extracted from the global model i.e.,θ n ⊂ θ .

Original: The local objective function F n ( · ) is thereby rewritten as:
Revised: The local objective function F n ( · ) is thereby rewritten as:

Original: [Equation]
Revised: [Equation]

Original: For simplicity, we abuse the notation of l by ignoring the other parameters in θ but not in θ n .
Revised: For simplicity, we abuse the notation of l by ignoring the other parameters in θ but not in θ n .

Original: Thecore of the optimization problem is to select subset θ n from the global model parameters θ based onmodel capacity C n .
Revised: Thecore of the optimization problem is to select subset θ n from the global model parameters θ based onmodel capacity C n .

Original: 3.2 Partial Training and FedRolex
Revised: 3.2 Partial Training and FedRolex

Original: As discussed in Section 2, PT-based methods train heterogeneous models by extracting a subset of theglobal model and training the smaller model on private data.
Revised: As discussed in Section 2, PT-based methods train heterogeneous models by extracting a subset of theglobal model and training the smaller model on private data.

Original: The primary step in PT-based methodsis this sub-model extraction.
Revised: The primary step in PT-based methodsis this sub-model extraction.

Original: Consider θ ( j ) n selected from global model weights, θ for client n at thej th communication round.
Revised: Consider θ ( j ) n selected from global model weights, θ for client n at thej th communication round.

Original: Assume that the proportion of nodes extracted from each layer in θ forclient n be β n , which belongs to the set of unique model capacities β .
Revised: Assume that the proportion of nodes extracted from each layer in θ forclient n be β n , which belongs to the set of unique model capacities β .

Original: Let the total number of nodesin the i th layer be K i and the selected nodes are S ( j ) i .
Revised: Let the total number of nodesin the i th layer be K i and the selected nodes are S ( j ) i .

Original: Then the parameters θ ( j ) n, [ i, · ] for the i th layer inextracted model parameters θ ( j ) n consist of { θ ( j ) n, [ i, S ( j ) i, 0 ] , θ ( j ) n, [ i, S ( j ) i, 1 ] , ..., θ ( j ) n, [ i, S ( j ) i,Ki ] } .
Revised: Then the parameters θ ( j ) n, [ i, · ] for the i th layer inextracted model parameters θ ( j ) n consist of { θ ( j ) n, [ i, S ( j ) i, 0 ] , θ ( j ) n, [ i, S ( j ) i, 1 ] , ..., θ ( j ) n, [ i, S ( j ) i,Ki ] } .

Original: Let, M ⊂ N be the set of selected clients from the client pool from which the server pulls modelparameters at round j .
Revised: Let, M ⊂ N be the set of selected clients from the client pool from which the server pulls modelparameters at round j .

Original: Let θ [ i,k ] be the k th trainable weight of the i th layer of global model andθ m, [ i,k ] ; m ∈ M be the k th trainable weight of the i th layer of the client m .
Revised: Let θ [ i,k ] be the k th trainable weight of the i th layer of global model andθ m, [ i,k ] ; m ∈ M be the k th trainable weight of the i th layer of the client m .

Original: The model parametersare aggregated as follows:
Revised: The model parametersare aggregated as follows:

Original: [Equation]
Revised: [Equation]

Original: Here, p m is an empirical weight given to a client.
Revised: Here, p m is an empirical weight given to a client.

Original: This can be based on a variety of factors like theclient model capability, the number of data points a client has, etc.
Revised: This can be based on a variety of factors like theclient model capability, the number of data points a client has, etc.

Original: However, there is no theoreticalfoundation in the literature to quantify the relative importance of a client update nor are thereempirical studies that accurately study this.
Revised: However, there is no theoreticalfoundation in the literature to quantify the relative importance of a client update nor are thereempirical studies that accurately study this.

Original: We consider this tangent to our goals in this study and so,throughout the paper, unless otherwise stated, the weight of all clients is assumed to be the same, i.e,p m = 1 .
Revised: We consider this tangent to our goals in this study and so,throughout the paper, unless otherwise stated, the weight of all clients is assumed to be the same, i.e,p m = 1 .

Original: For different PT-based methods, the method in which the subset of selected parameters, S ischosen varies according to their design.
Revised: For different PT-based methods, the method in which the subset of selected parameters, S ischosen varies according to their design.

Original: Finally, the optimal sub-model is extracted from the globalmodel for each client according to client capability.
Revised: Finally, the optimal sub-model is extracted from the globalmodel for each client according to client capability.

Original: 3.3 Comparing Sub-model Extraction Techniques
Revised: 3.3 Comparing Sub-model Extraction Techniques

Original: FedRolex proposes a new way to extract the sub-models.
Revised: FedRolex proposes a new way to extract the sub-models.

Original: In general, sub-models are created bypulling a subset of weights from the global model.
Revised: In general, sub-models are created bypulling a subset of weights from the global model.

Original: This subset can be dynamic (Federated Dropout,
Revised: This subset can be dynamic (Federated Dropout,

Original: FedRolex ) in each round or remain static for all rounds (HeteroFL).
Revised: FedRolex ) in each round or remain static for all rounds (HeteroFL).

Original: In this section, we compare
Revised: In this section, we compare

Original: FedRolex with two existing methods: static sub-model extraction (HeteroFL) and random sub-modelextraction (Federated Dropout).
Revised: FedRolex with two existing methods: static sub-model extraction (HeteroFL) and random sub-modelextraction (Federated Dropout).

Original: For simplicity, we focus on the i th layer at the n th client during thej th communication round only.
Revised: For simplicity, we focus on the i th layer at the n th client during thej th communication round only.

Original: Let K i denote the number of nodes in the i th layer of the globalmodel and β n denote the capacity of the client model.
Revised: Let K i denote the number of nodes in the i th layer of the globalmodel and β n denote the capacity of the client model.

Original: Static Sub-Model Extraction Scheme
Revised: Static Sub-Model Extraction Scheme

Original: In this scheme (used by HeteroFL), the global model remainsstatic and the extracted sub-models are always the same.
Revised: In this scheme (used by HeteroFL), the global model remainsstatic and the extracted sub-models are always the same.

Original: This is shown in the left column of Figure 1.
Revised: This is shown in the left column of Figure 1.

Original: Unused nodes do not see the full data distribution.
Revised: Unused nodes do not see the full data distribution.

Original: The subset of node indices is given by:
Revised: The subset of node indices is given by:

Original: [Equation]
Revised: [Equation]

Original: K
Revised: K

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Random Sub-Model Extraction Scheme.
Revised: Random Sub-Model Extraction Scheme.

Original: In this scheme (used by Federated Dropout), the submodels are extracted by sampling nodes uniformly and randomly.
Revised: In this scheme (used by Federated Dropout), the submodels are extracted by sampling nodes uniformly and randomly.

Original: This is shown in the middle columnof Figure 1.
Revised: This is shown in the middle columnof Figure 1.

Original: The subset is given by:
Revised: The subset is given by:

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: where ⌊ β n K i ⌋ nodes are randomly chosen from the global model.
Revised: where ⌊ β n K i ⌋ nodes are randomly chosen from the global model.

Original: Unlike static sub-model extraction,since nodes are shuffled, all nodes end up seeing the entire data given enough communication rounds.
Revised: Unlike static sub-model extraction,since nodes are shuffled, all nodes end up seeing the entire data given enough communication rounds.

Original: However, this may require a large number of communication rounds.
Revised: However, this may require a large number of communication rounds.

Original: Rolling Sub-Model Extraction.
Revised: Rolling Sub-Model Extraction.

Original: In this scheme (used by FedRolex ), the global model is rolledbefore the subset is extracted.
Revised: In this scheme (used by FedRolex ), the global model is rolledbefore the subset is extracted.

Original: This is shown in the right column of Figure 1.
Revised: This is shown in the right column of Figure 1.

Original: Let ˆ j = j mod K i , thenthe subset is given by:
Revised: Let ˆ j = j mod K i , thenthe subset is given by:

Original: [Equation]
Revised: [Equation]

Original: That is, the selected nodes start at ˆ j and increase until ⌊ β n K i ⌋ nodes are chosen.
Revised: That is, the selected nodes start at ˆ j and increase until ⌊ β n K i ⌋ nodes are chosen.

Original: When the nodeindex reaches K i , it goes back to one again.
Revised: When the nodeindex reaches K i , it goes back to one again.

Original: Why Rolling Sub-model Extraction is better?
Revised: Why Rolling Sub-model Extraction is better?

Original: PT-based algorithms HeteroFL and FjORD extractthe model parameters θ n from specific locations in the global model.
Revised: PT-based algorithms HeteroFL and FjORD extractthe model parameters θ n from specific locations in the global model.

Original: Here the client model size isbased on the client capacity, and the nodes are selected starting from the first index.
Revised: Here the client model size isbased on the client capacity, and the nodes are selected starting from the first index.

Original: The extractedsub-models are then trained on selected devices from the client pool, and the updates for eachparameter are aggregated and averaged across participating clients for that parameter.
Revised: The extractedsub-models are then trained on selected devices from the client pool, and the updates for eachparameter are aggregated and averaged across participating clients for that parameter.

Original: The nodes inthe beginning indices of each layer will therefore see updates that were averaged across more clientsthan end indices.
Revised: The nodes inthe beginning indices of each layer will therefore see updates that were averaged across more clientsthan end indices.

Original: In comparison, the nodes at the end indices will see updates averaged across muchfewer clients, and therefore the updates to these nodes will have more uncertainty.
Revised: In comparison, the nodes at the end indices will see updates averaged across muchfewer clients, and therefore the updates to these nodes will have more uncertainty.

Original: Dropout-basedalgorithms like Federated Dropout Caldas et al.
Revised: Dropout-basedalgorithms like Federated Dropout Caldas et al.

Original: (2018) propose random dropout to create smallersub-models.
Revised: (2018) propose random dropout to create smallersub-models.

Original: Though the expected value of the frequency for updating each index is the same for allthe indices in Federated Dropout, their exact frequencies are not the same in each experiment becauseof the randomness.
Revised: Though the expected value of the frequency for updating each index is the same for allthe indices in Federated Dropout, their exact frequencies are not the same in each experiment becauseof the randomness.

Revised: This could potentially cause client drift.

Original: In fact, given I indices, if one index is chosen at each round, the probability toupdate all indices once in I rounds is (cid:81) Ii =1 iI .
Revised: In fact, given I indices, if one index ischosen at each round, the probability to update all indices once in I rounds is (cid:81) Ii =1 iI .

Original: Furthermore, the expected number of rounds to gothrough all indices at least once is close to I ln
Revised: Furthermore,the expected number of rounds to go through all indices at least once is close to I ln

Original: ( I ) , see Appendix A.1.
Revised: ( I ) , see Appendix A.1.

Original: It shows that random dropoutcan not balance the update frequencies of the indices, and it takes a large number of rounds to updateall the indices.
Revised: It shows that random dropout can not balance the update frequencies of the indices, and it takesa large number of rounds to update all the indices.

Original:  Experiments and Analysis
Revised:  Experiments and Analysis

Original: 4.1 Experimental Setup
Revised: 4.1 Experimental Setup

Original: Datasets and Platform.
Revised: Datasets and Platform.

Original: We validate the performance of FedRolex on three datasets – CIFAR-10,
Revised: We validate the performance of FedRolex on three datasets – CIFAR-10,

Original: CIFAR-100, and StackOverflow.
Revised: CIFAR-100, and StackOverflow.

Original: We apply bounding box crop (Zoph et al., 2020) to augment theimages.
Revised: We apply bounding box crop (Zoph et al., 2020) to augment theimages.

Original: To simulate the non-iid nature of the data, we use balanced non-iid data distribution following
Revised: To simulate the non-iid nature of the data, we use balanced non-iid data distribution following

Original: HeteroFL.
Revised: HeteroFL.

Original: We skew the data distribution by restricting the clients to having a fixed number of uniquelabels L .
Revised: We skew the data distribution by restricting the clients to having a fixed number of uniquelabels L .

Original: We define L = 2 as high data heterogeneity (i.e., high non-iid) and L = 5 as low dataheterogeneity (i.e., low non-iid) for CIFAR-10.
Revised: We define L = 2 as high data heterogeneity (i.e., high non-iid) and L = 5 as low dataheterogeneity (i.e., low non-iid) for CIFAR-10.

Original: Similarly, we use L = 20 as high non-iid and L = 50as low non-iid for CIFAR-100.
Revised: Similarly, we use L = 20 as high non-iid and L = 50as low non-iid for CIFAR-100.

Original: These roughly correspond to Dirichlet distribution Dir K ( α ) with αset to 0 .
Revised: These roughly correspond to Dirichlet distribution Dir K ( α ) with αset to 0 .

Original: 1 and 0 . 5 , respectively.
Revised: 1 and 0 . 5 , respectively.

Original: The data is partitioned into 100 non-iid partitions and assigned to theclient pool.
Revised: The data is partitioned into 100 non-iid partitions and assigned to theclient pool.

Original: In each communication round, 10% of the clients are randomly selected from the clientpool of 100 .
Revised: In each communication round, 10% of the clients are randomly selected from the clientpool of 100 .

Original: For StackOverflow, the data is partitioned over user ids.
Revised: For StackOverflow, the data is partitioned over user ids.

Original: The total number of clients,in this case, was 342 , 477 and in each round, 200 clients were sampled.
Revised: The total number of clients,in this case, was 342 , 477 and in each round, 200 clients were sampled.

Original: We use a vocabulary size
Revised: We use a vocabulary size

Original: Algorithm
Revised: Algorithm

Original: 1 Initialization ; θ (0) , N
Revised: 1 Initialization ; θ (0) , N

Original: Input : D n β n ∀ n ∈ N ,
Revised: Input : D n β n ∀ n ∈ N ,

Original: Output : θ J 2 Server Executes 3 for j ← 0 to J do
Revised: Output : θ J 2 Server Executes 3 for j ← 0 to J do

Original: Broadcast θ ( j ) n, [ i, S ( j ) i,k ] ∀ i, S from Equation 6 , 1 ≤ k ≤ K i and n ∈
Revised: Broadcast θ ( j ) n, [ i, S ( j ) i,k ] ∀ i, S from Equation 6 , 1 ≤ k ≤ K i and n ∈

Original: N
Revised: N

Original: Sample subset M from N for each client m ∈ M do clientStep( θ ( j ) n , D n ) end
Revised: Sample subset M from N for each client m ∈ M do clientStep( θ ( j ) n , D n ) end

Original: Aggregate θ ( j +1)
Revised: Aggregate θ ( j +1)

Original: [ i,k ] = 1 (cid:80) m ∈M 1 (cid:80)
Revised: [ i,k ] = 1 (cid:80) m ∈M 1 (cid:80)

Original: m ∈M θ m, [ i,k ] 11 end 12 Subroutine clientStep( θ ( j ) n , D n )
Revised: m ∈M θ m, [ i,k ] 11 end 12 Subroutine clientStep( θ ( j ) n , D n )

Original: m n ←− len
Revised: m n ←− len

Original: ( D n ) for k ← 0 to m n do θ n ←− θ n − η ∇ l ( θ n ; d n,k ) end return θ n of 10 , 000 words.
Revised: ( D n ) for k ← 0 to m n do θ n ←− θ n − η ∇ l ( θ n ; d n,k ) end return θ n of 10 , 000 words.

Original: We run our experiments on 8 Nvidia A 6000 GPUs using the PyTorch library formachine learning and the Ray library for simulating the distributed system.
Revised: We run our experiments on 8 Nvidia A 6000 GPUs using the PyTorch library formachine learning and the Ray library for simulating the distributed system.

Original: Models.
Revised: Models.

Original: We use pre-activated ResNet18 (PreResNet18) models (He et al., 2016) for CIFAR-10and CIFAR-100.
Revised: We use pre-activated ResNet18 (PreResNet18) models (He et al., 2016) for CIFAR-10and CIFAR-100.

Original: We replace the batch Normalization in PreResNet18 with static batch normalization (Diao et al., 2020;
Revised: We replace the batch Normalization in PreResNet18 with static batch normalization (Diao et al., 2020;

Original: Andreux et al., 2020) and add a scalar module after each convolutionlayer (Diao et al., 2020).
Revised: Andreux et al., 2020) and add a scalar module after each convolutionlayer (Diao et al., 2020).

Original: For StackOverflow experiments, we trained a modified 3-layer Transformermodel, where the dimension of the token embeddings was 128 , and the hidden dimension of thefeed-forward network(FFN) block is 2048 .
Revised: For StackOverflow experiments, we trained a modified 3-layer Transformermodel, where the dimension of the token embeddings was 128 , and the hidden dimension of thefeed-forward network(FFN) block is 2048 .

Original: We use 8 heads for the multi-head attention, where eachhead is based on 12-dimensional (query, key, value) vectors.
Revised: We use 8 heads for the multi-head attention, where eachhead is based on 12-dimensional (query, key, value) vectors.

Original: We use ReLU activation and set thedropout rate to 0.1.
Revised: We use ReLU activation and set thedropout rate to 0.1.

Original: To simplify our experiments and ablation studies, the model sizes are restricted toa set of 5 , i.e, β n would take any value in { 1 , 1 / 2 , 1 / 4 , 1 / 8 , 1 / 16 } .
Revised: To simplify our experiments and ablation studies, the model sizes are restricted toa set of 5 , i.e, β n would take any value in { 1 , 1 / 2 , 1 / 4 , 1 / 8 , 1 / 16 } .

Original: Note that we only vary the numberof kernels in convolution layers and keep the nodes in the output layers the same in both our methodand the PT-based baselines to keep the comparison fair.
Revised: Note that we only vary the numberof kernels in convolution layers and keep the nodes in the output layers the same in both our methodand the PT-based baselines to keep the comparison fair.

Original: Similarly, for the Transformer model, weonly vary the number of nodes in the hidden layer of the attention heads.
Revised: Similarly, for the Transformer model, weonly vary the number of nodes in the hidden layer of the attention heads.

Original: We provide pseudocodefor both HeteroFL and our Federated Dropout variant in Appendix A.7.
Revised: We provide pseudocodefor both HeteroFL and our Federated Dropout variant in Appendix A.7.

Original: Unless otherwise stated, theclient capacity distribution is assumed to be uniform over the entire client pool for all methods.
Revised: Unless otherwise stated, theclient capacity distribution is assumed to be uniform over the entire client pool for all methods.

Original: Welist the exact hyperparameters and architecture of the models in Appendix A.

Revised: Welist the exact hyperparameters and architecture of the models in Appendix A.

Original: Baselines.
Revised: Baselines.

Original: We compare FedRolex against state-of-the-art PT-based methods HeteroFL and Federated
Revised: We compare FedRolex against state-of-the-art PT-based methods HeteroFL and Federated

Original: Dropout and state-of-the-art KD-based methods FedDF, DS-FL, and Fed-ET obtained from Lin et al.(2020).
Revised: Dropout and state-of-the-art KD-based methods FedDF, DS-FL, and Fed-ET obtained from Lin et al.(2020).

Original: For a fair comparison, all the baselines from the PT-based methods are trained on the samelearning rate, the same number of communication rounds, and the same multi-step learning rate decayschedule.
Revised: For a fair comparison, all the baselines from the PT-based methods are trained on the samelearning rate, the same number of communication rounds, and the same multi-step learning rate decayschedule.

Original: The exact schedule for each dataset and experiment is given in Appendix A.6.
Revised: The exact schedule for each dataset and experiment is given in Appendix A.6.

Original: Evaluation Metrics.
Revised: Evaluation Metrics.

Original: We use local and global model accuracy as our evaluation metrics.
Revised: We use local and global model accuracy as our evaluation metrics.

Original: Specifically,global model accuracy is defined as the mean accuracy on the test set totally held out during training;local model accuracy is defined as the mean accuracy of the server model on each of the clientdatasets.
Revised: Specifically,global model accuracy is defined as the mean accuracy on the test set totally held out during training;local model accuracy is defined as the mean accuracy of the server model on each of the clientdatasets.

Original: For vision datasets CIFAR-10 and CIFAR-100, we measure classification accuracy andfor the StackOverflow dataset, we measure next word prediction accuracy.
Revised: For vision datasets CIFAR-10 and CIFAR-100, we measure classification accuracy andfor the StackOverflow dataset, we measure next word prediction accuracy.

Original: Since there is somerandomness due to the non-iid partitioning, we run our experiments on 5 different seeds and list themean and standard deviation of our metrics.
Revised: Since there is somerandomness due to the non-iid partitioning, we run our experiments on 5 different seeds and list themean and standard deviation of our metrics.

Original: Note that the results of KD-based methods were obtainedfrom Cho et al.
Revised: Note that the results of KD-based methods were obtainedfrom Cho et al.

Original: (2022) using 3 different seeds.
Revised: (2022) using 3 different seeds.

Original: 4.2 Performance Comparison with State-of-the-Art Model-Heterogeneous FL Methods
Revised: 4.2 Performance Comparison with State-of-the-Art Model-Heterogeneous FL Methods

Original: shows the global model accuracy comparison between FedRolex and the baselines underboth high and low data heterogeneity settings.
Revised: shows the global model accuracy comparison between FedRolex and the baselines underboth high and low data heterogeneity settings.

Original: We also include the global model accuracy of twomodel-homogeneous cases where all clients have the highest capacity ( β = 1 . 0 ) and lowest capacity
Revised: We also include the global model accuracy of twomodel-homogeneous cases where all clients have the highest capacity ( β = 1 . 0 ) and lowest capacity

Original: [Table]
Revised: [Table]

Original: ( β = 0 .
Revised: ( β = 0 .

Original: representing the upper and lower-bound performance, respectively.
Revised: representing the upper and lower-bound performance, respectively.

Original: We have threekey observations.
Revised: We have threekey observations.

Original: In comparison with state-of-the-art PT-based methods, FedRolex consistentlyoutperforms HeteroFL and Federated Dropout in both high and low data heterogeneity scenarios.
Revised: In comparison with state-of-the-art PT-based methods, FedRolex consistentlyoutperforms HeteroFL and Federated Dropout in both high and low data heterogeneity scenarios.

Original: Inparticular, under high data heterogeneity, Federated Dropout that uses random dropout has drasticallyworse performance than FedRolex and HeteroFL which both extract sub-models in a deterministicmanner.
Revised: Inparticular, under high data heterogeneity, Federated Dropout that uses random dropout has drasticallyworse performance than FedRolex and HeteroFL which both extract sub-models in a deterministicmanner.

Original: In comparison with state-of-the-art KD-based methods, FedRolex only performs worsethan Fed-ET and FedDF on CIFAR-10 under high data heterogeneity, but outperforms all the state-ofthe-art in more challenging CIFAR-100 dataset that has many more classes than CIFAR-10 underboth high and low data heterogeneity scenarios.
Revised: In comparison with state-of-the-art KD-based methods, FedRolex only performs worsethan Fed-ET and FedDF on CIFAR-10 under high data heterogeneity, but outperforms all the state-ofthe-art in more challenging CIFAR-100 dataset that has many more classes than CIFAR-10 underboth high and low data heterogeneity scenarios.

Original: It is important to emphasize that KD-based methodsleverage public data to boost their model accuracy while FedRolex does not.
Revised: It is important to emphasize that KD-based methodsleverage public data to boost their model accuracy while FedRolex does not.

Original: In comparison withmodel-homogeneous cases, compared to other PT-based methods, FedRolex considerably reducesthe gap between model-heterogeneous and model-homogeneous settings, particularly on CIFAR-10under low data heterogeneity.
Revised: In comparison withmodel-homogeneous cases, compared to other PT-based methods, FedRolex considerably reducesthe gap between model-heterogeneous and model-homogeneous settings, particularly on CIFAR-10under low data heterogeneity.

Original: Note that Fed-ET achieves a higher global model accuracy than themodel-homogeneous upper bound on CIFAR-10 under high data heterogeneity, which showcases theadvantage of having access to public data.
Revised: Note that Fed-ET achieves a higher global model accuracy than themodel-homogeneous upper bound on CIFAR-10 under high data heterogeneity, which showcases theadvantage of having access to public data.

Original: 4.3 Impact of Model Capacity Distribution
Revised: 4.3 Impact of Model Capacity Distribution

Original: In our previous experiment, in line with existing model-heterogeneous FL studies, we have performedour experiment where the distributions of model capacities are uniform for a fair comparison.
Revised: In our previous experiment, in line with existing model-heterogeneous FL studies, we have performedour experiment where the distributions of model capacities are uniform for a fair comparison.

Original: Inpractical scenarios, however, the distributions of model capacities will likely be skewed.
Revised: Inpractical scenarios, however, the distributions of model capacities will likely be skewed.

Original: Therefore,we conduct an experiment to understand how global model accuracy changes when the proportion ofhigher capacity clients varies.
Revised: Therefore,we conduct an experiment to understand how global model accuracy changes when the proportion ofhigher capacity clients varies.

Original: To do so, we set the proportion of large to small models to ρ and theset of unique client capacities is β
Revised: To do so, we set the proportion of large to small models to ρ and theset of unique client capacities is β

Original: = { 1 , 1 / 16 } where ρ = 1 represents the edge case in which themodels of all clients have the architecture of the full ResNet18 model, and ρ = 0 is the other edgecase in which all clients have the smallest model.
Revised: = { 1 , 1 / 16 } where ρ = 1 represents the edge case in which themodels of all clients have the architecture of the full ResNet18 model, and ρ = 0 is the other edgecase in which all clients have the smallest model.

Original: Figure 2 shows how global model accuracy changes when ρ varies from 0 to 1 for CIFAR-10 and
Revised: Figure 2 shows how global model accuracy changes when ρ varies from 0 to 1 for CIFAR-10 and

Original: CIFAR-100 under both high and low data heterogeneity.
Revised: CIFAR-100 under both high and low data heterogeneity.

Original: We have three key observations.
Revised: We have three key observations.

Original: (1) For
Revised: (1) For

Original: CIFAR-10, there is a large gap in global model accuracy between high and low data heterogeneityfor a wide range of ρ (from 0.1 to 1).
Revised: CIFAR-10, there is a large gap in global model accuracy between high and low data heterogeneityfor a wide range of ρ (from 0.1 to 1).

Original: This is because CIFAR-10 is a relatively simple task andhence the global model accuracy is bottlenecked by the level of data heterogeneity instead of modelcapacity.
Revised: This is because CIFAR-10 is a relatively simple task andhence the global model accuracy is bottlenecked by the level of data heterogeneity instead of modelcapacity.

Original: This result indicates that having more high-capacity models in the cohort has limitedcontribution to global model accuracy.
Revised: This result indicates that having more high-capacity models in the cohort has limitedcontribution to global model accuracy.

Original: For the more challenging CIFAR-100, the gap in globalmodel accuracy is much lower between high and low data heterogeneity.
Revised: For the more challenging CIFAR-100, the gap in globalmodel accuracy is much lower between high and low data heterogeneity.

Original: In contrast to CIFAR-10, theglobal model accuracy is bottlenecked by the highest capacity of the models rather than the level ofdata heterogeneity.
Revised: In contrast to CIFAR-10, theglobal model accuracy is bottlenecked by the highest capacity of the models rather than the level ofdata heterogeneity.

Original: For both CIFAR-10 and CIFAR-100, we observe that having a small fractionof high-capacity models is enough to significantly boost the global model accuracy.
Revised: For both CIFAR-10 and CIFAR-100, we observe that having a small fractionof high-capacity models is enough to significantly boost the global model accuracy.

Original: 4.4 Scalability
Revised: 4.4 Scalability

Original: One limitation of HeteroFL and FjORD is that the server model size is restricted to the highestcapacity client model.
Revised: One limitation of HeteroFL and FjORD is that the server model size is restricted to the highestcapacity client model.

Original: In contrast, FedRolex is flexible to train a global model that can be much
Revised: In contrast, FedRolex is flexible to train a global model that can be much

Original: [Figure]
Revised: [Figure]

Original: larger than the largest client model.
Revised: larger than the largest client model.

Original: This is more akin to real-world settings where the server has amuch higher capacity.
Revised: This is more akin to real-world settings where the server has amuch higher capacity.

Original: We consider the case where the global model size is γ times the size of the highest capacity clientmodel.
Revised: We consider the case where the global model size is γ times the size of the highest capacity clientmodel.

Original: Figure 3 shows the global model accuracy of γ = { 2 , 4 , 8 , 16 } for CIFAR-10 and CIFAR-100under high and low data heterogeneity.
Revised: Figure 3 shows the global model accuracy of γ = { 2 , 4 , 8 , 16 } for CIFAR-10 and CIFAR-100under high and low data heterogeneity.

Original: As shown, for both CIFAR-10 and CIFAR-100, the globalmodel accuracy drastically drops when γ increases from 2 to 4 but stay relatively stable when γincreases from 4 to 8 and from 8 to 16 .
Revised: As shown, for both CIFAR-10 and CIFAR-100, the globalmodel accuracy drastically drops when γ increases from 2 to 4 but stay relatively stable when γincreases from 4 to 8 and from 8 to 16 .

Original: This result indicates that there is a minimum client modelcapacity below which the server model cannot be trained effectively.
Revised: This result indicates that there is a minimum client modelcapacity below which the server model cannot be trained effectively.

Original: [Figure]
Revised: [Figure]

Original: 4.5 Enhance Inclusiveness of FL in Real-world Distribution
Revised: 4.5 Enhance Inclusiveness of FL in Real-world Distribution

Original: A primary goal of our method is to enhance the inclusiveness of FL.
Revised: A primary goal of our method is to enhance the inclusiveness of FL.

Original: If the highest model capacity isused for deployment, a lot of data from the low-end devices will be unused.
Revised: If the highest model capacity isused for deployment, a lot of data from the low-end devices will be unused.

Original: The exclusion of low-enddevices limits the usefulness of the global model for these low-end devices.
Revised: The exclusion of low-enddevices limits the usefulness of the global model for these low-end devices.

Original: For example, low-enddevices may have more grainy, low-resolution photos which will not be used in training, and so, theserver model cannot make accurate predictions on such images and hence limiting its usefulnessfor these devices.
Revised: For example, low-enddevices may have more grainy, low-resolution photos which will not be used in training, and so, theserver model cannot make accurate predictions on such images and hence limiting its usefulnessfor these devices.

Original: If we consider homogeneous FL where we want to include all the client data intraining, the smallest model would need to be used.
Revised: If we consider homogeneous FL where we want to include all the client data intraining, the smallest model would need to be used.

Original: The accuracy of the model is then limited by thelowest capacity client.
Revised: The accuracy of the model is then limited by thelowest capacity client.

Original: In this experiment, we show that FedRolex is a valid solution that enhances the inclusiveness offederated learning.
Revised: In this experiment, we show that FedRolex is a valid solution that enhances the inclusiveness offederated learning.

Original: To illustrate our point, we conducted an experiment that uses real-world householdincome distribution to emulate real-world device distribution.
Revised: To illustrate our point, we conducted an experiment that uses real-world householdincome distribution to emulate real-world device distribution.

Original: Specifically, we retrieve householdincome distribution information from Bureau (2021).
Revised: Specifically, we retrieve householdincome distribution information from Bureau (2021).

Original: We map β n = 1 / 16 with the income group withearning less than $ 75000 and assign proportions of remaining groups in $ 25000 increments with
Revised: We map β n = 1 / 16 with the income group withearning less than $ 75000 and assign proportions of remaining groups in $ 25000 increments with

Original: [Figure]
Revised: [Figure]

Original: increasing values of β n in the statistic.
Revised: increasing values of β n in the statistic.

Original: Detailed mapping of this distribution to the correspondingincome distribution is given in Appendix A.
Revised: Detailed mapping of this distribution to the correspondingincome distribution is given in Appendix A.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Table 3 shows the global and local model accuracy of FedRolex on this emulated setting comparedto two model-homogeneous cases where all clients have the smallest and largest model capacities,representing lower and upper bound on accuracy respectively.
Revised: Table 3 shows the global and local model accuracy of FedRolex on this emulated setting comparedto two model-homogeneous cases where all clients have the smallest and largest model capacities,representing lower and upper bound on accuracy respectively.

Original: We make two key observations.
Revised: We make two key observations.

Original: FedRolex consistently and significantly outperforms the lower bound model-homogeneous casein terms of local model accuracy for CIFAR-10 and CIFAR-100 under both high and low dataheterogeneity.
Revised: FedRolex consistently and significantly outperforms the lower bound model-homogeneous casein terms of local model accuracy for CIFAR-10 and CIFAR-100 under both high and low dataheterogeneity.

Original: This result indicates that FedRolex can effectively boost the local model accuracy oflow-end devices, which would otherwise not benefit from FL.
Revised: This result indicates that FedRolex can effectively boost the local model accuracy oflow-end devices, which would otherwise not benefit from FL.

Original: (2) FedRolex achieves a local modelaccuracy that is close to the upper bound model-homogeneous case, especially for CIFAR-10 underboth high and low data heterogeneity.
Revised: (2) FedRolex achieves a local modelaccuracy that is close to the upper bound model-homogeneous case, especially for CIFAR-10 underboth high and low data heterogeneity.

Original: This result indicates that FedRolex can significantly reducethe gap with the best-performing scenario without being constrained to only using high-end devices.
Revised: This result indicates that FedRolex can significantly reducethe gap with the best-performing scenario without being constrained to only using high-end devices.

Original: Finally, Figure 4 provides a detailed illustration of how the local model accuracy shifts for CIFAR-10and CIFAR-100 under low and high data heterogeneity when FedRolex is used compared to thesmallest homogeneous case with the same client outreach.
Revised: Finally, Figure 4 provides a detailed illustration of how the local model accuracy shifts for CIFAR-10and CIFAR-100 under low and high data heterogeneity when FedRolex is used compared to thesmallest homogeneous case with the same client outreach.

Original: As shown, the local model accuracy ofindividual clients is significantly boosted by FedRolex .
Revised: As shown, the local model accuracy ofindividual clients is significantly boosted by FedRolex .

Original:  Conclusion and Future Directions
Revised:  Conclusion and Future Directions

Original: In this work, we present a partial training (PT)-based model-heterogeneous method named FedRolexthat tackles the fundamental issue of existing PT-based methods.
Revised: In this work, we present a partial training (PT)-based model-heterogeneous method named FedRolexthat tackles the fundamental issue of existing PT-based methods.

Original: At the core of FedRolex is a simpleyet effective rolling sub-model extraction scheme that enables parameters of the global model to beevenly trained to minimize client drift.
Revised: At the core of FedRolex is a simpleyet effective rolling sub-model extraction scheme that enables parameters of the global model to beevenly trained to minimize client drift.

Original: Our experimental results show that FedRolex consistentlyoutperforms state-of-the-art PT-based methods in both high and low data heterogeneity scenarios.
Revised: Our experimental results show that FedRolex consistentlyoutperforms state-of-the-art PT-based methods in both high and low data heterogeneity scenarios.

Original: Wedemonstrate its effectiveness in real-world scenarios by showing its performance on a more practicalmodel capacity distribution and show FedRolex contributes to making FL more inclusive.
Revised: Wedemonstrate its effectiveness in real-world scenarios by showing its performance on a more practicalmodel capacity distribution and show FedRolex contributes to making FL more inclusive.

Original: In themeantime, we note that although we provide a statistical analysis of FedRolex , we do not derive arigorous convergence analysis.
Revised: In themeantime, we note that although we provide a statistical analysis of FedRolex , we do not derive arigorous convergence analysis.

Original: This is not a trivial task and therefore left for future work.
Revised: This is not a trivial task and therefore left for future work.

Original: In addition,our goal in this paper is to train a large global model using a federation of clients with heterogeneouson-device resources.
Revised: In addition,our goal in this paper is to train a large global model using a federation of clients with heterogeneouson-device resources.

Original: Determining what models to deploy onto each heterogeneous client after theglobal model has finished training during the federated learning process is a separate task.
Revised: Determining what models to deploy onto each heterogeneous client after theglobal model has finished training during the federated learning process is a separate task.

Original: This canbe achieved by a few techniques such as compressing/quantizing/KD the global model into smallermodels to fit the resources of the target client.
Revised: This canbe achieved by a few techniques such as compressing/quantizing/KD the global model into smallermodels to fit the resources of the target client.

Original: However, this is not the focus of this work and is aseparate study in itself.
Revised: However, this is not the focus of this work and is aseparate study in itself.

Original: We will pursue it in our future work as well.
Revised: We will pursue it in our future work as well.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] Our work’ claims are verified through experiment inSection 4.2 and statistical study in Appendix A.1(b)
Revised: [Yes] Our work’ claims are verified through experiment inSection 4.2 and statistical study in Appendix A.1(b)

Original: Did you describe the limitations of your work?
Revised: Did you describe the limitations of your work?

Original: [Yes] The limitations are mentioned inSection 5(c)
Revised: [Yes] The limitations are mentioned inSection 5(c)

Original: Did you discuss any potential negative societal impacts of your work?
Revised: Did you discuss any potential negative societal impacts of your work?

Original: [No](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: [No](d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]2.
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] We do notshow any rigorous theoretical results nor claim as such
Revised: [N/A] We do notshow any rigorous theoretical results nor claim as such

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A] We do not show any
Revised: [N/A] We do not show any

Original: rigorous theoretical results nor claim as such
Revised: rigorous theoretical results nor claim as such

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes] Code is madeavailable to reviewers.
Revised: [Yes] Code is madeavailable to reviewers.

Original: Data is open source.
Revised: Data is open source.

Original: Instructions to run are available in code(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: Instructions to run are available in code (b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes] All major training details are mentioned in Section 4 and finedetails provided in Appendix A.6(c)
Revised: [Yes] All major training details are mentioned in Section 4 and fine details provided in Appendix A.6(c)

Original: Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?

Original: [Yes] All experiments were performed with 5 different seedsand the mean and standard deviations were reported when applicable.
Revised: [Yes] All experiments were performed with 5 different seeds and the mean and standard deviations were reported when applicable.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes]
Revised: [Yes]

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets.
Revised: If you are using existing assets (e g., code, data, models) or curating/releasing new assets..

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]