Original: FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-model Extraction
Revised: FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-model Extraction

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Federated learning (FL) is a collaborative machine learning paradigm to trainmodels from decentralized private data.
Revised: Federated learning (FL) is a collaborative machine learning paradigm to trainmodels from decentralized private data.

Original: Most FL research focuses on modelhomogeneous setting where models deployed across all the participating clientsand the server are required to be identical.
Revised: Most FL research focuses on modelhomogeneous setting where models deployed across all the participating clientsand the server are required to be identical.

Original: However, in real-world scenarios,such a requirement acts as a constraint that restricts the outreach to clients withheterogeneous device resources and unfairly excludes users with low-end deviceswho would otherwise beneﬁt from FL.
Revised: However, in real-world scenarios,such a requirement acts as a constraint that restricts the outreach to clients withheterogeneous device resources and unfairly excludes users with low-end deviceswho would otherwise benefit from FL.

Original: In this work, we propose a simple yeteffective model-heterogeneous FL method named FedRolex to tackle this constraint.
Revised: In this work, we propose a simple yeteffective model-heterogeneous FL method named FedRolex to tackle this constraint.

Original: Unlike the model-homogeneous scenario, the fundamental challenge ofmodel heterogeneity in FL is that different parameters of the global model aretrained on heterogeneous data distributions.
Revised: Unlike the model-homogeneous scenario, the fundamental challenge ofmodel heterogeneity in FL is that different parameters of the global model aretrained on heterogeneous data distributions.

Original: Our method addresses this challengeby rolling the sub-model in each federated iteration so that the parameters of theglobal model are evenly trained on the global data distribution across all devices,making it more akin to model-homogeneous training.
Revised: Our method addresses this challengeby rolling the sub-model in each federated iteration so that the parameters of theglobal model are evenly trained on the global data distribution across all devices,making it more akin to model-homogeneous training.

Original: Our experiments show that
Revised: Our experiments show that

Original: FedRolex outperforms other state-of-the-art model-heterogeneous FL methods,especially under high data-heterogeneity scenarios.
Revised: FedRolex outperforms other state-of-the-art model-heterogeneous FL methods,especially under high data-heterogeneity scenarios.

Original: We have conducted ablationstudies to show that submodel rolling is an effective technique to reduce the gapbetween model-heterogeneous and standard model-homogeneous settings.
Revised: We have conducted ablationstudies to show that submodel rolling is an effective technique to reduce the gapbetween model-heterogeneous and standard model-homogeneous settings.

Original: Lastly,we consider the distribution of client capabilities that is similar to real-world income distribution instead of the uniform distribution used in existing works.
Revised: Lastly,we consider the distribution of client capabilities that is similar to real-world income distribution instead of the uniform distribution used in existing works.

Original: Ourresults show a consistent improvement in the accuracies on low-end devices, whichenhances the inclusiveness of federated learning.
Revised: Ourresults show a consistent improvement in the accuracies on low-end devices, whichenhances the inclusiveness of federated learning.

Original:  Introduction
Revised:  Introduction

Original: Cross-device federated learning (FL) was originally proposed by McMahan et al.
Revised: Cross-device federated learning (FL) was originally proposed by McMahan et al.

Original: (2017) as a privacypreserving machine learning paradigm to train a machine learning model on a federation of resourceconstrained edge devices without accessing their data.
Revised: (2017) as a privacypreserving machine learning paradigm to train a machine learning model on a federation of resourceconstrained edge devices without accessing their data.

Original: This idea has far-reaching applications, whichare made urgent by modern demands for privacy.
Revised: This idea has far-reaching applications, whichis made urgent by modern demands for privacy.

Original: Indeed, the goal of strong data privacy is a centralmotivation for FL.
Revised: Indeed, the goal of strong data privacy is a central motivation for FL.

Original: By storing data locally, instead of replicating them on a remote server, the attacksurface of the system is decreased (Jere et al., 2020), and by using focused ephemeral updates andearly aggregation, the communication cost is also minimized (Nishio and Yonetani, 2019).
Revised: By storing data locally, instead of replicating them on a remote server, the attacksurface of the system is decreased (Jere et al., 2020), and by using focused ephemeral updates andearly aggregation, the communication cost is also minimized (Nishio and Yonetani, 2019).

Original: Majority ofthe existing cross-device FL studies focus on model-homogeneous setting, where modelsdeployed across all the participating clients and the server are required to be identical.
Revised: The majority of existing cross-device FL studies focus on model-homogeneous setting, where models deployed across all the participating clients and the server are required to be identical.

Original: In practice,device heterogeneity is a  realistic constraint to consider for deploying in real-world environments.
Revised: In practice, device heterogeneity is a more realistic constraint to consider for deploying in real-world environments

Original: Different edge devices have different on-device resources and, hence, are only capable of trainingmodels with capacities that match their on-device resources.
Revised: Different edge devices have different on-device resources and, hence, are only capable of training models with capacities that match their on-device resources.

Original: Having the same model on all thedevices forces a limitation that restricts the outreach to clients with heterogeneous device resources and unfairly excludes users with low-end devices who would otherwise beneﬁt from FL
Revised: Having the same model on all the devices forces a limitation that restricts the outreach to clients with heterogeneous device resources and unfairly excludes users with low-end devices who would otherwise benefit from FL

Original: [Table]
Revised: [Table]

Original: To tackle this  constraint 
Revised: To relax this fundamental constraint in standard FL

Original: [Table]
Revised: [Table]

Original: recent works proposed model-heterogeneous FL
Revised: recent works proposed model-heterogeneous FL

Original: [Table]
Revised: [Table]

Original: where models with differentcapacities are trained on edge devices with heterogeneous resources during the federated trainingprocess.
Revised: where models with different capacities are trained on edge devices with heterogeneous resourcesduring the federated training process.

Original: One fundamental challenge in model-heterogeneous FL is the aggregation of heterogeneousclient models.
Revised: One fundamental challenge in model-heterogeneous FL isthe aggregation of heterogeneous client models.

Original: Lopes et al.
Revised: Lopes et al.

Original: ; He et al.
Revised: ; He et al.

Original:  Cho et al. (2022) proposed to use knowledgedistillation (KD) (Hinton et al., 2015)
Revised: ; Cho et al.(2022) proposed to use knowledge distillation (KD) (Hinton et al., 2015)

Original: [Table]
Revised: [Table]

Original: where the client models serve as teachers and  server ensemblesthe knowledge distilled from the individual client models.
Revised: where the client modelsserve as teachers and the server ensembles the knowledge distilled from the individual client models.

Original: However, KD-based methods generallyrequire access to public data to achieve competitive model accuracy [Table] where the accessibility of highquality public data is not always practical.
Revised: However, KD-based methods generally require access to public data to achieve competitive modelaccuracy, where the accessibility of high-quality public data is not always practical.

Original: Moreover, KD becomes computationally expensive whenthe cohort size in each FL round scales up.
Revised: Moreover, KDbecomes computationally expensive when the cohort size in each FL round scales up.

Original: To remove the dependency on public data, another set ofstudies focuses on designing partial training (PT) based techniques such as federated dropout (Caldaset al., 2018), ordered dropout (Horvath et al., 2021), and static sub-model extraction (Diao et al.
Revised: To remove thedependency on public data, another set of studies focuses on designing partial training (PT) basedtechniques such as federated dropout (Caldas et al., 2018), ordered dropout (Horvath et al., 2021),and static sub-model extraction (Diao et al., 2020), where each client trains a smaller sub-model ofthe larger global model on the server.

Original: ,2020), where each client trains a smaller sub-model of the larger global model on theserver. However,the fundamental issue of existing PT-based methods is that the sub-models are extracted in ways suchthat the parameters of the global model obtained by aggregating the extracted sub-models are notevenly trained, making the global model vulnerable to a phenomenon called client drift (Wang et al.,2021).
Revised: However, the fundamental issue of existing PT-based methodsis that the sub-models are extracted in ways such that the parameters of the global model obtained byaggregating the extracted sub-models are not evenly trained, making the global model vulnerable to aphenomenon called client drift (Wang et al., 2021) where it is used to describe the error in gradientestimation due to the differences in data distributions between clients. However, this estimation errorcan originate from any discrepancy across the federation.

Revised: For instance, in model heterogeneousfederated learning, client drift will also occur due to the inconsistency between the clients’ individualmodel architectures in addition to the difference in private data distributions.

Original: This phenomenon becomes more prominent when the cohort size in each communicationround is small or data heterogeneity across clients becomes more severe, resulting in lower globalmodel accuracy.
Revised: This also becomes moreprominent when the cohort size in each communication round is small or data heterogeneity acrossclients becomes more severe, resulting in lower global model accuracy.

Original: In this work, we propose a simple yet effective PT-based model-heterogeneous FL method named
Revised: In this work, we propose a simple yet effective PT-based model-heterogeneous FL method named

Original: FedRolex that tackles the fundamental issue of existing PT-based methods.
Revised: FedRolex that tackles the fundamental issue of existing PT-based methods.

Original: Compared to existing
Revised: Compared to existing

Original: PT-based methods
Revised: PT-based methods

Original: , FedRolex proposes a rolling sub-model extraction scheme, where the sub-modelis extracted from the global model using a rolling window that advances in each communication round.
Revised: , FedRolex proposes a rolling sub-model extraction scheme, where the sub-modelis extracted from the global model using a rolling window that advances in each communicationround.

Original: As such, sub-models are extracted from different parts of the global model in different rounds.
Revised: As such, sub-models are extracted from different parts of the global model in differentrounds.

Original: Sincethe extraction window is rolling in each round, parameters of the global model are evenly trained tominimize the client drift.
Revised: Since the extraction window is rolling in each round, parameters of the global model areevenly trained to minimize the client drift.

Original: Throughout communication rounds, all the parameters of the global modelare updated over the entire data distribution.
Revised: Throughout communication rounds, all the parametersof the global model are updated over the entire data distribution.

Original: As we show in §4.2, such rolling sub-model extractionstrategy signiﬁcantly reduces the gap between model-heterogeneous and model-homogeneous setting.
Revised: As we show in §4.2, such arolling sub-model extraction strategy significantly reduces the gap between model-heterogeneous andmodel-homogeneous settings.

Original: We evaluate the performance of FedRolex on two datasets – CIFAR-10 and CIFAR-100 – andcompare it against state-of-the-art KD-based model-heterogeneous FL methods FedDF (Lin et al.,2020), DS-FL (Itahara et al., 2020) and Fed-ET (Cho et al., 2021) as well as PT-based modelheterogeneous FL methods HeteroFL (Diao et al., 2020) and Federated Dropout (Caldas et al., 2018) 1 .
Revised: We evaluate the performance of FedRolex on two datasets – CIFAR-10 and CIFAR-100 – andcompare it against state-of-the-art KD-based model-heterogeneous FL methods FedDF (Lin et al.,2020), DS-FL (Itahara et al., 2020) and Fed-ET (Cho et al., 2021) as well as PT-based modelheterogeneous FL methods HeteroFL (Diao et al., 2020) and Federated Dropout (Caldas et al., 2018) 1 .

Original: Our results show that FedRolex consistently outperforms state-of-the-art PT-based methods in bothhigh and low data heterogeneity scenarios.
Revised: Our results show that FedRolex consistently outperforms state-of-the-art PT-based methods in bothhigh and low data heterogeneity scenarios.

Original: FedRolex also outperforms state-of-the-art KD-basedmethods in low data heterogeneity scenario as well as on the more challenging CIFAR-100 dataset in high data heterogeneity scenario without public data.
Revised: FedRolex also outperforms state-of-the-art KD-basedmethods in low data heterogeneity scenarios as well as on the more challenging CIFAR-100 datasetin high data heterogeneity scenarios without public data.

Original: Furthermore, we show that our proposed rolling sub-model extraction scheme is ﬂexible to train a global model that can be much larger than the largest client model.
Revised: Furthermore, we show that our proposedrolling sub-model extraction scheme is flexible to train a global model that can be much larger thanthe largest client model.

Original: Lastly, we conducted an experiment that uses real-world household income distribution to emulate real-world device distribution.
Revised: Lastly, we conducted an experiment that uses real-world household income distribution to emulate real-world device distribution.

Original: Our results show that FedRolex effectively boosts the model accuracy of low-end devices, which enhances the inclusiveness of federated learning
Revised: Our results show that FedRolex effectivelyboosts the model accuracy of low-end devices, which enhances the inclusiveness of federated learning.

Original: The code of FedRolex will be publicly available on Github.
Revised: The code of FedRolex will be publicly available on Github.

Original:  Related Work
Revised:  Related Work

Original: Our work focuses on model-heterogeneous FL where existing methods can be generally grouped intotwo categories: knowledge distillation (KD) based and partial training (PT) based methods.
Revised: Our work focuses on model-heterogeneous FL. Existing methods can be generally grouped into twocategories: knowledge distillation (KD) based and partial training (PT) based methods.

Original: Knowledge Distillation (KD) based Model-Heterogeneous FL.
Revised: Knowledge Distillation (KD) based Model-Heterogeneous FL.

Original: In knowledge distillation (Hintonet al., 2015; Mirzadeh et al., 2020; Liu et al., 2019), the core idea is to compress a large pre-trainedmodel by teaching a smaller network, step by step, exactly what to do using the larger pre-trainednetwork.
Revised: The key idea of knowledge distillation (Hinton et al., 2015; Mirzadeh et al., 2020; Liu et al., 2019) is to compress a large pre-trainedmodel by teaching a smaller network, step by step, exactly what to do using the larger pre-trainednetwork.

Original: However, this has been adapted in  FLsystems to train a single model from a federation ofclient models trained on private distributed data.
Revised: This mechanism has been adopted in model-heterogeneous FL to train the server modelfrom a federation of client models with different architectures.

Original: In FedDF, (Lin et al., 2020) uses KD from a set ofclassiﬁers pooled from a federation of client devices with their own private data.
Revised: For example, in FedDF, (Lin et al.,2020) uses KD from a set of classifiers pooled from a federation of client devices with their ownprivate data.

Original: The logit outputsof each of the classiﬁers against a public dataset input are then used to train a student model at theserver and thereby distill knowledge.
Revised: The logit outputs of each of the classifiers against a public dataset input are then usedto train a student model at the server and thereby distill knowledge.

Original: In DS-FL (Itahara et al., 2020), an unlabeled public datasetis similarly used.
Revised: In DS-FL (Itahara et al., 2020),an unlabeled public dataset is similarly used.

Original: The logit output for this dataset is aggregated and averaged at the server and isbroadcast back to the clients.
Revised: The logit output for this dataset is aggregated andaveraged at the server and is broadcast back to the clients.

Original: Local models now train on the additional pseudo-labeled data, andperformance is enhanced owing to the data augmentation effect.
Revised: Local models now train on the additionalpseudo-labeled data, and performance is enhanced owing to the data augmentation effect.

Original: Fed-ET (Cho et al., 2022) considersa weighted consensus distillation approach with diversity regularization that enables training of alarge server model with smaller models at the clients.
Revised: Fed-ET(Cho et al., 2022) considers a weighted consensus distillation approach with diversity regularizationthat enables training of a large server model with smaller models at the clients.

Original: These models have several drawbacks.
Revised: These models haveseveral drawbacks.

Original: KD isgenerally a computationally expensive task and so the costs blow up as the system scales up for largermodels and more clients.
Revised: KD is generally a computationally expensive task and so the costs blow upas the system scales up for larger models and more clients.

Original: These are also typically restricted to classiﬁcation tasks as they depend onlogits for implementing KD.
Revised: These are also typically restricted toclassification tasks as they depend on logits for implementing KD.

Original: The methods are also likely to face deployment issues, especially fortasks with a large number of possible output classes.
Revised: The methods are also likely toface deployment issues, especially for tasks with a large number of possible output classes.

Original: Partial Training (PT) based Model-Heterogeneous FL.
Revised: Partial Training (PT) based Model-Heterogeneous FL.

Original: There have been several studies employingdifferent methods to train models of different sizes.
Revised: There have been several studies employingdifferent methods to train models of different sizes.

Original: These generally follow PT.
Revised: These generally follow PT.

Original: HeteroFL (Diaoet al., 2020) uses this concept to aggregate models of different sizes.
Revised: HeteroFL (Diaoet al., 2020) uses this concept to aggregate models of different sizes.

Original: Fjord (Horvath et al., 2021) usesordered dropout, which similarly samples submodels from a global model, trains them, and then doesa weighted aggregation.
Revised: Fjord (Horvath et al., 2021) usesordered dropout, which similarly samples submodels from a global model, trains them, and then doesa weighted aggregation.

Original: PruneFL (Jiang et al., 2022) uses model pruning to extract submodels.
Revised: PruneFL (Jiang et al., 2022) uses model pruning to extract submodels.

Original: Thesealgorithms are easily scalable and have minimum overheads on the server.
Revised: Thesealgorithms are easily scalable and have minimum overheads on the server.

Original: However, these methodshave a signiﬁcant drawback in the sense that different parts of the model are restricted to seeingupdates from a ﬁxed pool of client data.
Revised: However, these methodshave a significant drawback in the sense that different parts of the model are restricted to seeingupdates from a fixed pool of client data.

Original: This limits the performance as the global knowledge isconcentrated on a smaller portion of the model.
Revised: This limits the performance as the global knowledge isconcentrated on a smaller portion of the model.

Original: Furthermore, the server model capability is restrictedto the same capability as the largest client model.
Revised: Furthermore, the server model capability is restrictedto the same capability as the largest client model.

Original: Servers are usually much more computationallycapable and so should be able to use a larger model.
Revised: Servers are usually much more computationallycapable and so should be able to use a larger model.

Original: In comparison, Federated Dropout (Caldaset al., 2018; Guliani et al., 2022; Ro et al., 2022) can train on larger global models and use its entirecapability, especially when the dataset and the number of clients are large.
Revised: In comparison, Federated Dropout (Caldaset al., 2018; Guliani et al., 2022; Ro et al., 2022) can train on larger global models and use its entirecapability, especially when the dataset and the number of clients are large.

Original: Here, a portion of theparameters in each layer of the global model is dropped, leaving a smaller submodel.
Revised: Here, a portion of theparameters in each layer of the global model is dropped, leaving a smaller submodel.

Original: Submodels aretrained on the client devices according to their capacities and the updates are aggregated via weightedaveraging similar to traditional dropout.
Revised: Submodels aretrained on the client devices according to their capacities and the updates are aggregated via weightedaveraging similar to traditional dropout.

Original: However, as reported by Bouacida et al.
Revised: However, as reported by Bouacida et al.

Original: (2020), the approachworks well only with less heterogeneous data and large client pools.
Revised: (2020), the approachworks well only with less heterogeneous data and large client pools.

Original: This is because of high variancein heterogeneous data when only a small subset of the client pool is selected.
Revised: This is because of high variancein heterogeneous data when only a small subset of the client pool is selected.

Original:  Our Method
Revised:  Our Method

Original: 3.1 Problem Formulation of Model-Heterogeneous FL
Revised: 3.1 Problem Formulation of Model-Heterogeneous FL

Original: Consider N different client devices, N , with different compute capacities { C 1 , C 2 , ..., C N } and private data { D 1 , D 2 , ..., D N } , each with different distributions.
Revised: Consider N different client devices, N , with different compute capacities { C 1 , C 2 , ..., C N } and private data { D 1 , D 2 , ..., D N } , each with different distributions.

Original: The objective is to train a large globalmodel characterized by the set of trainable model parameters ✓ .
Revised: The objective is to train a large globalmodel characterized by the set of trainable model parameters θ .

Original: Therefore, model-heterogeneous FLcan be formulated as the following distributed optimization problem:
Revised: Therefore, model-heterogeneous FLcan be formulated as the following distributed optimization problem:

Original: [Equation]
Revised: [Equation]

Original: where F ( ✓ ) =
Revised: where F ( θ ) =

Original: E ⇠ ⇠ D l ( ✓ ; ⇠ ) , F n ( · ) is the local objective function on the client dataset D n ={ d n, 1 , d n, 2 , d n, 3 ...
Revised: E ξ ∼ D l ( θ ; ξ ) , F n ( · ) is the local objective function on the client dataset D n ={ d n, 1 , d n, 2 , d n, 3 ...

Original: d n,m n } with respect to the user deﬁned loss function l ( · ; · ) and p n is the corresponding weight of the n th client such that p n  0 and 
Revised: d n,m n } with respect to the user defined loss function l ( · ; · ) and p n is the corre- sponding weight of the n th client such that p n ≥ 0 and (cid:80) Nn

Original: P Nn =1 p n = 1 .
Revised: =1 p n = 1 .

Original: The n th device contains m ndata points and has model parameters ✓ n (this can be changed from one iteration to another).
Revised: The n th device contains m ndata points and has model parameters θ n (this can be changed from one iteration to another).

Original: Thesize of ✓ n depends on the client’s model capacity C n and are extracted from the global model i.e., [Figure] ✓ n ⇢ ✓ .
Revised: Thesize of θ n depends on the client’s model capacity C n and are extracted from the global model i.e.,θ n ⊂ θ .

Original: The local objective function F n ( · ) is thereby rewritten as:
Revised: The local objective function F n ( · ) is thereby rewritten as:

Original: [Equation]
Revised: [Equation]

Original: For simplicity, we abuse the notation of l by ignoring the other parameters in ✓ but not in ✓ n .
Revised: For simplicity, we abuse the notation of l by ignoring the other parameters in θ but not in θ n .

Original: Thecore of the optimization problem is to select subset ✓ n from the global model parameters ✓ based onmodel capacity C n .
Revised: Thecore of the optimization problem is to select subset θ n from the global model parameters θ based onmodel capacity C n .

Original: 3.2 FedRolex: Model-Heterogeneous FL with Rolling Sub-model Extraction
Revised: 3.2 Partial Training and FedRolex

Original: We propose a more optimal way to extract and train the sub-models that are more consistent withpractical applications than previous methods. In our work, we extract subsets from a rolling windowthat advances and loops over all the kernels of each convolution layer in the model.
Revised: As discussed in Section 2, PT-based methods train heterogeneous models by extracting a subset of theglobal model and training the smaller model on private data. The primary step in PT-based methodsis this sub-model extraction.

Original: Consider✓ ( j ) n selected from ✓ for client n at the j th communication round.
Revised: Consider θ ( j ) n selected from global model weights, θ for client n at thej th communication round.

Original: Assume that the proportion ofkernels extracted from global parameter set ✓ at client n be  n , which belongs to the set of uniquemodel capacities  .
Revised: Assume that the proportion of nodes extracted from each layer in θ forclient n be β n , which belongs to the set of unique model capacities β .

Original: Let the total number of kernels in the i thconvolution layer be K i  .
Revised: Let the total number of nodesin the i th layer be K i and the selected nodes are S ( j ) i .

Original: Thenthe parameters ✓ ( j ) n, [ i, · ] for the i thconvolution layer in extracted model parameters ✓ ( j ) n consist of{ ✓ ( j ) n, [ i,j mod K i ] , ✓ ( j ) n, [ i,  ( j +1) mod K i ] , ... ✓ ( j ) n, [ i,  ( j+  n K i ) mod K i ] }where mod is the modulus operator. For j = 0 , the window starts from the very ﬁrst kernel index and kernel parameters for the sub-modelare extracted starting from index j = 0 .
Revised: Then the parameters θ ( j ) n, [ i, · ] for the i th layer inextracted model parameters θ ( j ) n consist of { θ ( j ) n, [ i, S ( j ) i, 0 ] , θ ( j ) n, [ i, S ( j ) i, 1 ] , ..., θ ( j ) n, [ i, S ( j ) i,Ki ] } .

Original: After each round, the value j is incremented by 1 , and so thewindow advances forward over each round and loops back from the beginning after one completecycle through all the kernels of a particular convolution layer.

Original: Over a sufﬁciently large number ofcommunication rounds, all parts of the model will see the entire data distribution across the N clients.

Original: The process over three rounds is illustrated in Figure 1.

Original: Let, M ⇢ N be the set of selected clients from the client pool from which the server pulls model 146 parameters at round j .
Revised: Let, M ⊂ N be the set of selected clients from the client pool from which the server pulls modelparameters at round j .

Original: Let ✓ [ i,k ] be the k th trainable weight of the i th layer of global model and✓ m, [ i,k ] ; m 2 M be the k th trainable weight of the i th layer of the client m .
Revised: Let θ [ i,k ] be the k th trainable weight of the i th layer of global model andθ m, [ i,k ] ; m ∈ M be the k th trainable weight of the i th layer of the client m .

Original: The model parametersare aggregated as follows:
Revised: The model parametersare aggregated as follows:

Original: [Equation]
Revised: [Equation]

Original: Here, p m is an empirical weight given to a client.
Revised: Here, p m is an empirical weight given to a client.

Original: This can be based on a variety of factors like theclient model capability, the number of data points a client has, etc.
Revised: This can be based on a variety of factors like theclient model capability, the number of data points a client has, etc.

Original: However, there is no theoreticalfoundation in the literature to quantify the relative importance of a client update nor are thereempirical studies that accurately study this.
Revised: However, there is no theoreticalfoundation in the literature to quantify the relative importance of a client update nor are thereempirical studies that accurately study this.

Original: We consider this tangent to our goals in this study and so,throughout the paper, unless otherwise stated, the weight of all clients are assumed to be the same, i.e,p m = 1 .
Revised: We consider this tangent to our goals in this study and so,throughout the paper, unless otherwise stated, the weight of all clients is assumed to be the same, i.e,p m = 1 .

Revised: For different PT-based methods, the method in which the subset of selected parameters, S ischosen varies according to their design.

Original: Finally, the optimal sub-model is extracted from the global model for each client accordingto client capability.
Revised: Finally, the optimal sub-model is extracted from the globalmodel for each client according to client capability.

Original: Why Rolling Sub-model Extracting is better?
Revised: FedRolex proposes a new way to extract the sub-models. In general, sub-models are created bypulling a subset of weights from the global model. This subset can be dynamic (Federated Dropout,

Revised: FedRolex ) in each round or remain static for all rounds (HeteroFL).

Revised: In this section, we compare

Revised: FedRolex with two existing methods: static sub-model extraction (HeteroFL) and random sub-modelextraction (Federated Dropout).

Revised: For simplicity, we focus on the i th layer at the n th client during thej th communication round only.

Revised: Let K i denote the number of nodes in the i th layer of the globalmodel and β n denote the capacity of the client model.

Revised: Static Sub-Model Extraction Scheme

Revised: In this scheme (used by HeteroFL), the global model remainsstatic and the extracted sub-models are always the same.

Revised: This is shown in the left column of Figure 1.

Revised: Unused nodes do not see the full data distribution.

Revised: The subset of node indices is given by:

Revised: [Equation]

Revised: K

Revised: [Equation]

Revised: [Equation]

Revised: Random Sub-Model Extraction Scheme.

Revised: In this scheme (used by Federated Dropout), the submodels are extracted by sampling nodes uniformly and randomly.

Revised: This is shown in the middle columnof Figure 1.

Revised: The subset is given by:

Revised: [Equation]

Revised: [Figure]

Revised: where ⌊ β n K i ⌋ nodes are randomly chosen from the global model.

Revised: Unlike static sub-model extraction,since nodes are shuffled, all nodes end up seeing the entire data given enough communication rounds.

Revised: However, this may require a large number of communication rounds.

Original: PT-based algorithms HeteroFL and FjORD extractthe model parameters ✓ n from speciﬁc locations in the global model.
Revised: Rolling Sub-Model Extraction. In this scheme (used by FedRolex ), the global model is rolledbefore the subset is extracted. This is shown in the right column of Figure 1.

Original: Here the number of kernelsextracted from each convolution layer for creating a client model is based on theclient capacity, andthe kernels are selected starting from the ﬁrst index.
Revised: Let ˆ j = j mod K i , thenthe subset is given by: [Equation] That is, the selected nodes start at ˆ j and increase until ⌊ β n K i ⌋ nodes are chosen. When the nodeindex reaches K i , it goes back to one again.

Original:  The extracted sub-models are then trained onselected client devices from the client pool, and the updates for each parameter are aggregated andaveraged based on the number of clientsthat participated in training for that parameter.
Revised: Why Rolling Sub-model Extraction is better? PT-based algorithms HeteroFL and FjORD extractthe model parameters θ n from specific locations in the global model. Here the client model size isbased on the client capacity, and the nodes are selected starting from the first index. The extractedsub-models are then trained on selected devices from the client pool, and the updates for eachparameter are aggregated and averaged across participating clients for that parameter.

Original: The kernelsin the beginning indices of eachconvolution layer will therefore see updates that were averagedacross more clients than end indices.
Revised: The nodes inthe beginning indices of each layer will therefore see updates that were averaged across more clientsthan end indices.

Original: This means the expected value of the updates will tend moretoward the optimal update.

Original: In comparison, the kernels at the end indices will see updates averagedacross much fewer clients, and therefore the updates to these kernels will have more uncertainty.Essentially, the kernels will be unevenly trained.
Revised: In comparison, the nodes at the end indices will see updates averaged across muchfewer clients, and therefore the updates to these nodes will have more uncertainty.

Original: Dropout-based algorithms like Federated Dropout Caldas et al.
Revised: Dropout-basedalgorithms like Federated Dropout Caldas et al.

Original: (2018) propose random dropout to create smaller sub-models.
Revised: (2018) propose random dropout to create smallersub-models.

Original: Though the expectedvalue of the frequency for updating each index is the same for all the indices in Federated Dropout,their exact frequencies are not the same in each experiment because of the randomness.
Revised: Though the expected value of the frequency for updating each index is the same for allthe indices in Federated Dropout, their exact frequencies are not the same in each experiment becauseof the randomness.

Original: In fact, given I indices, if one index is chosen at each round, the probability to update all indices once in I rounds is Q Ii =1 iI .
Revised: In fact, given I indices, if one index is chosen at each round, the probability toupdate all indices once in I rounds is (cid:81) Ii =1 iI .

Original: Furthermore, the expected number of rounds to go through all indices at least once is closeto I ln
Revised: Furthermore, the expected number of rounds to gothrough all indices at least once is close to I ln

Original: ( I ) , see Appendix A.1.
Revised: ( I ) , see Appendix A.1.

Original: It shows that random dropout can not balance the update frequenciesof the indices, and it takes a large number of rounds to update all the indices.
Revised: It shows that random dropoutcan not balance the update frequencies of the indices, and it takes a large number of rounds to updateall the indices.

Original:  Experiments and Analysis
Revised:  Experiments and Analysis

Original: 4.1 Experimental Setup
Revised: 4.1 Experimental Setup

Original: Datasets and Platform.
Revised: Datasets and Platform.

Original: We validate the performance of FedRolex on two public open-sourcedatasets – CIFAR-10 and CIFAR-100 (Krizhevsky, 2009).
Revised: We validate the performance of FedRolex on three datasets – CIFAR-10, CIFAR-100, and StackOverflow.

Original: We apply bounding box crop (Zoph et al. ,2020) to augment the images.
Revised: We apply bounding box crop (Zoph et al., 2020) to augment theimages.

Original: To simulate the non-iid nature of the data, we use balanced non-iid datadistribution following HeteroFL.
Revised: To simulate the non-iid nature of the data, we use balanced non-iid data distribution following HeteroFL.

Original: We skew the data distribution by restricting the clients to having a ﬁxed number of unique labels L .
Revised: We skew the data distribution by restricting the clients to having a fixed number of uniquelabels L .

Original: We deﬁne L = 2 as high data heterogeneity (i.e., high non-iid)and L = 5 as low data heterogeneity (i.e., low non-iid) for CIFAR-10.
Revised: We define L = 2 as high data heterogeneity (i.e., high non-iid) and L = 5 as low dataheterogeneity (i.e., low non-iid) for CIFAR-10.

Original: Similarly, we use L = 20as high non-iid and L = 50 as low non-iid for CIFAR-100.
Revised: Similarly, we use L = 20 as high non-iid and L = 50as low non-iid for CIFAR-100.

Original: These roughly correspond to Dirichletdistribution Dir K ( ↵ ) with ↵ set to 0 .
Revised: These roughly correspond to Dirichlet distribution Dir K ( α ) with αset to 0 .

Original: 1 and 0 . 5 , respectively.
Revised: 1 and 0 . 5 , respectively.

Original: The data is partitioned into 100 non-iidpartitions and assigned to the client pool.
Revised: The data is partitioned into 100 non-iid partitions and assigned to theclient pool.

Original: In each communication round, 10% of the clients arerandomly selected from the client pool of 100 .
Revised: In each communication round, 10% of the clients are randomly selected from the clientpool of 100 .

Original: We run our experiments on 8 Nvidia A 6000 GPUsusing the PyTorch library for machine learning and the Ray library for simulating the distributedsystem.
Revised: For StackOverflow, the data is partitioned over user ids. The total number of clients,in this case, was 342 , 477 and in each round, 200 clients were sampled. We use a vocabulary size Algorithm

Revised: 1 Initialization ; θ (0) , N

Revised: Input : D n β n ∀ n ∈ N ,

Revised: Output : θ J 2 Server Executes 3 for j ← 0 to J do

Revised: Broadcast θ ( j ) n, [ i, S ( j ) i,k ] ∀ i, S from Equation 6 , 1 ≤ k ≤ K i and n ∈

Revised: N

Revised: Sample subset M from N for each client m ∈ M do clientStep( θ ( j ) n , D n ) end

Revised: Aggregate θ ( j +1)

Revised: [ i,k ] = 1 (cid:80) m ∈M 1 (cid:80)

Revised: m ∈M θ m, [ i,k ] 11 end 12 Subroutine clientStep( θ ( j ) n , D n )

Revised: m n ←− len

Revised: ( D n ) for k ← 0 to m n do θ n ←− θ n − η ∇ l ( θ n ; d n,k ) end return θ n of 10 , 000 words.

Revised: We run our experiments on 8 Nvidia A 6000 GPUs using the PyTorch library formachine learning and the Ray library for simulating the distributed system.

Original: Models.
Revised: Models.

Original: We use pre-activated ResNet18 (PreResNet18) models (He et al., 2016).
Revised: We use pre-activated ResNet18 (PreResNet18) models (He et al., 2016) for CIFAR-10and CIFAR-100.

Original: We replace thebatch Normalization in PreResNet18 with static batch normalization (Diao et al., 2020; Andreuxet al., 2020) and add a scaler module after each convolution layer (Diao et al., 2020).
Revised: We replace the batch Normalization in PreResNet18 with static batch normalization (Diao et al., 2020; Andreux et al., 2020) and add a scalar module after each convolutionlayer (Diao et al., 2020).

Original: We list the exactarchitecture of the model in Appendix A.2.
Revised: For StackOverflow experiments, we trained a modified 3-layer Transformermodel, where the dimension of the token embeddings was 128 , and the hidden dimension of thefeed-forward network(FFN) block is 2048 . We use 8 heads for the multi-head attention, where eachhead is based on 12-dimensional (query, key, value) vectors. We use ReLU activation and set thedropout rate to 0.1.

Original: To simplify our experiments and ablation studies, themodel sizes are restricted to a set of 5 , i.e,  n would take any value in { 1 , 1 / 2 , 1 / 4 , 1 / 8 , 1 / 16 } .
Revised: To simplify our experiments and ablation studies, the model sizes are restricted toa set of 5 , i.e, β n would take any value in { 1 , 1 / 2 , 1 / 4 , 1 / 8 , 1 / 16 } .

Original: Note thatwe only vary the number of kernels in convolution layers  in  our method, HeteroFL, and Federated Dropout to keep the comparison fair.
Revised: Note that we only vary the numberof kernels in convolution layers and keep the nodes in the output layers the same in both our methodand the PT-based baselines to keep the comparison fair.

Revised: Similarly, for the Transformer model, weonly vary the number of nodes in the hidden layer of the attention heads.

Original: We provide pseudocode for both HeteroFL and our Federated Dropout variant in Appendix A.3.
Revised: We provide pseudocodefor both HeteroFL and our Federated Dropout variant in Appendix A.7.

Original: Unless otherwise stated, the client capacity distribution is assumedto be uniform over the entire client pool for all methods.
Revised: Unless otherwise stated, theclient capacity distribution is assumed to be uniform over the entire client pool for all methods.

Original: {+Welist the exact hyperparameters and architecture of the models in Appendix A.
+}
Revised: {+Welist the exact hyperparameters and architecture of the models in Appendix A.
+}

Original: Baselines.
Revised: Baselines.

Original: We compare FedRolex against state-of-the-art PT-based methods HeteroFL and Federated
Revised: We compare FedRolex against state-of-the-art PT-based methods HeteroFL and Federated

Original: Dropout and state-of-the-art KD-based methods FedDF, DS-FL, and Fed-ET obtained from Lin et al.(2020).
Revised: Dropout and state-of-the-art KD-based methods FedDF, DS-FL, and Fed-ET obtained from Lin et al.(2020).

Original: For a fair comparison, all the baselines from the PT-based methods are trained on the samelearning rate, the same number of communication rounds, and the same multi-step learning rate decayschedule.
Revised: For a fair comparison, all the baselines from the PT-based methods are trained on the samelearning rate, the same number of communication rounds, and the same multi-step learning rate decayschedule.

Original: The exact schedule for each dataset and experiment is given in Appendix A.2.
Revised: The exact schedule for each dataset and experiment is given in Appendix A.6.

Original: Evaluation Metrics.
Revised: Evaluation Metrics.

Original: We use local and global model accuracy as our evaluation metrics.
Revised: We use local and global model accuracy as our evaluation metrics.

Original: Speciﬁcally,global model accuracy is deﬁned as the mean accuracy on the test set totally held out during training;local model accuracy is deﬁned as the mean accuracy of the server model on each of the clientdatasets.
Revised: Specifically,global model accuracy is defined as the mean accuracy on the test set totally held out during training;local model accuracy is defined as the mean accuracy of the server model on each of the clientdatasets.

Revised: For vision datasets CIFAR-10 and CIFAR-100, we measure classification accuracy andfor the StackOverflow dataset, we measure next word prediction accuracy.

Original: Since there is some randomness due to the non-iid partitioning, we run our experimentson 5 different seeds and list the mean and standard deviation of our metrics.
Revised: Since there is somerandomness due to the non-iid partitioning, we run our experiments on 5 different seeds and list themean and standard deviation of our metrics.

Original: Note that the results of KD-based methods were obtained from Cho et al.
Revised: Note that the results of KD-based methods were obtainedfrom Cho et al.

Original: (2022) using 3 different seeds.
Revised: (2022) using 3 different seeds.

Original: 4.2 Performance Comparison with State-of-the-Art Model-Heterogeneous FL Methods
Revised: 4.2 Performance Comparison with State-of-the-Art Model-Heterogeneous FL Methods

Original: Table 2 shows the global model accuracy comparison between FedRolex and the baselines underboth high and low data heterogeneity settings.
Revised: shows the global model accuracy comparison between FedRolex and the baselines underboth high and low data heterogeneity settings.

Original: We also include the global model accuracy of twomodel-homogeneous cases where all clients have the highest capacity (  = 1 . 0 ) and lowest capacity(  = 0 .0625 ) representing the upper and lower-bound performance, respectively.
Revised: We also include the global model accuracy of twomodel-homogeneous cases where all clients have the highest capacity ( β = 1 . 0 ) and lowest capacity [Table] ( β = 0 . representing the upper and lower-bound performance, respectively.

Original: We have threekey observations.
Revised: We have threekey observations.

Original: In comparison with state-of-the-art PT-based methods, FedRolex consistentlyoutperforms HeteroFL and Federated Dropout in both high and low data heterogeneity scenarios.
Revised: In comparison with state-of-the-art PT-based methods, FedRolex consistentlyoutperforms HeteroFL and Federated Dropout in both high and low data heterogeneity scenarios.

Original: Inparticular, under high data heterogeneity, Federated Dropout that uses random dropout has drasticallyworse performance than FedRolex and HeteroFL which both extract sub-models in a deterministicmanner.
Revised: Inparticular, under high data heterogeneity, Federated Dropout that uses random dropout has drasticallyworse performance than FedRolex and HeteroFL which both extract sub-models in a deterministicmanner.

Original: In comparison with state-of-the-art KD-based methods, FedRolex only performs worsethan Fed-ET and FedDF on CIFAR-10 under high data heterogeneity, but outperforms all the state-ofthe-arts in more challenging CIFAR-100 dataset that has many more classes than CIFAR-10 underboth high and low data heterogeneity scenarios.
Revised: In comparison with state-of-the-art KD-based methods, FedRolex only performs worsethan Fed-ET and FedDF on CIFAR-10 under high data heterogeneity, but outperforms all the state-ofthe-art in more challenging CIFAR-100 dataset that has many more classes than CIFAR-10 underboth high and low data heterogeneity scenarios.

Original: It is important to emphasize that KD-based methodsleverage public data to boost their model accuracy while FedRolex does not.
Revised: It is important to emphasize that KD-based methodsleverage public data to boost their model accuracy while FedRolex does not.

Original: In comparison withmodel-homogeneous cases, compared to other PT-based methods, FedRolex considerably reducesthe gap between model-heterogeneous and model-homogeneous setting, particularly on CIFAR-10under low data heterogeneity.
Revised: In comparison withmodel-homogeneous cases, compared to other PT-based methods, FedRolex considerably reducesthe gap between model-heterogeneous and model-homogeneous settings, particularly on CIFAR-10under low data heterogeneity.

Original: Note that Fed-ET achieves a higher global model accuracy than themodel-homogeneous upper bound on CIFAR-10 under high data heterogeneity, which showcases theadvantage of having access to public data.
Revised: Note that Fed-ET achieves a higher global model accuracy than themodel-homogeneous upper bound on CIFAR-10 under high data heterogeneity, which showcases theadvantage of having access to public data.

Original: 4.3 Impact of Model Capacity Distribution
Revised: 4.3 Impact of Model Capacity Distribution

Original: In our previous experiment, in line with existing model-heterogeneous FL studies, we have performedour experiment where the distributions of model capacities are uniform for a fair comparison.
Revised: In our previous experiment, in line with existing model-heterogeneous FL studies, we have performedour experiment where the distributions of model capacities are uniform for a fair comparison.

Original: Inpractical scenarios, however, the distributions of model capacities will likely be skewed.
Revised: Inpractical scenarios, however, the distributions of model capacities will likely be skewed.

Original: Therefore,we conduct an experiment to understand how global model accuracy changes when the proportion ofhigher capacity clients varies.
Revised: Therefore,we conduct an experiment to understand how global model accuracy changes when the proportion ofhigher capacity clients varies.

Original: To do so, we set the proportion of large to small models to ⇢ and the
Revised: To do so, we set the proportion of large to small models to ρ and theset of unique client capacities is β

Original: [Table] set of unique client capacities is  = { 1 , 1 / 16 } where ⇢ = 1 represents the edge case in which themodels of all clients have the architecture of the full ResNet18 model [Table] and ⇢ = 0 is the other edgecase in which all clients have the smallest model.
Revised: = { 1 , 1 / 16 } where ρ = 1 represents the edge case in which themodels of all clients have the architecture of the full ResNet18 model, and ρ = 0 is the other edgecase in which all clients have the smallest model.

Original: Figure 2 shows how global model accuracy changes when ⇢ varies from 0 to 1 for CIFAR-10 and
Revised: Figure 2 shows how global model accuracy changes when ρ varies from 0 to 1 for CIFAR-10 and

Original: CIFAR-100 under both high and low data heterogeneity.
Revised: CIFAR-100 under both high and low data heterogeneity.

Original: We have three key observations.
Revised: We have three key observations.

Original: (1) For
Revised: (1) For

Original: CIFAR-10, there is a large gap in global model accuracy between high and low data heterogeneityfor a wide range of ⇢ (from 0.1 to 1).
Revised: CIFAR-10, there is a large gap in global model accuracy between high and low data heterogeneityfor a wide range of ρ (from 0.1 to 1).

Original: This is because CIFAR-10 is a relatively simple task andhence the global model accuracy is bottlenecked by the level of data heterogeneity instead of modelcapacity.
Revised: This is because CIFAR-10 is a relatively simple task andhence the global model accuracy is bottlenecked by the level of data heterogeneity instead of modelcapacity.

Original: This result indicates that having more high-capacity models in the cohort has limitedcontribution to global model accuracy.
Revised: This result indicates that having more high-capacity models in the cohort has limitedcontribution to global model accuracy.

Original: For the more challenging CIFAR-100, the gap in globalmodel accuracy is much lower between high and low data heterogeneity.
Revised: For the more challenging CIFAR-100, the gap in globalmodel accuracy is much lower between high and low data heterogeneity.

Original: In contrast to CIFAR-10, theglobal model accuracy is bottlenecked by the highest capacity of the models rather than the level ofdata heterogeneity.
Revised: In contrast to CIFAR-10, theglobal model accuracy is bottlenecked by the highest capacity of the models rather than the level ofdata heterogeneity.

Original: For both CIFAR-10 and CIFAR-100, we observe that having a small fractionof high-capacity models is enough to signiﬁcantly boost the global model accuracy.
Revised: For both CIFAR-10 and CIFAR-100, we observe that having a small fractionof high-capacity models is enough to significantly boost the global model accuracy.

Original: [Figure] 4.4 Training Large Server Models
Revised: 4.4 Scalability

Original: One limitation of HeteroFL and FjORD is that the server model size is restricted to the highestcapacity client model.
Revised: One limitation of HeteroFL and FjORD is that the server model size is restricted to the highestcapacity client model.

Original: In contrast, FedRolex is ﬂexible to train a global model that can be muchlarger than the largest client model.
Revised: In contrast, FedRolex is flexible to train a global model that can be much [Figure] larger than the largest client model.

Original: This is more akin to real-world settings where the server has amuch higher capacity.
Revised: This is more akin to real-world settings where the server has amuch higher capacity.

Original: We consider the case where the global model size is  times the size of the highest capacity clientmodel.
Revised: We consider the case where the global model size is γ times the size of the highest capacity clientmodel.

Original: Figure 3 shows the global model accuracy of  = { 2 , 4 , 8 , 16 } for CIFAR-10 and CIFAR-100under high and low data heterogeneity.
Revised: Figure 3 shows the global model accuracy of γ = { 2 , 4 , 8 , 16 } for CIFAR-10 and CIFAR-100under high and low data heterogeneity.

Original: As shown, for both CIFAR-10 and CIFAR-100, the globalmodel accuracy drastically drops when  increases from 2 to 4 but stay relatively stable when  increases from 4 to 8 and from 8 to 16 .
Revised: As shown, for both CIFAR-10 and CIFAR-100, the globalmodel accuracy drastically drops when γ increases from 2 to 4 but stay relatively stable when γincreases from 4 to 8 and from 8 to 16 .

Original: This result indicates that there is a minimum client modelcapacity below which the server model cannot be trained effectively.
Revised: This result indicates that there is a minimum client modelcapacity below which the server model cannot be trained effectively.

Original: [Figure]
Revised: [Figure]

Original: 4.5 Enhance Inclusiveness of FL in Real-world Distribution
Revised: 4.5 Enhance Inclusiveness of FL in Real-world Distribution

Original: A primary goal of our method is to enhance the inclusiveness of FL.
Revised: A primary goal of our method is to enhance the inclusiveness of FL.

Original: If the highest model capacity isused for deployment, a lot of data from the low-end devices will be unused.
Revised: If the highest model capacity isused for deployment, a lot of data from the low-end devices will be unused.

Original: The exclusion of low-enddevices limits the usefulness of the global model for these low-end devices.
Revised: The exclusion of low-enddevices limits the usefulness of the global model for these low-end devices.

Original: For example, low-enddevices may have more grainy, low-resolution photos which will not be used in training, and so, theserver model cannot make accurate predictions on such images and hence limiting its usefulnessfor these devices.
Revised: For example, low-enddevices may have more grainy, low-resolution photos which will not be used in training, and so, theserver model cannot make accurate predictions on such images and hence limiting its usefulnessfor these devices.

Original: If we consider homogeneous FL where we want to include all the client data intraining, the smallest model would need to be used.
Revised: If we consider homogeneous FL where we want to include all the client data intraining, the smallest model would need to be used.

Original: The accuracy of the model is then limited by thelowest capacity client.
Revised: The accuracy of the model is then limited by thelowest capacity client.

Original: In this experiment, we show that FedRolex is a valid solution that enhances the inclusiveness offederated learning.
Revised: In this experiment, we show that FedRolex is a valid solution that enhances the inclusiveness offederated learning.

Original: To illustrate our point, we conducted an experiment that uses real-world householdincome distribution to emulate real-world device distribution.
Revised: To illustrate our point, we conducted an experiment that uses real-world householdincome distribution to emulate real-world device distribution.

Original: Speciﬁcally, we retrieve householdincome distribution information from Bureau (2021).
Revised: Specifically, we retrieve householdincome distribution information from Bureau (2021).

Original: We map  n = 1 / 16 with the income group withearning less than $ 75000 and assign proportions of remaining groups in $ 25000 increments withincreasing values of  n in the statistic.
Revised: We map β n = 1 / 16 with the income group withearning less than $ 75000 and assign proportions of remaining groups in $ 25000 increments with [Figure] increasing values of β n in the statistic.

Original: Detailed mapping of this distribution to the correspondingincome distribution is given in Appendix A.2.
Revised: Detailed mapping of this distribution to the correspondingincome distribution is given in Appendix A.

Original: [Table]
Revised: [Table]

Original:  Table 3 shows the global and local model accuracy of FedRolex on this emulated setting comparedto two model-homogeneous cases where all clients have the smallest and largest model capacities,representing lower and upper bound on accuracy respectively.
Revised: [Table] Table 3 shows the global and local model accuracy of FedRolex on this emulated setting comparedto two model-homogeneous cases where all clients have the smallest and largest model capacities,representing lower and upper bound on accuracy respectively.

Original: We make two key observations (1)
Revised: We make two key observations.

Original: FedRolex consistently and signiﬁcantly outperforms the lower bound model-homogeneous casein terms of local model accuracy for CIFAR-10 and CIFAR-100 under both high and low dataheterogeneity.
Revised: FedRolex consistently and significantly outperforms the lower bound model-homogeneous casein terms of local model accuracy for CIFAR-10 and CIFAR-100 under both high and low dataheterogeneity.

Original: This result indicates that FedRolex is able to effectively boost the local modelaccuracy of low-end devices, which would otherwise not beneﬁt from FL.
Revised: This result indicates that FedRolex can effectively boost the local model accuracy oflow-end devices, which would otherwise not benefit from FL.

Original:  FedRolex achievesa local model accuracy that is close to the upper bound model-homogeneous case, especially for CIFAR-10 under both high and low data heterogeneity.
Revised: (2) FedRolex achieves a local modelaccuracy that is close to the upper bound model-homogeneous case, especially for CIFAR-10 underboth high and low data heterogeneity.

Original: This result indicates that FedRolex is ableto signiﬁcantly reduce the gap with the best-performing scenario without being constrained to onlyusing high-end devices.
Revised: This result indicates that FedRolex can significantly reducethe gap with the best-performing scenario without being constrained to only using high-end devices.

Original: [Figure] Finally, Figure 4 provides a detailed illustration of how the local model accuracy shifts for CIFAR-10and CIFAR-100 under low and high data heterogeneity when FedRolex is used compared to thesmallest homogeneous case with the same client outreach.
Revised: Finally, Figure 4 provides a detailed illustration of how the local model accuracy shifts for CIFAR-10and CIFAR-100 under low and high data heterogeneity when FedRolex is used compared to thesmallest homogeneous case with the same client outreach.

Original: As shown, the local model accuracy ofindividual clients is signiﬁcantly boosted by FedRolex .
Revised: As shown, the local model accuracy ofindividual clients is significantly boosted by FedRolex .

Original:  Conclusion 
Revised:  Conclusion and Future Directions

Original: In this work, we present a partial training (PT)-based model-heterogeneous method named FedRolexthat tackles the fundamental issue of existing PT-based methods.
Revised: In this work, we present a partial training (PT)-based model-heterogeneous method named FedRolexthat tackles the fundamental issue of existing PT-based methods.

Original: At the core of FedRolex is a rollingsub-model extraction scheme that enables parameters of the global model to be evenly trained tominimize the client drift.
Revised: At the core of FedRolex is a simpleyet effective rolling sub-model extraction scheme that enables parameters of the global model to beevenly trained to minimize client drift.

Original: Our experimental results show that FedRolex consistently outperformsstate-of-the-art PT-based methods in both high and low data heterogeneity scenarios.
Revised: Our experimental results show that FedRolex consistentlyoutperforms state-of-the-art PT-based methods in both high and low data heterogeneity scenarios.

Original: We demonstrateits effectiveness in real-world scenarios by showing its performance on a more practical modelcapacity distribution and show FedRolex contributes to making FL more inclusive. 
Revised: Wedemonstrate its effectiveness in real-world scenarios by showing its performance on a more practicalmodel capacity distribution and show FedRolex contributes to making FL more inclusive. In themeantime, we note that although we provide a statistical analysis of FedRolex , we do not derive arigorous convergence analysis. This is not a trivial task and therefore left for future work. In addition,our goal in this paper is to train a large global model using a federation of clients with heterogeneouson-device resources.

Original: FedRolexhowever is not tested on other tasks like language modeling and is left for future work.
Revised: Determining what models to deploy onto each heterogeneous client after theglobal model has finished training during the federated learning process is a separate task. This canbe achieved by a few techniques such as compressing/quantizing/KD the global model into smallermodels to fit the resources of the target client. However, this is not the focus of this work and is aseparate study in itself. We will pursue it in our future work as well.

Original: Futuredirections, therefore, include validating its performance on different kinds of tasks.
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope? [Yes] Our work’ claims are verified through experiment inSection 4.2 and statistical study in Appendix A.1(b)

Revised: Did you describe the limitations of your work?

Revised: [Yes] The limitations are mentioned inSection 5(c)

Revised: Did you discuss any potential negative societal impacts of your work?

Revised: [No](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Revised: [Yes]2.

Revised: If you are including theoretical results...

Revised: (a) Did you state the full set of assumptions of all theoretical results?

Revised: [N/A] We do notshow any rigorous theoretical results nor claim as such

Revised: (b) Did you include complete proofs of all theoretical results?

Revised: [N/A] We do not show any

Revised: rigorous theoretical results nor claim as such

Revised: If you ran experiments...

Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Revised: mental results (either in the supplemental material or as a URL)?

Revised: [Yes] Code is madeavailable to reviewers.

Revised: Data is open source.

Revised: Instructions to run are available in code(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Revised: [Yes] All major training details are mentioned in Section 4 and finedetails provided in Appendix A.6(c)

Revised: Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Revised: [Yes] All experiments were performed with 5 different seedsand the mean and standard deviations were reported when applicable.

Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Revised: [Yes]

Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets.

Revised: (a) If your work uses existing assets, did you cite the creators?

Revised: [Yes]

Revised: (b) Did you mention the license of the assets?

Revised: [N/A]

Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Revised: [N/A]

Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Revised: using/curating?

Revised: [N/A]

Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Revised: information or offensive content?

Revised: [N/A]

Revised: If you used crowdsourcing or conducted research with human subjects...

Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Revised: applicable?

Revised: [N/A]

Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Revised: Board (IRB) approvals, if applicable?

Revised: [N/A]

Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Revised: spent on participant compensation?

Revised: [N/A]