Original: D EPTH FL: D EPTHWISE F EDERATED L EARNING FOR H ETEROGENEOUS C LIENTS
Revised: D EPTH FL: D EPTHWISE F EDERATED L EARNING FOR H ETEROGENEOUS C LIENTS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Federated learning is for training a global model without collecting private local data from clients.
Revised: Federated learning is for training a global model without collecting private local data from clients.

Original: As they repeatedly need to upload locally-updated weights or gradients instead, clients require both computation and communication resources enough to participate in learning, but in reality their resources are heterogeneous.
Revised: As they repeatedly need to upload locally-updated weights or gradients instead, clients require both computation and communication resources enough to participate in learning, but in reality their resources are heterogeneous.

Original: To enable resource-constrained clients to train smaller local models, width scaling techniques have been used, which reduces the channels of a global model.
Revised: To enable resource-constrained clients to train smaller local models, width scaling techniques have been used, which prunes the channels of a global model.

Original: Unfortunately, width scaling suffers from heterogeneity of local models when averaging them, leading to a lower accuracy than when simply excluding resourceconstrained clients from training.
Revised: Unfortunately, width scaling suffers from parameter mismatches of channels when aggregating them, leading to a lower accuracy than when simply excluding resourceconstrained clients from training.

Original: This paper proposes a new approach based on depth scaling called DepthFL .
Revised: This paper proposes a new approach based on depth scaling called DepthFL to solve this issue.

Original: DepthFL defines local models of different depths by pruning the deepest layers off the global model, and allocates them to clients depending on theiravailable resources.
Revised: DepthFL defines local models of different depths by pruning the deepest layers off the global model, and allocates them to clients depending on their resources.

Original: Since many clients do not have enough resources to train deep local models, this would make deep layers partially-trained with insufficient data, unlike shallow layers that are fully trained.
Revised: Since many clients do not have enough resources to train deep local models, this would make deep layers partially-trained with insufficient data, unlike shallow layers that are fully trained.

Original: DepthFL alleviates this problem by mutual self-distillation of knowledge among the classifiers of various depths within a local model.
Revised: DepthFL alleviates this problem by mutual self-distillation of knowledge among the classifiers of various depths within a local model.

Original: Our experiments show that depth-scaled local models build a global model better than width-scaled ones, and that selfdistillation is highly effective in training data-insufficient deep layers.
Revised: Our experiments show that depth-scaled local models build a global model better than width-scaled ones, and that self-distillation is highly effective in training data-insufficient deep layers.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Federated learning is a type of distributed learning.
Revised: Federated learning is a type of distributed learning.

Original: It trains a shared global model by aggregating locally-updated model parameters without direct access to the data held by clients.
Revised: It trains a shared global model by aggregating locally-updated model parameters without direct access to the data held by clients.

Original: It is particularly suitable for training a model with on-device private data, such as next word prediction or on-device item ranking (Bonawitz et al., 2019).
Revised: It is particularly suitable for training a model with on-device private data, such as next word prediction or on-device item ranking (Bonawitz et al., 2019).

Original: Generally, federated learning demands client devices to have enough computing power to train a deep model as well as enough communication resources to exchange the model parameters with the server.
Revised: Generally, federated learning demands client devices to have enough computing power to train a deep model as well as enough communication resources to exchange the model parameters with the server.

Original: However, the computation and communication capability of each client is quite diverse, often changing dynamically depending on its current loads, which can make those clients with the smallest resources become a bottleneck for federated learning.
Revised: However, the computation and communication capability of each client is quite diverse, often changing dynamically depending on its current loads, which can make those clients with the smallest resources become a bottleneck for federated learning.

Original: To handle this issue, it would be appropriate for clients to have a different-sized local model depending on their available resources.
Revised: To handle this issue, it would be appropriate for clients to have a different-sized local model depending on their available resources.

Original: However, it is unclear how we can create local models of different sizes, without affecting the convergence of the global model and its performance.
Revised: However, it is unclear how we can create local models of different sizes, without affecting the convergence of the global model and its performance.

Original: There are various methods to prune a single global model to create heterogeneous local models, such as
Revised: There are various methods to prune a single global model to create heterogeneous local models, such as

Original: HeteroFL (Diao et al., 2021), FjORD (Horv´ath et al., 2021), and Split-Mix (Hong et al., 2022).
Revised: HeteroFL (Diao et al., 2021), FjORD (Horv ´ ath et al., 2021), and Split-Mix (Hong et al., 2022).

Original: They create a local model as a subset of the global model by pruning channels, that is, width -based scaling.
Revised: They create a local model as a subset of the global model by pruning channels, that is, width -based scaling.

Original: HeteroFL was a cornerstone research that could make different local models by dividing a global model based on width, yet still producing a global model successfully.
Revised: HeteroFL was a cornerstone research that could make different local models by dividing a global model based on width, yet still producing a global model successfully.

Original: However, we observed some issues of width scaling.
Revised: However, we observed some issues of width scaling.

Original: We evaluated HeteroFL compared to exclusive federated learning, which simply excludes those clients who do not have enough resources to train a given global model from training (see Section 4.2).
Revised: We evaluated HeteroFL compared to exclusive federated learning, which simply excludes those clients who do not have enough resources to train a given global model from training (see Section 4.2).

Original: The result shows that the global model of HeteroFL achieves a tangibly lower accuracy than the models of exclusive learning, due to parameter mismatch of channels when they are aggregated.
Revised: The result shows that the global model of HeteroFL achieves a tangibly lower accuracy than the models of exclusive learning, due to parameter mismatch of channels when they are aggregated.

Original: In this paper, we propose a different approach of making local models, called
Revised: In this paper, we propose a different approach of making local models, called

Original: DepthFL .
Revised: DepthFL .

Original: DepthFL divides s global model based on depth rather than width.
Revised: DepthFL divides a global model based on depth rather than width.

Original: We construct a global model that has several classifiers of different depths.
Revised: We construct a global model that has several classifiers of different depths.

Original: Then, we prune the highest-level layers of the global model to create local models with different depths, thus with a different number of classifiers.
Revised: Then, we prune the highest-level layers of the global model to create local models with different depths, thus with a different number of classifiers.

Original: We found that this depth-based scaling shows a better performance than exclusive learning in most cases, unlike in HeteroFL, since training a local model  directly supervises sub-classifierswithin it as well, obviating parameter mismatch of sub-classifiers when aggregated.
Revised: We found that this depth-based scaling shows a better performance than exclusive learning in most cases, unlike in HeteroFL, since training a local model can directly supervise its sub-classifiers as well as its output classifier, obviating parameter mismatch of sub-classifiers when aggregated.

Original: This means that depth scaling allows resource-constrained clients to  contribute to learning, with a small overhead of separate classifiers.
Revised: This means that depth scaling allows resource-constrained clients to participate and contribute to learning, although there is a small overhead of separate classifiers.

Original: We analyzed the root cause of this difference between depth scaling and width scaling.
Revised: We analyzed the root cause of this difference between depth scaling and width scaling.

Original: There is one issue in depth scaling, though.
Revised: There is one issue in depth scaling, though.

Original: There are only a few clients whose local models include deep classifiers, while many clients have shallow classifiers in their local model.
Revised: There are only a few clients whose local models include deep classifiers, while many clients have shallow classifiers in their local model.

Original: This would make deep classifiers be partially-trained only with a limited amount of data, so their accuracy might be inferior to fully-trained shallow classifiers.
Revised: This would make deep classifiers be partially-trained only with a limited amount of data, so their accuracy might be inferior to fully-trained shallow classifiers.

Original: That is, resource-constrained clients cannot train deep classifiers, so deep classifiers cannot be general enough to cover the unseen data of those clients.
Revised: That is, resource-constrained clients cannot train deep classifiers, so deep classifiers cannot be general enough to cover the unseen data of those clients.

Original: To moderate this issue, we make the deep classifiers of a local model learn from its shallow classifiers by knowledge distillation.
Revised: To moderate this issue, we make the deep classifiers of a local model learn from its shallow classifiers by knowledge distillation.

Original: This is similar to self distillation (Zhang et al., 2019), except that the direction of distillation is opposite.
Revised: This is similar to self distillation (Zhang et al., 2019), except that the direction of distillation is opposite.

Original: Actually, we make the classifiers collaborate with each other as in deep mutual learning (Zhang et al., 2018).
Revised: Actually, we make the classifiers collaborate with each other as in deep mutual learning (Zhang et al., 2018).

Original: So, a client not only trains the classifiers in its local model using its data, but distills each other’s knowledge at the same time.
Revised: So, a client not only trains the classifiers in its local model using its data, but distills each other’s knowledge at the same time.

Original: Our evaluation shows that deep classifiers can learn from shallow classifiers trained with otherwise unseen data, and that both classifiers actually help each other to improve the overall performance of the global model.
Revised: Our evaluation shows that deep classifiers can learn from shallow classifiers trained with otherwise unseen data, and that both classifiers actually help each other to improve the overall performance of the global model.

Original: We also analyzed the fundamental reason for the effectiveness of knowledge distillation in DepthFL, using an evaluation in a general teacher-student environment.
Revised: We also analyzed the fundamental reason for the effectiveness of knowledge distillation in DepthFL, using an evaluation in a general teacher-student environment.

Original: Recently, InclusiveFL (Liu et al., 2022) proposed a kind of depth-scaled method with a similar intuition to ours, yet with less elaboration.
Revised: Recently, InclusiveFL (Liu et al., 2022) proposed a kind of depth-scaled method with a similar intuition to ours, yet with less elaboration.

Original: We show that depth scaling alone in InclusiveFL without the companion objective of sub-classifiers cannot effectively solve parameter mismatches, and that its performance of deep classifiers is much lower due to no self distillation.
Revised: We show that depth scaling alone in InclusiveFL without the companion objective of sub-classifiers cannot effectively solve parameter mismatches, and that its performance of deep classifiers is much lower due to no self distillation.

Original: We employed the latest federated learning algorithm FedDyn (Acar et al., 2021) as the optimizer for proposed method.
Revised: We employed the latest federated learning algorithm FedDyn (Acar et al., 2021) as the optimizer for proposed method.

Original: Although DepthFL is a framework that focuses on resource heterogeneity of clients, we try to verify if the FedDyn optimizer that focuses on data heterogeneity is applicable to DepthFL seamlessly.
Revised: Although DepthFL is a framework that focuses on resource heterogeneity of clients, we try to verify if the FedDyn optimizer that focuses on data heterogeneity is applicable to DepthFL seamlessly.

Original: Our experiment shows that DepthFL works well for both resource heterogeneity and data heterogeneity.
Revised: Our experiment shows that DepthFL works well for both resource heterogeneity and data heterogeneity.

Original: Contributions To summarize, our contributions are threefold:
Revised: Contributions To summarize, our contributions are threefold:

Original: • We present a new depth scaling method to create heterogeneous local models with sub-classifiers, which  directly supervises sub-classifiers, obviating their parameter mismatch when aggregated.
Revised: • We present a new depth scaling method to create heterogeneous local models with sub-classifiers, which are directly supervised during training, thus no parameter mismatches when aggregated.

Original: • We show that knowledge distillation among the classifiers in a local model can effectively train deep classifiers that can see only a limited amount of data.
Revised: • We show that knowledge distillation among the classifiers in a local model can effectively train deep classifiers that can see only a limited amount of data.

Original: • We perform a comprehensive evaluation on the difference between depth scaling and width scaling using exclusive learning, and on the effectiveness of knowledge distillation in DepthFL.
Revised: • We perform a comprehensive evaluation on the difference between depth scaling and width scaling using exclusive learning, and on the effectiveness of knowledge distillation in DepthFL.

Original:  R ELATED W ORK 2. F EDERATED L EARNING
Revised:  R ELATED W ORK 2. F EDERATED L EARNING

Original: Based on FedAvg (McMahan et al., 2017), the standard federated learning method, there have been many studies to solve various problems of federated learning (Li et al., 2020a; Kairouz et al., 2019).
Revised: Based on FedAvg (McMahan et al., 2017), the standard federated learning method, there have been many studies to solve various problems of federated learning (Li et al., 2020a; Kairouz et al., 2019).

Original: One main research field deals with efficient learning algorithms considering the non-IID distribution of data among clients (Karimireddy et al., 2020; Li et al., 2021b;
Revised: One main research field deals with efficient learning algorithms considering the non-IID distribution of data among clients (Karimireddy et al., 2020; Li et al., 2021b;

Original: Wang et al., 2020; Acar et al., 2021;
Revised: Wang et al., 2020; Acar et al., 2021;

Original: Li et al., 2020b).
Revised: Li et al., 2020b).

Original: For example, FedDyn (Acar et al., 2021) performs exact minimization by making the local objective align with the global objective, which enables fast and stable convergence.
Revised: For example, FedDyn (Acar et al., 2021) performs exact minimization by making the local objective align with the global objective, which enables fast and stable convergence.

Original: There are also studies to deal with heterogeneous, resource-constrained clients in federated learning.
Revised: There are also studies to deal with heterogeneous, resource-constrained clients in federated learning.

Original: One popular method to create heterogeneous local models is to prune the channels of a global model for width scaling (Diao et al., 2021; Li et al., 2021a; Niu et al., 2020; Horv´ath et al., 2021; Hong et al., 2022; Zhu et al., 2022).
Revised: One popular method to create heterogeneous local models is to prune the channels of a global model for width scaling (Diao et al., 2021; Li et al., 2021a; Niu et al., 2020; Horv ´ ath et al., 2021; Hong et al., 2022; Zhu et al., 2022).

Original: Based on the general layer-pruning methods (Chen & Zhao, 2019;
Revised: Based on the general layer-pruning methods (Chen & Zhao, 2019;

Original: Sajjad et al., 2023) that create a shallow model, there is a recent work (Liu et al., 2022) to prune the layers of a global model for tacking system heterogeneity, but it partially exploits the benefit of depth scaling and knowledge distillation unlike DepthFL (see Appendix A.3).
Revised: Sajjad et al., 2023) that create a shallow model, there is a recent work (Liu et al., 2022) to prune the layers of a global model for tackling system heterogeneity, but it partially exploits the benefit of depth scaling and knowledge distillation unlike DepthFL (see Appendix A.3).

Original: Another direction is utilizing knowledge distillation to reduce the burden on clients (He et al., 2020) or to aggregate heterogeneous local models (Zhang & Yuan, 2021; Lin et al., 2020).
Revised: Another direction is utilizing knowledge distillation to reduce the burden on clients (He et al., 2020) or to aggregate heterogeneous local models (Zhang & Yuan, 2021; Lin et al., 2020).

Original: Separately, there are researches to reduce the communication cost by compressing the local models (Rothchild et al., 2020; Had- dadpour et al., 2021; Reisizadeh et al., 2020).
Revised: Separately, there are researches to reduce the communication cost by compressing the local models (Rothchild et al., 2020; Haddadpour et al., 2021; Reisizadeh et al., 2020).

Original: There are also research directions on personalized federated learning to handle heterogeneity of clients (Li et al., 2021c; Zhang et al., 2021; Hanzely et al., 2020; T. Dinh et al., 2020).
Revised: There are also research directions on personalized federated learning to handle heterogeneity of clients (Li et al., 2021c; Zhang et al., 2021; Hanzely et al., 2020; T. Dinh et al., 2020).

Original: 2. K NOWLEDGE D ISTILLATION
Revised: 2. K NOWLEDGE D ISTILLATION

Original: Knowledge distillation (Hinton et al., 2015) was introduced as a way of training a small student model with the help of a big teacher model.
Revised: Knowledge distillation (Hinton et al., 2015) was introduced as a way of training a small student model with the help of a big teacher model.

Original: It was initially thought that the role of knowledge distillation is transferring high-quality, similarity information among categories, but researches are still on going to understand its impact such as the relationship with label smoothing regularization (Yuan et al., 2020; Tang et al., 2020).
Revised: It was initially thought that the role of knowledge distillation is transferring high-quality, similarity information among categories, but researches are still on going to understand its impact such as the relationship with label smoothing regularization (Yuan et al., 2020; Tang et al., 2020).

Original: As such, variants of knowledge distillation techniques have been proposed (Huang et al., 2021;
Revised: As such, variants of knowledge distillation techniques have been proposed (Huang et al., 2021;

Original: Park et al., 2019; Zhang et al., 2018; 2019).
Revised: Park et al., 2019; Zhang et al., 2018; 2019).

Original: Deep mutual learning (Zhang et al., 2018) allows student models to mutually distill each other’s knowledge without a cumbersome teacher model.
Revised: Deep mutual learning (Zhang et al., 2018) allows student models to mutually distill each other’s knowledge without a cumbersome teacher model.

Original: Self distillation (Zhang et al., 2019) distills knowledge within a multi-exit network (Teerapittayanon et al., 2016), from its deep layers to shallow layers for higher performance, or for faster and accurate inference (Phuong & Lampert, 2019).
Revised: Self distillation (Zhang et al., 2019) distills knowledge within a multi-exit network (Teerapittayanon et al., 2016), from its deep layers to shallow layers for higher performance, or for faster and accurate inference (Phuong & Lampert, 2019).

Original: DepthFL also employs self distillation within a local model, yet in an opposite direction, mainly for distilling from fully-trained shallow layers to partially-trained deep layers.
Revised: DepthFL also employs self distillation within a local model, yet in an opposite direction, mainly for distilling from fully-trained shallow layers to partially-trained deep layers.

Original:  D EPTH FL M ETHOD
Revised:  D EPTH FL M ETHOD

Original: This section describes our proposed DepthFL method in detail.
Revised: This section describes our proposed DepthFL method in detail.

Original: We assume that a total of m heterogeneous clients participate in federated learning, and that each client k has d k resource capability.
Revised: We assume that a total of m heterogeneous clients participate in federated learning, and that each client k has d k resource capability.

Original: Randomly selected P t ∈ [ m ] clients participate in each communication round.
Revised: Randomly selected P t ∈ [ m ] clients participate in each communication round.

Original: 3. L OCAL M ODELS
Revised: 3. L OCAL M ODELS

Original: We create heterogeneous local models by pruning zero or more of the highest-level layers of the global model, instead of pruning some channels for all layers as in HeteroFL.
Revised: We create heterogeneous local models by pruning zero or more of the highest-level layers of the global model, instead of pruning some channels for all layers as in HeteroFL.

Original: Each client can contribute to the training of the global model by selecting a local model suitable for its resource amount, in a fixed or dynamic way.
Revised: Each client can con- tribute to the training of the global model by selecting a local model suitable for its resource amount, in a fixed or dynamic way.

Original: Since the highest-level layers are entirely pruned, an additional bottleneck layer is needed for a local model to become an independent classifier.
Revised: Since the highest-level layers are entirely pruned, an additional bottleneck layer is needed for a local model to become an independent classifier.

Original: This means that the global model should have a different classifier at a different depth.
Revised: This means that the global model should have a different classifier at a different depth.

Original: The structure of a global model that satisfies these conditions can be found in Deeply-Supervised Nets (DSN) (Lee et al., 2015).
Revised: The structure of a global model that satisfies these conditions can be found in Deeply-Supervised Nets (DSN) (Lee et al., 2015).

Original: Each classifier shares the blocks that create feature maps, and an additional bottleneck layer is attached to each block so that the local models and the global model have several classifiers inside.
Revised: Each classifier shares the blocks that create feature maps, and an additional bottleneck layer is attached to each block so that the local models and the global model have several classifiers inside.

Original: Accordingly, the local model parameters of the client whose resource capability is d k is W d k l = W g
Revised: Accordingly, the local model parameters of the client whose resource capability is d k is W d k l = W g

Original: [: d k ] , and it has d k classifiers inside.
Revised: [: d k ] , and it has d k classifiers inside.

Original: The overall model structure can be found in Figure 1.
Revised: The overall model structure can be found in Figure 1.

Original: DSN can provide integrated direct supervision to a sub-classifier in addition to the output classifier, since back-propagation with the overall loss as well as the companion loss will update its hidden layers, as shown in Figure 1.
Revised: DSN can provide integrated direct supervision to sub-classifiers in addition to the output classifier.

Original: DSN exploits it for better performance, but DepthFL exploits it for consistent training of a sub-classifier across all local models that includes it to fully exhibit its performance.
Revised: DSN exploits it for better performance, but DepthFL exploits it for consistent training of sub-classifiers across all local models that includes it to fully exhibit its performance.

Original: This way of creating heterogeneous local models is likely to make deep classifiers learn from a partial amount of data, because only resource-rich clients can train local models with deep classifiers.
Revised: This way of creating heterogeneous local models is likely to make deep classifiers learn from a partial amount of data, because only resource-rich clients can train local models with deep classifiers.

Original: On the other hand, shallow classifiers are likely to be fully trained since many clients can train local models with them.
Revised: On the other hand, shallow classifiers are likely to be fully trained since many clients can train local models with them.

Original: To mitigate the variance of the performance of different classifiers, it is natural to use their ensemble for inference.
Revised: To mitigate the variance of the performance of different classifiers, it is natural to use their ensemble for inference.

Original: One interesting question is if partially-trained deep classifiers can really contribute to learning when many clients cannot afford a local model with them.
Revised: One interesting question is if partially-trained deep classifiers can really contribute to learning when many clients cannot afford a local model with them.

Original: In our experimental result in Table 4, deep classifiers perform similarly or even worse than shallow classifiers unless they receive  help, which is self distillation .
Revised: In our experimental result in Table 4, deep classifiers perform similarly or even worse than shallow classifiers unless they receive a help, in the form of self distillation described below.

Original: 3. S ELF D ISTILLATION
Revised: 3. S ELF D ISTILLATION

Original: To mitigate the low performance of deep classifiers, DepthFL utilizes the concept of self distillation.
Revised: To mitigate the low performance of deep classifiers, DepthFL utilizes the concept of self distillation.

Original: Several classifiers inside a local model are trained through the cross-entropy loss with labels, as well as the KL loss with the output of other classifiers, as depicted in Figure 1.
Revised: Several classifiers inside a local model are trained through the cross-entropy loss with labels, as well as the KL loss with the output of other classifiers, as depicted in Figure 1.

Original: Previously, self distillation made shallow classifiers imitate deep classifiers (Zhang et al., 2019; Phuong & Lampert, 2019).
Revised: Previously, self distillation made shallow classifiers imitate deep classifiers (Zhang et al., 2019; Phuong & Lampert, 2019).

Original: Whereas, DepthFL makes deep classifiers imitate shallow classifiers, or more precisely, learn collaboratively.
Revised: Whereas, DepthFL makes deep classifiers imitate shallow classifiers, or more precisely, learn collaboratively.

Original: The local objective function of the client is as follows:
Revised: The local objective function of the client is as follows:

Original: [Equation]
Revised: [Equation]

Original: where L ice is a cross-entropy loss of i th classifier, and p i are softmax outputs of i th classifier’s logits.
Revised: where L ice is a cross-entropy loss of i th classifier, and p i are softmax outputs of i th classifier’s logits.

Original: Although FjORD (Horv ´ ath et al., 2021) utilized self distillation already, the motivation of self distillation in FjORD was to allow for the bigger capacity supermodel to teach the width-scaled submodel.
Revised: Although FjORD (Horv ´ ath et al., 2021) utilized self distillation already, the motivation of self distillation in FjORD was to allow for the bigger capacity supermodel to teach the width-scaled submodel.

Original: In contrast, the purpose of self distillation in DepthFL is that depth-scaled submodels help training of bigger capacity supermodels.
Revised: In contrast, the purpose of self distillation in DepthFL is that depth-scaled submodels help training of bigger capacity supermodels.

Original: Also, unlike distillation between width scaled models, which should run forward pass through teacher supermodel and student submodel independently, there is no overhead of distillation between depth scaled models except for the bottleneck layers because they share feature maps.
Revised: Also, unlike distillation between width scaled models, which should run forward pass through teacher supermodel and student submodel independently, there is no overhead of distillation between depth scaled models except for the bottleneck layers because they share feature maps.

Original: We use FedDyn (Acar et al., 2021) instead of FedAvg (McMahan et al., 2017) as the default optimizer for the above local objective function 1, to make a fast convergence even when there is a data heterogeneity of clients.
Revised: We use FedDyn (Acar et al., 2021) instead of FedAvg (McMahan et al., 2017) as the default optimizer for the above local objective function 1, to make a fast convergence even when there is a data heterogeneity of clients.

Original: When applying dynamic regularization, we replaced the client’s local objective by the modified heterogeneous local objective of 1.
Revised: When applying dynamic regularization, we replaced the client’s local objective by the modified heterogeneous local objective of 1.

Original: Also, ∇ L k ( θ tk ) and h values required for dynamic regularization are used and updated in consideration of the heterogeneity of the local models.
Revised: Also, ∇ L k ( θ tk ) and h values required for dynamic regularization are used and updated in consideration of the heterogeneity of the local models.

Original: Therefore, the penalized local objective function of the client is as follows.
Revised: Therefore, the penalized local objective function of the client is as follows.

Original: [Equation]
Revised: [Equation]

Original: where ˜ θ is the local model parameters, ∇ L k (˜ θ tk ) is the gradient of the local objective function in the previous round, and ˜ θ t is parts of the current global model’s parameters corresponding to the local model parameters ˜ θ .
Revised: where ˜ θ is the local model parameters, ∇ L k (˜ θ tk ) is the gradient of the local objective function in the previous round, and ˜ θ t is parts of the current global model’s parameters corresponding to the local model parameters ˜ θ .

Original: Considering the case where the complexity d k of the client changes dynamically depending on its current loads, ∇ L k ( θ tk ) is stored in the same shape as the entire global model parameters, and only a subset corresponding to the current local model parameters is used for actual training.
Revised: Considering the case where the complexity d k of the client changes dynamically depending on its current loads, ∇ L k ( θ tk ) is stored in the same shape as the entire global model parameters, and only a subset corresponding to the current local model parameters is used for actual training.

Original: So, the value of ∇ L k ( θ tk ) is updated as in the following equation.
Revised: So, the value of ∇ L k ( θ tk ) is updated as in the following equation.

Original: [Equation]
Revised: [Equation]

Original: When the local models of different depths are aggregated into a single global model, the local model parameters of the same depths will be aggregated together.

Original: This naturally makes shallow layers close to the input aggregate a large number of local model parameters, while deep layers close to the output aggregate a small number of local model parameters.

Original: Algorithm 1 shows the full algorithm.

Original: Replacing FedDyn optimizer by FedAvg is straightforward.

Original: Initialization : θ 0 , h 0 = 0 , ∇ L k ( θ 0 k ) =
Revised: Initialization : θ 0 , h 0 = 0 , ∇ L k ( θ 0 k ) =

Original: Server executes: for round t = 0 , 1 , . . .
Revised: Server executes: for round t = 0 , 1 , . . .

Original: T − 1 do
Revised: T − 1 do

Original: P t ← Random Clients θ t +1 ←h t +1 ← h t for each client k ∈ P t , and in parallel do ˜ θ t ← θ t
Revised: P t ← Random Clients θ t +1 ←h t +1 ← h t for each client k ∈ P t , and in parallel do ˜ θ t ← θ t

Original: [: d k ] ˜ θ t +1 k ← Client Update( k, ˜ θ t ) h t +

Revised: [: d k ] ˜ θ t +1 k ← Client Update( k, ˜ θ t ) h t +

Original: [: d k ] ← h t +

Revised: [: d k ] ← h t +

Original: [: d k ] − α 1 m (˜ θ t +1 k − θ t
Revised: [: d k ] − α 1 m (˜ θ t +1 k − θ t

Original: [: d k ])
Revised: [: d k ])

Original: θ t +1 [: d k ] ← θ t +

Revised: θ t +1 [: d k ] ← θ t +

Original: [: d k ] + ˜ θ t +1 k endfor each resource capability d i do θ
Revised: [: d k ] + ˜ θ t +1 k endfor each resource capability d i do θ

Original: t +

Revised: t +

Original: [ d i ] = 1 | P dk ≥ di t | θ t +

Revised: [ d i ] = 1 | P dk ≥ di t | θ t +

Original: [ d i ] − 1 α h t +1 [ d i ] end end
Revised: [ d i ] − 1 α h t +1 [ d i ] end end

Original: Client Update( k, ˜ θ t ): ˜ θ t +1 k ← ˜ θ t ∇ L k (˜ θ tk ) ← ∇ L k ( θ latest updated k )
Revised: Client Update( k, ˜ θ t ): ˜ θ t +1 k ← ˜ θ t ∇ L k (˜ θ tk ) ← ∇ L k ( θ latest updated k )

Original: [: d k ] for local epoch e = 1 , 2 , . . .
Revised: [: d k ] for local epoch e = 1 , 2 , . . .

Original: E do for each mini batch b do
Revised: E do for each mini batch b do

Original: L k = d k (cid:80) i =1 L ice + 1 d k −d k (cid:80) i =
Revised: L k = d k (cid:80) i =1 L ice + 1 d k −d k (cid:80) i =

Original: d k (cid:80) j =1 ,j ̸ = i D KL ( p j ∥ p i )
Revised: d k (cid:80) j =1 ,j ̸ = i D KL ( p j ∥ p i )

Original: L ′ k
Revised: L ′ k

Original: (˜ θ )
Revised: (˜ θ )

Original: =
Revised: =

Original: L k (˜ θ ) −⟨∇
Revised: L k (˜ θ ) −⟨∇

Original: L k (˜ θ tk ) , ˜ θ ⟩ + α 2 || ˜ θ − ˜ θ t ||˜ θ t +1 k ← ˜ θ t +1 k − η ∇ L ′ k
Revised: L k (˜ θ tk ) , ˜ θ ⟩ + α 2 || ˜ θ − ˜ θ t ||˜ θ t +1 k ← ˜ θ t +1 k − η ∇ L ′ k

Original: (˜ θ t +1 k ; b ) end end ∇ L k ( θ t +1 k )
Revised: (˜ θ t +1 k ; b ) end end ∇ L k ( θ t +1 k )

Original: [: d k ] ← ∇ L k ( θ tk )[: d k ] − α (˜ θ t +1 k − ˜ θ t ) return ˜ θ t +1 k 
Revised: [: d k ] ← ∇ L k ( θ tk )[: d k ] − α (˜ θ t +1 k − ˜ θ t ) return ˜ θ t +1 k [Table]

Revised: When the local models of different depths are aggregated into a single global model

Original: [Table]
Revised: [Table]

Revised: the local model parameters of the same depths will be aggregated together.

Revised: This naturally makes shallow layers close to the input aggregate a large number of local model parameters

Revised: [Table]

Revised: while deep layers close to the output aggregate a small number of local model parameters.

Revised: Algorithm 1 shows the full algorithm.

Revised: Replacing FedDyn optimizer by FedAvg is straightforward

Revised: [Table]

Original:  E XPERIMENTS 4. S ETTINGS
Revised:  E XPERIMENTS 4. S ETTINGS

Original: Datasets and models We used MNIST, CIFAR-100, and Tiny ImageNet datasets for the image classification task
Revised: Datasets and models We used MNIST, CIFAR-100, and Tiny ImageNet datasets for the image classification task

Original: [Table]
Revised: [Table]

Original: and WikiText-2 dataset for the masked language modeling task.
Revised: and WikiText-2 dataset for the masked language modeling task.

Original: CNN composed of 4 convolution layers, Resnet
Revised: CNN composed of 4 convolution layers, Resnet

Original: [Table]
Revised: [Table]

Original: and Resnet34 models were used for MNIST, CIFAR- [Table] and
Revised: and Resnet34 models were used for MNIST, CIFAR-100, and

Original: Tiny ImageNet, respectively.
Revised: Tiny ImageNet, respectively.

Original: The transformer model was used for Wikitext-2.
Revised: The transformer model was used for Wikitext-2.

Original: We create four local models with DepthFL [Table] and four local models with HeteroFL by dividing the channels into four equal parts following its division method.
Revised: We create four local models with DepthFL, and four local models with HeteroFL by dividing the channels into four equal parts following its division method.

Original: Table 1 depicts the model size and number of MACs for the four local models of HeteroFL and DepthFL.
Revised: Table 1 depicts the model size and number of MACs for the four local models of HeteroFL and DepthFL.

Original: For example, the smallest model a has only the classifier 1 / 4 in DepthFL, while having 1 / 4 channels in HeteroFL.
Revised: For example, the smallest model a has only the classifier 1 / 4 in DepthFL, while having 1 / 4 channels in HeteroFL.

Original: For inference, DepthFL uses the ensemble of all internal classifiers, while HeteroFL uses the global model with all channels.
Revised: For inference, DepthFL uses the ensemble of all internal classifiers, while HeteroFL uses the global model with all channels.

Original: Local model
Revised: Local model

Original: If scaling is done as mentioned above, each depth-scaled local model would have fewer parameters and more MACs than the corresponding width-scaled one, since shallow layers often have fewer parameters yet require more computations.
Revised: If scaling is done as mentioned above, each depth-scaled local model would have fewer parameters and more MACs than the corresponding width-scaled one, since shallow layers often have fewer parameters yet require more computations.

Original: So, for the Resnet global model, the number of MACs decreases linearly as the depth decreases, but the number of model parameters decreases more rapidly, as shown in Table 1 (the global model of DepthFL has slightly more parameters and MACs than that of HeteroFL due to the additional bottleneck layers).
Revised: So, for the Resnet global model, the number of MACs decreases linearly as the depth decreases, but the number of model parameters decreases more rapidly, as shown in Table 1 (the global model of DepthFL has slightly more parameters and MACs than that of HeteroFL due to the additional bottleneck layers).

Original: This means that the average communication overhead of depth-scaled clients would be lower than that of width-scaled clients, but it would be opposite for the average computation overhead.
Revised: This means that the average communication overhead of depth-scaled clients would be lower than that of width-scaled clients, but it would be opposite for the average computation overhead.

Original: Also, depth scaling is less fine-grained than width scaling since depth scaling is more dependent on the layer structure of the global model, and may not be effective if the size of the bottleneck layer is too large.
Revised: Also, depth scaling is less fine-grained than width scaling since depth scaling is more dependent on the layer structure of the global model, and may not be effective if the size of the bottleneck layer is too large.

Original: Despite these limitations, depth-scaled local models do not seriously affect the performance of the global model when aggregated, unlike width-scaled local models, as will be explained shortly.
Revised: Despite these limitations, depth-scaled local models do not seriously affect the performance of the global model when aggregated, unlike width-scaled local models, as will be explained shortly.

Original: Default Settings Unless otherwise stated, the same number of clients are allocated to each of the four different local models (i.e., 25% of the clients are allocated to each of a , b , c , and d local models in Table 1).
Revised: Default Settings Unless otherwise stated, the same number of clients are allocated to each of the four different local models (i.e., 25% of the clients are allocated to each of a , b , c , and d local models in Table 1).

Original: Also, the data is distributed in an IID manner, FedDyn is used as the optimizer, and randomly sampled 10% of the clients among a total of 100 clients participate in each communication round.
Revised: Also, the data is distributed in an IID manner, FedDyn is used as the optimizer, and randomly sampled 10% of the clients among a total of 100 clients participate in each communication round.

Original: When the data is distributed in a non-IID manner, as in FedMA (Wang et al., 2020), a
Revised: When the data is distributed in a non-IID manner, as in FedMA (Wang et al., 2020), a

Original: Dirichlet distribution p c ∼ Dir k
Revised: Dirichlet distribution p c ∼ Dir k

Original: ( β = 0 . 5) was used to allocate p c,k ratio of data samples belonging to class c to client k .
Revised: ( β = 0 . 5) was used to allocate p c,k ratio of data samples belonging to class c to client k .

Original: We ran all our experiments on Nvidia RTX 2080 Ti GPUs.

Original: 4. C OMPARISON WITH E XCLUSIVE L EARNING
Revised: 4. C OMPARISON WITH E XCLUSIVE L EARNING

Original: We first evaluate if it is beneficial for resource-constrained clients to participate in learning with depth-scaled local models even if they cannot accommodate a given global model.
Revised: We first evaluate if it is beneficial for resource-constrained clients to participate in learning with depth-scaled local models even if they cannot accommodate a given global model.

Original: For this evaluation, we compare the accuracy of DepthFL with the accuracy of its exclusive learning , which excludes those clients who do not have enough resources to run a given global model from learning.
Revised: For this evaluation, we compare the accuracy of DepthFL with the accuracy of its exclusive learning , which excludes those clients who do not have enough resources to run a given global model from learning.

Original: In DepthFL, d is the global model but we still allow those 75% clients whose resource cannot accommodate d to participate in learning.
Revised: In DepthFL, d is the global model but we still allow those 75% clients whose resource cannot accommodate d to participate in learning.

Original: In exclusive learning, however, if d is the global model, we allow only those 25% clients who can run d to participate in learning.
Revised: In exclusive learning, however, if d is the global model, we allow only those 25% clients who can run d to participate in learning.

Original: Similarly, if c is the global model, we allow those 50% clients who can run c (including those who can run d but should run c since c is the global model now) to participate in learning.
Revised: Similarly, if c is the global model, we allow those 50% clients who can run c (including those who can run d but should run c since c is the global model now) to participate in learning.

Original: We also evaluate DepthFL without self-distillation and FedDyn (use FedAvg instead), which we call DepthFL(FedAvg), by comparing with its corresponding exclusive learning.
Revised: We also evaluate DepthFL without self-distillation and FedDyn (use FedAvg instead), which we call DepthFL(FedAvg), by comparing with its corresponding exclusive learning.

Original: Finally, we evaluate HeteroFL by comparing with its corresponding exclusive learning.
Revised: Finally, we evaluate HeteroFL by comparing with its corresponding exclusive learning.

Original: For a fair comparison, exclusive learning is trained and tested in the same way as the corresponding scaled method.
Revised: For a fair comparison, exclusive learning is trained and tested in the same way as the corresponding scaled method.

Original: For example, when comparing DepthFL with its exclusive learning, the local objective function of both includes the self distillation and regularization terms.
Revised: For example, when comparing DepthFL with its exclusive learning, the local objective function of both includes the self distillation and regularization terms.

Original: Also, both are tested with the ensemble inference of the internal classifiers.
Revised: Also, both are tested with the ensemble inference of the internal classifiers.

Original: It should be noted that we cannot compare the accuracy of DepthFL and HeteroFL directly, since their local models have a different size, so comparing only with their corresponding exclusive learning is meaningful.
Revised: It should be noted that we cannot compare the accuracy of DepthFL and HeteroFL directly, since their local models have a different size, so comparing only with their corresponding exclusive learning is meaningful.

Original: Table 2 shows the results comparing HeteroFL, DepthFL(FedAvg), and DepthFL to their corresponding exclusive learning.
Revised: Table 2 shows the results comparing HeteroFL, DepthFL(FedAvg), and DepthFL to their corresponding exclusive learning.

Original: In case of HeteroFL, exclusive learning with b as the global model that prunes half of the channels, and with only 75% of the clients participating in learning, shows a tangibly better accuracy than HeteroFL in CIFAR-100.
Revised: In case of HeteroFL, exclusive learning with b as the global model that prunes half of the channels, and with only 75% of the clients participating in learning, shows a tangibly better accuracy than HeteroFL in CIFAR-100.

Original: This means that although clients with insufficient resources can participate in learning on the HeteroFL framework, heterogeneous local models appear to deteriorate the global model and produce a worse result.
Revised: This means that although clients with insufficient resources can participate in learning on the HeteroFL framework, heterogeneous local models appear to deteriorate the global model and produce a worse result.

Original: DepthFL(FedAvg) shows a better result than any exclusive learning for MNIST and CIFAR-100, but not for Tiny Imagenet.
Revised: DepthFL(FedAvg) shows a better result than any exclusive learning for MNIST and CIFAR-100, but not for Tiny Imagenet.

Original: On the other hand, DepthFL performs better than any of exclusive learning in all datasets.
Revised: On the other hand, DepthFL performs better than any of exclusive learning in all datasets.

Original: This result indicates that DepthFL can better train the global model using heterogeneous local models.
Revised: This result indicates that DepthFL can better train the global model using heterogeneous local models.

Original: One question is why HeteroFL shows a tangibly lower performance than exclusive learning, and why this is not the case for DepthFL(FedAvg).
Revised: One question is why HeteroFL shows a tangibly lower performance than exclusive learning, and why this is not the case for DepthFL(FedAvg).

Original: To answer these questions, we measured the performance of each global sub-model of HeteroFL separately, by running 1/4, 2/4, 3/4, and 4/4 channels of the global model.
Revised: To answer these questions, we measured the performance of each global sub-model of HeteroFL separately, by running 1/4, 2/4, 3/4, and 4/4 channels of the global model.

Original: We also measured the same for InclusiveFL
Revised: We also measured the same for InclusiveFL

Original: (Liu et al., 2022)  and DepthFL(FedAvg), i.e., the performance of classifiers at 1/4, 2/4, 3/4, and 4/4 of the global model.
Revised: (Liu et al., 2022) without momentum distillation and DepthFL(FedAvg), i.e., the performance of classifiers at 1/4, 2/4, 3/4, and 4/4 of the global model.

Original: Table 3 shows the result compared to exclusive learning result of Table 2 (named Classifier 1/4 ∼ 4/4).
Revised: Table 3 shows the result compared to exclusive learning result of Table 2 (named Classifier 1/4 ∼ 4/4).

Original: We can see that a global sub-model of HeteroFL mostly performs worse than the corresponding exclusive leaning model (although they have the same number of channels and clients), while the global submodels of DepthFL(FedAvg) perform better than exclusive learning.
Revised: We can see that a global sub-model of HeteroFL mostly performs worse than the corresponding exclusive learning model, while the global sub-models of DepthFL(FedAvg) perform better than exclusive learning.

Original: In HeteroFL, for example, only 25% clients train Classifier 1/4 directly as in exclusive learning, while the rest 75% clients train it indirectly during the training of their 2/4, 3/4, or 4/4 Classifiers, so we suspect that this causes a parameter mismatch for Classifier 1/4 when being averaged, degrading the performance of the global model.
Revised: In HeteroFL, for example, only 25% clients train Classifier 1/ directly as in exclusive learning, while the rest 75% clients train it indirectly during the training of their 2/4, 3/4, or 4/4 Classifiers, so we suspect that this causes a parameter mismatch for Classifier 1/4 when being averaged, degrading the performance of the global model.

Original: To confirm this suspicion, we evaluated a different version of HeteroFL called SHeteroFL , proposed in Split-Mix (Hong et al., 2022), where a client trains all affordable sub-models.
Revised: To confirm this suspicion, we evaluated a different version of HeteroFL called SHeteroFL , proposed in Split-Mix (Hong et al., 2022), where a client trains all affordable sub-models.

Original: For example, when a client with Classifier 2/4 trains, it not only trains Classifier 1/4 indirectly when training Classifier 2/4, but trains  Classifier 1/4 directly.
Revised: For example, when a client with Classifier 2/4 trains, it not only trains Classifier 1/4 indirectly when training Classifier 2/4, but trains [Table] [Table] Classifier 1/4 directly.

Original: Table 3 shows that SHeteroFL differs from HeteroFL in that the accuracy of its global sub-models is similar or higher than that of exclusive learning, as in DepthFL.
Revised: Table 3 shows that SHeteroFL differs from HeteroFL in that the accuracy of its global sub-models is similar or higher than that of exclusive learning, as in DepthFL.

Original: This can also explain the better performance of DepthFL(FedAvg) than InclusiveFL.
Revised: This can also explain the better performance of DepthFL(FedAvg) than its exclusive learning.

Original: That is, when a client trains its 2/4 Classifier, it  trains 1/4 Classifieras well with the companion objective (see Apeendix A.3), as explained in Section 3.1.
Revised: That is, when a client trains its 2/4 Classifier, it also trains 1/4 Classifier with the companion objective, as explained in Section 3.1.

Revised: InclusiveFL has no companion objective, showing a worse accuracy in Table 3 (see also Appendix A.3).

Original: Actually, DepthFL does not require redundant training of sub-models when training their super-model, unlike SHeteroFL.
Revised: Actually, DepthFL does not require redundant training of sub-models when training their super-model, unlike SHeteroFL.

Original: Moreover, additional training of depth-scaled sub-models can further improve the global model[Table] [Table] performance, unlike additional training of width-scaled sub-models (see Appendix A.1).
Revised: Moreover, additional training of depth-scaled sub-models can further improve the global model performance, unlike additional training of widthscaled sub-models (see Appendix A.1).

Original: DepthFL also works better than Split-Mix for the same amount of MACs (see Appendix A.2).
Revised: DepthFL also works better than Split-Mix for the same amount of MACs (see Appendix A.2).

Original: All these results indicate that DepthFL is more effective than other methods for handling parameter mismatches.
Revised: All these results indicate that DepthFL is more effective than other methods for handling parameter mismatches.

Original: 4. A BLATION S TUDY
Revised: 4. A BLATION S TUDY

Original: Knowledge Distillation To analyze the impact of mutual self-distillation on the performance of the global model, we turn on and off self distillation and measure the accuracy of the global model.
Revised: Knowledge Distillation To analyze the impact of mutual self-distillation on the performance of the global model, we turn on and off self distillation and measure the accuracy of the global model.

Original: We experiment with both IID and non-IID data distribution, whose results are depicted in Table 4.
Revised: We experiment with both IID and non-IID data distribution, whose results are depicted in Table 4.

Original: Regardless of data distribution, it shows that self distillation plays a key role in enhancing the accuracy of deep classifiers.
Revised: Regardless of data distribution, it shows that self distillation plays a key role in enhancing the accuracy of deep classifiers.

Original: That is, deep classifiers tend to perform worse than shallow classifiers when self distillation is off, yet they perform similarly or better in most cases when self distillation is on.
Revised: That is, deep classifiers tend to perform worse than shallow classifiers when self distillation is off, yet they perform similarly or better in most cases when self distillation is on.

Original: Table 4 shows that self distillation improves the accuracy of shallow classifiers as well, which is encouraging because resource-constrained clients should do inference using shallow classifiers.
Revised: Table 4 shows that self distillation improves the accuracy of shallow classifiers as well, which is encouraging because resource-constrained clients should do inference using shallow classifiers.

Original: The impact on the performance of the ensemble model is relatively small compared to individual classifiers since the variance among the classifiers is reduced when self-distillation is used.

Original: Maximum & Dynamic Complexity
Revised: Maximum & Dynamic Complexity

Original: We evaluate why self-distillation enhances the accuracy of deep classifiers.
Revised: We evaluate why self-distillation enhances the accuracy of deep classifiers.

Original: Our conjecture was that fully-supervised shallow classifiers can train undersupervised deep classifiers that were partially trained with a limited amount of data.
Revised: Our conjecture was that fully-supervised shallow classifiers can train undersupervised deep classifiers that were partially trained with a limited amount of data.

Original: For this evaluation, we constructed a Maximum experimental environment where all clients can train the full global model (i.e., the d model in Table 1).
Revised: For this evaluation, we constructed a Maximum experimental environment where all clients can train the full global model (i.e., the d model in Table 1).

Original: We performed mutual self-distillation on the Maximum environment and measure the accuracy as in Table 5, which includes the previous IID result of Table 4 (marked Fixed ) for comparison.
Revised: We performed mutual self-distillation on the Maximum environment and measure the accuracy as in Table 5, which includes the previous IID result of Table 4 (marked Fixed ) for comparison.

Original: Since all clients in Maximum have enough resources to train every classifier of the global model, self-distillation would work only as general optimization.

Original: We can actually see that the impact of self-distillation on deep  classifiers in Maximum is tangibly smaller than in Fixed.
Revised: We can actually see that the impact of self-distillation on deep [Table] [Table] classifiers in Maximum is tangibly smaller than in Fixed.

Original: This means that when resources are heterogeneous, self-distillation achieves additional performance by transmitting the domain knowledge of shallow classifiers to deep classifiers.
Revised: This means that when resources are heterogeneous, self-distillation achieves additional performance by transmitting the domain knowledge of shallow classifiers to deep classifiers.

Original: We also constructed a Dynamic experimental environment where the resources of the clients can change randomly in every round, yet the average amount of resources of those clients participating[Table] [Table] in each round is the same as in Fixed.
Revised: We also constructed a Dynamic experimental environment where the resources of the clients can change randomly in every round, yet the average amount of resources of those clients participating in each round is the same as in Fixed.

Original: So, unlike Fixed, deeper layers in Dynamic can learn from all data from every client as in Maximum, albeit less frequently.
Revised: So, unlike Fixed, deeper layers in Dynamic can learn from all data from every client as in Maximum, albeit less frequently.

Original: In Table 5, we can see that the impact of self-distillation on deep classifiers in Dynamic is similar to Maximum.
Revised: In Table 5, we can see that the impact of self-distillation on deep classifiers in Dynamic is similar to Maximum.

Original: This indicates that the lower accuracy of deep classifiers in Fixed is indeed due to those unseen data of resourceconstrained clients, and that self-distillation can alleviate the problem.
Revised: This indicates that the lower accuracy of deep classifiers in Fixed is indeed due to those unseen data of resource-constrained clients, and that self-distillation can alleviate the problem.

Original: 4. U NDERSTANDING S ELF -D ISTILLATION E FFECT OF D EPTH FL
Revised: 4. U NDERSTANDING S ELF -D ISTILLATION E FFECT OF D EPTH FL

Original: As we saw above, self-distillation is highly effective for DepthFL.

Original: This section attempts to understand the fundamental reason behind the effectiveness of self-distillation for depthFL.
Revised: This section attempts to understand the fundamental reason behind the effectiveness of selfdistillation for depthFL.

Original: For this we evaluate knowledge distillation in a general setup of teacher-student models in a centralized learning environment, except that the student model is trained with insufficient data (as the partially-trained, deep layers of DepthFL).
Revised: For this we evaluate knowledge distillation in a general setup of teacherstudent models in a centralized learning environment, except that the student model is trained with insufficient data (as the partially-trained, deep layers of DepthFL).

Original: We first evaluate if the existing analysis methods of knowledge distillation in (Tang et al., 2020), which are label smoothing (LS), gradient rescaling (KD-pt), and domain knowledge of class relationships (KD-sim), can explain the effectiveness of self-distillation in DepthFL (see A.6 for detailed explanation).
Revised: We first evaluate if the existing analysis methods of knowledge distillation in (Tang et al., 2020), which are label smoothing (LS), gradient rescaling (KD-pt), and domain knowledge of class relationships (KD-sim), can explain the effectiveness of self-distillation in DepthFL (see A.6 for detailed explanation).

Original: The Resnet18 model is used as the student, while the Resnet101 model, which is fully trained with all data, is used as the teacher.
Revised: The Resnet model is used as the student, while the Resnet101 model, which is fully trained with all data, is used as the teacher.

Original: Additionally, we analyze the effect of knowledge distillation by the poor teacher (PKD), which is trained only with 25% of the data, as well as by the light teacher (LKD), whichis trained with all data but is sized smaller than the student model (as the fully-trained, shallow layers of DepthFL). As
Revised: Additionally, we analyze the effect of knowledge distillation by the poor teacher (PKD), which is trained only with 25% of the data, as well as by the light teacher (LKD), which

Original: [Table]
Revised: [Table]

Revised: is trained with all data but is sized smaller than the student model (as the fully-trained, shallow layers of DepthFL).

Original:  the light teacher of LKD, we use × 0 .
Revised: As the light teacher of LKD, we use × 0 .

Original: 25 depth-scaled local model a = W 1 l of Resnet18.
Revised: 25 depth-scaled local model a = W 1 l of Resnet18.

Original: We perform these experiments with a variable amount of data the student learns as in Table 6.
Revised: We perform these experiments with a variable amount of data the student learns as in Table 6.

Original: When the student model learns all data (100%), most effects of knowledge distillation could be explained by the three existing effects.
Revised: When the student model learns all data (100%), most effects of knowledge distillation could be explained by the three existing effects.

Original: However, when the student model learns partly (50% or 25%), the effect is not fully explainable by the composition of those three effects, neither by PKD.
Revised: However, when the student model learns partly (50% or 25%), the effect is not fully explainable by the composition of those three effects, neither by PKD.

Original: On the other hand, LKD is quite effective, achieving high accuracy even for 50% and 25%.
Revised: On the other hand, LKD is quite effective, achieving high accuracy even for 50% and 25%.

Original: This means that even if the size of the light teacher is smaller than that of the student, the class relationship for each input instance can be transferred by the light teacher, which effectively helps for the generalization of the data-insufficient student model.
Revised: This means that even if the size of the light teacher is smaller than that of the student, the class relationship for each input instance can be transferred by the light teacher, which effectively helps for the generalization of the data-insufficient student model.

Original: This additional role appears to be the reason why self-distillation in DepthFL is effective, especially for data-insufficient deep layers.
Revised: This additional role appears to be the reason why self-distillation in DepthFL is effective, especially for data-insufficient deep layers.

Original: 4. R OBUSTNESS T EST
Revised: 4. R OBUSTNESS T EST

Original: The performance of DepthFL would inevitably be different depending on the distribution of each client’s resource capability.
Revised: The performance of DepthFL would inevitably be different depending on the distribution of each client’s resource capability.

Original: To evaluate the robustness of DepthFL, we changed the distribution of resource capabilities and measured the performance of the classifiers, as in Figure 2.
Revised: To evaluate the robustness of DepthFL, we changed the distribution of resource capabilities and measured the performance of the classifiers, as in Figure 2.

Original: As expected, as the ratio of resource-constrained clients increases, the performance of deep classifiers gets lower, even seriously when self distillation is off.
Revised: As expected, as the ratio of resource-constrained clients increases, the performance of deep classifiers gets lower, even seriously when self distillation is off.

Original: With self distillation on, however, even if the deepest classifier is trained only by 10% of the clients, the performance drop is small, due to the help of other classifiers.
Revised: With self distillation on, however, even if the deepest classifier is trained only by 10% of the clients, the performance drop is small, due to the help of other classifiers.

Original: Since shallow classifiers will be used for fast inference while an ensemble model will be used for high performance, deep classifiers do not always have to be better than shallow ones.
Revised: Since shallow classifiers will be used for fast inference while an ensemble model will be used for high performance, deep classifiers do not always have to be better than shallow ones.

Original: Appendix A.5.1 explains and evaluates whether deep classifiers are really needed for DepthFL.
Revised: Appendix A.5.1 explains and evaluates whether deep classifiers are really needed for DepthFL.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We presented DepthFL, a new federated learning framework considering resource heterogeneity of clients.
Revised: We presented DepthFL, a new federated learning framework considering resource heterogeneity of clients.

Original: DepthFL creates local models of different sizes by scaling the depth of the global model, and allocates them to the clients depending on their available resources.
Revised: DepthFL creates local models of different sizes by scaling the depth of the global model, and allocates them to the clients depending on their available resources.

Original: During local training, a client trains several classifiers within its local model, and at the same time, distills their knowledge with each other.
Revised: During local training, a client trains several classifiers within its local model, and at the same time, distills their knowledge with each other.

Original: As a result, both deep classifiers trained with limited data and shallow classifiers trained by most clients can help one another and collaboratively build the global model.
Revised: As a result, both deep classifiers trained with limited data and shallow classifiers trained by most clients can help one another to build the global model, with no parameter mismatch.

Original: We also evaluated depth scaling compared to width scaling thoroughly, and self-distillation in DepthFL.
Revised: We also evaluated depth scaling compared to width scaling thoroughly, and self-distillation in DepthFL.

Revised: A CKNOWLEDGMENTS

Revised: This work was supported by Institute of Information & communications Technology Planning &

Revised: Evaluation (IITP) grant funded by the Korea government (MSIT)

Revised: (No. 2021-0-00180, 25%) and (No. 2021-0-00136, 25%), by the ITRC (Information Technology Research Center) support program (IITP-2021-0-01835, 25%) supervised by the IITP, and by the National Research Foundation of

Revised: Korea (NRF) grant funded by the Korea government (MSIT)

Original:  A A PPENDIX A. L IMITATION OF SH ETERO FL
Revised: (No. 2020R1A2B5B02001845, 25%). A A PPENDIX A. L IMITATION OF SH ETERO FL

Original: As mentioned in Section 4.2, although SHeteroFL includes additional computation overhead of locally training all possible sub-models, the performance of its global model that leverages all these sub-models can improve.
Revised: As mentioned in Section 4.2, although SHeteroFL includes additional computation overhead of locally training all possible sub-models, the performance of its global model that leverages all these sub-models can improve.

Original: However, we want to check whether additional training of sub-models of different widths is always helpful for the global model.
Revised: However, we want to check whether additional training of sub-models of different widths is always helpful for the global model.

Original: We compared the performance of FedAvg,
Revised: We compared the performance of FedAvg,

Original: SHeteroFL, and DepthFL(FedAvg) in the Maximum case introduced in Section 4.3 where all clients have enough resources to train the global model.
Revised: SHeteroFL, and DepthFL(FedAvg) in the Maximum case introduced in Section 4.3 where all clients have enough resources to train the global model.

Original: Table 7 shows the experimental result on CIFAR100 using Resnet18 as the global model.
Revised: Table 7 shows the experimental result on CIFAR100 using Resnet18 as the global model.

Original: SHeteroFL does not show better performance than FedAvg despite its additional training of width-scaled sub-models.
Revised: SHeteroFL does not show better performance than FedAvg despite its additional training of width-scaled sub-models.

Original: On the other hand, DepthFL could improve the performance of the global model by additionally training depth-scaled sub-models, which is in line with the result in Lee et al. (2015).
Revised: On the other hand, DepthFL could improve the performance of the global model by additionally training depth-scaled sub-models, which is in line with the result in Lee et al. (2015).

Original: In other words, although SHeteroFL could alleviate the problem of HeteroFL, DepthFL is a better approach to heterogeneous federated learning since it can better enhance the performance of global model with less computation cost.
Revised: In other words, although SHeteroFL could alleviate the problem of HeteroFL, DepthFL is a better approach to heterogeneous federated learning since it can better enhance the performance of global model with less computation cost.

Original: [Table]
Revised: [Table]

Original: A. C OMPARISON WITH S PLIT -M IX (H ONG ET AL ., 2022)
Revised: A. C OMPARISON WITH S PLIT -M IX (H ONG ET AL ., 2022)

Original: Split-Mix provides a customizable global model by mixing the base split models of the same sizes.
Revised: Split-Mix provides a customizable global model by mixing the base split models of the same sizes.

Original: While HeteroFL and DepthFL prune a single global model to create local models of different sizes,
Revised: While HeteroFL and DepthFL prune a single global model to create local models of different sizes,

Original: Split-Mix can create a local model by combining small base models.
Revised: Split-Mix can create a local model by combining small base models.

Original: To compare Split-Mix with
Revised: To compare Split-Mix with

Original: DepthFL, we combine its base models to create four local models whose number of MACs is the same as that of the four local models of DepthFL.
Revised: DepthFL, we combine its base models to create four local models whose number of MACs is the same as that of the four local models of DepthFL.

Original: For the CIFAR-100 dataset, Resnet18 was used as a global model.
Revised: For the CIFAR-100 dataset, Resnet18 was used as a global model.

Original: The number of MACs and the number of parameters of four local models are shown in Table 8.
Revised: The number of MACs and the number of parameters of four local models are shown in Table 8.

Original: As the base model of Split-Mix, half of the channels of the global model were pruned.
Revised: As the base model of Split-Mix, half of the channels of the global model were pruned.

Original: For a fair comparison, DepthFL was trained without regularization and self distillation.
Revised: For a fair comparison, DepthFL was trained without regularization and self distillation.

Original: [Table]
Revised: [Table]

Original: The accuracy of DethpFL and Split-Mix in Fixed and Dynamic cases are shown in Figure 3.
Revised: The accuracy of DethpFL and Split-Mix in Fixed and Dynamic cases are shown in Figure 3.

Original: In Fixed
Revised: In Fixed

Original: [Table]
Revised: [Table]

Original: the learning curves of DepthFL and Split-Mix are almost same.
Revised: the learning curves of DepthFL and Split-Mix are almost same.

Original: Since the number of parameters of local models of DepthFL is much smaller than Split-Mix, DepthFL is more efficient in terms of communication overhead.
Revised: Since the number of parameters of local models of DepthFL is much smaller than Split-Mix, DepthFL is more efficient in terms of communication overhead.

Original: In Dynamic, DepthFL shows faster convergence and higher accuracy than Split-Mix.
Revised: In Dynamic, DepthFL shows faster convergence and higher accuracy than Split-Mix.

Original: The reason is that Split-Mix allows the client to train multiple base models alternately, so there is no big difference between Fixed and Dynamic, whereas in DepthFL, the deep classifier can train more data in Dynamic, as explained in Section 4. [Table]
Revised: The reason is that Split-Mix allows the client to train multiple base models alternately, so there is no big difference between Fixed and Dynamic, whereas in DepthFL, the deep classifier can train more data in Dynamic, as explained in Section 4.3.

Original: A. C OMPARISON WITH I NCLUSIVE FL (L IU ET AL ., 2022)
Revised: A. C OMPARISON WITH I NCLUSIVE FL (L IU ET AL ., 2022)

Original: Recently, InclusiveFL (Liu et al., 2022) proposed a kind of depth-scaled method for heterogeneous federated learning.
Revised: Recently, InclusiveFL (Liu et al., 2022) proposed a kind of depth-scaled method for heterogeneous federated learning.

Original: However, there are two major differences from DepthFL.
Revised: However, there are two major differences from DepthFL.

Original: First, unlike DepthFL,
Revised: First, unlike DepthFL,

Original: InclusiveFL trains only the last classifier of the local model without trainingsub-models of the local model.
Revised: InclusiveFL trains only the last classifier of the local model without training the sub-models.

Original: As we saw in Table 3, trainingall possible sub-models during local training has a decisive impact on performance.
Revised: As we saw in Table 3, training sub-models during local training has a decisive impact on the performance of global sub-models, hence the overall performance (its ensemble result is 63.79, lower than the ensemble result 72.34 of DepthFL).

Original: Similar to the comparison result between SHeteroFL andHeteroFL,
Revised: That is, similar to the comparison result between SHeteroFL and

Original:  DepthFL has superior performance of sub-classifiers compared to InclusiveFL, and this has a direct impact on the performance of the global model.
Revised: HeteroFL, DepthFL has superior performance of sub-classifiers compared to InclusiveFL, and this has a direct impact on the performance of the global model.

Original: Unlike DepthFL, InclusiveFL uses only the output of the last classifier during inference, soin InclusiveFL, the performance of the sub classifiers is not that important, and unlike DepthFL, there is no overhead for bottleneck layers.
Revised: Actually, unlike DepthFL, InclusiveFL uses only the output of the last classifier during inference, so the performance of the sub classifiers is not that important, and InclusiveFL has no overhead for bottleneck layers.

Original: Nevertheless,since the performance of Classifier 4/4 in InclusiveFL islow even compared to exclusive learning, sub-modelsexisting in the local model must be trained together through the companion objective during local training.
Revised: Nevertheless, the performance of Classifier 4/4 in InclusiveFL is even lower than exclusive learning 1/4 ∼ 3/4, which indicates that training all sub-models in the local model using the companion objective is critical.

Original: Second, since InclusiveFL locally trains only the last classifier in the local model, knowledge distillation between sub-models is impossible.
Revised: Second, since InclusiveFL locally trains only the last classifier in the local model, knowledge distillation between sub-models is impossible.

Original: InclusiveFL uses a special technique called momentum distillation to transfer the knowledge of a larger model to a smaller model, which has a limitation since the size of each layer must be kept constant.
Revised: InclusiveFL uses a special technique called momentum distillation to transfer the knowledge of a larger model to a smaller model, which has a limitation since the size of each layer must be kept constant.

Original: Also, its role is different from self-distillation in DepthFL, which transfers knowledge of a smaller model to a larger model.
Revised: Also, its role is different from self-distillation in DepthFL, which transfers knowledge of a smaller model to a larger model.

Original: In summary, since
Revised: In summary, since

Original: InclusiveFL does not train sub-models in the local model through companion objective, InclusiveFL does not have many advantages of depth scaling unlike DepthFL.
Revised: InclusiveFL does not train sub-models in the local model through companion objective nor employ self distillation, InclusiveFL does not fully exploit the advantages of depth scaling unlike DepthFL.

Original: A. O PTIMIZER
Revised: A. O PTIMIZER

Original: DepthFL employs FedDyn (Acar et al., 2021) as the default optimizer.
Revised: DepthFL employs FedDyn (Acar et al., 2021) as the default optimizer.

Original: To analyze the impact of the optimizer, we replaced FedDyn with FedAvg (McMahan et al., 2017).
Revised: To analyze the impact of the optimizer, we replaced FedDyn with FedAvg (McMahan et al., 2017).

Original: And, we performed the same experiments in Table 4 and Table 5, whose results are shown in Table 9 and Table 10, respectively.
Revised: And, we performed the same experiments in Table 4 and Table 5, whose results are shown in Table 9 and Table 10, respectively.

Original: Even if FedAvg is used as the optimizer, the overall trend of the experimental results is almost the same, but we can see that the overall accuracy is lowered.
Revised: Even if FedAvg is used as the optimizer, the overall trend of the experimental results is almost the same, but we can see that the overall accuracy is lowered.

Original: Also, we can see that the impact of self-distillation is reduced compared to when FedDyn is used.
Revised: Also, we can see that the impact of self-distillation is reduced compared to when FedDyn is used.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A. M ORE A BLATION S TUDIES A.5. D EEP C LASSIFIERS ARE R EALLY N EEDED ?
Revised: A. M ORE A BLATION S TUDIES A.5. D EEP C LASSIFIERS ARE R EALLY N EEDED ?

Original: Table 4 showed that the performance of the deepest classifiers is similar or slightly lower than that of shallow classifiers even with self-distillation.
Revised: Table 4 showed that the performance of the deepest classifiers is similar or slightly lower than that of shallow classifiers even with self-distillation.

Original: So, one might question if deep classifiers are really needed for resource-heterogeneous federated learning, since they are likely to have fewer clients to train although they allow more generalization of the model.
Revised: So, one might question if deep classifiers are really needed for resource-heterogeneous federated learning, since they are likely to have fewer clients to train although they allow more generalization of the model.

Original: That is, if most clients have insufficient resources, deeper layers might not help; otherwise, they might contribute to enhancing the performance of the global model.
Revised: That is, if most clients have insufficient resources, deeper layers might not help; otherwise, they might contribute to enhancing the performance of the global model.

Original: On the other hand
Revised: On the other hand

Original: [Table]
Revised: [Table]

Original: as in the Dynamic case in Table
Revised: as in the Dynamic case in Table

Original: [Table]
Revised: [Table]

Original: when the client resources change dynamically over time so that all clients can train deep layers even occasionally, deep layers can have a higher performance.
Revised: when the client resources change dynamically over time so that all clients can train deep layers even occasionally, deep layers can have a higher performance.

Original: In reality
Revised: In reality

Original: [Table]
Revised: [Table]

Original: the performance of classifiers with different depths would inevitably be different depending on various factors such as the distribution of client resources, the nature of the learning task, the number of data each client has, and the size of global model/sub-models.
Revised: the performance of classifiers with different depths would inevitably be different depending on various factors such as the distribution of client resources, the nature of the learning task, the number of data each client has, and the size of global model/sub-models.

Original: We tried to test whether deep classifiers are really necessary by performing an experiment excluding deep classifiers, whose result is in Figure 4.
Revised: We tried to test whether deep classifiers are really necessary by performing an experiment excluding deep classifiers, whose result is in Figure 4.

Original: For example, DepthFL (exclude #4 Classifier) reduces the depth of the global model to 3/4, so that those 25% clients whose resource can train Classifier 4/4 trains only up to Classifier 3/4.
Revised: For example, DepthFL (exclude #4 Classifier) reduces the depth of the global model to 3/4, so that those 25% clients whose resource can train Classifier 4/4 trains only up to Classifier 3/4.

Original: We conducted the experiment with big and small global models (Resnet18 and ConvNet) for two learning environments (Fixed and Dynamic) on CIFAR-100.
Revised: We conducted the experiment with big and small global models (Resnet18 and ConvNet) for two learning environments (Fixed and Dynamic) on CIFAR-100.

Original: We can see that if the size of the global model is smaller compared to the task, and if the client resources change dynamically so the deep classifier can see more data, deep classifiers gets more important for high performance.
Revised: We can see that if the size of the global model is smaller compared to the task, and if the client resources change dynamically so the deep classifier can see more data, deep classifiers gets more important for high performance.

Original: Actually, this paper presented a general framework that can flexibly deal with such diverse situations.
Revised: Actually, this paper presented a general framework that can flexibly deal with such diverse situations.

Original: A.5. N ON -IID D EGREE AND N UMBER OF C LIENTS
Revised: A.5. N ON -IID D EGREE AND N UMBER OF C LIENTS

Original: We performed ablation study for diverse non-IID degrees and for different number of clients, whose results are in Table 11 and Table 12, respectively.
Revised: We performed ablation study for diverse non-IID degrees and for different number of clients, whose results are in Table 11 and Table 12, respectively.

Original: They show a similar behavior as previously.
Revised: They show a similar behavior as previously.

Original: A. P ARTIAL K NOWLEDGE D ISTILLATION M ETHODS
Revised: A. P ARTIAL K NOWLEDGE D ISTILLATION M ETHODS

Original: The experiment in Section 4.4 was conducted as follows.
Revised: The experiment in Section 4.4 was conducted as follows.

Original: In KD-pt method, we synthesize teacher distribution ρ pt as ρ pti =
Revised: In KD-pt method, we synthesize teacher distribution ρ pt as ρ pti =

Original: p t if i =
Revised: p t if i =

Original: t , (1 − p t ) /
Revised: t , (1 − p t ) /

Original: ( K − 1) otherwise, where p t is prediction on ground truth class from the teacher’s probability distribution.
Revised: ( K − 1) otherwise, where p t is prediction on ground truth class from the teacher’s probability distribution.

Original: In KD-sim method, we synthesize teacher distribution ρ sim as the softmax over cosine similarity between the teacher’s last logit layer’s weights: ρ sim = softmax ( relu ( ˆ w t ˆ W T ) α /β ) , where ˆ W ∈ R K × d is the normalized logit layer
Revised: In KD-sim method, we synthesize teacher distribution ρ sim as the softmax over cosine similarity between the teacher’s last logit layer’s weights: ρ sim = softmax ( relu ( ˆ w t ˆ W T ) α /β ) , where ˆ W ∈ R K × d is the normalized logit layer

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: weights, ˆ w t is the t -th row of ˆ W corresponding to the ground truth
Revised: weights, ˆ w t is the t -th row of ˆ W corresponding to the ground truth

Original: [Table]
Revised: [Table]

Original: and α, β are hyper-parameters for resolution of cosine similarities
Revised: and α, β are hyper-parameters for resolution of cosine similarities

Original: [Table]
Revised: [Table]

Original: A. L EARNING CURVES
Revised: A. L EARNING CURVES

Original: The learning curves of Table 2 for comparing HeteroFL, DepthFL(FedAvg), and DepthFL with their corresponding exclusive learning are in Figure 5.
Revised: The learning curves of Table 2 for comparing HeteroFL, DepthFL(FedAvg), and DepthFL with their corresponding exclusive learning are in Figure 5.

Original: The learning curves of Table 4 for the ablation study of knowledge distillation are in Figure 6.
Revised: The learning curves of Table 4 for the ablation study of knowledge distillation are in Figure 6.

Original: The x-axis is the communication round, and the y-axis is the moving average of the test accuracy.
Revised: The x-axis is the communication round, and the y-axis is the moving average of the test accuracy.

Original: B H YPERPARAMETERS
Revised: B H YPERPARAMETERS

Original: Most of the hyperparameters used in our experiments are depicted in Table 13.
Revised: Most of the hyperparameters used in our experiments are depicted in Table 13.

Original: We set both α, β hyper-parameters to 0 .
Revised: We set both α, β hyper-parameters to 0 .

Original: 3 for KD-sim (Tang et al., 2020) experiments.
Revised: 3 for KD-sim (Tang et al., 2020) experiments.

Original: In the masked language modeling task with the transformer model, we randomly selected 15% of the input tokens.
Revised: In the masked language modeling task with the transformer model, we randomly selected 15% of the input tokens.

Original: Among the selected tokens, 80% were modified to [mask] tokens, 10% to random tokens [Table] and 10% to remain unchanged.
Revised: Among the selected tokens, 80% were modified to [mask] tokens, 10% to random tokens, and 10% to remain unchanged.

Original: [Table]
Revised: [Table]