Original: F ED CL: C RITICAL L EARNING P ERIODS - AWARE A DAPTIVE C LIENT S ELECTION IN F EDERATED L EARNING
Revised: F ED CL: C RITICAL L EARNING P ERIODS - AWARE A DAPTIVE C LIENT S ELECTION IN F EDERATED L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Federated learning (FL) is a distributed optimization paradigm that learns from data samples distributed across a number of clients.
Revised: Federated learning (FL) is a distributed optimization paradigm that learns from data samples distributed across a number of clients.

Original: Adaptive client selection that is cognizant of the training progress of clients has become a major trend to improve FL efﬁciency but not yet well-understood.
Revised: Adaptive client selection that is cognizant of the training progress of clients has become a major trend to improve FL efﬁciency but not yet well-understood.

Original: Most existing FL methods such as FedAvg and its state-of-the-art variants implicitly assume that all learning phases during the FL training process are equally important.
Revised: Most existing FL methods such as FedAvg and its state-of-the-art variants implicitly assume that all learning phases during the FL training process are equally important.

Original: Unfortunately, this assumption has been revealed to be invalid due to recent ﬁndings on critical learning (CL) periods, in which small gradient errors may lead to an irrecoverable deﬁciency on ﬁnal test accuracy.
Revised: Unfortunately, this assumption has been revealed to be invalid due to recent ﬁndings on critical learning (CL) periods, in which small gradient errors may lead to an irrecoverable deﬁciency on ﬁnal test accuracy.

Original: In this paper, we develop FedCL, a CL periods-aware FL framework to reveal that adaptively augmenting exiting FL methods with CL periods, the resultant performance is signiﬁcantly improved when the client selection is guided by the discovered CL periods.
Revised: In this paper, we develop FedCL, a CL periods-aware FL framework to reveal that adaptively augmenting exiting FL methods with CL periods, the resultant performance is signiﬁcantly improved when the client selection is guided by the discovered CL periods.

Original: Experiments based on various machine learning models and datasets validate that the proposed FedCL framework consistently achieves an improved model accuracy while maintains comparable or even better communication efﬁciency as compared to state-of-the-art methods, demonstrating a promising and easily adopted method for tackling the heterogeneity of FL training.
Revised: Experiments based on various machine learning models and datasets validate that the proposed FedCL framework consistently achieves an improved model accuracy while maintains comparable or even better communication efﬁciency as compared to state-of-the-art methods, demonstrating a promising and easily adopted method for tackling the heterogeneity of FL training.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Federated learning (FL) (McMahan et al., 2017) has emerged as an attractive distributed learning paradigm that leverages a large number of clients to collaboratively learn a joint model with decentralized training data under the coordination of a centralized server.
Revised: Federated learning (FL) (McMahan et al., 2017) has emerged as an attractive distributed learning paradigm that leverages a large number of clients to collaboratively learn a joint model with decentralized training data under the coordination of a centralized server.

Original: In contrast with centralized learning, the FL architecture allows for preserving clients’ privacy and reducing the communication burden caused by transmitting data to the server.
Revised: In contrast with centralized learning, the FL architecture allows for preserving clients’ privacy and reducing the communication burden caused by transmitting data to the server.

Original: While there is a rich literature in distributed optimization in the context of machine learning, FL distinguishes itself from traditional distributed optimization in two key challenges: high degrees of system and statistical heterogeneity (Kairouz et al., 2019).
Revised: While there is a rich literature in distributed optimization in the context of machine learning, FL distinguishes itself from traditional distributed optimization in two key challenges: high degrees of system and statistical heterogeneity (Kairouz et al., 2019).

Original: In an attempt to address the heterogeneity and improve the efﬁciency of FL, various optimization methods have been developed for FL.
Revised: In an attempt to address the heterogeneity and improve the efﬁciency of FL, various optimization methods have been developed for FL.

Original: In particular, the federated averaging algorithm (FedAvg) (McMahan et al., 2017) is the current state-of-the-art method for FL.
Revised: In particular, the federated averaging algorithm (FedAvg) (McMahan et al., 2017) is the current state-of-the-art method for FL.

Original: In each communication round,
Revised: In each communication round,

Original: FedAvg leverages local computation at each client and employs a centralized server to aggregate and update the global model parameter.
Revised: FedAvg leverages local computation at each client and employs a centralized server to aggregate and update the global model parameter.

Original: While FedAvg has demonstrated empirical success in heterogeneous settings, it fails to fully address the underlying challenges associated with heterogeneity.
Revised: While FedAvg has demonstrated empirical success in heterogeneous settings, it fails to fully address the underlying challenges associated with heterogeneity.

Original: For example, FedAvg randomly selects a subset of clients in each iteration regardless of their statistical heterogeneity, which has been shown to diverge empirically in settings where data samples of each client follow a non-identical and independent distribution (non-IID).
Revised: For example, FedAvg randomly selects a subset of clients in each iteration regardless of their statistical heterogeneity, which has been shown to diverge empirically in settings where data samples of each client follow a non-identical and independent distribution (non-IID).

Original: A recent trend of improving FL efﬁciency focuses on adaptive client selection during the FL training process, such as (Ruan et al., 2021;
Revised: A recent trend of improving FL efﬁciency focuses on adaptive client selection during the FL training process, such as (Ruan et al., 2021;

Original: Karimireddy et al., 2020; Li et al., 2020a; Wang et al., 2020c;b;
Revised: Karimireddy et al., 2020; Li et al., 2020a; Wang et al., 2020c;b;

Original: Cho et al., 2020; Wang et al., 2020a; Rothchild et al., 2020; Lai et al., 2021).
Revised: Cho et al., 2020; Wang et al., 2020a; Rothchild et al., 2020; Lai et al., 2021).

Original: However, these studies implicitly assume that all learning phases during the FL training process are equally important.
Revised: However, these studies implicitly assume that all learning phases during the FL training process are equally important.

Original: Unfortunately, this assumption has recently been revealed to be invalid due to the existence of critical learning (CL) periods , i.e., the ﬁnal quality of a deep neural network (DNN) model is determined by the ﬁrst few training epochs, in which deﬁcits such as low quality or quantity of training data will cause irreversible model degradation.
Revised: Unfortunately, this assumption has recently been revealed to be invalid due to the existence of critical learning (CL) periods , i.e., the ﬁnal quality of a deep neural network (DNN) model is determined by the ﬁrst few training epochs, in which deﬁcits such as low quality or quantity of training data will cause irreversible model degradation.

Original: Notably, this phenomenon was revealed in the latest series of works (Achille et al., 2019; Jastrzebski et al., 2019; Golatkar et al., 2019; Jastrzebski et al., 2021) for centralized learning, and in (Yan et al., 2022) for FL settings.
Revised: Notably, this phenomenon was revealed in the latest series of works (Achille et al., 2019; Jastrzebski et al., 2019; Golatkar et al., 2019; Jastrzebski et al., 2021) for centralized learning, and in (Yan et al., 2022) for FL settings.

Original: Despite their insightful ﬁndings, there remains to be a major gap between the observation of CL periods in FL and the goal of more efﬁcient training and improved model accuracy , since existing client selection methods in state-of-the-art FL algorithms are unaware of the existence of CL periods in FL, which were only identiﬁed using a computationally expensive metric that emerges after the full training process.
Revised: Despite their insightful ﬁndings, there remains to be a major gap between the observation of CL periods in FL and the goal of more efﬁcient training and improved model accuracy , since existing client selection methods in state-of-the-art FL algorithms are unaware of the existence of CL periods in FL, which were only identiﬁed using a computationally expensive metric that emerges after the full training process.

Original: In this paper, we close this gap by demonstrating the importance of CL periods awareness for client selection in state-of-the-art FL algorithms.
Revised: In this paper, we close this gap by demonstrating the importance of CL periods awareness for client selection in state-of-the-art FL algorithms.

Original: Through a range of carefully designed experiments on different machine learning models and datasets, we observe a consistently improved model accuracy without sacriﬁcing communication efﬁciency by augmenting state-of-the-art FL algorithms with CL periods.
Revised: Through a range of carefully designed experiments on different machine learning models and datasets, we observe a consistently improved model accuracy without sacriﬁcing communication efﬁciency by augmenting state-of-the-art FL algorithms with CL periods.

Original: We build upon recent work by (Yan et al., 2022), who showed that if the training dataset for each client is not recovered to the entire training dataset early enough in the training process, the test accuracy of FL is permanently impaired.
Revised: We build upon recent work by (Yan et al., 2022), who showed that if the training dataset for each client is not recovered to the entire training dataset early enough in the training process, the test accuracy of FL is permanently impaired.

Original: We extend this notation to client selection in FL and show that a larger number of clients are only required during these CL periods.
Revised: We extend this notation to client selection in FL and show that a larger number of clients are only required during these CL periods.

Original: As a result, an adaptive and efﬁcient client selection scheme is akin to ﬁnding CL periods in the FL training process.
Revised: As a result, an adaptive and efﬁcient client selection scheme is akin to ﬁnding CL periods in the FL training process.

Original: These
Revised: These

Original: CL periods can be detected in an online manner using a new metric called Federated Gradient Norm (FGN).
Revised: CL periods can be detected in an online manner using a new metric called Federated Gradient Norm (FGN).

Original: To the best of our knowledge, this is the ﬁrst step taken towards exploiting CL periods for adaptive client selection in FL to mitigate heterogeneity.
Revised: To the best of our knowledge, this is the ﬁrst step taken towards exploiting CL periods for adaptive client selection in FL to mitigate heterogeneity.

Original: Our main contributions in this paper are summarized as follows:
Revised: Our main contributions in this paper are summarized as follows:

Original: We propose a practical, easy-to-compute Federated Gradient Norm (FGN) metric to identify CL periods in an online manner, ﬁxing a major paradox for connecting CL periods with client selection for the efﬁcient FL training goal.
Revised: We propose a practical, easy-to-compute Federated Gradient Norm (FGN) metric to identify CL periods in an online manner, ﬁxing a major paradox for connecting CL periods with client selection for the efﬁcient FL training goal.

Original: We propose a simple but powerful CL periods-aware FL framework, dubbed as FedCL, that is generic across and orthogonal to different FL methods.
Revised: We propose a simple but powerful CL periods-aware FL framework, dubbed as FedCL, that is generic across and orthogonal to different FL methods.

Original: In particular, we use FedAvg as our building block since it is the ﬁrst and the most widely used one.
Revised: In particular, we use FedAvg as our building block since it is the ﬁrst and the most widely used one.

Original: FedCL inspects the changes in FGN to detect CL periods in FL training process, and adaptively determines the number of clients to participate in each FL training round.
Revised: FedCL inspects the changes in FGN to detect CL periods in FL training process, and adaptively determines the number of clients to participate in each FL training round.

Original: With extensive empirical evaluation on different machine learning models with different datasets, we show that FedCL consistently achieves up to 11% accuracy improvement while maintaining comparable or even better communication efﬁciency compared to FedAvg. 3.
Revised: With extensive empirical evaluation on different machine learning models with different datasets, we show that FedCL consistently achieves up to 11% accuracy improvement while maintaining comparable or even better communication efﬁciency compared to FedAvg. 3.

Original: We show that the CL periods awareness can be easily combined with state-of-the-art FL methods, such as FedProx (Li et al., 2020a),
Revised: We show that the CL periods awareness can be easily combined with state-of-the-art FL methods, such as FedProx (Li et al., 2020a),

Original: VRL-SGD (Liang et al., 2019) and FedNova (Wang et al., 2020c).
Revised: VRL-SGD (Liang et al., 2019) and FedNova (Wang et al., 2020c).

Original: When augmented by FedCL via manipulating the client selection, existing methods achieve up to 11%, 13% and 10% accuracy improvement, respectively, compared to training without the awareness of CL periods.
Revised: When augmented by FedCL via manipulating the client selection, existing methods achieve up to 11%, 13% and 10% accuracy improvement, respectively, compared to training without the awareness of CL periods.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Critical Learning (CL) Periods.
Revised: Critical Learning (CL) Periods.

Original: The presence of CL periods in centralized neural network training was ﬁrst highlighted in (Achille et al., 2019; Jastrzebski et al., 2019).
Revised: The presence of CL periods in centralized neural network training was ﬁrst highlighted in (Achille et al., 2019; Jastrzebski et al., 2019).

Original: Some other works (Golatkar et al., 2019;
Revised: Some other works (Golatkar et al., 2019;

Original: Jastrzebski et al., 2021; Frankle et al., 2020; Jastrzebski et al., 2020) have also highlighted the importance of early training phase in centralized learning.
Revised: Jastrzebski et al., 2021; Frankle et al., 2020; Jastrzebski et al., 2020) have also highlighted the importance of early training phase in centralized learning.

Original: The existence of CL periods in FL was recently discovered in (Yan et al., 2022).
Revised: The existence of CL periods in FL was recently discovered in (Yan et al., 2022).

Original: However, studying CL phenomena hinged on costly information metric (e.g., eigenvalues of the Hessian) that emerges after the full training, limiting their practical beneﬁts.
Revised: However, studying CL phenomena hinged on costly information metric (e.g., eigenvalues of the Hessian) that emerges after the full training, limiting their practical beneﬁts.

Original: We differ from existing works by developing an easy-to-compute metric to identify
Revised: We differ from existing works by developing an easy-to-compute metric to identify

Original: CL periods during the training process in an online manner.
Revised: CL periods during the training process in an online manner.

Original: Federated Learning and Client Selection.
Revised: Federated Learning and Client Selection.

Original: The state-of-the-art method for FL is FedAvg, which was ﬁrst proposed in (McMahan et al., 2017) and has sparked many follow-ups (Stich, 2018;
Revised: The state-of-the-art method for FL is FedAvg, which was ﬁrst proposed in (McMahan et al., 2017) and has sparked many follow-ups (Stich, 2018;

Original: Wang &
Revised: Wang &

Original: Joshi, 2021; Yu et al., 2019) with full client participation.
Revised: Joshi, 2021; Yu et al., 2019) with full client participation.

Original: In practice, only a small fraction of clients participate in each training round, which exacerbates the effect of data heterogeneity.
Revised: In practice, only a small fraction of clients participate in each training round, which exacerbates the effect of data heterogeneity.

Original: As a result, solutions with partial client participation and effect of data heterogeneity have been developed and analyzed (Katharopoulos & Fleuret, 2018;
Revised: As a result, solutions with partial client participation and effect of data heterogeneity have been developed and analyzed (Katharopoulos & Fleuret, 2018;

Original: Ruan et al., 2021; Karimireddy et al., 2020; Li et al., 2020a;
Revised: Ruan et al., 2021; Karimireddy et al., 2020; Li et al., 2020a;

Original: Wang et al., 2020c; Li et al., 2020b; Wang et al., 2020b; Ribero & Vikalo, 2020; Cho et al., 2020;
Revised: Wang et al., 2020c; Li et al., 2020b; Wang et al., 2020b; Ribero & Vikalo, 2020; Cho et al., 2020;

Original: Wang et al., 2020a; Yang et al., 2021; Cho et al., 2022; Reddi et al., 2021; Haddadpour & Mahdavi, 2019; Khaled et al., 2020; Stich & Karimireddy, 2020;
Revised: Wang et al., 2020a; Yang et al., 2021; Cho et al., 2022; Reddi et al., 2021; Haddadpour & Mahdavi, 2019; Khaled et al., 2020; Stich & Karimireddy, 2020;

Original: Woodworth et al., 2020; Horv´ath & Richtarik, 2021; Nishio & Yonetani, 2019; Malinovskiy et al., 2020; Pathak & Wainwright, 2020).
Revised: Woodworth et al., 2020; Horv´ath & Richtarik, 2021; Nishio & Yonetani, 2019; Malinovskiy et al., 2020; Pathak & Wainwright, 2020; Goetz et al., 2019; Tang et al., 2022).

Original: For a comprehensive introduction to FL and other algorithmic variants in FL, we refer interested readers to
Revised: For a comprehensive introduction to FL and other algorithmic variants Input M , η , E θ 0) , T do

Original: Server aggregates the θ ’s as θ ( t +1) :
Revised: Server aggregates the θ ’s as θ ( t +1) :

Original: = (cid:80) k ∈M  t )
Revised: = (cid:80) k ∈M ( t )

Original: N k (cid:80) k ∈M ( t ) N k θ ( t +1) k 7:  for  (Kairouz et al., 2019).
Revised: N k (cid:80) k ∈M ( t ) N k θ ( t +1) k 7: end for in FL, we refer interested readers to (Kairouz et al., 2019).

Original: Unlike these works that are agnostic to the existence of CL periods, we design a novel CL periods-aware FL framework.
Revised: Unlike these works that are agnostic to the existence of CL periods, we design a novel CL periods-aware FL framework.

Original: Importantly, we remark that our CL periods-aware FL framework, FedCL is orthogonal to these methods, sinceFedCL merely augments a state-of-the-art
Revised: Importantly, we remark that our CL periods-aware FL framework, FedCL is orthogonal to these methods, since

Original:  FL method to adaptively determine the number of clients that participate in each FL training round, rather than changing the way how the FL method selects clients.
Revised: FedCL merely augments a state-of-the-art FL method to adaptively determine the number of clients that participate in each FL training round, rather than changing the way how the FL method selects clients.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: The Federated Optimization Setting.
Revised: The Federated Optimization Setting.

Original: Consider the federated architecture where M clients jointly solve the optimization problem: min θ ∈ R d F ( θ ) :
Revised: Consider the federated architecture where M clients jointly solve the optimization problem: min θ ∈ R d F ( θ ) :

Original: = (cid:80) Mk =1 p k F k ( θ ) , where p k = N k /N represents the relative sample size, and F k ( θ ) =
Revised: = (cid:80) Mk =1 p k F k ( θ ) , where p k = N k /N represents the relative sample size, and F k ( θ ) =

Original: 1 N k (cid:80) ξ ∈D k
Revised: 1 N k (cid:80) ξ ∈D k

Original: (cid:96) k
Revised: (cid:96) k

Original: ( θ ; ξ ) is the local objective function at the k -th client.
Revised: ( θ ; ξ ) is the local objective function at the k -th client.

Original: Here (cid:96) k denotes the loss function deﬁned by the learning model, ξ represents a data sample from local dataset D k , and M denotes the set of clients.
Revised: Here (cid:96) k denotes the loss function deﬁned by the learning model, ξ represents a data sample from local dataset D k , and M denotes the set of clients.

Original: The Federated Averaging Algorithm.
Revised: The Federated Averaging Algorithm.

Original: Federated Averaging (FedAvg) (McMahan et al., 2017) is the ﬁrst and most common algorithm used to solve the above optimization problem through aggregating the locally trained models at the central server at the end of each communication round.
Revised: Federated Averaging (FedAvg) (McMahan et al., 2017) is the ﬁrst and most common algorithm used to solve the above optimization problem through aggregating the locally trained models at the central server at the end of each communication round.

Original: At the initial step, the central server randomly initializes a global model θ (0) .
Revised: At the initial step, the central server randomly initializes a global model θ (0) .

Original: At each round, a ﬁxed number of randomly selected clients run E iterations of local solver, e.g., the stochastic gradient descent (SGD) (Yu et al., 2019; Wang & Joshi, 2019; 2021), and then the resulting model updates are averaged.
Revised: At each round, a ﬁxed number of randomly selected clients run E iterations of local solver, e.g., the stochastic gradient descent (SGD) (Yu et al., 2019; Wang & Joshi, 2019; 2021), and then the resulting model updates are averaged.

Original: The details of FedAvg are summarized in Algorithm 1, where M ( t ) ⊆
Revised: The details of FedAvg are summarized in Algorithm 1, where M ( t ) ⊆

Original: M and m :=
Revised: M and m :=

Original: |M ( t ) | ≤ M , ∀ t .
Revised: |M ( t ) | ≤ M , ∀ t .

Original: Although the performance of FedAvg has been improved in both theory and practice by recent literature such as FedProx (Li et al., 2020a), FedNova (Wang et al., 2020c), SCAFFOLD (Karimireddy et al., 2020), VRL-SGD (Liang et al., 2019), FedBoost (Hamer et al., 2020), FedMA (Wang et al., 2020b) and FetchSGD (Rothchild et al., 2020), FedAvg is the ﬁrst and the most widely used one.
Revised: Although the performance of FedAvg has been improved in both theory and practice by recent literature such as FedProx (Li et al., 2020a), FedNova (Wang et al., 2020c), SCAFFOLD (Karimireddy et al., 2020), VRL-SGD (Liang et al., 2019), FedBoost (Hamer et al., 2020), FedMA (Wang et al., 2020b) and FetchSGD (Rothchild et al., 2020), FedAvg is the ﬁrst and the most widely used one.

Original: As a result, we see FedAvg as our basic block.
Revised: As a result, we see FedAvg as our basic block.

Original: Our proposed CL periods-aware FL framework FedCL is orthogonal to and can be easily combined with these methods (see Section 5).
Revised: Our proposed CL periods-aware FL framework FedCL is orthogonal to and can be easily combined with these methods (see Section 5).

Original: Moreover, FedCL is also compatible with and complementary to other techniques such as gradient compression/quantization (Basu et al., 2019; Haddadpour et al., 2021).
Revised: Moreover, FedCL is also compatible with and complementary to other techniques such as gradient compression/quantization (Basu et al., 2019; Haddadpour et al., 2021).

Original: CL Periods in FL.
Revised: CL Periods in FL.

Original: The latest series of studies have identiﬁed CL periods or the initial training phase that are important for training a high-quality model in both centralized training (Achille et al., 2019;
Revised: The latest series of studies have identiﬁed CL periods or the initial training phase that are important for training a high-quality model in both centralized training (Achille et al., 2019;

Original: Jastrzebski et al., 2019) and decentralized training (Yan et al., 2022).
Revised: Jastrzebski et al., 2019) and decentralized training (Yan et al., 2022).

Original: We build upon recent work by (Yan et al., 2022), who showed that if the training dataset for each client is not recovered to the entire training dataset early enough in the training process, the testing accuracy of FL is permanently impaired.
Revised: We build upon recent work by (Yan et al., 2022), who showed that if the training dataset for each client is not recovered to the entire training dataset early enough in the training process, the testing accuracy of FL is permanently impaired.

Original: We extend these ideas to aid in the design of adaptive client selection in FL and show that more clients are only required during these CL periods.
Revised: We extend these ideas to aid in the design of adaptive client selection in FL and show that more clients are only required during these CL periods.

Original:  F ED CL: A CL P ERIODS -A WARE C LIENT S ELECTION F RAMEWORK
Revised:  F ED CL: A CL P ERIODS -A WARE C LIENT S ELECTION F RAMEWORK

Original: This section describes our proposed approach to efﬁciently detect CL periods in federated settings that lay out the rationale behind our method.
Revised: This section describes our proposed approach to efﬁciently detect CL periods in federated settings that lay out the rationale behind our method.

Original: The rest of this section focuses on our proposed framework
Revised: The rest of this section focuses on our proposed framework

Original: FedCL that augments client selection in state-of-the-art methods with CL periods.
Revised: FedCL that augments client selection in state-of-the-art methods with CL periods.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: F
Revised: F

Original: [Figure]
Revised: [Figure]

Original: 1e−
Revised: 1e−

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: (Yan et al., 2022) showed that the ﬁnal test accuracy of FL is dramatically affected by early training phases.
Revised: (Yan et al., 2022) showed that the ﬁnal test accuracy of FL is dramatically affected by early training phases.

Original: They setup experiments where only partial datasets are available for the ﬁrst few communication rounds and then continue training the model with entire training datasets for the rest of communication rounds.
Revised: They setup experiments where only partial datasets are available for the ﬁrst few communication rounds and then continue training the model with entire training datasets for the rest of communication rounds.

Original: Surprisingly, the FL model trained in this way showed a permanent impaired test accuracy performance no matter how many additional training rounds are performed after CL periods.
Revised: Surprisingly, the FL model trained in this way showed a permanent impaired test accuracy performance no matter how many additional training rounds are performed after CL periods.

Original: We extend these ideas to aid in the design of adaptive client selection in FL since it is a major trend of improving FL efﬁciency and handling heterogeneity.
Revised: We extend these ideas to aid in the design of adaptive client selection in FL since it is a major trend of improving FL efﬁciency and handling heterogeneity.

Original: As motivated by aforementioned works, it is clear that ﬁnding an adaptive client selection scheme is akin to ﬁnding CL periods in FL training process.
Revised: As motivated by aforementioned works, it is clear that ﬁnding an adaptive client selection scheme is akin to ﬁnding CL periods in FL training process.

Original: To this end, we begin with how to efﬁciently detect CL periods in FL training process.
Revised: To this end, we begin with how to efﬁciently detect CL periods in FL training process.

Original: Detecting Critical Learning Periods.
Revised: Detecting Critical Learning Periods.

Original: Prior works use the changes in eigenvalues of the Hessian or approximating the Hessian using (federated) Fisher information (Achille et al., 2019; Jastrzebski et al., 2019; Yan et al., 2022) as an indicator to detect CL periods.
Revised: Prior works use the changes in eigenvalues of the Hessian or approximating the Hessian using (federated) Fisher information (Achille et al., 2019; Jastrzebski et al., 2019; Yan et al., 2022) as an indicator to detect CL periods.

Original: We deviate from these works and develop an approach based on federated gradient norm (FGN), which can be efﬁciently computed.
Revised: We deviate from these works and develop an approach based on federated gradient norm (FGN), which can be efﬁciently computed.

Original: Considering the difference in training loss for an individual data sample ξ , let g ( θ ; ξ ) =
Revised: Considering the difference in training loss for an individual data sample ξ , let g ( θ ; ξ ) =

Original: ∂∂θ (cid:96) ( θ ; ξ ) denote the gradient of the loss function evaluated on ξ .
Revised: ∂∂θ (cid:96) ( θ ; ξ ) denote the gradient of the loss function evaluated on ξ .

Original: After performing a step SGD on this sample, the training loss ∆ (cid:96) =
Revised: After performing a step SGD on this sample, the training loss ∆ (cid:96) =

Original: (cid:96) ( θ − ηg ( θ ; ξ ); ξ ) − (cid:96) ( θ ; ξ ) can be approximated by its gradient norm using
Revised: (cid:96) ( θ − ηg ( θ ; ξ ); ξ ) − (cid:96) ( θ ; ξ ) can be approximated by its gradient norm using

Original: Taylor expansion, i.e., ∆ (cid:96) ≈ − η (cid:107) g
Revised: Taylor expansion, i.e., ∆ (cid:96) ≈ − η (cid:107) g

Original: ( θ ; ξ ) (cid:107) 2 .
Revised: ( θ ; ξ ) (cid:107) 2 .

Original: As a result, the overall training loss at the t -th round, which we deﬁne as the FGN, can be approximated using the weighted average of training loss across all selected clients, i.e.,
Revised: As a result, the overall training loss at the t -th round, which we deﬁne as the FGN, can be approximated using the weighted average of training loss across all selected clients, i.e.,

Original: [Equation]
Revised: [Equation]

Original: We compare the CL periods identiﬁed by our FGN approach with the federated Fisher information (FedFIM) approach in (Yan et al., 2022).
Revised: We compare the CL periods identiﬁed by our FGN approach with the federated Fisher information (FedFIM) approach in (Yan et al., 2022).

Original: From Figure 1, we observe that these two approaches yield similar results, but our FGN approach is much more computationally efﬁcient (being orders of magnitude faster to compute) and can be easily leveraged for client selection during the training process in an online manner.
Revised: From Figure 1, we observe that these two approaches yield similar results, but our FGN approach is much more computationally efﬁcient (being orders of magnitude faster to compute) and can be easily leveraged for client selection during the training process in an online manner.

Original: For example, the FedFIM approach takes up to 9 × more computation time and consumes 40 × more memory than our FGN approach (See Appendix A.2.1 for details).
Revised: For example, the FedFIM approach takes up to 9 × more computation time and consumes 40 × more memory than our FGN approach (See Appendix A.2.1 for details).

Original: 4. T HE D ESIGN OF F ED CL F RAMEWORK
Revised: 4. T HE D ESIGN OF F ED CL F RAMEWORK

Original: We now describe FedCL, our proposed framework that adaptively determines the number of selected clients for FL training by leveraging identiﬁed CL periods.
Revised: We now describe FedCL, our proposed framework that adaptively determines the number of selected clients for FL training by leveraging identiﬁed CL periods.

Original: Again, we use FedAvg as the building block, and our framework can be easily combined with other existing methods, which we will illustrate in Section 5.
Revised: Again, we use FedAvg as the building block, and our framework can be easily combined with other existing methods, which we will illustrate in Section 5.

Original: Our FedCL builds on the identiﬁed CL periods, which can be efﬁciently detected by
Revised: Our FedCL builds on the identiﬁed CL periods, which can be efﬁciently detected by

Original: FGN as discussed previously (see Figure 1).
Revised: FGN as discussed previously (see Figure 1).

Original: To this end, we develop a simple threshold-based rule to detect the CL periods based on FGN as follows: if
Revised: To this end, we develop a simple threshold-based rule to detect the CL periods based on FGN as follows: if

Original: [Equation]
Revised: [Equation]

Original: then the current round t is in CL periods, where δ is the threshold used to declare CL periods.
Revised: then the current round t is in CL periods, where δ is the threshold used to declare CL periods.

Original: We set δ = 0 .
Revised: We set δ = 0 .

Original: 01 as the default value in our experiments and will investigate its impact in Section 5.
Revised: 01 as the default value in our experiments and will investigate its impact in Section 5.

Original: Per our discussions on CL periods, the ﬁnal model accuracy will be permanently impaired if not enough clients are involved in CL periods no matter how much additional training is performed after the period (Yan et al., 2022).
Revised: Per our discussions on CL periods, the ﬁnal model accuracy will be permanently impaired if not enough clients are involved in CL periods no matter how much additional training is performed after the period (Yan et al., 2022).

Original: Therefore, our FedCL framework increases the number of selected clients of FedAvg from n 0 to 2 n 0 , implying that more clients now participate in improving the global model in the next round during the CL periods.
Revised: Therefore, our FedCL framework increases the number of selected clients of FedAvg from n 0 to 2 n 0 , implying that more clients now participate in improving the global model in the next round during the CL periods.

Original: Using the model learned from the previous round θ n 0 as the initial model, the 2 n 0 selected clients employ the FedAvg and continue the learning procedure to reach a global model θ 2 n 0 .
Revised: Using the model learned from the previous round θ n 0 as the initial model, the 2 n 0 selected clients employ the FedAvg and continue the learning procedure to reach a global model θ 2 n 0 .

Original: The procedure of geometrically increasing the number of selected clients continues till the set of selected clients contains all the available M clients when the communication rounds are still in CL periods.
Revised: The procedure of geometrically increasing the number of selected clients continues till the set of selected clients contains all the available M clients when the communication rounds are still in CL periods.

Original: Since the ﬁnal accuracy of using partial datasets is similar to that of using all dataset after CL periods (Yan et al., 2022), FedCL starts to gradually decrease the number of selected clients after CL periods for the sake of communication efﬁciency.
Revised: Since the ﬁnal accuracy of using partial datasets is similar to that of using all dataset after CL periods (Yan et al., 2022), FedCL starts to gradually decrease the number of selected clients after CL periods for the sake of communication efﬁciency.

Original: Algorithm 2 summarizes the CL periods-aware FedAvg algorithm, dubbed as FedCL.
Revised: Algorithm 2 summarizes the CL periods-aware FedAvg algorithm, dubbed as FedCL.

Original: From a high-level perspective, FedCL exploits more clients in the initial phase of the learning procedure than a ﬁxed number of clients for FedAvg in each round, to promptly reach a global model with higher accuracy since the initial learning phase plays a critical role in FL performance.
Revised: From a high-level perspective, FedCL exploits more clients in the initial phase of the learning procedure than a ﬁxed number of clients for FedAvg in each round, to promptly reach a global model with higher accuracy since the initial learning phase plays a critical role in FL performance.

Original: By doing so, we hypothesize that the SGD is navigating to the steeper parts of the loss surface of the global model during CL periods since a larger amount of data samples have contributed to the global model.
Revised: By doing so, we hypothesize that the SGD is navigating to the steeper parts of the loss surface of the global model during CL periods since a larger amount of data samples have contributed to the global model.

Original: However, the communication overhead of such an approach is relatively large since more clients are involved in FL training in each communication round.
Revised: However, the communication overhead of such an approach is relatively large since more clients are involved in FL training in each communication round.

Original: By gradually decreasing the number of selected clients after CL periods, the communication overhead of FedCL improves without hurting the ﬁnal model accuracy.
Revised: By gradually decreasing the number of selected clients after CL periods, the communication overhead of FedCL improves without hurting the ﬁnal model accuracy.

Original: The key point is that more clients join the training process in the initial learning phase, and only a smaller number of clients are needed after CL periods.
Revised: The key point is that more clients join the training process in the initial learning phase, and only a smaller number of clients are needed after CL periods.

Original: As a result, FedCL consistently improves the model accuracy while maintains comparable or even better communication efﬁciency than FedAvg.
Revised: As a result, FedCL consistently improves the model accuracy while maintains comparable or even better communication efﬁciency than FedAvg.

Original: As the proposed FedCL in Algorithm 2 provides a general framework to augment client selection with identiﬁed CL periods in federated settings, one needs to specify the inner optimization subroutine (e.g., line 4 in Algorithm 2) to quantify the improvement of the proposed approach.
Revised: As the proposed FedCL in Algorithm 2 provides a general framework to augment client selection with identiﬁed CL periods in federated settings, one needs to specify the inner optimization subroutine (e.g., line 4 in Algorithm 2) to quantify the improvement of the proposed approach.

Original: In particular, we set the subroutine to be FedAvg in Algorithm 2 since it is the most common algorithm and the building block of many variants in federated settings.
Revised: In particular, we set the subroutine to be FedAvg in Algorithm 2 since it is the most common algorithm and the building block of many variants in federated settings.

Original: This subroutine could be any federated learning algorithms (with possible variants), such as FedProx (Li et al., 2020a), VRL-SGD (Liang et al., 2019) and
Revised: This subroutine could be any federated learning algorithms (with possible variants), such as FedProx (Li et al., 2020a), VRL-SGD (Liang et al., 2019) and

Original: FedNova (Wang et al., 2020c), which we will numerically illustrate in Section 5.
Revised: FedNova (Wang et al., 2020c), which we will numerically illustrate in Section 5.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We ﬁrst present an empirical study of FedCL in Section 5.2, and then illustrate the generalization of our proposed framework by combining CL periods with other methods such as FedProx, VRLSGD and FedNova in Section 5.3.
Revised: We ﬁrst present an empirical study of FedCL in Section 5.2, and then illustrate the generalization of our proposed framework by combining CL periods with other methods such as FedProx, VRLSGD and FedNova in Section 5.3.

Original: We study classiﬁcation problems using two representative DNN models: AlexNet (Krizhevsky et al., 2012) and VGG-11 (Simonyan & Zisserman, 2015) on non-IID partitioned CIFAR-10 and CIFAR-100 (Krizhevsky et al., 2009), and
Revised: We study classiﬁcation problems using two representative DNN models: AlexNet (Krizhevsky et al., 2012) and VGG-11 (Simonyan & Zisserman, 2015) on non-IID partitioned CIFAR-10 and CIFAR-100 (Krizhevsky et al., 2009), and

Original: Fashion-MNIST (Xiao et al., 2017) datasets.
Revised: Fashion-MNIST (Xiao et al., 2017) datasets.

Original: We further investigate the task of next-character prediction on the dataset of The
Revised: We further investigate the task of next-character prediction on the dataset of The

Original: Complete Works of William Shakespeare (Shakespeare) (McMahan et al., 2017).
Revised: Complete Works of William Shakespeare (Shakespeare) (McMahan et al., 2017).

Original: We relegate details of datasets and models, and additional experimental results, particularly on CIFAR-100 and Shakespeare, to Appendix A.
Revised: We relegate details of datasets and models, and additional experimental results, particularly on CIFAR-100 and Shakespeare, to Appendix A.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: on non-IID CIFAR-
Revised: on non-IID CIFAR-

Original: [Figure]
Revised: [Figure]

Original: 5. E XPERIMENT S ETUP
Revised: 5. E XPERIMENT S ETUP

Original: We implement FedCL and considered baselines in PyTorch (Paszke et al., 2017) on Python 3 with three NVIDIA RTX A6000 GPUs.
Revised: We implement FedCL and considered baselines in PyTorch (Paszke et al., 2017) on Python 3 with three NVIDIA RTX A6000 GPUs.

Original: For CIFAR-10, CIFAR-100 and Fashion-MNIST datasets, we simulate the non-IID FL scenario by considering a heterogeneous partition for which the number of data points and class proportions are unbalanced.
Revised: For CIFAR-10, CIFAR-100 and Fashion-MNIST datasets, we simulate the non-IID FL scenario by considering a heterogeneous partition for which the number of data points and class proportions are unbalanced.

Original: In particular, we simulate a heterogeneous partition into M clients by sampling p k ∼ Dir M ( α ) , where α is the parameter of the Dirichlet distribution.
Revised: In particular, we simulate a heterogeneous partition into M clients by sampling p k ∼ Dir M ( α ) , where α is the parameter of the Dirichlet distribution.

Original: We choose α = 0 .
Revised: We choose α = 0 .

Original: 3 in our experiments as done in (Wang et al.
Revised: 3 in our experiments as done in (Wang et al.

Original: , 2020b;c).
Revised: , 2020b;c).

Original: The level of heterogeneity among local datasets across different clients can be reduced when α increases.
Revised: The level of heterogeneity among local datasets across different clients can be reduced when α increases.

Original: We consider the total number of clients to be 128.
Revised: We consider the total number of clients to be 128.

Original: The local learning rate η is initialized as 0 . 01 and decayed by a constant factor after each communication round.
Revised: The local learning rate η is initialized as 0 . 01 and decayed by a constant factor after each communication round.

Original: We set the weight decay to be 10 − 5 .
Revised: We set the weight decay to be 10 − 5 .

Original: The detection threshold is δ = 0 .
Revised: The detection threshold is δ = 0 .

Original: 01 and the number of local training epochs is E = 2 .
Revised: 01 and the number of local training epochs is E = 2 .

Original: An ablation study is conducted in Section 5.4 to investigate the impact of these hyperparameters.
Revised: An ablation study is conducted in Section 5.4 to investigate the impact of these hyperparameters.

Original: We run each experiment with three independent trials and report the mean results.
Revised: We run each experiment with three independent trials and report the mean results.

Original: 5. I MPORTANCE OF CL P ERIODS A WARENESS : F ED CL v.s. F ED A VG
Revised: 5. I MPORTANCE OF CL P ERIODS A WARENESS : F ED CL v.s. F ED A VG

Original: In this experiment, we study the performance of FedCL with accuracy and communication efﬁciency
Revised: In this experiment, we study the performance of FedCL with accuracy and communication efﬁciency

Original: Our goal is to compare FedCL to FedAvg in terms of the ﬁnal test accuracy and the number of communication rounds needed for the global model to achieve good performance on the test data.
Revised: Our goal is to compare FedCL to FedAvg in terms of the ﬁnal test accuracy and the number of communication rounds needed for the global model to achieve good performance on the test data.

Original: Test Accuracy.
Revised: Test Accuracy.

Original: The test accuracy comparisons on non-IID partitioned CIFAR-10 and FashionMNIST with FedAvg selecting 16 clients in each round are shown in Figures 2 and 3, respectively.
Revised: The test accuracy comparisons on non-IID partitioned CIFAR-10 and FashionMNIST with FedAvg selecting 16 clients in each round are shown in Figures 2 and 3, respectively.

Original: Obviously, FedCL consistently outperforms FedAvg in all scenarios with an improved ﬁnal test accuracy up to 11%.
Revised: Obviously, FedCL consistently outperforms FedAvg in all scenarios with an improved ﬁnal test accuracy up to 11%.

Original: Its advantage is especially pronounced when the dataset is partitioned across clients using a Dirichlet distribution with parameter 0.1, i.e., the datasets across clients are highly non-IID.
Revised: Its advantage is especially pronounced when the dataset is partitioned across clients using a Dirichlet distribution with parameter 0.1, i.e., the datasets across clients are highly non-IID.

Original: Not surprisingly, we observe the importance of CL periods awareness in training efﬁciency, which is fully reﬂected via the test accuracy.
Revised: Not surprisingly, we observe the importance of CL periods awareness in training efﬁciency, which is fully reﬂected via the test accuracy.

Original: For example in Figures 2(a) and 3(a), FedCL exhibits a dramatic accuracy increase in the early phase of the FL training process.
Revised: For example in Figures 2(a) and 3(a), FedCL exhibits a dramatic accuracy increase in the early phase of the FL training process.

Original: This coincides with the fact that FedCL selects a larger number of clients in each round in the early phase due to the awareness of CL periods (lines 3-6 in Algorithm 2).
Revised: This coincides with the fact that FedCL selects a larger number of clients in each round in the early phase due to the awareness of CL periods (lines 3-6 in Algorithm 2).

Original: Though the accuracy slightly decreases in a short period due to the decreased number of selected clients (lines 7-9 in Algorithm 2), the ﬁnal test accuracy is signiﬁcantly improved.
Revised: Though the accuracy slightly decreases in a short period due to the decreased number of selected clients (lines 7-9 in Algorithm 2), the ﬁnal test accuracy is signiﬁcantly improved.

Original: Our ﬁndings on the importance of CL periods awareness in the FL training
Revised: Our ﬁndings on the importance of CL periods awareness in the FL training

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: process, e.g., for client selection, seem to be consistent with recently reported observations that the initial learning phase plays a key role in determining the outcome of the training process.
Revised: process, e.g., for client selection, seem to be consistent with recently reported observations that the initial learning phase plays a key role in determining the outcome of the training process.

Original: Communication Efﬁciency.
Revised: Communication Efﬁciency.

Original: The beneﬁt of CL periods awareness for FL training is further reﬂected via communication efﬁciency.
Revised: The beneﬁt of CL periods awareness for FL training is further reﬂected via communication efﬁciency.

Original: In Figures 4 and 5, we report the communication rounds required for
Revised: In Figures 4 and 5, we report the communication rounds required for

Original: FedCL and FedAvg to achieve a targeted accuracy, which is chosen to be the ﬁnal test accuracy of
Revised: FedCL and FedAvg to achieve a targeted accuracy, which is chosen to be the ﬁnal test accuracy of

Original: FedAvg as reported in Figures 2 and 3, respectively.
Revised: FedAvg as reported in Figures 2 and 3, respectively.

Original: Comparisons on other targeted test accuracy can be found in Appendix A.2.
Revised: Comparisons on other targeted test accuracy can be found in Appendix A.2.

Original: It is clear that FedCL requires fewer rounds to achieve the same test accuracy.
Revised: It is clear that FedCL requires fewer rounds to achieve the same test accuracy.

Original: Again this advantage is pronounced on highly non-IID data partitions.
Revised: Again this advantage is pronounced on highly non-IID data partitions.

Original: We further compute the average number of clients involved in each round for FedCL and FedAvg to achieve its ﬁnal test accuracy.
Revised: We further compute the average number of clients involved in each round for FedCL and FedAvg to achieve its ﬁnal test accuracy.

Original: We observe that FedCL consistently improves the model accuracy (as shown Figuresand 3) with its average number of clients involved in each round being 0.95 × to 1.1 × that of FedAvg.
Revised: We observe that FedCL consistently improves the model accuracy (as shown Figuresand 3) with its average number of clients involved in each round being 0.95 × to 1.1 × that of FedAvg.

Original: This is due to the fact that more clients are only needed during CL periods and a smaller number of clients are need afterwards in our FedCL framework.
Revised: This is due to the fact that more clients are only needed during CL periods and a smaller number of clients are need afterwards in our FedCL framework.

Original: 5. G ENERALIZATION
Revised: 5. G ENERALIZATION

Original: As mentioned earlier, our proposed CL periods-aware FL framework, FedCL is orthogonal to existing state-of-the-art methods, and hence can be easily combined with these methods by simply replacing the inner optimization subroutine (FedAvg) in Algorithm 2 with the corresponding methods.
Revised: As mentioned earlier, our proposed CL periods-aware FL framework, FedCL is orthogonal to existing state-of-the-art methods, and hence can be easily combined with these methods by simply replacing the inner optimization subroutine (FedAvg) in Algorithm 2 with the corresponding methods.

Original: To this end, we study the generalization of FedCL and consider three state of the arts, i.e., FedProx (Li et al., 2020a),
Revised: To this end, we study the generalization of FedCL and consider three state of the arts, i.e., FedProx (Li et al., 2020a),

Original: VRL-SGD (Liang et al., 2019) and FedNova (Wang et al., 2020c).
Revised: VRL-SGD (Liang et al., 2019) and FedNova (Wang et al., 2020c).

Original: We call the corresponding
Revised: We call the corresponding

Original: CL periods-aware methods as FedProx-CL, VRL-SGD-CL, and FedNova-CL, respectively.
Revised: CL periods-aware methods as FedProx-CL, VRL-SGD-CL, and FedNova-CL, respectively.

Original: We notice that the performance of FedProx depends on the hyperparameter µ , i.e., the coefﬁcient associated with the proximal term of each local objective.
Revised: We notice that the performance of FedProx depends on the hyperparameter µ , i.e., the coefﬁcient associated with the proximal term of each local objective.

Original: We tune this parameter using grid search and report the best value of µ = 0 .
Revised: We tune this parameter using grid search and report the best value of µ = 0 .

Original: 01 for AlexNet experiments and µ = 0 .
Revised: 01 for AlexNet experiments and µ = 0 .

Original: 001 for VGGexperiments.
Revised: 001 for VGGexperiments.

Original: We present results on non-IID datasets across 128 clients with FedProx, VRL-SGD and FedNova selecting 16 clients in each iteration.
Revised: We present results on non-IID datasets across 128 clients with FedProx, VRL-SGD and FedNova selecting 16 clients in each iteration.

Original: Again, the CL periods awareness signiﬁcantly improves the test performance of baseline methods, i.e., FedProx-CL,VRL-SGD-CL and FedNova-CL outperforms FedProx, VRL-SGD and FedNova, respectively, in all scenarios with an improved ﬁnal test accuracy up to 11%, 13% and 10%, respectively, as shown in Figures 6 and 7.
Revised: Again, the CL periods awareness signiﬁcantly improves the test performance of baseline methods, i.e., FedProx-CL,VRL-SGD-CL and FedNova-CL outperforms FedProx, VRL-SGD and FedNova, respectively, in all scenarios with an improved ﬁnal test accuracy up to 11%, 13% and 10%, respectively, as shown in Figures 6 and 7.

Original: Its advantage is
Revised: Its advantage is

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: especially pronounced on highly non-IID dataset across clients in Figures 6(a) and 7(a).
Revised: especially pronounced on highly non-IID dataset across clients in Figures 6(a) and 7(a).

Original: Likewise, the CL periods augmented method, e.g., FedProx-CL requires fewer rounds to achieve the ﬁnal test accuracy of the corresponding baseline FedProx, as shown in Figures 8 and 9, while maintaining a comparable average number of clients involved in each round.
Revised: Likewise, the CL periods augmented method, e.g., FedProx-CL requires fewer rounds to achieve the ﬁnal test accuracy of the corresponding baseline FedProx, as shown in Figures 8 and 9, while maintaining a comparable average number of clients involved in each round.

Original: Similar observations can be made for
Revised: Similar observations can be made for

Original: VRL-SGD-CL vs. VRL-SGD and FedNova-CL vs. FedNova, which can be found in Appendix A.2.
Revised: VRL-SGD-CL vs. VRL-SGD and FedNova-CL vs. FedNova, which can be found in Appendix A.2.

Original: A BLATION S TUDY
Revised: A BLATION S TUDY

Original: Detection Thresholds.
Revised: Detection Thresholds.

Original: As discussed previously in Figure 1, our experiments reveal that CL periods can be efﬁciently identiﬁed using the easy-to-compute FGN via a simple threshold-type rule in
Revised: As discussed previously in Figure 1, our experiments reveal that CL periods can be efﬁciently identiﬁed using the easy-to-compute FGN via a simple threshold-type rule in

Original: Equation (2).
Revised: Equation (2).

Original: We now evaluate the sensitivity of the threshold value δ used to declare CL periods.
Revised: We now evaluate the sensitivity of the threshold value δ used to declare CL periods.

Original: The candidate threshold values we consider are { 0 , 0 . 01 , 0 .
Revised: The candidate threshold values we consider are { 0 , 0 . 01 , 0 .

Original: 35 , 0 . 5 } , and the corresponding ﬁnal test accuracy of FedCL using AlexNet on non-IID partitioned CIFAR-10 and Fashion-MNIST is illustrated in Figure 10.
Revised: 35 , 0 . 5 } , and the corresponding ﬁnal test accuracy of FedCL using AlexNet on non-IID partitioned CIFAR-10 and Fashion-MNIST is illustrated in Figure 10.

Original: When data partitions are highly non-IID (i.e., α = 0 . 1 ), the CL periods declaration determined by δ has an observable effect on the ﬁnal accuracy.
Revised: When data partitions are highly non-IID (i.e., α = 0 . 1 ), the CL periods declaration determined by δ has an observable effect on the ﬁnal accuracy.

Original: This is because as δ becomes larger, fewer rounds in the initial phase are declared as CL periods by Equation (2).
Revised: This is because as δ becomes larger, fewer rounds in the initial phase are declared as CL periods by Equation (2).

Original: As a result, the effect of CL periods awareness on the ﬁnal test accuracy is shallowed since FedCL only uses a larger number of clients in fewer rounds compared to FedAvg according to Algorithm 2.
Revised: As a result, the effect of CL periods awareness on the ﬁnal test accuracy is shallowed since FedCL only uses a larger number of clients in fewer rounds compared to FedAvg according to Algorithm 2.

Original: On the other hand, our CL periods-aware framework FedCL is robust to the detection process, i.e., tolerant to detection errors with different threshold values when data partitions are not highly non-IID.
Revised: On the other hand, our CL periods-aware framework FedCL is robust to the detection process, i.e., tolerant to detection errors with different threshold values when data partitions are not highly non-IID.

Original: Similar observations can be made for FedProx-CL,VRL-SGD-CL and FedNova-CL and hence are relegated to Appendix A.2.
Revised: Similar observations can be made for FedProx-CL,VRL-SGD-CL and FedNova-CL and hence are relegated to Appendix A.2.

Original: To this end, we set δ = 0 .
Revised: To this end, we set δ = 0 .

Original: 01 in all of our experiments.
Revised: 01 in all of our experiments.

Original: Non-IID Degree.
Revised: Non-IID Degree.

Original: We simulate a heterogeneous data partition into M clients using the Dirichlet distribution with parameter α .
Revised: We simulate a heterogeneous data partition into M clients using the Dirichlet distribution with parameter α .

Original: From Figure 11, we observe that the CL periods awareness consistently improves the ﬁnal test accuracy of a state-of-the-art method across all values of α in consideration.
Revised: From Figure 11, we observe that the CL periods awareness consistently improves the ﬁnal test accuracy of a state-of-the-art method across all values of α in consideration.

Original: For example, FedCL always outperforms FedAvg, and FedProx-CL always outperforms FedProx.
Revised: For example, FedCL always outperforms FedAvg, and FedProx-CL always outperforms FedProx.

Original: The beneﬁts of CL periods awareness are especially pronounced when the datasets across clients are highly non-IID (i.e., a smaller value of α ).
Revised: The beneﬁts of CL periods awareness are especially pronounced when the datasets across clients are highly non-IID (i.e., a smaller value of α ).

Original: Hence, we choose α = 0 .
Revised: Hence, we choose α = 0 .

Original: 3 for illustrations in above experiments.
Revised: 3 for illustrations in above experiments.

Original: For ease of readability, we set α = 0 .
Revised: For ease of readability, we set α = 0 .

Original: 1 in the rest of ablation studies and relegate results on α = 0 .
Revised: 1 in the rest of ablation studies and relegate results on α = 0 .

Original: 3 to Appendix A.2.
Revised: 3 to Appendix A.2.

Original: In addition, as the non-IID degree decreases (as α increases), the ﬁnal test accuracy of FedCL, FedProx-CL, VRL-SGD-CL and FedNova-CL increases.
Revised: In addition, as the non-IID degree decreases (as α increases), the ﬁnal test accuracy of FedCL, FedProx-CL, VRL-SGD-CL and FedNova-CL increases.

Original: This is consistent with recently reported observations, e.g., in (Lin et al., 2020; Achituve et al., 2021;
Revised: This is consistent with recently reported observations, e.g., in (Lin et al., 2020; Achituve et al., 2021;

Original: Gong et al., 2021) that non-IID degree degrades the model ﬁnal test accuracy.
Revised: Gong et al., 2021) that non-IID degree degrades the model ﬁnal test accuracy.

Original: Local Training Epochs.
Revised: Local Training Epochs.

Original: We note that the number of local training epochs (denoted E ) is a common parameter shared by considered baselines, which reportedly has an impact on the performance of
Revised: We note that the number of local training epochs (denoted E ) is a common parameter shared by considered baselines, which reportedly has an impact on the performance of

Original: FedAvg (McMahan et al., 2017; Wang et al., 2020b).
Revised: FedAvg (McMahan et al., 2017; Wang et al., 2020b).

Original: To this end, we evaluate the effect of E using
Revised: To this end, we evaluate the effect of E using

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: AlexNet on non-IID partitioned CIFAR-10 and Fashion-MNIST with α = 0 .
Revised: AlexNet on non-IID partitioned CIFAR-10 and Fashion-MNIST with α = 0 .

Original: The candidate local epochs we consider are E ∈ { 1 , 2 , 3 , 4 , 5 } as done in (Wang et al., 2020c).
Revised: The candidate local epochs we consider are E ∈ { 1 , 2 , 3 , 4 , 5 } as done in (Wang et al., 2020c).

Original: From Figure 12, we observe that increasing the number of local epochs improves the ﬁnal test accuracy in general, and the CL periods awareness consistently improves the ﬁnal test accuracy of state-of-the-art methods across all values of E .
Revised: From Figure 12, we observe that increasing the number of local epochs improves the ﬁnal test accuracy in general, and the CL periods awareness consistently improves the ﬁnal test accuracy of state-of-the-art methods across all values of E .

Original: Since the gains in test accuracy exhibit the “diminishing return effect” as the number of local epochs increases, we set E = 2 in all of our experiments.
Revised: Since the gains in test accuracy exhibit the “diminishing return effect” as the number of local epochs increases, we set E = 2 in all of our experiments.

Original: Weight Decay.
Revised: Weight Decay.

Original: Though the CL periods in FL are robust to the values of weight decays as reported in (Yan et al., 2022), the ﬁnal test accuracy using AlexNet on non-IID partitioned CIFAR-10 and
Revised: Though the CL periods in FL are robust to the values of weight decays as reported in (Yan et al., 2022), the ﬁnal test accuracy using AlexNet on non-IID partitioned CIFAR-10 and

Original: Fashion-MNIST with α = 0 .
Revised: Fashion-MNIST with α = 0 .

Original: 1 is still affected with the values of weight decays as shown in Figure 13.
Revised: 1 is still affected with the values of weight decays as shown in Figure 13.

Original: Again, we consistently observe the beneﬁts of CL periods awareness across all values of weight decays.
Revised: Again, we consistently observe the beneﬁts of CL periods awareness across all values of weight decays.

Original: Since the advantage decreases as the weight decay increases, we set the weight decay to be 10 − 5 in our experiments.
Revised: Since the advantage decreases as the weight decay increases, we set the weight decay to be 10 − 5 in our experiments.

Original: Number of Clients.
Revised: Number of Clients.

Original: In all of our above experiments, we consider a FL setting with 128 clients in total.
Revised: In all of our above experiments, we consider a FL setting with 128 clients in total.

Original: We now consider the same experimental settings as above besides varying the total number of clients in the system using AlexNet on non-IID partitioned CIFAR-10 and Fashion-MNIST with α = 0 .
Revised: We now consider the same experimental settings as above besides varying the total number of clients in the system using AlexNet on non-IID partitioned CIFAR-10 and Fashion-MNIST with α = 0 .

Original: As shown in Figure 14, the advantage of CL periods awareness exhibits across all settings, i.e., FedCL (resp.
Revised: As shown in Figure 14, the advantage of CL periods awareness exhibits across all settings, i.e., FedCL (resp.

Original: FedProx-CL) outperforms FedAvg (resp.
Revised: FedProx-CL) outperforms FedAvg (resp.

Original: FedProx)
Revised: FedProx)

Original: regardless of the total number of clients.
Revised: regardless of the total number of clients.

Original: Without loss of generality, we choose M = 128 in all of our experiments.
Revised: Without loss of generality, we choose M = 128 in all of our experiments.

Original: Client Participation Rate.
Revised: Client Participation Rate.

Original: In all of our experiments, FedAvg, FedProx, VRL-SGD and FedNova select 16 out of 128 clients to participate in each training round, i.e., the participation rate is 12.5%.
Revised: In all of our experiments, FedAvg, FedProx, VRL-SGD and FedNova select 16 out of 128 clients to participate in each training round, i.e., the participation rate is 12.5%.

Original: We now investigate the impact of client participation rates on the model accuracy and the awareness of CL periods using AlexNet on non-IID partitioned CIFAR-10 and Fashion-MNIST with α = 0 . 1 .
Revised: We now investigate the impact of client participation rates on the model accuracy and the awareness of CL periods using AlexNet on non-IID partitioned CIFAR-10 and Fashion-MNIST with α = 0 . 1 .

Original: Again, when a state-of-the-art method is augmented with the CL periods, the ﬁnal test accuracy is consistently improved across all client participation rates.
Revised: Again, when a state-of-the-art method is augmented with the CL periods, the ﬁnal test accuracy is consistently improved across all client participation rates.

Original: The advantage is particularly pronounced with a low participation rate.
Revised: The advantage is particularly pronounced with a low participation rate.

Original: This is quite intuitive since in our CL periods aware framework, FedCL selects more clients during the CL periods than FedAvg (see line 5 in Algorithm 2), and hence the beneﬁts are more obvious when FedAvg has a low client participation rate.
Revised: This is quite intuitive since in our CL periods aware framework, FedCL selects more clients during the CL periods than FedAvg (see line 5 in Algorithm 2), and hence the beneﬁts are more obvious when FedAvg has a low client participation rate.

Original: We select 16 clients, i.e., a 12.5% participation rate for all state-of-the-art methods via considering the tradeoff between ﬁnal test accuracy and beneﬁts of CL periods awareness.
Revised: We select 16 clients, i.e., a 12.5% participation rate for all state-of-the-art methods via considering the tradeoff between ﬁnal test accuracy and beneﬁts of CL periods awareness.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we presented FedCL, a simple but powerful CL periods-aware FL framework for adaptive client selection.
Revised: In this paper, we presented FedCL, a simple but powerful CL periods-aware FL framework for adaptive client selection.

Original: FedCL worked by adaptively choosing more clients in CL periods during the FL training process and fewer clients elsewhere.
Revised: FedCL worked by adaptively choosing more clients in CL periods during the FL training process and fewer clients elsewhere.

Original: We proposed a practical and easy-to-compute federated gradient norm (FGN) metric to identify such CL periods during the training process in an online manner.
Revised: We proposed a practical and easy-to-compute federated gradient norm (FGN) metric to identify such CL periods during the training process in an online manner.

Original: We showed that FedCL signiﬁcantly improved the ﬁnal test accuracy by up to 11% compared to its counterpart FedAvg using different models and datasets, while maintaining comparable or even better communication efﬁciency.
Revised: We showed that FedCL signiﬁcantly improved the ﬁnal test accuracy by up to 11% compared to its counterpart FedAvg using different models and datasets, while maintaining comparable or even better communication efﬁciency.

Original: Finally, we illustrated the generalization of our proposed CL periods aware framework via manipulating the client selection of state-of-the-art methods augmented by FedCL.
Revised: Finally, we illustrated the generalization of our proposed CL periods aware framework via manipulating the client selection of state-of-the-art methods augmented by FedCL.

Original: In the future work, we want to extend FedCL to improve FL of different machine learning models on other popular techniques such as gradient compression/quantization, fair aggregation, personalization, and adversarial attacks.
Revised: In the future work, we want to extend FedCL to improve FL of different machine learning models on other popular techniques such as gradient compression/quantization, fair aggregation, personalization, and adversarial attacks.

Original: We also believe that it is important to study the performance of FedCL on other models and datasets.
Revised: We also believe that it is important to study the performance of FedCL on other models and datasets.

Original: A E XTENSIVE R EVIEW AND R ESULTS OF E XPERIMENTS
Revised: A E XTENSIVE R EVIEW AND R ESULTS OF E XPERIMENTS

Original: We provide details of datasets and models in Appendix A.1 and additional experimental results in
Revised: We provide details of datasets and models in Appendix A.1 and additional experimental results in

Original: Appendix A.2.
Revised: Appendix A. [Table]

Revised: Parameter

Revised: [Table]

Original: A. S UMMARY OF D ATASETS AND M ODEL A RCHITECTURES
Revised: A. S UMMARY OF D ATASETS AND M ODEL A RCHITECTURES

Original: We implement FedCL and considered baselines in PyTorch (Paszke et al., 2017) on Python 3 with three NVIDIA RTX A6000 GPUs, 48GB with 128GB RAM.
Revised: We implement FedCL and considered baselines in PyTorch (Paszke et al., 2017) on Python 3 with three NVIDIA RTX A6000 GPUs, 48GB with 128GB RAM.

Original: We conduct experiments on the popular datasets CIFAR-10 and CIFAR-100 (Krizhevsky et al., 2009),
Revised: We conduct experiments on the popular datasets CIFAR-10 and CIFAR-100 (Krizhevsky et al., 2009),

Original: Fashion-MNIST (Xiao et al., 2017) and Shakespeare (McMahan et al., 2017).
Revised: Fashion-MNIST (Xiao et al., 2017) and Shakespeare (McMahan et al., 2017).

Original: The CIFAR-10 and CIFAR1-00 dataset consists of 60, 32 × 32 color images in 10 and 100 classes, respectively, where 50,000 samples are for training and the other 10,000 samples for testing.
Revised: The CIFAR-10 and CIFAR1-00 dataset consists of 60,000 32 × 32 color images in 10 and 100 classes, respectively, where 50,000 samples are for training and the other 10,000 samples for testing.

Original: The Fashion-MNIST datasets contain handwritten digits with 60,000 samples for training and 10,000 samples for testing, where each sample is an 28 × 28 grayscale images over 10 classes.
Revised: The Fashion-MNIST datasets contain handwritten digits with 60,000 samples for training and 10,000 samples for testing, where each sample is an 28 × 28 grayscale images over 10 classes.

Original: The Shakespeare dataset consists of 74 characters with 734,057 training data and 70,657 testing data.
Revised: The Shakespeare dataset consists of 74 characters with 734,057 training data and 70,657 testing data.

Revised: We simulate a heterogeneous partition into M clients by sampling p k ∼ Dir M ( α ) , where α is the parameter of the Dirichlet distribution (Wang et al., 2020b;c).

Revised: Speciﬁcally, for each class of samples, set the class probability in each client by sampling from a Dirichlet distribution with the same α parameter.

Revised: For instance, when α = 0 .

Revised: 5 , sampling p k ∼ Dir (0 . 5) and allocating a p k,j proportion of the training instances of class k to local client j .

Original: We summarize the details of VGG-11 (Simonyan & Zisserman, 2015) and AlexNet (Krizhevsky et al., 2012)
Revised: We summarize the details of VGG-11 (Simonyan & Zisserman, 2015) and AlexNet (Krizhevsky et al., 2012)

Original: architectures used in our experiments for classiﬁcation tasks in Tables 1 and 2, respectively.
Revised: architectures used in our experiments for classiﬁcation tasks in Tables 1 and 2, respectively.

Original: For the language task, we train a stacked character-level LSTM language model as in (Kim et al., 2016; McMahan et al., 2017), which is summarized in Table 
Revised: For the language task, we train a stacked character-level LSTM language model as in (Kim et al., 2016; McMahan et al., 2017), which is summarized in Table [Table] [Table]

Revised: hyper-parameter layer1.embeding 80 ×

Revised: N/A layer2.lstm 256 ×num layers

Original: [Table]
Revised: [Table]

Revised: True dropout

Revised: N/A p=5% layer3.fc.weight 512 ×

Original: [Table]
Revised: [Table]

Revised: N/

Original: [Table]
Revised: [Table]

Revised: bias

Revised: N/

Original: [Table]
Revised: [Table]

Original: A. A DDITIONAL E XPERIMENTAL R ESULTS
Revised: A. A DDITIONAL E XPERIMENTAL R ESULTS

Original: Results complementary to Section 5 on non-IID partitioned CIFAR-10 and Fashion-MNIST datasets are given in Appendix A.2.2.
Revised: Results complementary to Section 5 on non-IID partitioned CIFAR-10 and Fashion-MNIST datasets are given in Appendix A.2.2.

Original: Experimental results on non-IID partitioned CIFAR-100 and Shakespeare datasets are presented in Appendix A.2.3. 
Revised: Experimental results on non-IID partitioned CIFAR-100 and Shakespeare datasets are presented in Appendix A.2.3. [Figure]

Original: [Figure] 
Revised: [Figure] and FedFIM approaches to detect CL periods. A.2. C OMPARISON BETWEEN FGN AND F ED FIM BASED APPROACH TO DETECT CL PERIODS

Original: As discussed in Section 4, we propose a lightweight FGN based approach to detect CL periods that can be easily leveraged for client selection during the training process in an online manner.
Revised: As discussed in Section 4, we propose a lightweight FGN based approach to detect CL periods that can be easily leveraged for client selection during the training process in an online manner.

Original: Though the detection performance is similar between FGN and FedFIM approach as shown in Figure 1, our FGN approach is much more computationally efﬁcient.
Revised: Though the detection performance is similar between FGN and FedFIM approach as shown in Figure 1, our FGN approach is much more computationally efﬁcient.

Original: For example, consider the settings in Section 4, the computation time and memory consumption of FGN and 
Revised: For example, consider the settings in Section 4, the computation time and memory consumption of FGN and FedFIM under the settings in

Original: FedFIM under the settings in Appendix A.1 are presented in Figure 16, where C+A, C+V, M+F, and F+A represent AlexNet onCIFAR-10, VGG-11 on
Revised: Appendix A.1 are presented in Figure 16, where C+A, C+V, M+F, and F+A represent AlexNet on

Original: CIFAR-10,  FC on MNIST, and AlexNet on Fashion-MNIST, respectively.
Revised: CIFAR-10, VGG-11 on CIFAR-10, FC on MNIST, and AlexNet on Fashion-MNIST, respectively.

Revised: Since the identiﬁed CL periods by using FGN are almost the same as those identiﬁed by using

Revised: FedFIM (see Figure 1), the test accuracy of FedCL is expected to be the same.

Revised: For instance, the test accuracy of FedCL when leveraging the CL periods identiﬁed by FGN and FedFIM, which we denote as FedCL (FGN) and FedCL (FedFIM), respectively, using AlexNet on CIFAR-10 is reported as Figure 17.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: We further evaluate the impact of random seed when using FGN to detect CL periods. In particular, we randomly generate ﬁve seeds, and report the identiﬁed CL periods in Figure 18. We observe that our FGN can consistently identify the CL periods across different random seeds and the identiﬁed CL periods are almost the same.

Original: [Figure]
Revised: This performance is further pronounced when we compare the test accuracy as shown in Figure 19. These results further support the robustness of our proposed FGN metric.

Original: A.2. A DDITIONAL R ESULTS ON CIFAR-10 AND F ASHION -MNIST DATASETS
Revised: A.2. A DDITIONAL R ESULTS ON CIFAR-10 AND F ASHION -MNIST DATASETS

Revised: [Figure]

Revised: [Figure]

Original: Communication Efﬁciency.
Revised: Communication Efﬁciency.

Original: Similar to Figures 4 and 5, we report the communication rounds required for FedCL and FedAvg to achieve some common targeted accuracies in Figures 17 and 18.
Revised: Similar to Figures 4 and 5, we report the communication rounds required for FedCL and FedAvg to achieve some common targeted accuracies in Figures 20 and 21.

Original: Again, we observe that FedCL requires fewer rounds to achieve the same test accuracy as FedAvg.
Revised: Again, we observe that FedCL requires fewer rounds to achieve the same test accuracy as FedAvg.

Original: The communication comparison for FedProx, VRL-SGD and FedNova that is similar to Figuresand 9, are presented in Figures 19 and 20 for VRL-SGD vs. VRL-SGD-CL, and in Figures 21 andfor FedNova vs. FedNova-CL. 
Revised: The communication comparison for FedProx, VRL-SGD and FedNova that is similar to Figuresand 9, are presented in Figures 22 and 23 for VRL-SGD vs. VRL-SGD-CL, and in Figures 24 andfor FedNova vs. FedNova-CL. [Figure] [Figure] [Figure]

Revised: [Figure]

Original: Detection Thresholds.
Revised: Detection Thresholds.

Original: Complementary to the results shown in Figure 10, the detection thresholds also have the same impact on the ﬁnal test accuracy of FedProx-CL, VRL-SGD-CL and FedNova-CL using AlexNet on non-IID partitioned CIFAR-10 and Fashion-MNIST as illustrated in Figures 23,and 25.
Revised: Complementary to the results shown in Figure 10, the detection thresholds also have the same impact on the ﬁnal test accuracy of FedProx-CL, VRL-SGD-CL and FedNova-CL using AlexNet on non-IID partitioned CIFAR-10 and Fashion-MNIST as illustrated in Figures 26,and 28. [Figure] [Figure] [Figure]

Original: A.2. R ESULTS ON CIFAR-100 AND S HAKESPEARE DATASETS
Revised: A.2. R ESULTS ON CIFAR-100 AND S HAKESPEARE DATASETS

Original: Test Accuracy.
Revised: Test Accuracy.

Original: We present the test accuracy on non-IID partitioned CIFAR-100 and Shakespeare datasets in Figures 26 and 27, and Figures 28 and 29, respectively.
Revised: We present the test accuracy on non-IID partitioned CIFAR-100 and Shakespeare datasets in Figures 29 and 30, and Figures 31 and 32, respectively.

Original: Again we observe that the CL periods awareness consistently improves the ﬁnal test accuracy.
Revised: Again we observe that the CL periods awareness consistently improves the ﬁnal test accuracy.

Original: Communication Efﬁciency.
Revised: Communication Efﬁciency.

Original: The comparisons of communication rounds required for FedCL,
Revised: The comparisons of communication rounds required for FedCL,

Original: FedProx-CL, VRL-SGD-CL and FedNova-CL to achieve the ﬁnal test accuracy of FedAvg, FedProx,
Revised: FedProx-CL, VRL-SGD-CL and FedNova-CL to achieve the ﬁnal test accuracy of FedAvg, FedProx,

Original: VRL-SGD and FedNova, respectively, are presented in Figures30, 31, 32 and 33,  respectively, using the non-IID partitioned CIFAR-100 dataset; and Figures 34and 35, respectively, using the non-IID partitioned Shakespeare dataset.
Revised: VRL-SGD and FedNova, respectively, are presented in Figures 33, 34, 35 and 36, respectively, using the non-IID partitioned CIFAR-100 dataset; and Figures 37and 38, respectively, using the non-IID partitioned Shakespeare dataset.

Original: Similar conclusions can be made and hence we omit the discussions here. 
Revised: Similar conclusions can be made and hence we omit the discussions here. [Figure] [Figure] [Figure]

Revised: Shakespeare .

Original: [Figure] 
Revised: [Figure] [Figure] [Figure] [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: A. A DDITIONAL R ESULTS ON THE G ENERALIZATION OF F ED CL

Revised: In Section 5.3, we discuss the generalization of our proposed FedCL by considering three state of the arts, i.e., FedProx (Li et al., 2020a),

Revised: VRL-SGD (Liang et al., 2019) and FedNova (Wang et al., 2020c).

Revised: We now provide additional experimental results on the generalization of FedCL by considering

Revised: FedOPT (Reddi et al., 2021) with three other state-of-the-art methods, i.e., FedAdagrad, FedYogi, and

Original: [Figure]
Revised: FedAdam. We call the corresponding CL periods-aware methods as FedAdagrad-CL, FedYogi-CL, and FedAdam-CL, respectively. We consider the same setting as that in Section 5.3. From Figuresand 40, we again observe that CL periods awareness signiﬁcantly improves the test performance of baseline methods, i.e., FedAdagrad-CL, FedYogi-CL, and FedAdam-CL outperform FedAdagrad,

Original: [Figure]
Revised: FedYogi, and FedAdam, respectively, in all scenarios. Likewise, the CL periods augmented method, e.g., FedAdagrad-CL requires fewer rounds to achieve the ﬁnal test accuracy of the corresponding baseline FedAdagrad, as shown in Figures 41 and 42. Similar observations can be made for FedYogi in Figures 43 and 44, and FedAdam in Figures 45 and 46.

Original: [Figure] 
Revised: [Figure] [Figure] [Figure] [Figure]

Revised: A. R ANDOM I NCREASE AND D ECREASE THE N UMBER OF P ARTICIPATED C LIENTS

Revised: Besides deterministically increasing or decreasing the number of participated clients as in FedCL

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: we randomly increase or decrease the number. Speciﬁcally, we consider two settings. On the one

Original: [Figure] 
Revised: [Figure] [Figure]

Revised: hand

Original: [Figure]
Revised: [Figure]

Revised: we ﬁx the probability of decreasing the participated clients from m to m/2 to be 30% in each round

Original: [Figure]
Revised: [Figure]

Revised: and investigate the impact of the probability of increasing the participated clients from m to 2m in each round.

Revised: On the other hand

Original: [Figure]
Revised: [Figure]

Revised: we ﬁx the probability of increasing the participated clients from m to 2m to be 30% in each round

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: and investigate the impact of the probability of decreasing the participated clients from m to m/2 in each round. We report the ﬁnal test accuracy of the model, and compare it with the FedCL (see Algorithm 2) and the FedAvg. As shown in Figures 47 and 48, random increase or decrease may not necessarily improve the performance of FedAvg. This is due to the fact that the random strategy may not necessarily align with the ﬁndings of critical learning periods (e

Original: [Figure]
Revised: [Figure]

Revised: g., Achille et al.

Revised: 2019 and Yan et al. 2022) that more data/clients need to be involved in early training phases.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: -SGD-CL and FedNova vs. FedNovaCL on non-IID Shakespeare .
Revised: We report the number of participated clients and the FGN curve in Figure 49. We observe that more clients are involved in the early training phases where the CL periods occur [Figure]