Original: O N THE E FFICACY OF S ERVER -A IDED F EDERATED L EARNING AGAINST P ARTIAL C LIENT P ARTICIPATION
Revised: O N THE E FFICACY OF S ERVER -A IDED F EDERATED L EARNING AGAINST P ARTIAL C LIENT P ARTICIPATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Although federated learning (FL) has become a prevailing distributed learning framework in recent years due to its beneﬁts in scalability/privacy, there remain many signiﬁcant challenges in FL system design.
Revised: Although federated learning (FL) has become a prevailing distributed learning framework in recent years due to its beneﬁts in scalability/privacy, there remain many signiﬁcant challenges in FL system design.

Original: Notably, most existing works in the current FL literature assume either full client or uniformly distributed client participation.
Revised: Notably, most existing works in the current FL literature assume either full client or uniformly distributed client participation.

Original: Unfortunately, this idealistic assumption rarely holds in practice.
Revised: Unfortunately, this idealistic assumption rarely holds in practice.

Original: It has been frequently observed that some clients may never participate in FL training (aka partial/incomplete participation) due to a meld of system heterogeneity factors.
Revised: It has been frequently observed that some clients may never participate in FL training (aka partial/incomplete participation) due to a meld of system heterogeneity factors.

Original: To mitigate impacts of partial client participation, an increasingly popular approach in practical FL systems is the sever-aided federated learning (SA-FL) framework, where one equips the server with an auxiliary dataset.
Revised: To mitigate impacts of partial client participation, an increasingly popular approach in practical FL systems is the sever-aided federated learning (SA-FL) framework, where one equips the server with an auxiliary dataset.

Original: However, despite the fact that SA-FL has been empirically shown to be effective in addressing the partial client participation problem, there remains a lack of theoretical understanding for
Revised: However, despite the fact that SA-FL has been empirically shown to be effective in addressing the partial client participation problem, there remains a lack of theoretical understanding for

Original: SA-FL.
Revised: SA-FL.

Original: Worse yet, even the ramiﬁcations of partial worker participation are not clearly understood in conventional FL so far.
Revised: Worse yet, even the ramiﬁcations of partial worker participation are not clearly understood in conventional FL so far.

Original: These theoretical gaps motivate us to rigorously investigate SA-FL.
Revised: These theoretical gaps motivate us to rigorously investigate SA-FL.

Original: To this end, we ﬁrst reveal that conventional FL is not
Revised: To this end, we ﬁrst reveal that conventional FL is not

Original: PAC-learnable under partial client participation in the worst case, which advances our understanding of conventional FL.
Revised: PAC-learnable under partial client participation in the worst case, which advances our understanding of conventional FL.

Original: Then, we show that the PAC-learnability of FL with partial client participation can indeed be revived by SA-FL, which theoretically justiﬁes the use of SA-FL for the ﬁrst time.
Revised: Then, we show that the PAC-learnability of FL with partial client participation can indeed be revived by SA-FL, which theoretically justiﬁes the use of SA-FL for the ﬁrst time.

Original: Lastly, to further make
Revised: Lastly, to further make

Original: SA-FL communication-efﬁcient, we propose the SAFARI (server-aided federated averaging) algorithm that enjoys convergence guarantees and the same level of communication efﬁciency and privacy as the state-of-the-art FL.
Revised: SA-FL communication-efﬁcient, we propose the SAFARI (server-aided federated averaging) algorithm that enjoys convergence guarantees and the same level of communication efﬁciency and privacy as the state-of-the-art FL.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Since the seminal work by McMahan et al.
Revised: Since the seminal work by McMahan et al.

Original: (2017), federated learning (FL) has emerged as a powerful distributed learning paradigm that enables a large number of clients (e.g., edge devices) to collaboratively train a model under a central server’s coordination.
Revised: (2017), federated learning (FL) has emerged as a powerful distributed learning paradigm that enables a large number of clients (e.g., edge devices) to collaboratively train a model under a central server’s coordination.

Original: However, with FL gaining popularity, it has also become apparent that FL faces a key challenge that is unseen in traditional distributed learning in datacenter settings – system heterogeneity.
Revised: However, with FL gaining popularity, it has also become apparent that FL faces a key challenge that is unseen in traditional distributed learning in datacenter settings – system heterogeneity.

Original: Generally speaking, system heterogeneity in FL is caused by the massively different computation and communication capabilities at each client (computational power, communication capacity, drop-out rate, etc.).
Revised: Generally speaking, system heterogeneity in FL is caused by the massively different computation and communication capabilities at each client (computational power, communication capacity, drop-out rate, etc.).

Original: Studies have shown that system heterogeneity can signiﬁcantly impact client participation in a highly non-trivial fashion and severely degrade the learning performance of FL algorithms (Bonawitz et al., 2019; Yang et al., 2021a).
Revised: Studies have shown that system heterogeneity can signiﬁcantly impact client participation in a highly non-trivial fashion and severely degrade the learning performance of FL algorithms (Bonawitz et al., 2019; Yang et al., 2021a).

Original: For example, it is shown in (Yang et al., 2021a) that more than 30% clients never participate in FL, while only 30% of the clients contribute to 81% of the total computation even if the server uniformly samples the clients.
Revised: For example, it is shown in (Yang et al., 2021a) that more than 30% clients never participate in FL, while only 30% of the clients contribute to 81% of the total computation even if the server uniformly samples the clients.

Original: Exacerbating the problem is the fact that the client’s status could be unstable and time-varying due to the aforementioned computation and communication constraints.
Revised: Exacerbating the problem is the fact that the client’s status could be unstable and time-varying due to the aforementioned computation and communication constraints.

Original: To mitigate the impact of partial client participation, one approach called server-aided federated learning (SA-FL) has been increasingly adopted in practical FL systems in recent years (see, e.g., (Zhao et al., 2018;
Revised: To mitigate the impact of partial client participation, one approach called server-aided federated learning (SA-FL) has been increasingly adopted in practical FL systems in recent years (see, e.g., (Zhao et al., 2018;

Original: Wang et al., 2021b)).
Revised: Wang et al., 2021b)).

Original: The basic idea of SA-FL is to equip the server in FL with a small auxiliary dataset that approximately mimics the population distribution, so that the distribution deviation induced by partial client participation can be corrected.
Revised: The basic idea of SA-FL is to equip the server in FL with a small auxiliary dataset that approximately mimics the population distribution, so that the distribution deviation induced by partial client participation can be corrected.

Original: To date, even though SA-FL has been empirically shown to be quite effective in addressing the partial client participation problem in practice, there remains a lack of theoretical understanding for SA-FL.
Revised: To date, even though SA-FL has been empirically shown to be quite effective in addressing the partial client participation problem in practice, there remains a lack of theoretical understanding for SA-FL.

Original: This motivates us to rigorously investigate the efﬁcacy of SA-FL against partial client participation in FL in this paper.
Revised: This motivates us to rigorously investigate the efﬁcacy of SA-FL against partial client participation in FL in this paper.

Original: Somewhat counterintuitively, to fully understand SA-FL, one must be able to ﬁrst see what happens if SA-FL is not used and partial client participation is left untreated in conventional FL.
Revised: Somewhat counterintuitively, to fully understand SA-FL, one must be able to ﬁrst see what happens if SA-FL is not used and partial client participation is left untreated in conventional FL.

Original: In other words, we need to ﬁrst answer the following fundamental question: “1) What are the impacts of partial client participation on FL learning performance?”
Revised: In other words, we need to ﬁrst answer the following fundamental question: “1) What are the impacts of partial client participation on FL learning performance?”

Original: Upon answering this question, the next important follow-up question regarding SA-FL is: “2) What beneﬁts could SA-FL bring
Revised: Upon answering this question, the next important follow-up question regarding SA-FL is: “2) What beneﬁts could SA-FL bring

Original: and how could we theoretically characterize them?
Revised: and how could we theoretically characterize them?

Original: Also, since SA-FL still largely follows the server-client architecture that demands intensive communication between server and clients, the third fundamental question regarding SA-FL is: “3) Could we make SA-FL as communication-efﬁcient as conventional
Revised: Also, since SA-FL still largely follows the server-client architecture that demands intensive communication between server and clients, the third fundamental question regarding SA-FL is: “3) Could we make SA-FL as communication-efﬁcient as conventional

Original: FL?”
Revised: FL?”

Original: Indeed, answering these three questions constitutes the rest of this paper, where we address the ﬁrst two questions through the lens of PAC (probably approximately correct) learnability, while resolving the third question by proposing a communication-efﬁcient SA-FL algorithm.
Revised: Indeed, answering these three questions constitutes the rest of this paper, where we address the ﬁrst two questions through the lens of PAC (probably approximately correct) learnability, while resolving the third question by proposing a communication-efﬁcient SA-FL algorithm.

Original: Our major contributions in this work are summarized as follows:
Revised: Our major contributions in this work are summarized as follows:

Original: • By establishing a worst-case generalization error lower bound, we show that conventional FL is not PAC-learnable under partial client participation.
Revised: • By establishing a worst-case generalization error lower bound, we show that conventional FL is not PAC-learnable under partial client participation.

Original: In other words, no learning algorithm can approach zero generalization error under partial participation for conventional FL even in the limit of inﬁnitely many data samples and training iterations.
Revised: In other words, no learning algorithm can approach zero generalization error under partial participation for conventional FL even in the limit of inﬁnitely many data samples and training iterations.

Original: This insight, though being negative, warrants the necessity of developing new algorithmic techniques and system architectures (e
Revised: This insight, though being negative, warrants the necessity of developing new algorithmic techniques and system architectures (e

Original: g., SA-FL) to modify the conventional FL framework to mitigate partial client participation.
Revised: g., SA-FL) to modify the conventional FL framework to mitigate partial client participation.

Original: • Inspired by techniques from domain adaptation, we prove a new generalization error bound to show that SA-FL can indeed revive the PAC learnability of FL under partial client participation.
Revised: • Inspired by techniques from domain adaptation, we prove a new generalization error bound to show that SA-FL can indeed revive the PAC learnability of FL under partial client participation.

Original: We note that this bound could reach zero asymptotically as the number data samples increase.
Revised: We note that this bound could reach zero asymptotically as the number data samples increase.

Original: This is much stronger than previous results in domain adaptation with non-vanishing small error (see Section 2 for details).
Revised: This is much stronger than previous results in domain adaptation with non-vanishing small error (see Section 2 for details).

Original: • To make SA-FL communication-efﬁcient, we propose a new training algorithm for SA-FL called SAFARI (server-aided federated averaging).
Revised: • To make SA-FL communication-efﬁcient, we propose a new training algorithm for SA-FL called SAFARI (server-aided federated averaging).

Original: By carefully designing the update coordination between the server and the clients,
Revised: By carefully designing the update coordination between the server and the clients,

Original: we show that SAFARI achieves an O (1 / √ KT ) convergence rate, matching the convergence rate of state-of-the-art conventional FL algorithms.
Revised: we show that SAFARI achieves an O (1 / √ KT ) convergence rate, matching the convergence rate of state-of-the-art conventional FL algorithms.

Original: We also conduct extensive experiments to demonstrate the efﬁcacy and efﬁciency of our SAFARI algorithm.
Revised: We also conduct extensive experiments to demonstrate the efﬁcacy and efﬁciency of our SAFARI algorithm.

Original: The rest of this paper is organized as follows.
Revised: The rest of this paper is organized as follows.

Original: In Section 2, we review the literature to put our work in comparative perspectives.
Revised: In Section 2, we review the literature to put our work in comparative perspectives.

Original: Section 3 presents the PAC learning analysis of standard FL under partial participation and our proposed SA-FL framework.
Revised: Section 3 presents the PAC learning analysis of standard FL under partial participation and our proposed SA-FL framework.

Original: We then propose SAFARI algorithm with convergence guarantees in Section 4.
Revised: We then propose SAFARI algorithm with convergence guarantees in Section 4.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: 1) Client Participation in Federated Learning: The seminal FedAvg algorithm was ﬁrst proposed in (McMahan et al., 2017) as a heuristic to improve communication efﬁciency and data privacy for FL.
Revised: 1) Client Participation in Federated Learning: The seminal FedAvg algorithm was ﬁrst proposed in (McMahan et al., 2017) as a heuristic to improve communication efﬁciency and data privacy for FL.

Original: Since then, there have been many follow-ups (e.g., (Li et al., 2020a;
Revised: Since then, there have been many follow-ups (e.g., (Li et al., 2020a;

Original: Wang et al., 2020; Zhang et al., 2020; Acar et al., 2021; Karimireddy et al., 2020; Luo et al., 2021; Mitra et al., 2021; Karimireddy et al., 2021; Khanduri et al., 2021; Murata & Suzuki, 2021; Avdiukhin & Kasiviswanathan, 2021) and so on) on addressing the data heterogeneity challenge in FL.
Revised: Wang et al., 2020; Zhang et al., 2020; Acar et al., 2021; Karimireddy et al., 2020; Luo et al., 2021; Mitra et al., 2021; Karimireddy et al., 2021; Khanduri et al., 2021; Murata & Suzuki, 2021; Avdiukhin & Kasiviswanathan, 2021) and so on) on addressing the data heterogeneity challenge in FL.

Original: However, most of these works (e.g., (Li et al., 2020a; Wang et al., 2020; Zhang et al., 2020; Acar et al., 2021; Karimireddy et al., 2020;
Revised: However, most of these works (e.g., (Li et al., 2020a; Wang et al., 2020; Zhang et al., 2020; Acar et al., 2021; Karimireddy et al., 2020;

Original: Yang et al., 2021b)) are based on the full or uniform (i.e., sampling clients uniformly at random) client participation assumption.
Revised: Yang et al., 2021b)) are based on the full or uniform (i.e., sampling clients uniformly at random) client participation assumption.

Original: The full or uniform participation assumptions are essential since they are required to ensure that the stochastic gradient estimator is unbiased in each round of update.
Revised: The full or uniform participation assumptions are essential since they are required to ensure that the stochastic gradient estimator is unbiased in each round of update.

Original: Thus, even if “model drift” or “objective inconsistency” emerges due to local updates (Karimireddy et al., 2020; Wang et al., 2020), the full/uniform client participation in each communication round averages them out in the long run, therefore, guaranteeing convergence.
Revised: Thus, even if “model drift” or “objective inconsistency” emerges due to local updates (Karimireddy et al., 2020; Wang et al., 2020), the full/uniform client participation in each communication round averages them out in the long run, therefore, guaranteeing convergence.

Original: A related interesting line of works in FL different from full/uniform client participation focuses on proactively creating ﬂexible client participation (see, e.g., (Xie et al., 2019;
Revised: A related interesting line of works in FL different from full/uniform client participation focuses on proactively creating ﬂexible client participation (see, e.g., (Xie et al., 2019;

Original: Ruan et al., 2021;
Revised: Ruan et al., 2021;

Original: Gu et al., 2021; Avdiukhin & Kasiviswanathan, 2021;
Revised: Gu et al., 2021; Avdiukhin & Kasiviswanathan, 2021;

Original: Yang et al., 2022; Wang & Ji, 2022)).
Revised: Yang et al., 2022; Wang & Ji, 2022)).

Original: The main idea here is to allow asynchronous communication or ﬁxed participation pattern (e.g., given probability) for clients to ﬂexibly participate in training.
Revised: The main idea here is to allow asynchronous communication or ﬁxed participation pattern (e.g., given probability) for clients to ﬂexibly participate in training.

Original: Existing works in this area often require extra assumptions, such as bounded delay (Ruan et al., 2021;
Revised: Existing works in this area often require extra assumptions, such as bounded delay (Ruan et al., 2021;

Original: Gu et al., 2021;
Revised: Gu et al., 2021;

Original: Yang et al., 2022; Avdiukhin & Kasiviswanathan, 2021) and identical computation rate (Avdiukhin & Kasiviswanathan, 2021).
Revised: Yang et al., 2022; Avdiukhin & Kasiviswanathan, 2021) and identical computation rate (Avdiukhin & Kasiviswanathan, 2021).

Original: Under these assumptions, although the stochastic gradients are no longer unbiased estimators of the full gradients, the deviation in each communication round remains bounded.
Revised: Under these assumptions, although the stochastic gradients are no longer unbiased estimators of the full gradients, the deviation in each communication round remains bounded.

Original: For sufﬁciently many communication rounds, the impact of such deviation from full gradients vanishes asymptotically, since each client can still participate in FL in the long run.
Revised: For sufﬁciently many communication rounds, the impact of such deviation from full gradients vanishes asymptotically, since each client can still participate in FL in the long run.

Original: In contrast, this paper considers a more practical (or worst-case) scenario in FL – partial client participation even in the long run , which can be caused by many real-world system heterogeneity factors as mentioned in Section 1 .
Revised: In contrast, this paper considers a more practical (or worst-case) scenario in FL – partial client participation even in the long run , which can be caused by many real-world system heterogeneity factors as mentioned in Section 1 .

Original: 2) Domain Adaptation: Since partial client participation induces a gap between the dataset distribution used for FL training and the true data population distribution across all clients, our work is also related to the ﬁeld of domain adaptation in learning.
Revised: 2) Domain Adaptation: Since partial client participation induces a gap between the dataset distribution used for FL training and the true data population distribution across all clients, our work is also related to the ﬁeld of domain adaptation in learning.

Original: Domain adaptation focuses on the learnability of a model trained in one source domain but applied to a different and related target domain.
Revised: Domain adaptation focuses on the learnability of a model trained in one source domain but applied to a different and related target domain.

Original: The basic approach is to quantify the error in terms of the source domain plus the distance between source and target domains.
Revised: The basic approach is to quantify the error in terms of the source domain plus the distance between source and target domains.

Original: Speciﬁcally, let P and Q be the target and source distributions, respectively.
Revised: Speciﬁcally, let P and Q be the target and source distributions, respectively.

Original: Then, the generalization error is expressed as O ( A ( n Q ))
Revised: Then, the generalization error is expressed as O ( A ( n Q ))

Original: + dist ( P, Q ) , where A ( n Q ) is an upper bound of the error dependent on the total number of samples in Q , Widely-used distance measures include d A -divergence (Ben-David et al., 2010; David et al., 2010) and
Revised: + dist ( P, Q ) , where A ( n Q ) is an upper bound of the error dependent on the total number of samples in Q , Widely-used distance measures include d A -divergence (Ben-David et al., 2010; David et al., 2010) and

Original: Y -discrepancy
Revised: Y -discrepancy

Original: (Mansour et al., 2009;
Revised: (Mansour et al., 2009;

Original: Mohri & Medina, 2012).
Revised: Mohri & Medina, 2012).

Original: We note, however, that results in domain adaptation cannot be directly applied in FL with partial client participation, since doing so yields an overly pessimistic bound.
Revised: We note, however, that results in domain adaptation cannot be directly applied in FL with partial client participation, since doing so yields an overly pessimistic bound.

Original: Speciﬁcally, the error based on domain adaptation remains non-zero for asymptotically small distance dist ( P, Q ) between P and Q even with inﬁnite many samples in n Q (i.e., A ( n Q ) → 0 ).
Revised: Speciﬁcally, the error based on domain adaptation remains non-zero for asymptotically small distance dist ( P, Q ) between P and Q even with inﬁnite many samples in n Q (i.e., A ( n Q ) → 0 ).

Original: In this paper, rather than directly using results from domain adaptation, we establish a much sharper upper bound (see Section 3).
Revised: In this paper, rather than directly using results from domain adaptation, we establish a much sharper upper bound (see Section 3).

Original: Another line of work in domain adaptation is using importance weights deﬁned by the density ratios between P and Q to correct the bias and reduce the discrepancy (Sugiyama et al., 2007a;b; Cortes et al., 2008).
Revised: Another line of work in domain adaptation is using importance weights deﬁned by the density ratios between P and Q to correct the bias and reduce the discrepancy (Sugiyama et al., 2007a;b; Cortes et al., 2008).

Original: However, due to FL privacy constraints, such density ratios are difﬁcult to estimate, which renders importance-weights-based methods infeasible in FL.
Revised: However, due to FL privacy constraints, such density ratios are difﬁcult to estimate, which renders importance-weights-based methods infeasible in FL.

Original: A closely related work is (Hanneke & Kpotufe, 2019), which proposed a new notion of discrepancy between source and target distributions called transfer exponents.
Revised: A closely related work is (Hanneke & Kpotufe, 2019), which proposed a new notion of discrepancy between source and target distributions called transfer exponents.

Original: However, this work considers non-overlapping support between P and Q , while we focus on overlapping support (see Fig.
Revised: However, this work considers non-overlapping support between P and Q , while we focus on overlapping support (see Fig.

Original: 1 in Section 3.3).
Revised: 1 in Section 3.3).

Original:  PAC-L EARNABILITY OF C ONVENTIONAL F EDERATED L EARNING WITH P ARTIAL C LIENT P ARTICIPATION
Revised:  PAC-L EARNABILITY OF C ONVENTIONAL F EDERATED L EARNING WITH P ARTIAL C LIENT P ARTICIPATION

Original: In this section, we ﬁrst focus on understanding the impacts of partial client participation on conventional FL in terms of PAC-learnability in Section 3.2.
Revised: In this section, we ﬁrst focus on understanding the impacts of partial client participation on conventional FL in terms of PAC-learnability in Section 3.2.

Original: This will also pave the way for studying
Revised: This will also pave the way for studying

Original: SA-FL later in Section 3.3.
Revised: SA-FL later in Section 3.3.

Original: In what follows, we start with the conventional FL formulation and some deﬁnitions in statistical learning that are necessary to formulate and prove our main results.
Revised: In what follows, we start with the conventional FL formulation and some deﬁnitions in statistical learning that are necessary to formulate and prove our main results.

Original: 3. P RELIMINARIES
Revised: 3. P RELIMINARIES

Original: The goal of FL is to minimize the following loss function F ( x ) =
Revised: The goal of FL is to minimize the following loss function F ( x ) =

Original: E i ∼P
Revised: E i ∼P

Original: [ F i ( x )] , where F i ( x ) (cid:44) E ξ ∼ P i [ f i ( x , ξ )] .
Revised: [ F i ( x )] , where F i ( x ) (cid:44) E ξ ∼ P i [ f i ( x , ξ )] .

Original: Here, P represents the distribution of the entire client population, x ∈ R d is the model parameter, F i ( x ) represents the local loss function at client i , and P i is the underlying distribution of the local dataset at client i .
Revised: Here, P represents the distribution of the entire client population, x ∈ R d is the model parameter, F i ( x ) represents the local loss function at client i , and P i is the underlying distribution of the local dataset at client i .

Original: In general, P i (cid:54) =
Revised: In general, P i (cid:54) =

Original: P j , if i (cid:54) =
Revised: P j , if i (cid:54) =

Original: j due to data heterogeneity.
Revised: j due to data heterogeneity.

Original: However, the loss function F ( x ) or full gradient ∇ F ( x ) can not be directly computed as the exact distribution of data is unknown in general.
Revised: However, the loss function F ( x ) or full gradient ∇ F ( x ) can not be directly computed as the exact distribution of data is unknown in general.

Original: Instead, one often considers the following empirical risk minimization (ERM) problem in the form of ﬁnite-sum:
Revised: Instead, one often considers the following empirical risk minimization (ERM) problem in the form of ﬁnite-sum:

Original: [Equation]
Revised: [Equation]

Original: where ˆ F i ( x ) (cid:44) (1 / | S i | ) (cid:80) ξ ∈ S i f i ( x , ξ ) .
Revised: where ˆ F i ( x ) (cid:44) (1 / | S i | ) (cid:80) ξ ∈ S i f i ( x , ξ ) .

Original: Here, M is the total number of clients, S i is the local dataset with cardinality | S i | , which is i.i.d.
Revised: Here, M is the total number of clients, S i is the local dataset with cardinality | S i | , which is i.i.d.

Original: and sampled from distribution P i , α i = | S
Revised: and sampled from distribution P i , α i = | S

Original: i | / ( (cid:80) j ∈
Revised: i | / ( (cid:80) j ∈

Original: [ M ] | S j | ) (hence (cid:80) i ∈
Revised: [ M ] | S j | ) (hence (cid:80) i ∈

Original: [ M ] α i = 1 ).
Revised: [ M ] α i = 1 ).

Original: For ease of presentation, we consider the balanced dataset case: α i = 1 /M, ∀ i ∈
Revised: For ease of presentation, we consider the balanced dataset case: α i = 1 /M, ∀ i ∈

Original: [ M ] .
Revised: [ M ] .

Original: Next, we state several deﬁnitions from statistical learning theory (Mohri et al., 2018).
Revised: Next, we state several deﬁnitions from statistical learning theory (Mohri et al., 2018).

Original: Deﬁnition 1 (Generalization Error and Empirical Error) .
Revised: Deﬁnition 1 (Generalization Error and Empirical Error) .

Original: Given a hypothesis h ∈ H , a target concept f , an underlying distribution D and a dataset S i.i.d.
Revised: Given a hypothesis h ∈ H , a target concept f , an underlying distribution D and a dataset S i.i.d.

Original: sampled from D ( S ∼ D ), the generalization error and empirical error of h are deﬁned as follows: R D ( h, f ) =
Revised: sampled from D ( S ∼ D ), the generalization error and empirical error of h are deﬁned as follows: R D ( h, f ) =

Original: E ( x,y )
Revised: E ( x,y )

Original: ∼D l ( h ( x ) , f ( x )) and ˆ R D ( h, f ) = 1 | S | (cid:80) i ∈
Revised: ∼D l ( h ( x ) , f ( x )) and ˆ R D ( h, f ) = 1 | S | (cid:80) i ∈

Original: S l ( h ( x i ) , f ( x i )) , where l ( · ) is some valid loss function.
Revised: S l ( h ( x i ) , f ( x i )) , where l ( · ) is some valid loss function.

Original: With a slight abuse of notation, we omit f by using R D ( h ) and
Revised: With a slight abuse of notation, we omit f by using R D ( h ) and

Original: ˆ R D ( h ) for simplicity.
Revised: ˆ R D ( h ) for simplicity.

Original: Deﬁnition 2 (Optimal Hypothesis) .
Revised: Deﬁnition 2 (Optimal Hypothesis) .

Original: For a distribution D and a dataset S ∼ D , we deﬁne h ∗D = argmin h ∈H R D ( h ) and ˆ h ∗D = argmin h ∈H ˆ R D ( h ) .
Revised: For a distribution D and a dataset S ∼ D , we deﬁne h ∗D = argmin h ∈H R D ( h ) and ˆ h ∗D = argmin h ∈H ˆ R D ( h ) .

Original: Deﬁnition 3 (Excess Error) .
Revised: Deﬁnition 3 (Excess Error) .

Original: For hypothesis h and distribution D , the excess error and excess empirical error are deﬁned as ε D ( h )=
Revised: For hypothesis h and distribution D , the excess error and excess empirical error are deﬁned as ε D ( h )=

Original: R D ( h ) −R D ( h ∗D ) , and ˆ ε D ( h )=
Revised: R D ( h ) −R D ( h ∗D ) , and ˆ ε D ( h )=

Original: ˆ R D ( h ) − ˆ R D (ˆ h ∗D ) , respectively.
Revised: ˆ R D ( h ) − ˆ R D (ˆ h ∗D ) , respectively.

Original: 3. C ONVENTIONAL F EDERATED L EARNING UNDER P ARTIAL C LIENT P ARTICIPATION
Revised: 3. C ONVENTIONAL F EDERATED L EARNING UNDER P ARTIAL C LIENT P ARTICIPATION

Original: With the above notations, we consider conventional FL under partial client participation.
Revised: With the above notations, we consider conventional FL under partial client participation.

Original: Consider an FL system with M clients in total.
Revised: Consider an FL system with M clients in total.

Original: We let P denote the underlying joint distribution of the entire system, which can be decomposed into the summation of the local distributions at each client, i.e.,
Revised: We let P denote the underlying joint distribution of the entire system, which can be decomposed into the summation of the local distributions at each client, i.e.,

Original: P = (cid:80) i ∈
Revised: P = (cid:80) i ∈

Original: [ M ] λ i P i , where λ i > 0 and (cid:80) i ∈
Revised: [ M ] λ i P i , where λ i > 0 and (cid:80) i ∈

Original: [ M ] λ i = 1 .
Revised: [ M ] λ i = 1 .

Original: We assume that each client i has n training samples i.i.d.
Revised: We assume that each client i has n training samples i.i.d.

Original: drawn from P i , i.e., | S i | = n, ∀ i ∈ [ M ] .
Revised: drawn from P i , i.e., | S i | = n, ∀ i ∈ [ M ] .

Original: Then, S = { ( x i , y i ) , i ∈
Revised: Then, S = { ( x i , y i ) , i ∈

Original: [ M × n ] } can be viewed as the dataset i.i.d.
Revised: [ M × n ] } can be viewed as the dataset i.i.d.

Original: sampled from the joint distribution P .
Revised: sampled from the joint distribution P .

Original: We consider a partial client participation setting, where m ∈ [0 , M ) clients participate in the FL training as a result of some client sampling process F .
Revised: We consider a partial client participation setting, where m ∈ [0 , M ) clients participate in the FL training as a result of some client sampling process F .

Original: We let F ( S ) represent the data ensemble actually used in training and D denote the underlying distribution corresponding to F ( S ) .
Revised: We let F ( S ) represent the data ensemble actually used in training and D denote the underlying distribution corresponding to F ( S ) .

Original: For convenience, we deﬁne the notion α = mM as the FL system capacity (i.e., only m clients participate in the training).
Revised: For convenience, we deﬁne the notion α = mM as the FL system capacity (i.e., only m clients participate in the training).

Original: For FL with partial client participation, we establish the following fundamental performance limit of any learner in general.
Revised: For FL with partial client participation, we establish the following fundamental performance limit of any learner in general.

Original: For simplicity, we use binary classiﬁcation with zero-one loss here.
Revised: For simplicity, we use binary classiﬁcation with zero-one loss here.

Original: We state the following impossibility result in Theorem 1 in terms of PAC learnability:
Revised: We state the following impossibility result in Theorem 1 in terms of PAC learnability:

Original: Theorem 1 (Impossibility Theorem) .
Revised: Theorem 1 (Impossibility Theorem) .

Original: Let H be a non-trivial hypothesis space and L : ( X , Y ) ( m × n ) → H be the learner for an FL system.
Revised: Let H be a non-trivial hypothesis space and L : ( X , Y ) ( m × n ) → H be the learner for an FL system.

Original: There exists a client participation process F , a distribution P , and a target function f ∈ H with min h ∈H R P
Revised: There exists a client participation process F , a distribution P , and a target function f ∈ H with min h ∈H R P

Original: ( h, f )
Revised: ( h, f )

Original: = 0 , such that P
Revised: = 0 , such that P

Original: S ∼ P (cid:2) R P (
Revised: S ∼ P (cid:2) R P (

Original: [Equation]
Revised: [Equation]

Original: ( F ( S ) , f )
Revised: ( F ( S ) , f )

Original: [Equation]
Revised: [Equation]

Original: > 1 −
Revised: > 1 −

Original: [Equation]
Revised: [Equation]

Original: > 120 .
Revised: > 120 .

Original: Proof Sketch.
Revised: Proof Sketch.

Original: The proof is based on the method of induced distributions in (Bshouty et al., 2002;
Revised: The proof is based on the method of induced distributions in (Bshouty et al., 2002;

Original: Mohri et al., 2018; Konstantinov et al., 2020).
Revised: Mohri et al., 2018; Konstantinov et al., 2020).

Original: We ﬁrst show that the learnability of an FL system is equivalent to that of a system that arbitrarily selects mn out of Mn samples in the centralized learning.
Revised: We ﬁrst show that the learnability of an FL system is equivalent to that of a system that arbitrarily selects mn out of Mn samples in the centralized learning.

Original: Then, for any learning algorithm, there exists a distribution P such that dataset F ( S ) resulting from partial participation and seen by the algorithm is always distributed identically for any target functions.
Revised: Then, for any learning algorithm, there exists a distribution P such that dataset F ( S ) resulting from partial participation and seen by the algorithm is always distributed identically for any target functions.

Original: Thus, no algorithm can learn a better predictor than random guessing.
Revised: Thus, no algorithm can learn a better predictor than random guessing.

Original: Due to space limitation, we relegate the full proof to supplementary material.
Revised: Due to space limitation, we relegate the full proof to supplementary material.

Original: Given the system capacity α ∈ (0 , 1) , the above theorem characterizes the worst-case scenario for
Revised: Given the system capacity α ∈ (0 , 1) , the above theorem characterizes the worst-case scenario for

Original: FL with partial client participation.
Revised: FL with partial client participation.

Original: It says that for any learner (i.e., algorithm)
Revised: It says that for any learner (i.e., algorithm)

Original: L , there exists a bad client participation process F and distributions P
Revised: L , there exists a bad client participation process F and distributions P

Original: i , i ∈
Revised: i , i ∈

Original: [ M ] over target function f , for which the error of the hypotheses returned by L is constant with non-zero probability.
Revised: [ M ] over target function f , for which the error of the hypotheses returned by L is constant with non-zero probability.

Original: In other words, FL with partial client participation is not PAC-learnable .
Revised: In other words, FL with partial client participation is not PAC-learnable .

Original: One interesting observation here is that the lower bound is independent of the number of samples per client n .
Revised: One interesting observation here is that the lower bound is independent of the number of samples per client n .

Original: This indicates that even if each client has inﬁnitely many samples ( n → ∞ ), it is impossible to have a zero-generation-error learner under the partial client participation ( α ∈ (0 , 1) ).
Revised: This indicates that even if each client has inﬁnitely many samples ( n → ∞ ), it is impossible to have a zero-generation-error learner under the partial client participation ( α ∈ (0 , 1) ).

Original: Note that this fundamental result relies on two conditions: heterogeneous dataset and arbitrary client participation.
Revised: Note that this fundamental result relies on two conditions: heterogeneous dataset and arbitrary client participation.

Original: Under these two conditions, there exists a worst-case scenario where the underlying distribution D of the participating data S D
Revised: Under these two conditions, there exists a worst-case scenario where the underlying distribution D of the participating data S D

Original: = F ( S ) deviates from the ground-truth P , thus yielding a non-vanishing error.
Revised: = F ( S ) deviates from the ground-truth P , thus yielding a non-vanishing error.

Original: This result also sheds light on how to motivate client participation in FL effectively and efﬁciently: the participating client’s data should be comprehensive enough to model the complexity of the joint distribution P to close the gap between D and P .
Revised: This result also sheds light on how to motivate client participation in FL effectively and efﬁciently: the participating client’s data should be comprehensive enough to model the complexity of the joint distribution P to close the gap between D and P .

Original: Note that this result is not contradictory to previous works where the convergence of FedAvg is guaranteed, since this theorem is not applicable for homogeneous (i.i.d.)
Revised: Note that this result is not contradictory to previous works where the convergence of FedAvg is guaranteed, since this theorem is not applicable for homogeneous (i.i.d.)

Original: datasets or uniformly random client participation.
Revised: datasets or uniformly random client participation.

Original: As mentioned earlier, most of the existing works rely on at least one of these two assumptions.
Revised: As mentioned earlier, most of the existing works rely on at least one of these two assumptions.

Original: However, none of these two assumptions hold for conventional
Revised: However, none of these two assumptions hold for conventional

Original: FL with partial client participation in practice.
Revised: FL with partial client participation in practice.

Original: 3. T HE PAC-L EARNABILITY OF S ERVER - AIDED FEDERATED LEARNING (SA-FL)
Revised: 3. T HE PAC-L EARNABILITY OF S ERVER - AIDED FEDERATED LEARNING (SA-FL)

Original: The intuition of SA-FL is to utilize a dataset T i.i.d.
Revised: The intuition of SA-FL is to utilize a dataset T i.i.d.

Original: sampled from distribution P with cardinality n T as a vehicle to correct potential distribution derivations due to partial client participation.
Revised: sampled from distribution P with cardinality n T as a vehicle to correct potential distribution derivations due to partial client participation.

Original: By doing so, the server steers the learning by a small number of representative data, while the clients aid the learning by federation to leverage the huge amount of privately decentralized data ( n S (cid:29) n T ).
Revised: By doing so, the server steers the learning by a small number of representative data, while the clients aid the learning by federation to leverage the huge amount of privately decentralized data ( n S (cid:29) n T ).

Original: Note that the assumption of access to this dataset is not restrictive since such datasets are already available in many FL systems: although not always necessary for training, an auxiliary dataset is often needed for deﬁning FL tasks (e.g., simulation prototyping) before training and model checking after training (e.g., quality evaluation and sanity checking)
Revised: Note that the assumption of access to this dataset is not restrictive since such datasets are already available in many FL systems: although not always necessary for training, an auxiliary dataset is often needed for deﬁning FL tasks (e.g., simulation prototyping) before training and model checking after training (e.g., quality evaluation and sanity checking)

Original: (McMahan et al., 2021;
Revised: (McMahan et al., 2021;

Original: Wang et al., 2021a).
Revised: Wang et al., 2021a).

Original: Also, obtaining an auxiliary dataset is affordable since the number of data points required is relatively small (of the order of hundreds, see our experimental results), and hence the cost is low.
Revised: Also, obtaining an auxiliary dataset is affordable since the number of data points required is relatively small (of the order of hundreds, see our experimental results), and hence the cost is low.

Original: Then, SA-FL can be easily achieved or even with manually labelled data thanks to its small size.
Revised: Then, SA-FL can be easily achieved or even with manually labelled data thanks to its small size.

Original: It is also worth noting that many works use such auxiliary datasets in FL for security (Cao et al., 2021), incentive design (Wang et al., 2019), and knowledge distillation (Cho et al., 2021).
Revised: It is also worth noting that many works use such auxiliary datasets in FL for security (Cao et al., 2021), incentive design (Wang et al., 2019), and knowledge distillation (Cho et al., 2021).

Original: For SA-FL, we consider the same partial client participation setting that induces a dataset S D ∼ D with cardinality n S and D (cid:54) =
Revised: For SA-FL, we consider the same partial client participation setting that induces a dataset S D ∼ D with cardinality n S and D (cid:54) =

Original: P .
Revised: P .

Original: As a result, the learning process is to minimize R P ( h ) by utilizing ( X , Y ) n T + n S to learn a hypothesis h ∈ H .
Revised: As a result, the learning process is to minimize R P ( h ) by utilizing ( X , Y ) n T + n S to learn a hypothesis h ∈ H .

Original: For notional clarity, we assume the joint dataset
Revised: For notional clarity, we assume the joint dataset

Original: S Q = ( S D ∪ T ) ∼ Q with cardinality n T + n S for some distribution Q .
Revised: S Q = ( S D ∪ T ) ∼ Q with cardinality n T + n S for some distribution Q .

Original: Before deriving the generalization error bound for SA-FL, we state the following assumption and deﬁnition.
Revised: Before deriving the generalization error bound for SA-FL, we state the following assumption and deﬁnition.

Original: Assumption 1 (Noise Condition) .
Revised: Assumption 1 (Noise Condition) .

Original: Suppose that h ∗ P and h ∗ Q exist.
Revised: Suppose that h ∗ P and h ∗ Q exist.

Original: There exist β P , β Q ∈ [0 , 1] and α P , α Q > 0 such that the following hold: P x ∼ P ( h ( x ) (cid:54) =
Revised: There exist β P , β Q ∈ [0 , 1] and α P , α Q > 0 such that the following hold: P x ∼ P ( h ( x ) (cid:54) =

Original: h ∗ P ( x )) ≤ α P [ ε P ( h )]
Revised: h ∗ P ( x )) ≤ α P [ ε P ( h )]

Original: β P , P x ∼ Q ( h ( x ) (cid:54) =
Revised: β P , P x ∼ Q ( h ( x ) (cid:54) =

Original: [Equation]
Revised: [Equation]

Original: This assumption is a traditional noise model known as the Bernstein class condition, which has been widely used in the literature (Massart & Nédélec, 2006; Koltchinskii, 2006; Hanneke, 2016).
Revised: This assumption is a traditional noise model known as the Bernstein class condition, which has been widely used in the literature (Massart & Nédélec, 2006; Koltchinskii, 2006; Hanneke, 2016).

Original: Deﬁnition 4 ( ( α, β ) -Positively-Related) .
Revised: Deﬁnition 4 ( ( α, β ) -Positively-Related) .

Original: Distributions P and Q are said to be ( α, β ) -positivelyrelated
Revised: Distributions P and Q are said to be ( α, β ) -positivelyrelated

Original: if there exist constants α ≥ 0 and β ≥ 0 such that | ε P ( h ) − ε Q ( h ) | ≤ α [ ε Q ( h )] β , ∀ h ∈ H .
Revised: if there exist constants α ≥ 0 and β ≥ 0 such that | ε P ( h ) − ε Q ( h ) | ≤ α [ ε Q ( h )] β , ∀ h ∈ H .

Original: Deﬁnition 4 speciﬁes a stronger constraint between distributions P and Q .
Revised: Deﬁnition 4 speciﬁes a stronger constraint between distributions P and Q .

Original: It indicates that the difference of excess error for one hypothesis h ∈ H between P and Q is bounded by the excess error of Q in some exponential form.
Revised: It indicates that the difference of excess error for one hypothesis h ∈ H between P and Q is bounded by the excess error of Q in some exponential form.

Original: With the above assumption and deﬁnition, we have the following generation error bound for SA-FL, which shows that SA-FL is PAC-learnable:
Revised: With the above assumption and deﬁnition, we have the following generation error bound for SA-FL, which shows that SA-FL is PAC-learnable:

Original: Theorem 2 (Generalization Error Bound for SA-FL) .
Revised: Theorem 2 (Generalization Error Bound for SA-FL) .

Original: For an SA-FL system with arbitrary system and data heterogeneity, if distributions P and Q satisfy Assumption 1 and are ( α, β ) -positively-related, then with probability at least 1 − δ for any δ ∈ (0 , 1) , it holds that
Revised: For an SA-FL system with arbitrary system and data heterogeneity, if distributions P and Q satisfy Assumption 1 and are ( α, β ) -positively-related, then with probability at least 1 − δ for any δ ∈ (0 , 1) , it holds that

Original: [Equation]
Revised: [Equation]

Original: where d H denotes the ﬁnite VC dimension for hypotheses class H , and parameters { P, Q, n T , n S , β , β Q } are deﬁned the same as before.
Revised: where d H denotes the ﬁnite VC dimension for hypotheses class H , and parameters { P, Q, n T , n S , β , β Q } are deﬁned the same as before.

Original: It is known that (see, e.g., (Hanneke, 2016))
Revised: It is known that (see, e.g., (Hanneke, 2016))

Original: the generalization error bound of centralized learning is (cid:101) O
Revised: the generalization error bound of centralized learning is (cid:101) O

Original: (( 1 n ) 1 2 − βQ )
Revised: (( 1 n ) 1 2 − βQ )

Original: (hiding logarithmic factors) with n samples in total and noise parameter β Q .
Revised: (hiding logarithmic factors) with n samples in total and noise parameter β Q .

Original: Note that when β ≥ 1 , the ﬁrst term in Eq.
Revised: Note that when β ≥ 1 , the ﬁrst term in Eq.

Original: (1) dominates.
Revised: (1) dominates.

Original: Hence, Theorem 2 implies that the generalization error bound for SA-FL matches that of centralized learning (with dataset size n T + n S ).
Revised: Hence, Theorem 2 implies that the generalization error bound for SA-FL matches that of centralized learning (with dataset size n T + n S ).

Original: Meanwhile, compared with solely training on server’s dataset T , SA-FL exhibits an improvement from (cid:101) O
Revised: Meanwhile, compared with solely training on server’s dataset T , SA-FL exhibits an improvement from (cid:101) O

Original: (( 1 n T ) 1 2 − βQ ) to (cid:101) O
Revised: (( 1 n T ) 1 2 − βQ ) to (cid:101) O

Original: (( 1 n T + n S ) β 2 − βQ ) .
Revised: (( 1 n T + n S ) β 2 − βQ ) .

Original: Note that SA-FL shares some similarity with the domain adaptation problem, where the learning is on Q but the results will be adapted to P .
Revised: Note that SA-FL shares some similarity with the domain adaptation problem, where the learning is on Q but the results will be adapted to P .

Original: In what follows, we offer some deeper insights between the two by answering two key questions: 1) What is the difference between SA-FL and domain adaptation (a.k.a. transfer learning)?
Revised: In what follows, we offer some deeper insights between the two by answering two key questions: 1) What is the difference between SA-FL and domain adaptation (a.k.a. transfer learning)?

Original: and 2) Why is SA-FL from Q to P PAC-learnable, but FL from D to P with partial client participation not PAC-learnable (as indicated in Theorem 1)?
Revised: and 2) Why is SA-FL from Q to P PAC-learnable, but FL from D to P with partial client participation not PAC-learnable (as indicated in Theorem 1)?

Original: To answer these questions, we illustrate the distribution relationships for domain adaptation and federated learning, in Fig. 1, respectively.
Revised: To answer these questions, we illustrate the distribution relationships for domain adaptation and federated learning, in Fig. 1, respectively.

Original: In domain adaptation, the target P and source Q distributions often have overlapping support but there also exists distinguishable difference .
Revised: In domain adaptation, the target P and source Q distributions often have overlapping support but there also exists distinguishable difference .

Original: In contrast, the two distributions P and Q in SA-FL happen to share exactly the same support with different density, since
Revised: In contrast, the two distributions P and Q in SA-FL happen to share exactly the same support with different density, since

Original: Q is a mixture of D and P .
Revised: Q is a mixture of D and P .

Original: As a result, the known bounds in domain adaptation (or transfer learning) are pessimistic in SA-FL.
Revised: As a result, the known bounds in domain adaptation (or transfer learning) are pessimistic in SA-FL.

Original: For example, the dist ( P, Q ) in d A -divergence and Y -divergence both have non-negligible gaps when applied to SA-FL.
Revised: For example, the dist ( P, Q ) in d A -divergence and Y -divergence both have non-negligible gaps when applied to SA-FL.

Original: Here in Theorem 2, we provide a generalization error bound in terms of the total sample size n T + n S , thus showing the beneﬁt of SA-FL.
Revised: Here in Theorem 2, we provide a generalization error bound in terms of the total sample size n T + n S , thus showing the beneﬁt of SA-FL.

Original: [Figure]
Revised: [Figure]

Original: P
Revised: P

Original: D
Revised: D

Original: Moreover, for SA-FL, only the auxiliary dataset T i.i.d.
Revised: Moreover, for SA-FL, only the auxiliary dataset T i.i.d.

Original: ∼
Revised: ∼

Original: P is directly available for the server.
Revised: P is directly available for the server.

Original: The clients’ datasets could be used in SA-FL training, but they are not directly accessible due to privacy constraints.
Revised: The clients’ datasets could be used in SA-FL training, but they are not directly accessible due to privacy constraints.

Original: Thus, previous methods in domain adaptation (e.g., importance weights-based methods in covariate shift adaptation (Sugiyama et al., 2007a;b)) are not applicable since they require the knowledge of density ratio between training and test datasets.
Revised: Thus, previous methods in domain adaptation (e.g., importance weights-based methods in covariate shift adaptation (Sugiyama et al., 2007a;b)) are not applicable since they require the knowledge of density ratio between training and test datasets.

Original: The key difference between FL and SA-FL lies in relations among D, P and Q .
Revised: The key difference between FL and SA-FL lies in relations among D, P and Q .

Original: For FL, the distance between
Revised: For FL, the distance between

Original: D and P under partial participation could be large due to system and data heterogeneity in the worst-case.
Revised: D and P under partial participation could be large due to system and data heterogeneity in the worst-case.

Original: More speciﬁcally, the support of D could be narrow enough to miss some part of P , resulting in non-vanishing error as indicated in Theorem 1.
Revised: More speciﬁcally, the support of D could be narrow enough to miss some part of P , resulting in non-vanishing error as indicated in Theorem 1.

Original: For SA-FL, distribution Q is a mixture of P and D ( Q = λ
Revised: For SA-FL, distribution Q is a mixture of P and D ( Q = λ

Original: 1 D + λ 2 P , with λ 1 , λ 2 ≥ 0 , λ 1 + λ 2 = 1 ), thus having the same support with P .
Revised: 1 D + λ 2 P , with λ 1 , λ 2 ≥ 0 , λ 1 + λ 2 = 1 ), thus having the same support with P .

Original: Hence, under Assumption 4, the PAC-learnability is guaranteed.
Revised: Hence, under Assumption 4, the PAC-learnability is guaranteed.

Original: Although we provide a promising bound to show the PAC-learnability of SA-FL in Theorem 2, the superiority of SA-FL over training solely with dataset T in server (i.e., (cid:101) O (( 1 n T ) 1 2 − βP ) ) is not always guaranteed as β → 0 (i.e., Q becomes increasingly different from P ).
Revised: Although we provide a promising bound to show the PAC-learnability of SA-FL in Theorem 2, the superiority of SA-FL over training solely with dataset T in server (i.e., (cid:101) O (( 1 n T ) 1 2 − βP ) ) is not always guaranteed as β → 0 (i.e., Q becomes increasingly different from P ).

Original: In what follows, we reveal under what conditions between P and Q could SA-FL perform no worse than centralized learning in terms of generalization error.
Revised: In what follows, we reveal under what conditions between P and Q could SA-FL perform no worse than centralized learning in terms of generalization error.

Original: Theorem 3 (SA-FL Being No Worse Than Centralized Learning) .
Revised: Theorem 3 (SA-FL Being No Worse Than Centralized Learning) .

Original: Consider an SA-FL system with arbitrary system and data heterogeneity.
Revised: Consider an SA-FL system with arbitrary system and data heterogeneity.

Original: If Assumption 1 holds and additionally ˆ R P
Revised: If Assumption 1 holds and additionally ˆ R P

Original: (ˆ h ∗ Q ) ≤ ˆ R P ( h ∗ Q ) and ε P ( h ∗ Q ) =
Revised: (ˆ h ∗ Q ) ≤ ˆ R P ( h ∗ Q ) and ε P ( h ∗ Q ) =

Original: O ( A ( n T , δ )) , where A ( n T , δ ) =
Revised: O ( A ( n T , δ )) , where A ( n T , δ ) =

Original: d H n T log( n T d H + 1 n T log( 1 δ )) , then with probability at least 1 − δ for any δ ∈ (0 , 1) , it holds that ε P (ˆ h ∗ Q ) =
Revised: d H n T log( n T d H + 1 n T log( 1 δ )) , then with probability at least 1 − δ for any δ ∈ (0 , 1) , it holds that ε P (ˆ h ∗ Q ) =

Original: (cid:101) O (cid:16) ( d H /n T ) 1 2 − βP (cid:17) .
Revised: (cid:101) O (cid:16) ( d H /n T ) 1 2 − βP (cid:17) .

Original: Other parameters are the same as deﬁned in Theorem 2.
Revised: Other parameters are the same as deﬁned in Theorem 2.

Original: Here, we remark that ε P ( h ∗ Q )
Revised: Here, we remark that ε P ( h ∗ Q )

Original: =
Revised: =

Original: O ( A ( n T , δ )) is a weaker condition than the ε P ( h ∗ Q ) =
Revised: O ( A ( n T , δ )) is a weaker condition than the ε P ( h ∗ Q ) =

Original: 0 condition and the covariate shift assumption ( P Y | X = Q Y | X ) used in the transfer learning literatures (Hanneke & Kpotufe, 2019; 2020).
Revised: 0 condition and the covariate shift assumption ( P Y | X = Q Y | X ) used in the transfer learning literatures (Hanneke & Kpotufe, 2019; 2020).

Original: Together with the condition ˆ R P (ˆ h ∗ Q ) ≤ ˆ R P ( h ∗ Q ) , the following intermediate result holds: ˆ R P (ˆ h ∗ Q ) − ˆ R P ( h ∗ P )
Revised: Together with the condition ˆ R P (ˆ h ∗ Q ) ≤ ˆ R P ( h ∗ Q ) , the following intermediate result holds: ˆ R P (ˆ h ∗ Q ) − ˆ R P ( h ∗ P )

Original: =
Revised: =

Original: O ( A ( n T , δ ))
Revised: O ( A ( n T , δ ))

Original: (see Lemma 2 in the supplementary material).
Revised: (see Lemma 2 in the supplementary material).

Original: Intuitively, this states that “if P and Q share enough similarity, then the difference of excess empirical error between ˆ h ∗ Q and h ∗ P on P can be bounded.”
Revised: Intuitively, this states that “if P and Q share enough similarity, then the difference of excess empirical error between ˆ h ∗ Q and h ∗ P on P can be bounded.”

Original: Thus, the excess error of ˆ h ∗ Q shares the same upper bound as that of ˆ h ∗ P in centralized learning.
Revised: Thus, the excess error of ˆ h ∗ Q shares the same upper bound as that of ˆ h ∗ P in centralized learning.

Original: Therefore, Theorem 3 implies that, under mild conditions,
Revised: Therefore, Theorem 3 implies that, under mild conditions,

Original: SA-FL guarantees the same generalization error upper bound as that of centralized learning with dataset T , hence being “no worse than” centralized learning with dataset T .
Revised: SA-FL guarantees the same generalization error upper bound as that of centralized learning with dataset T , hence being “no worse than” centralized learning with dataset T .

Original:  T HE SAFARI ALGORITHM
Revised:  T HE SAFARI ALGORITHM

Original: Although we have shown that SA-FL is PAC-learnable under partial client participation, we note that
Revised: Although we have shown that SA-FL is PAC-learnable under partial client participation, we note that

Original: SA-FL still follows the server-client architecture, which demands intensive communication between the server and clients.
Revised: SA-FL still follows the server-client architecture, which demands intensive communication between the server and clients.

Original: Hence, it is important to design communication-efﬁcient algorithms for SA-FL with a comparable level of communication complexity as that of conventional FL.
Revised: Hence, it is important to design communication-efﬁcient algorithms for SA-FL with a comparable level of communication complexity as that of conventional FL.

Original: In this section, we propose a new algorithm called SAFARI (server-aided federated averaging) algorithm for SA-FL and characterize its convergence guarantees.
Revised: In this section, we propose a new algorithm called SAFARI (server-aided federated averaging) algorithm for SA-FL and characterize its convergence guarantees.

Original: As shown in Algorithm 1, SAFARI iteratively performs the following three steps: 1) Server samples a subset of clients as in conventional FL and synchronize the latest global model x t with each participating clients (Line 3); 2) The server and all participating clients train the model based on local dataset (Lines 4-8).
Revised: As shown in Algorithm 1, SAFARI iteratively performs the following three steps: 1) Server samples a subset of clients as in conventional FL and synchronize the latest global model x r with each participating clients (Line 3); 2) The server and all participating clients train the model based on local dataset (Lines 4-8).

Original: Speciﬁcally, client initializes its local model with x t and then performs K local steps by the stochastic gradient descent method.
Revised: Speciﬁcally, client initializes its local model with x r and then performs K local steps by the stochastic gradient descent method.

Original: Then, each client sends its locally accumulated
Revised: Then, each client sends its locally accumulated

Original: Initialize do  5:
Revised: Initialize do 3: do 5:

Revised: Synchronization

Revised: Local

Original:  1 : x ir k{+ir,k +
+}
Revised: − 1 : x {+ir,k +
+}

Original: = 
Revised: = x

Original: ir k − η ∇ F i ( x ir k ξ 
Revised: ir,k − η ∇ F i ( x ir,k , ξ ir,k ) .

Original: Send ∆ it = − (cid:80) K − 1 k =0 ∇ F i ( x ir,k , ξ ir,k ) to server.
Revised: Send ∆ ir = − (cid:80) K − 1 k =0 ∇ F i ( x ir,k , ξ ir,k ) to server.

Original: 8: end for 9: for Server do 10:
Revised: 8: end for 9: for Server do 10:

Original: Local updates: 
Revised: Local updates: for

Original:  1 : x 0 r,k +1{++
+} = x 0 r,k − η ∇ F ( x 0 r,k , ξ 0 r,k ) , ∆ 0 t = − (cid:80) K − 1 k =0 ∇ F ( x 0 r,k , ξ 0 r,k ) .
Revised: K − 1 : x 0 r,k {++
+} = x 0 r,k − η ∇ F ( x 0 r,k , ξ 0 r,k ) , ∆ 0 r = − (cid:80) K − 1 k =0 ∇ F ( x 0 r,k , ξ 0 r,k ) .

Original: Receive ∆ ir , i ∈ S r , and normalize it: ˆ∆ir
Revised: Receive ∆ ir , i ∈ S r , and normalize it: ˆ∆

Original: [Equation] c [Equation]
Revised: ir =

Original:  ir[Equation] [Equation] (cid:107)
Revised: c r ∆ ir

Original: [Equation]
Revised: [Equation]

Original:  ir [Equation] ∆ 0 t (cid:107) .
Revised: ∆ 0 r (cid:107) ∆ ir − ∆ 0 r (cid:107) .

Original: Server Update: x r +1 = x
Revised: Server Update: x r +1 = x

Original: r + η (cid:16)

Original: [Equation]

Original: t

Original: [Equation] 1 | S [Equation] (cid:80)
Revised: r + η (cid:16) ∆ 0 r + 1 | S r | (cid:80)

Original: i [Equation] r ˆ∆ ir (cid:17) .
Revised: i ∈ S r ˆ∆ ir (cid:17) .

Original: 13: end for 14: end for update ∆ it back to the server.
Revised: 13: end for 14: end for update ∆ ir back to the server.

Original: Note the server simultaneously takes K local steps based on its auxiliary dataset (in Line 10).
Revised: Note the server simultaneously takes K local steps based on its auxiliary dataset (in Line 10).

Original: 3) Server aggregates and updates the global model (Lines 11-12).
Revised: 3) Server aggregates and updates the global model (Lines 11-12).

Original: Upon receiving the local update ∆ it , the server normalizes and rescales it by a hyper-parameter c t .
Revised: Upon receiving the local update ∆ ir , the server normalizes and rescales it by a hyper-parameter c r .

Original: Then, the server updates the global model by aggregating the normalized update ˆ∆ it and the server’s update ∆ 0 t based on its own auxiliary dataset.
Revised: Then, the server updates the global model by aggregating the normalized update ˆ∆ ir and the server’s update ∆ 0 r based on its own auxiliary dataset.

Original: Compared to FedAvg (McMahan et al., 2017) in FL, SAFARI shares the same communication and computation process from the client’s perspective.
Revised: Compared to FedAvg (McMahan et al., 2017) in FL, SAFARI shares the same communication and computation process from the client’s perspective.

Original: Hence, it enjoys the same level of communication efﬁciency and privacy beneﬁts.
Revised: Hence, it enjoys the same level of communication efﬁciency and privacy beneﬁts.

Original: Before conducting the convergence performance analysis, we ﬁrst state two commonly used assumptions in the FL literature.
Revised: Before conducting the convergence performance analysis, we ﬁrst state two commonly used assumptions in the FL literature.

Original: Assumption 2.
Revised: Assumption 2.

Original: ( L -Lipschitz Continuous Gradient)
Revised: ( L -Lipschitz Continuous Gradient)

Original: There exists a constant L > 0 , such that
Revised: There exists a constant L > 0 , such that

Original: [Equation]
Revised: [Equation]

Original: Assumption 3.
Revised: Assumption 3.

Original: (Unbiased Stochastic Gradient with Bounded Variance)
Revised: (Unbiased Stochastic Gradient with Bounded Variance)

Original: The stochastic gradient is unbiased, i.e., E [ ∇ F ( x , ξ )]
Revised: The stochastic gradient is unbiased, i.e., E [ ∇ F ( x , ξ )]

Original: = ∇ F ( x ) and E [ (cid:107)∇ F ( x , ξ ) − ∇ F ( x ) (cid:107) 2 ] ≤ σ 2 .partial
Revised: = ∇ F ( x ) and E [ (cid:107)∇ F ( x , ξ ) − ∇ F ( x ) (cid:107) 2 ] ≤ σ 2 .partial

Original: With the assumptions above, we are now in the position to analyze the convergence of SAFARI .
Revised: With the assumptions above, we are now in the position to analyze the convergence of SAFARI .

Original: Theorem 4 (Convergence Rate for SAFARI ) .
Revised: Theorem 4 (Convergence Rate for SAFARI ) .

Original: Under Assumptions 2 and 3, let constant learning rate η satisfy ( 12 − 4 LKη − 20 K
Revised: Under Assumptions 2 and 3, let constant learning rate η satisfy ( 12 − 4 LKη − 20 K

Original: ( L + 4 KL 3 η ) η 2 ) > 0 .
Revised: ( L + 4 KL 3 η ) η 2 ) > 0 .

Original: Then, the sequence { x r } generated by the
Revised: Then, the sequence { x r } generated by the

Original: SAFARI algorithm satisﬁes:
Revised: SAFARI algorithm satisﬁes:

Original: [Equation]
Revised: [Equation]

Original: where c is a constant and x ∗ denotes an optimal solution.
Revised: where c is a constant and x ∗ denotes an optimal solution.

Original: Theorem 4 implies an O (1 /R ) convergence rate to a neighborhood of a stationary point.
Revised: Theorem 4 implies an O (1 /R ) convergence rate to a neighborhood of a stationary point.

Original: Furthermore, by choosing parameters { c r } and the learning rate η appropriately, we have the following convergence rate to a stationary point:
Revised: Furthermore, by choosing parameters { c r } and the learning rate η appropriately, we have the following convergence rate to a stationary point:

Original: Corollary 1.
Revised: Corollary 1.

Original: If (cid:80) R − 1 t =

Revised: If (cid:80) R − 1 t =

Original: c 2 r is bounded and learning rate η = 1 √ KR , the convergence rate of SAFARI is:
Revised: c 2 r is bounded and learning rate η = 1 √ KR , the convergence rate of SAFARI is:

Original: [Equation]
Revised: [Equation]

Original: In Theorem 4 and Corollary 1, we show the convergence guarantee of SAFARI under no extra assumptions on the data and system heterogeneity (client participation), which corroborates the learnability
Revised: In Theorem 4 and Corollary 1, we show the convergence guarantee of SAFARI under no extra assumptions on the data and system heterogeneity (client participation), which corroborates the learnability

Original: [Figure]
Revised: [Figure]

Original: analysis in Section 3.
Revised: analysis in Section 3.

Original: Hence, in the worst-case scenarios, convergence rate O (1 / ( K 1 / 2 R 1 / 2 )) is achieved for sufﬁciently large R and K ≤ R .
Revised: Hence, in the worst-case scenarios, convergence rate O (1 / ( K 1 / 2 R 1 / 2 )) is achieved for sufﬁciently large R and K ≤ R .

Original: In comparison, a non-vanishing error term emerges consistently for the same setting in FL (Yang et al., 2022).
Revised: In comparison, a non-vanishing error term emerges consistently for the same setting in FL (Yang et al., 2022).

Original: This veriﬁes the superiority of SA-FL over conventional FL with partial client participation.
Revised: This veriﬁes the superiority of SA-FL over conventional FL with partial client participation.

Original: Note that Corollary 1 requires a convergent series { c 2 r } .
Revised: Note that Corollary 1 requires a convergent series { c 2 r } .

Original: This can be relaxed to (cid:80) R − 1 r =

Revised: This can be relaxed to (cid:80) R − 1 r =

Original: c 2 r
Revised: c 2 r

Original: =
Revised: =

Original: O
Revised: O

Original: (min { KR, K 3 / 2 R 1 / 2 } ) to maintain the same
Revised: (min { KR, K 3 / 2 R 1 / 2 } ) to maintain the same

Original: O (1 / ( K 1 / 2 R 1 / 2 )) rate.
Revised: O (1 / ( K 1 / 2 R 1 / 2 )) rate.

Original: It can be readily veriﬁed that p -series ( c r = r − p ) satisﬁes the condition.
Revised: It can be readily veriﬁed that p -series ( c r = r − p ) satisﬁes the condition.

Original: Also, we will see later in numerical results that, compared to FedAvg, our SAFARI has a faster convergence as the size of auxiliary server dataset increases.
Revised: Also, we will see later in numerical results that, compared to FedAvg, our SAFARI has a faster convergence as the size of auxiliary server dataset increases.

Original: It is also worth noting that SA-FL can be used to mitigate Byzantine attacks to FL systems.
Revised: It is also worth noting that SA-FL can be used to mitigate Byzantine attacks to FL systems.

Original: For example, empirical success of utilizing an auxiliary dataset and normalization at the server has been shown in (Cao et al., 2021) in defending against Byzantine attacks in FL systems.
Revised: For example, empirical success of utilizing an auxiliary dataset and normalization at the server has been shown in (Cao et al., 2021) in defending against Byzantine attacks in FL systems.

Original: Lastly, note that the convergence rate bound in Corollarydoes not explicitly depend on the number of participating clients m .
Revised: Lastly, note that the convergence rate bound in Corollarydoes not explicitly depend on the number of participating clients m .

Original: It is thus an interesting future work to study whether there exist algorithms for SA-FL that theoretically guarantee a “linear speedup effect” as the number of participating clients increases under partial client participation.
Revised: It is thus an interesting future work to study whether there exist algorithms for SA-FL that theoretically guarantee a “linear speedup effect” as the number of participating clients increases under partial client participation.

Original:  N UMERICAL RESULTS
Revised:  N UMERICAL RESULTS

Original: In this section, we conduct numerical experiments to verify our theoretical results using 1) logistic regression (LR) on MNIST dataset (LeCun et al., 1998) and 2) convolutional neural network (CNN) on CIFAR-10 dataset (Krizhevsky et al., 2009).
Revised: In this section, we conduct numerical experiments to verify our theoretical results using 1) logistic regression (LR) on MNIST dataset (LeCun et al., 1998) and 2) convolutional neural network (CNN) on CIFAR-10 dataset (Krizhevsky et al., 2009).

Original: To simulate data heterogeneity, we distribute the data into each client evenly in a label-based partition, following the same process as in previous works (e.g., (McMahan et al., 2017;
Revised: To simulate data heterogeneity, we distribute the data into each client evenly in a label-based partition, following the same process as in previous works (e.g., (McMahan et al., 2017;

Original: Yang et al., 2021b; Li et al., 2020b)).
Revised: Yang et al., 2021b; Li et al., 2020b)).

Original: As a result, we can use a parameter p to represent the classes of labels in each client’s dataset, which serves as an index of data heterogeneity level (non-i.i.d. index).
Revised: As a result, we can use a parameter p to represent the classes of labels in each client’s dataset, which serves as an index of data heterogeneity level (non-i.i.d. index).

Original: The smaller p -value, the more heterogeneous the data among clients.
Revised: The smaller p -value, the more heterogeneous the data among clients.

Original: To mimic partial client participation, we force s clients to be excluded.
Revised: To mimic partial client participation, we force s clients to be excluded.

Original: We can use s as an index to represent the degree of such partial client participation.
Revised: We can use s as an index to represent the degree of such partial client participation.

Original: In our experiments, there are
Revised: In our experiments, there are

Original: M = 10 clients in total, and m = 5 clients participate in the training in each communication round, who are uniformly sampled from the M − s clients.
Revised: M = 10 clients in total, and m = 5 clients participate in the training in each communication round, who are uniformly sampled from the M − s clients.

Original: We use two algorithms as baselines to compare with our algorithm: i) the FedAvg algorithm without any auxiliary dataset at the server; and ii) the
Revised: We use two algorithms as baselines to compare with our algorithm: i) the FedAvg algorithm without any auxiliary dataset at the server; and ii) the

Original: SGD algorithm for centralized learning with dataset only at the server (without any clients).
Revised: SGD algorithm for centralized learning with dataset only at the server (without any clients).

Original: Due to space limitation, we highlight four key observations in this section, and relegate the details of the experiments and additional experiments to the supplementary material.
Revised: Due to space limitation, we highlight four key observations in this section, and relegate the details of the experiments and additional experiments to the supplementary material.

Original: ) Performance Degradation of Partial Client Participation: We ﬁrst show the test accuracy
Revised: ) Performance Degradation of Partial Client Participation: We ﬁrst show the test accuracy

Original: of
Revised: of

Original: FedAvg on MNIST for different values of non-i.i.d.
Revised: FedAvg on MNIST for different values of non-i.i.d.

Original: index p and partial client index s in Fig. 2(a).
Revised: index p and partial client index s in Fig. 2(a).

Original: For nearly homogeneous data (e.g., from p = 10 to p = 5 ), partial client participation has negligible impacts on test accuracy.
Revised: For nearly homogeneous data (e.g., from p = 10 to p = 5 ), partial client participation has negligible impacts on test accuracy.

Original: However, for highly non-i.i.d.
Revised: However, for highly non-i.i.d.

Original: cases, partial client participation results in dramatic performance degradation.
Revised: cases, partial client participation results in dramatic performance degradation.

Original: Speciﬁcally, for p = 1 , the test accuracy for s = 4 is only 57% ,
Revised: Speciﬁcally, for p = 1 , the test accuracy for s = 4 is only 57% ,

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: yielding a large degradation ( 35% ) compared to that of s = 0 .
Revised: yielding a large degradation ( 35% ) compared to that of s = 0 .

Original: This is consistent with the worst-case analysis in Theorem 1 and also the main motivation of SA-FL.
Revised: This is consistent with the worst-case analysis in Theorem 1 and also the main motivation of SA-FL.

Original: 2) Improvement of the SAFARI Algorithm under Partial Client Participation: In Table 1, we show the test accuracy improvement of our SAFARI algorithm compared with that of FedAvg in standard FL.
Revised: 2) Improvement of the SAFARI Algorithm under Partial Client Participation: In Table 1, we show the test accuracy improvement of our SAFARI algorithm compared with that of FedAvg in standard FL.

Original: The key observation is that, with a small amount of auxiliary data at the server, there is a signiﬁcant increase of test accuracy for our SAFARI algorithm.
Revised: The key observation is that, with a small amount of auxiliary data at the server, there is a signiﬁcant increase of test accuracy for our SAFARI algorithm.

Original: For example, with only 50 data samples at the server ( 0 . 1% of the total training data), there is a 12 .
Revised: For example, with only 50 data samples at the server ( 0 . 1% of the total training data), there is a 12 .

Original: 32% test accuracy increase.
Revised: 32% test accuracy increase.

Original: With 500 data samples, the improvement reaches 27 .
Revised: With 500 data samples, the improvement reaches 27 .

Original: This veriﬁes the effectiveness of our SA-FL framework and our SAFARI algorithm.
Revised: This veriﬁes the effectiveness of our SA-FL framework and our SAFARI algorithm.

Original: Another observation is that for nearly homogeneous case (e.g., from p = 10 to p = 5 ), there is no statistical difference with or without auxiliary data at the server (denoted by ‘-’ in Table 1.
Revised: Another observation is that for nearly homogeneous case (e.g., from p = 10 to p = 5 ), there is no statistical difference with or without auxiliary data at the server (denoted by ‘-’ in Table 1.

Original: This is consistent with the previous observations of negligible degradation in cases with homogeneous data across clients.
Revised: This is consistent with the previous observations of negligible degradation in cases with homogeneous data across clients.

Original: 3) Beneﬁts of Collaboration Even with Partial Client Participation: In Table 2, we show the comparison between our SAFARI algorithm under partial client participation ( s = 4 ) and SGD for centralized learning (i.e., only with data at the server).
Revised: 3) Beneﬁts of Collaboration Even with Partial Client Participation: In Table 2, we show the comparison between our SAFARI algorithm under partial client participation ( s = 4 ) and SGD for centralized learning (i.e., only with data at the server).

Original: This comparison is to illustrate the beneﬁt of collaborations from the clients.
Revised: This comparison is to illustrate the beneﬁt of collaborations from the clients.

Original: We can see these collaborations from the clients signiﬁcantly improve the performance for the nearly homogeneous case, especially when the size of the server dataset is small (cf.
Revised: We can see these collaborations from the clients signiﬁcantly improve the performance for the nearly homogeneous case, especially when the size of the server dataset is small (cf.

Original: the columns of p = 10 and p = 5 ).
Revised: the columns of p = 10 and p = 5 ).

Original: However, for highly heterogeneous cases, it shows no obvious improvement from the collaboration of clients if participation is partial.
Revised: However, for highly heterogeneous cases, it shows no obvious improvement from the collaboration of clients if participation is partial.

Original: This conﬁrms our theoretical analysis in Theorem 2.
Revised: This conﬁrms our theoretical analysis in Theorem 2.

Original: Speciﬁcally, highly partial client participation and heterogeneous data render a small β -value, and so there is no clear beneﬁt of collaboration in FL over centralized learning only on small auxiliary dataset due to the signiﬁcant distribution deviation.
Revised: Speciﬁcally, highly partial client participation and heterogeneous data render a small β -value, and so there is no clear beneﬁt of collaboration in FL over centralized learning only on small auxiliary dataset due to the signiﬁcant distribution deviation.

Original: 4) Convergence Speedup of SAFARI with Larger Server Dataset: In this experiment, we illustrate the speedup effect of SAFARI numerically as the size of server dataset increases.
Revised: 4) Convergence Speedup of SAFARI with Larger Server Dataset: In this experiment, we illustrate the speedup effect of SAFARI numerically as the size of server dataset increases.

Original: In Fig. 2(b), we show the convergence processes of SAFARI on CIFAR-10 for partial client participation ( s = 4 ) and non-i.i.d.
Revised: In Fig. 2(b), we show the convergence processes of SAFARI on CIFAR-10 for partial client participation ( s = 4 ) and non-i.i.d.

Original: data ( p = 1 ).
Revised: data ( p = 1 ).

Original: We can see clearly that the convergence of SAFARI is accelerating and the test accuracy increases as more data are employed at the server.
Revised: We can see clearly that the convergence of SAFARI is accelerating and the test accuracy increases as more data are employed at the server.

Original: In this experiment setting, we also plot the convergence of FedAvg in Fig. 2(b).
Revised: In this experiment setting, we also plot the convergence of FedAvg in Fig. 2(b).

Original: It can be seen that all three cases of SAFARI converge faster than FedAvg in this experiment.
Revised: It can be seen that all three cases of SAFARI converge faster than FedAvg in this experiment.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we rigorously investigated the server-aided federated learning (SA-FL) framework, which has been increasingly adopted in practice to mitigate the impacts of partial client participation in conventional FL.
Revised: In this paper, we rigorously investigated the server-aided federated learning (SA-FL) framework, which has been increasingly adopted in practice to mitigate the impacts of partial client participation in conventional FL.

Original: In SA-FL, the key idea is to deploy an auxiliary dataset at the server to reduce the distribution deviation induced by partial client participation.
Revised: In SA-FL, the key idea is to deploy an auxiliary dataset at the server to reduce the distribution deviation induced by partial client participation.

Original: To characterize the beneﬁts of
Revised: To characterize the beneﬁts of

Original: SA-FL, we ﬁrst showed that conventional FL is not PAC-learnable under partial client participation by establishing a fundamental generalization error lower bound.
Revised: SA-FL, we ﬁrst showed that conventional FL is not PAC-learnable under partial client participation by establishing a fundamental generalization error lower bound.

Original: Then, we showed that SA-FL is able to revive the PAC-learnability of conventional FL under partial client participation.
Revised: Then, we showed that SA-FL is able to revive the PAC-learnability of conventional FL under partial client participation.

Original: Upon resolving the PAC-learnability challenge, we proposed a new SAFARI (server-aided federated averaging) algorithm that enjoys convergence guarantee and the same level of communication efﬁciency as that of conventional FL.
Revised: Upon resolving the PAC-learnability challenge, we proposed a new SAFARI (server-aided federated averaging) algorithm that enjoys convergence guarantee and the same level of communication efﬁciency as that of conventional FL.

Original: Extensive numerical results also validated our theoretical ﬁndings.
Revised: Extensive numerical results also validated our theoretical ﬁndings.