Original: G ENERALIZATION B OUNDS FOR F EDERATED L EARNING : F AST R ATES , U NPARTICIPATING C LIENTS AND U NBOUNDED L OSSES
Revised: G ENERALIZATION B OUNDS FOR F EDERATED L EARNING : F AST R ATES , U NPARTICIPATING C LIENTS AND U NBOUNDED L OSSES

Original: A BSTRACT
Revised: A BSTRACT

Original: In federated learning, the underlying data distributions may be different across clients.
Revised: In federated learning, the underlying data distributions may be different across clients.

Original: This paper provides a theoretical analysis of generalization error of federated learning, which captures both heterogeneity and relatedness of the distributions.
Revised: This paper provides a theoretical analysis of generalization error of federated learning, which captures both heterogeneity and relatedness of the distributions.

Original: In particular, we assume that the heterogeneous distributions are sampled from a meta-distribution.
Revised: In particular, we assume that the heterogeneous distributions are sampled from a meta-distribution.

Original: In this two-level distribution framework, we characterize the generalization error not only for clients participating in the training but also for unparticipating clients.
Revised: In this two-level distribution framework, we characterize the generalization error not only for clients participating in the training but also for unparticipating clients.

Original: We first show that the generalization error for unparticipating clients can be bounded by participating generalization error and participating gap caused by clients’ sampling.
Revised: We first show that the generalization error for unparticipating clients can be bounded by participating generalization error and participating gap caused by clients’ sampling.

Original: We further establish fast learning bounds of order O ( 1 mn + 1 m ) for unparticipating clients, where m is the number of clients and n is the sample size at each client.
Revised: We further establish fast learning bounds of order O ( 1 mn + 1 m ) for unparticipating clients, where m is the number of clients and n is the sample size at each client.

Original: To our knowledge, the obtained fast bounds are state-of-the-art in the two-level distribution framework.
Revised: To our knowledge, the obtained fast bounds are state-of-the-art in the two-level distribution framework.

Original: Moreover, previous theoretical results mostly require the loss function to be bounded.
Revised: Moreover, previous theoretical results mostly require the loss function to be bounded.

Original: We derive convergence bounds of order
Revised: We derive convergence bounds of order

Original: O ( 1 √ mn + 1 √ m ) under unbounded assumptions, including sub-exponential and sub-Weibull losses
Revised: O ( 1 √ mn + 1 √ m ) under unbounded assumptions, including sub-exponential and sub-Weibull losses

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In federated learning, a common model is trained based on the collaboration of the participating clients holding local data samples (McMahan et al., 2017).
Revised: In federated learning, a common model is trained based on the collaboration of the participating clients holding local data samples (McMahan et al., 2017).

Original: Typically, the underlying distributions vary across clients since the data-generating processes are affected by the local environment.
Revised: Typically, the underlying distributions vary across clients since the data-generating processes are affected by the local environment.

Original: Federated learning is heterogeneous in the scenario where local distributions are different (Wang et al., 2021).
Revised: Federated learning is heterogeneous in the scenario where local distributions are different (Wang et al., 2021).

Original: Most existing experimental and theoretical results focus on the convergence of optimization on training datasets (Li et al., 2020b; Karimireddy et al., 2020; Mitra et al., 2021; Mishchenko et al., 2022;
Revised: Most existing experimental and theoretical results focus on the convergence of optimization on training datasets (Li et al., 2020b; Karimireddy et al., 2020; Mitra et al., 2021; Mishchenko et al., 2022;

Original: Yun et al., 2022).
Revised: Yun et al., 2022).

Original: The generalization error, which is more natural and important in machine leanring, seems not to have been carefully examined in heterogeneous federated learning.
Revised: The generalization error, which is more natural and important in machine leanring, seems not to have been carefully examined in heterogeneous federated learning.

Original: As a key performance indicator of the machine learning model, generalization error measures the performance of a trained model by its population risk with the corresponding distribution.
Revised: As a key performance indicator of the machine learning model, generalization error measures the performance of a trained model by its population risk with the corresponding distribution.

Original: However, existing generalization results are generally derived for clients participating in the training, which only captures the performance of the learned model on seen distributions during training (Mohri et al., 2019;
Revised: However, existing generalization results are generally derived for clients participating in the training, which only captures the performance of the learned model on seen distributions during training (Mohri et al., 2019;

Original: Chen et al., 2021; Masiha et al., 2021).
Revised: Chen et al., 2021; Masiha et al., 2021).

Original: In practice, the probability that a client participates in the federated training is affected by many factors such as the reliability of network connections or the availability of the client.
Revised: In practice, the probability that a client participates in the federated training is affected by many factors such as the reliability of network connections or the availability of the client.

Original: The realistic participation ratio may be slow and a variety of clients never have a chance to participate during the training process (Kairouz et al., 2021; Li et al., 2020a; Yuan et al., 2021).
Revised: The realistic participation ratio may be slow and a variety of clients never have a chance to participate during the training process (Kairouz et al., 2021; Li et al., 2020a; Yuan et al., 2021).

Original: Though the training process is operated only on participating clients, the trained model will be used by both unparticipating and participating clients.
Revised: Though the training process is operated only on participating clients, the trained model will be used by both unparticipating and participating clients.

Original: Since the data distributions of unparticipating clients are different from that of participating clients, it is natural and emergent to ask the following question:
Revised: Since the data distributions of unparticipating clients are different from that of participating clients, it is natural and emergent to ask the following question:

Original: Would the unparticipating clients benefit from the model trained by participating clients?
Revised: Would the unparticipating clients benefit from the model trained by participating clients?

Original: ∗ Corresponding Author.
Revised: ∗ Corresponding Author.

Original: To answer this question theoretically, we take the participation gap into account in the analysis of generalization error, which is generally ignored by existing works.
Revised: To answer this question theoretically, we take the participation gap into account in the analysis of generalization error, which is generally ignored by existing works.

Original: In addition to the ignored participating gap, existing theoretical results on the generalization error of heterogeneous federated learning have two more limitations to our knowledge.
Revised: In addition to the ignored participating gap, existing theoretical results on the generalization error of heterogeneous federated learning have two more limitations to our knowledge.

Original: First, all previous learning rates in probability form are of the order O ( 1 √ mn ) , where m is the number of clients and n is the sample size at each client (Mohri et al., 2019).
Revised: First, all previous learning rates in probability form are of the order O ( 1 √ mn ) , where m is the number of clients and n is the sample size at each client (Mohri et al., 2019).

Original: We note that faster rates of order O ( 1 mn ) are derived in (Chen et al., 2021).
Revised: We note that faster rates of order O ( 1 mn ) are derived in (Chen et al., 2021).

Original: However, their learning rates are in expectation form.
Revised: However, their learning rates are in expectation form.

Original: Faster learning rates in probability form haven’t been derived even only for participating clients.
Revised: Faster learning rates in probability form haven’t been derived even only for participating clients.

Original: The guarantees in-expectation form reflect the average performance of the model trained based on the randomly sampled datasets.
Revised: The guarantees in-expectation form reflect the average performance of the model trained based on the randomly sampled datasets.

Original: The theoretical bounds in probability form, which we focus on in this paper, reflect the performance of a single sampling on datasets (Klochkov & Zhivotovskiy, 2021;
Revised: The theoretical bounds in probability form, which we focus on in this paper, reflect the performance of a single sampling on datasets (Klochkov & Zhivotovskiy, 2021;

Original: Kanade et al., 2022; Sefidgaran et al., 2022a).
Revised: Kanade et al., 2022; Sefidgaran et al., 2022a).

Original: Second, most previous generalization bounds are derived by assuming that the loss function is bounded.
Revised: Second, most previous generalization bounds are derived by assuming that the loss function is bounded.

Original: However, there are a variety of learning problems that do not satisfy this assumption.
Revised: However, there are a variety of learning problems that do not satisfy this assumption.

Original: This includes regression problems where unbounded noise is added to labels (Kuchibhotla & Patra, 2022; Kuchibhotla & Chakrabortty, 2018; Zhang & Zhou, 2018), clustering tasks with heavy-tailed distribution (Paul et al., 2021;
Revised: This includes regression problems where unbounded noise is added to labels (Kuchibhotla & Patra, 2022; Kuchibhotla & Chakrabortty, 2018; Zhang & Zhou, 2018), clustering tasks with heavy-tailed distribution (Paul et al., 2021;

Original: Vellal et al., 2022), domain adaptation, and so on.
Revised: Vellal et al., 2022), domain adaptation, and so on.

Original: Notable exception works in this direction include (Barnes et al., 2022) and (Sefidgaran et al., 2022b).
Revised: Notable exception works in this direction include (Barnes et al., 2022) and (Sefidgaran et al., 2022b).

Original: However, their results are established under the assumption that local clients are homogeneous, which is highly restrictive in the general federated scenario.
Revised: However, their results are established under the assumption that local clients are homogeneous, which is highly restrictive in the general federated scenario.

Original: In this paper, we assume that data distributions of participating and unparticipating clients are drawn from a meta-distribution P .
Revised: In this paper, we assume that data distributions of participating and unparticipating clients are drawn from a meta-distribution P .

Original: We argue that this assumption is reasonable in practice.
Revised: We argue that this assumption is reasonable in practice.

Original: For instance, in cross-device federated learning, the number of total clients is generally large and it is natural to assume that there exists a meta-distribution (Reisizadeh et al., 2020; Wang et al., 2021).
Revised: For instance, in cross-device federated learning, the number of total clients is generally large and it is natural to assume that there exists a meta-distribution (Reisizadeh et al., 2020; Wang et al., 2021).

Original: In this learning scenario, we assume that the total number of clients is M .
Revised: In this learning scenario, we assume that the total number of clients is M .

Original: Among all these M clients, only m clients have a chance to participate in the training phase, which means that the training process only involves the m distributions { D i } mi =1 .
Revised: Among all these M clients, only m clients have a chance to participate in the training phase, which means that the training process only involves the m distributions { D i } mi =1 .

Original: Note that the total number M and the number of unparticipating clients/distributions is generally larger than m (Hu et al., 2022; Xu & Wang, 2020;
Revised: Note that the total number M and the number of unparticipating clients/distributions is generally larger than m (Hu et al., 2022; Xu & Wang, 2020;

Original: Yang et al., 2020).
Revised: Yang et al., 2020).

Original: Practically, the model is trained based on datasets { S i } mi =1 , where S i is the dataset located in client i and is sampled from D i .
Revised: Practically, the model is trained based on datasets { S i } mi =1 , where S i is the dataset located in client i and is sampled from D i .

Original: This two-level framework not only captures the heterogeneity of clients’ distributions but also reflects the relatedness of the distributions.
Revised: This two-level framework not only captures the heterogeneity of clients’ distributions but also reflects the relatedness of the distributions.

Original: Thanks to this framework, we are allowed to characterize the generalization performance of both participating distributions and unparticipating distributions.
Revised: Thanks to this framework, we are allowed to characterize the generalization performance of both participating distributions and unparticipating distributions.

Original: A similar framework has been used by several recent literature (Yuan et al., 2021; Reisizadeh et al., 2020; Wang et al., 2021).
Revised: A similar framework has been used by several recent literature (Yuan et al., 2021; Reisizadeh et al., 2020; Wang et al., 2021).

Original: However, these works mainly focus on the optimization performance or only involve experimental results on the generalization.
Revised: However, these works mainly focus on the optimization performance or only involve experimental results on the generalization.

Original: The objective of this work is to provide theoretical results on generalization error in this framework.
Revised: The objective of this work is to provide theoretical results on generalization error in this framework.

Original: Our contributions are summarized as follows
Revised: Our contributions are summarized as follows

Original: • We provide a systematic analysis of the generalization error of federated learning in the two-level framework, which captures the missed participating gap in the existing works.
Revised: • We provide a systematic analysis of the generalization error of federated learning in the two-level framework, which captures the missed participating gap in the existing works.

Original: This two-level framework captures both heterogeneity and relatedness of clients’ distributions.
Revised: This two-level framework captures both heterogeneity and relatedness of clients’ distributions.

Original: Moreover, all learning bounds presented in this paper are in probability form instead of expectation form.
Revised: Moreover, all learning bounds presented in this paper are in probability form instead of expectation form.

Original: • We derive fast learning rates in the empirical risk minimization setting.
Revised: • We derive fast learning rates in the empirical risk minimization setting.

Original: The unparticipating error is bounded by two terms.
Revised: The unparticipating error is bounded by two terms.

Original: One is participating error.
Revised: One is participating error.

Original: The other is the participation gap results from missing clients in the training.
Revised: The other is the participation gap results from missing clients in the training.

Original: Our participating bounds and unparticipating bounds are of order O ( 1 mn ) and O ( 1 mn + 1 m ) , respectively.
Revised: Our participating bounds and unparticipating bounds are of order O ( 1 mn ) and O ( 1 mn + 1 m ) , respectively.

Original: • We study the learning bounds for unbounded loss functions, including sub-gaussian, subexponential, and heavy-tailed losses.
Revised: • We study the learning bounds for unbounded loss functions, including sub-gaussian, subexponential, and heavy-tailed losses.

Original: Small-ball methods and concentration inequalities for unbounded random variables are used in the unbounded setting.
Revised: Small-ball methods and concentration inequalities for unbounded random variables are used in the unbounded setting.

Original: Our bounds are comparable with the existing results with bounded assumptions.
Revised: Our bounds are comparable with the existing results with bounded assumptions.

Original: The rest of the paper is organized as follows.
Revised: The rest of the paper is organized as follows.

Original: In Section 2, we describe the two-level distribution framework and provide basic theoretical results in this framework.
Revised: In Section 2, we describe the two-level distribution framework and provide basic theoretical results in this framework.

Original: In Section 3, we derive fast generalization bounds.
Revised: In Section 3, we derive fast generalization bounds.

Original: In Section 4, we go beyond the bounded assumption and provide the generalization bounds for unbounded losses such as sub-exponential and sub-Weibull losses.
Revised: In Section 4, we go beyond the bounded assumption and provide the generalization bounds for unbounded losses such as sub-exponential and sub-Weibull losses.

Original: In Section 5, we discuss related work in the directly of generalization analysis of heterogeneous federated learning.
Revised: In Section 5, we discuss related work in the directly of generalization analysis of heterogeneous federated learning.

Original: Finally, we conclude this paper in Section 6.
Revised: Finally, we conclude this paper in Section 6.

Original: All proofs are postponed to the appendix.
Revised: All proofs are postponed to the appendix.

Original: [Figure]
Revised: [Figure]

Original:  T WO - LEVEL D ISTRIBUTION F RAMEWORK
Revised:  T WO - LEVEL D ISTRIBUTION F RAMEWORK

Original: Let X denote the input space and Y ⊂ R the output space.
Revised: Let X denote the input space and Y ⊂ R the output space.

Original: For simplicity, we denote Z =
Revised: For simplicity, we denote Z =

Original: ( X, Y ) the random variable with support Z = X ×Y .
Revised: ( X, Y ) the random variable with support Z = X ×Y .

Original: Let D denote the set of all probability distributions on
Revised: Let D denote the set of all probability distributions on

Original: Z and P is a meta-distribution on D .
Revised: Z and P is a meta-distribution on D .

Original: The assumption of meta-distribution is reasonable especially in cross-device federated learning scenario, where the local devices may be a large population of mobile phones.
Revised: The assumption of meta-distribution is reasonable especially in cross-device federated learning scenario, where the local devices may be a large population of mobile phones.

Original: As shown in Figure 1, in this two-level distribution framework, we assume the total number of clients is M (may be infinity) and the number of clients participating in training is m .
Revised: As shown in Figure 1, in this two-level distribution framework, we assume the total number of clients is M (may be infinity) and the number of clients participating in training is m .

Original: It is worth emphasizing that M is generally larger than m owing to unreliable network connections.
Revised: It is worth emphasizing that M is generally larger than m owing to unreliable network connections.

Original: We denote by D i the distribution associated to client i and assume { D 1 , · · · , D m } are independently sampled from D according to P .
Revised: We denote by D i the distribution associated to client i and assume { D 1 , · · · , D m } are independently sampled from D according to P .

Original: Data sample S i
Revised: Data sample S i

Original: = { Z ji } nj =1 located on participating client i is made of n i.i.d realizations of Z following D i .
Revised: = { Z ji } nj =1 located on participating client i is made of n i.i.d realizations of Z following D i .

Original: The global model is trained based on { S i } mi = and will be used by all M clients.
Revised: The global model is trained based on { S i } mi = and will be used by all M clients.

Original: Two-level distribution framework allows us to measure the performance of the global model with respect to clients’ distribution P , which quantifies both the participation gap (caused by client sampling) and participating error (caused by data sampling from participating distributions).
Revised: Two-level distribution framework allows us to measure the performance of the global model with respect to clients’ distribution P , which quantifies both the participation gap (caused by client sampling) and participating error (caused by data sampling from participating distributions).

Original: Throughout the paper we denote F by F =
Revised: Throughout the paper we denote F by F =

Original: { z (cid:55)→ ℓ ( h, z ) : h ∈ H} .
Revised: { z (cid:55)→ ℓ ( h, z ) : h ∈ H} .

Original: Moreover, we use Z i =
Revised: Moreover, we use Z i =

Original: ( X i , Y i ) to represent the random variables across two-level framework.
Revised: ( X i , Y i ) to represent the random variables across two-level framework.

Original: That is, E [ Z i ] =
Revised: That is, E [ Z i ] =

Original: E D i ∼ P
Revised: E D i ∼ P

Original: [ E Z i ∼ D
Revised: [ E Z i ∼ D

Original: i [ Z i ]] .
Revised: i [ Z i ]] .

Original: Let the hypothesis space H be a family of real-valued functions defined on X .
Revised: Let the hypothesis space H be a family of real-valued functions defined on X .

Original: The loss function ℓ : Y × Y → R
Revised: The loss function ℓ : Y × Y → R

Original: + is a non-negative function.
Revised: + is a non-negative function.

Original: We denote the population risk L P ( h ) by L P ( h ) =
Revised: We denote the population risk L P ( h ) by L P ( h ) =

Original: E
Revised: E

Original: D i ∼ P
Revised: D i ∼ P

Original: [ E Z ∼ D i [ ℓ ( h ( X ) , Y )]] , where h ∈ H represents the global hypothesis shared by all local clients.
Revised: [ E Z ∼ D i [ ℓ ( h ( X ) , Y )]] , where h ∈ H represents the global hypothesis shared by all local clients.

Original: The population risk minimizer h ∗ associated to population risk L P ( h ) is define as h ∗ =
Revised: The population risk minimizer h ∗ associated to population risk L P ( h ) is define as h ∗ =

Original: arg min h ∈H L P ( h ) .
Revised: arg min h ∈H L P ( h ) .

Original: However, it is impossible to minimize L P ( h ) directly because the exact meta distribution and client local distributions are unknown to us.
Revised: However, it is impossible to minimize L P ( h ) directly because the exact meta distribution and client local distributions are unknown to us.

Original: We have access to only a finite number of clients and finite training data at each client.
Revised: We have access to only a finite number of clients and finite training data at each client.

Original: The global objective function defined as population risk is often optimized by the form of empirical risk minimization (ERM) objective function defined as: L S ( h ) = 1 m (cid:80) mi
Revised: The global objective function defined as population risk is often optimized by the form of empirical risk minimization (ERM) objective function defined as: L S ( h ) = 1 m (cid:80) mi

Original: =1 1 n (cid:80) nj =1 ℓ ( h ( X ji ) , Y ji ) , where ( X ji , Y ji ) represents the j -th training data point at i -th participating client.
Revised: =1 1 n (cid:80) nj =1 ℓ ( h ( X ji ) , Y ji ) , where ( X ji , Y ji ) represents the j -th training data point at i -th participating client.

Original: For simplicity, we denote Z ji =
Revised: For simplicity, we denote Z ji =

Original: ( X ji , Y ji ) the data point.
Revised: ( X ji , Y ji ) the data point.

Original: Let
Revised: Let

Original: S i
Revised: S i

Original: =
Revised: =

Original: { Z ji } nj =1 denotes the local training set at i -th participating client and S =
Revised: { Z ji } nj =1 denotes the local training set at i -th participating client and S =

Original: S i (cid:83) · · · (cid:83) S m represent the global training set across all participating clients.
Revised: S i (cid:83) · · · (cid:83) S m represent the global training set across all participating clients.

Original: The empirical risk minimizer (cid:98) h condition on dataset S is define as (cid:98) h = arg min h ∈H L S ( h ) .
Revised: The empirical risk minimizer (cid:98) h condition on dataset S is define as (cid:98) h = arg min h ∈H L S ( h ) .

Original: To analyze the generalization in our two-level framework, we further define semi-empirical distribution D and the corresponding semiempirical risk L D ( h ) by D = 1 m (cid:80) mi =1 D i and L D ( h )
Revised: To analyze the generalization in our two-level framework, we further define semi-empirical distribution D and the corresponding semiempirical risk L D ( h ) by D = 1 m (cid:80) mi =1 D i and L D ( h )

Original: = 1 m (cid:80) mi
Revised: = 1 m (cid:80) mi

Original: =1 E Z ∼ D i [ ℓ ( h, Z )] .
Revised: =1 E Z ∼ D i [ ℓ ( h, Z )] .

Original: We extend the previous definitions and denote by (cid:98) h ∗ the semi-empirical risk minimizer (cid:98) h ∗ =
Revised: We extend the previous definitions and denote by (cid:98) h ∗ the semi-empirical risk minimizer (cid:98) h ∗ =

Original: arg min h ∈H L D ( h ) .
Revised: arg min h ∈H L D ( h ) .

Original: The semi-excess risk for participating clients is defined as: L D ( (cid:98) h ) − L D ( (cid:98) h ∗ ) .
Revised: The semi-excess risk for participating clients is defined as: L D ( (cid:98) h ) − L D ( (cid:98) h ∗ ) .

Original: Semi-excess risk indicates the performance of the learned model (cid:98) h on the unseen data associated with semi-empirical distribution D .
Revised: Semi-excess risk indicates the performance of the learned model (cid:98) h on the unseen data associated with semi-empirical distribution D .

Original: The excess risk for unparticipating clients is defined as: L P ( (cid:98) h ) − L P ( h ∗ ) .
Revised: The excess risk for unparticipating clients is defined as: L P ( (cid:98) h ) − L P ( h ∗ ) .

Original: Excess risk indicates the performance of the learned model (cid:98) h on the unseen clients distributed according to
Revised: Excess risk indicates the performance of the learned model (cid:98) h on the unseen clients distributed according to

Original: P .
Revised: P .

Original: Note that the the excess risk L P ( (cid:98) h ) −L P ( h ∗ ) is defined across two-level distribution framework.
Revised: Note that the the excess risk L P ( (cid:98) h ) −L P ( h ∗ ) is defined across two-level distribution framework.

Original: It will be shown that, in our analysis, all upper bound of excess risk L P ( (cid:98) h ) −L P ( h ∗ ) involves semi- excess risk L D ( (cid:98) h ) − L D ( (cid:98) h ∗ ) or its upper bound.
Revised: It will be shown that, in our analysis, all upper bound of excess risk L P ( (cid:98) h ) −L P ( h ∗ ) involves semi- excess risk L D ( (cid:98) h ) − L D ( (cid:98) h ∗ ) or its upper bound.

Original: To understand this framework better, we present our basic results of excess risk as follows:
Revised: To understand this framework better, we present our basic results of excess risk as follows:

Original: Definition 1 (VC dimension) .
Revised: Definition 1 (VC dimension) .

Original: Let ( X , H ) be a set system that consists of a set and a class H of subsets of X. A set system ( X , H ) shatters a set A if each subset of A can be expressed as A ∩ h for some h in H .
Revised: Let ( X , H ) be a set system that consists of a set and a class H of subsets of X. A set system ( X , H ) shatters a set A if each subset of A can be expressed as A ∩ h for some h in H .

Original: The VC-dimension of H is the size of the largest set shattered by H .
Revised: The VC-dimension of H is the size of the largest set shattered by H .

Original: Definition 2 (VC subgraph of real valued function) .
Revised: Definition 2 (VC subgraph of real valued function) .

Original: The subgraph of a function h ( ∈ H ) :
Revised: The subgraph of a function h ( ∈ H ) :

Original: X → R is the subset of X × R given by { ( x, t ) : t < h ( x ) } .
Revised: X → R is the subset of X × R given by { ( x, t ) : t < h ( x ) } .

Original: Then the V C -dimension of the function class
Revised: Then the V C -dimension of the function class

Original: F is defined as the V C -dimension of the set of subgraphs of functions in H .
Revised: F is defined as the V C -dimension of the set of subgraphs of functions in H .

Original: Theorem 1 (Generalization error for unparticipating clients) .
Revised: Theorem 1 (Generalization error for unparticipating clients) .

Original: Let F be a family of functions related to hypothesis space H : F = { z (cid:55)→ ℓ ( h, z ) : h ∈ H} .
Revised: Let F be a family of functions related to hypothesis space H : F = { z (cid:55)→ ℓ ( h, z ) : h ∈ H} .

Original: For the VC subgraph class F with VC dimension d .
Revised: For the VC subgraph class F with VC dimension d .

Original: If the loss function ℓ is bounded by b , it follows that with probability at least 1 − 2 δ ,
Revised: If the loss function ℓ is bounded by b , it follows that with probability at least 1 − 2 δ ,

Original: [Equation]
Revised: [Equation]

Original: where c 1 and c 2 are constants.
Revised: where c 1 and c 2 are constants.

Original: Remark 1.
Revised: Remark 1.

Original: Assume the total number of clients is M and P is a concrete meta-distribution on M different clients’ distributions.
Revised: Assume the total number of clients is M and P is a concrete meta-distribution on M different clients’ distributions.

Original: The global model (cid:98) h is trained based on m participating clients.
Revised: The global model (cid:98) h is trained based on m participating clients.

Original: The excess risk measures the average performance of (cid:98) h on total M clients, which include participating and unparticipating clients.
Revised: The excess risk measures the average performance of (cid:98) h on total M clients, which include participating and unparticipating clients.

Original: Theorem 1 indicates that, increasing the number of participating clients m leads to the decrease of excess risk L P ( (cid:98) h ) − L P ( h ∗ ) .
Revised: Theorem 1 indicates that, increasing the number of participating clients m leads to the decrease of excess risk L P ( (cid:98) h ) − L P ( h ∗ ) .

Original: In cross-device federated learning, the number of participating clients m may be large enough such that the excess risk approaches zero.
Revised: In cross-device federated learning, the number of participating clients m may be large enough such that the excess risk approaches zero.

Original: Based on these discussions, we can give a positive answer to the question asked in Introduction.
Revised: Based on these discussions, we can give a positive answer to the question asked in Introduction.

Original: This is, from the perspective of average performance, unparticipating clients would benefit from the model trained by participating clients .
Revised: This is, from the perspective of average performance, unparticipating clients would benefit from the model trained by participating clients .

Original: Remark 2.
Revised: Remark 2.

Original: Our theoretical results show that under some assumptions we are able to bound the excess risk L P ( (cid:98) h ) − L P ( h ∗ ) .
Revised: Our theoretical results show that under some assumptions we are able to bound the excess risk L P ( (cid:98) h ) − L P ( h ∗ ) .

Original: In the cases when every client is completely different, L P ( h ∗ ) will be large.
Revised: In the cases when every client is completely different, L P ( h ∗ ) will be large.

Revised: Thus, the generalization error for unparticipating clients L P ( h ∗ ) will be large.

Revised: This observations indicates that, we can not expect one commen model works well when heterogeneity is high.

Revised: We provide experimental results on EMINIST (Cohen et al., 2017) and synthetic data in appendix.

Revised: Though Theorem 1 is derived for VC class, experiments of neural network justify our theory.

Original:  F AST L EARNING R ATES IN T WO - LEVEL D ISTRIBUTION F RAMEWORK
Revised:  F AST L EARNING R ATES IN T WO - LEVEL D ISTRIBUTION F RAMEWORK

Original: In this section, we present fast learning rates in our two-level distribution framework.
Revised: In this section, we present fast learning rates in our two-level distribution framework.

Original: Recall that (cid:98) h is the empirical risk minimizer and h ∗ is the population risk minimizer.
Revised: Recall that (cid:98) h is the empirical risk minimizer and h ∗ is the population risk minimizer.

Original: Our goal is to bound the semi-excess risk for participating clients L D ( (cid:98) h ) −L D ( (cid:98) h ∗ ) and excess risk for unparticipating clients
Revised: Our goal is to bound the semi-excess risk for participating clients L D ( (cid:98) h ) −L D ( (cid:98) h ∗ ) and excess risk for unparticipating clients

Original: L P ( (cid:98) h ) − L P ( h ∗ ) .
Revised: L P ( (cid:98) h ) − L P ( h ∗ ) .

Original: To get faster learning rates in our two-level distribution framework, we start by making some assumptions on loss function ℓ , hypothesis space H , semi-empirical distribution D , and meta distribution P .
Revised: To get faster learning rates in our two-level distribution framework, we start by making some assumptions on loss function ℓ , hypothesis space H , semi-empirical distribution D , and meta distribution P .

Original: Assumption 1.
Revised: Assumption 1.

Original: Loss function ℓ is L -Lipschitz in its first argument: | ℓ ( y 1 , y ) − ℓ ( y 2 , y ) | ≤
Revised: Loss function ℓ is L -Lipschitz in its first argument: | ℓ ( y 1 , y ) − ℓ ( y 2 , y ) | ≤

Original: L | y 1 − y 2 | .
Revised: L | y 1 − y 2 | .

Original: Definition 3 (Bernstein condition) .
Revised: Definition 3 (Bernstein condition) .

Original: Let µ be a distribution supported on X × Y and let ℓ be a loss function with domain Y × Y .
Revised: Let µ be a distribution supported on X × Y and let ℓ be a loss function with domain Y × Y .

Original: The tuple ( µ, ℓ, H , h ∗ ) satisfies the ( β, B ) -Bernstein condition with parameter B > 0 if the following holds for any h ∈ H :
Revised: The tuple ( µ, ℓ, H , h ∗ ) satisfies the ( β, B ) -Bernstein condition with parameter B > 0 if the following holds for any h ∈ H :

Original: [Equation]
Revised: [Equation]

Revised: It is well known that fast learning rates require extra assumptions.

Revised: Bernstein condition is widely used to get fast learning rates in the learning theory community (Xu & Zeevi, 2020; van Erven et al., 2015; Wu et al., 2022).

Revised: We emphasize that it is not too restrictive.

Revised: For example, it is directly implied by the boundedness property of functions with any probability distribution (Bartlett et al., 2004).

Revised: Moreover, regression problems with strictly convex loss function satisfy the Bernstein condition if the function class is convex (Lecu´e & Mendelson, 2013).

Revised: Other examples include excess risk functions with minimizer of population risk when the loss function is strongly convex and Lipschitz (Klochkov & Zhivotovskiy, 2021).

Original: Assumption 2.
Revised: Assumption 2.

Original: Theoretical analyses in our two-level distribution framework involve different types of Bernstein conditons:
Revised: Theoretical analyses in our two-level distribution framework involve different types of Bernstein conditons:

Original: (a) The tuple ( D, ℓ, H , (cid:98) h ∗ ) satisfies the Bernstein condition with parameter B ′ ≥ 1 , 0 < β ′ ≤ 1 .
Revised: (a) The tuple ( D, ℓ, H , (cid:98) h ∗ ) satisfies the Bernstein condition with parameter B ′ ≥ 1 , 0 < β ′ ≤ 1 .

Original: That is, for any h ∈ H , 1 m (cid:80) m i =1 E [ h ( X 1 i ) − (cid:98) h ∗ ( X 1 i )] 2 ≤ B ′ ( L D ( h ) − L D ( (cid:98) h ∗ ))
Revised: That is, for any h ∈ H , 1 m (cid:80) mi =1 E [ h ( X 1 i ) − (cid:98) h ∗ ( X 1 i )] 2 ≤ B ′ ( L D ( h ) − L D ( (cid:98) h ∗ ))

Original: β ′ .
Revised: β ′ .

Original: (b) The tuple ( P, ℓ, H , h ∗ ) satisfies the Bernstein condition with parameter B ′′ ≥ 1 , 0 < β ′′ ≤ 1 .
Revised: (b) The tuple ( P, ℓ, H , h ∗ ) satisfies the Bernstein condition with parameter B ′′ ≥ 1 , 0 < β ′′ ≤ 1 .

Original: That is, for any h ∈ H , E D
Revised: That is, for any h ∈ H , E D

Original: i ∼ P
Revised: i ∼ P

Original: [ E X ∼ D i
Revised: [ E X ∼ D i

Original: [ h ( X ) − h ∗ ( X )] 2 ]
Revised: [ h ( X ) − h ∗ ( X )] 2 ]

Original: ≤ B ′′
Revised: ≤ B ′′

Original: ( L P ( h ) − L P ( h ∗ )) β ′′ .
Revised: ( L P ( h ) − L P ( h ∗ )) β ′′ .

Original: It is well known that fast learning rates require extra assumptions.

Original: Bernstein condition is widely used to get fast learning rates in the learning theory community (Xu & Zeevi, 2020; van Erven et al., 2015;

Original: Wu et al., 2022).

Original: We emphasize that it is not too restrictive.

Original: For example, it is directly implied by the boundedness property of functions with any probability distribution (Bartlett et al., 2004).

Original: Moreover, regression problems with strictly convex loss function satisfy the Bernstein condition if the function class is convex (Lecu´e & Mendelson, 2013).

Original: Other examples include excess risk functions with minimizer of population risk when the loss function is strongly convex and Lipschitz (Klochkov &

Original: Zhivotovskiy, 2021).

Original: For our purposes, we need to check that both (a) and (b) in Assumptionhold.
Revised: For our purposes, we need to check that both (a) and (b) in Assumption 2 hold.

Original: A typical example satisfying Assumption 2 is quadratic loss with convex function class H .
Revised: A typical example satisfying Assumption 2 is quadratic loss with convex function class H .

Original: We provide some examples satisfying Assumption 2 in appendix.
Revised: We provide some examples satisfying Assumption 2 in appendix.

Original: For more details, we refer to (Xu & Zeevi, 2020; Wu et al., 2022; van Erven et al., 2015).
Revised: For more details, we refer to (Xu & Zeevi, 2020; Wu et al., 2022; van Erven et al., 2015).

Original: Assumption 3 (Uniform entropy number 1 ) .
Revised: Assumption 3 (Uniform entropy number 1 ) .

Original: Let H be a family of bounded functions with uniformly entropy number log N ( ϵ, H , ∥ · ∥ 2 ) .
Revised: Let H be a family of bounded functions with uniformly entropy number log N ( ϵ, H , ∥ · ∥ 2 ) .

Original: Assume that there exist positive numbers γ, d and p such that log N ( ϵ, H , ∥ · ∥ 2 ) ≤ d log p ( γ/ϵ ) for any 0 < ϵ ≤ γ .
Revised: Assume that there exist positive numbers γ, d and p such that log N ( ϵ, H , ∥ · ∥ 2 ) ≤ d log p ( γ/ϵ ) for any 0 < ϵ ≤ γ .

Original: Assumption 3 is a mild assumption if the function classes are bounded.
Revised: Assumption 3 is a mild assumption if the function classes are bounded.

Original: We list some popular function classes satisfying Assumption 3: (a) If the VC-dimension of H is finite, then H satisfies assumption 3.
Revised: We list some popular function classes satisfying Assumption 3: (a) If the VC-dimension of H is finite, then H satisfies assumption 3.

Original: For instance, the function class of k -means methods has finite VC dimension.
Revised: For instance, the function class of k -means methods has finite VC dimension.

Original: For more details, we refer the reader to (Devroye et al., 2013).
Revised: For more details, we refer the reader to (Devroye et al., 2013).

Original: (b) When we set ϵ ∈ (0 , 1) , then all the unit Euclidean ball B ⊂ R d satisfy assumption 3.
Revised: (b) When we set ϵ ∈ (0 , 1) , then all the unit Euclidean ball B ⊂ R d satisfy assumption 3.

Original: (c) If H is a RKHS with kernel k and the rank of k is d , then H satisfies assumption 3.
Revised: (c) If H is a RKHS with kernel k and the rank of k is d , then H satisfies assumption 3.

Original: 3. F AST L EARNING RATES FOR PARTICIPATING CLIENTS
Revised: 3. F AST L EARNING RATES FOR PARTICIPATING CLIENTS

Original: In this subsection, we provide fast learning rates for participating clients in high probability.
Revised: In this subsection, we provide fast learning rates for participating clients in high probability.

Original: To obtain faster convergence rates, we focus on semi-excess risk. Let (cid:98) F ∗ := { f : ( X, Y ) (cid:55)→ ℓ (  h( X ) , Y ) − ℓ ( (cid:98) h ∗( X ) , Y ) , h ∈ H} , where (cid:98) h ∗ is the semi-empirical risk minimizer. We denote R mn
Revised: To obtain faster convergence rates, we focus on semi-excess risk L D ( (cid:98) h ) − L D ( (cid:98) h ∗ ) .

Original: ( (cid:98) F ∗ , r ) by the local empirical Rademacher complexity associated with semi-empirical distribution D : R mn ( (cid:98) F ∗ , r ) =

Original: E S,σ [ 1 mn sup T mn ( f ) ≤ r (cid:80) mi =1 (cid:80) nj =1 σ ji f ( Z ji )] , where T mn ( f ) = 1 m (cid:80) mi

Original: =

Original: E

Original: Z ∼ D i

Original: [ f ( Z )] 2 .

Original: Theorem 2 (Semi-excess risk for participating clients) .
Revised: Theorem 2 (Semi-excess risk for participating clients) .

Original: Let F be a family of functions bounded by b .
Revised: Let F be a family of functions bounded by b .

Original: Under assumptions 1, 3 and (a) of Assumption 2, when mn ≥ cd log p ( mn ) , it follows that with probability at least 1 − δ ,
Revised: Under assumptions 1, 3 and (a) of Assumption 2, when mn ≥ cd log p ( mn ) , it follows that with probability at least 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: where c 1 and c 2 are constants depending on γ, p, L, β ′ and B 1 , b, β ′ respectively.
Revised: where c 1 and c 2 are constants depending on γ, p, L, β ′ and B 1 , b, β ′ respectively.

Original: Remark 3.
Revised: Remark 3.

Original: Theorem 2 shows that the convergnece rate of semi-empirical excess risk ranges from
Revised: Theorem 2 shows that the convergnece rate of semi-empirical excess risk ranges from

Original: O ( 1 √ mn ) to faster order O ( 1 mn ) , which corresponds to β ′ = 0 and β ′ = 1 , respectively.
Revised: O ( 1 √ mn ) to faster order O ( 1 mn ) , which corresponds to β ′ = 0 and β ′ = 1 , respectively.

Original: It indicates that, under Bernstein condition, the semi-empirical excess risk convergences faster when we increase number of clients m or the size n of local dataset.
Revised: It indicates that, under Bernstein condition, the semi-empirical excess risk convergences faster when we increase number of clients m or the size n of local dataset.

Original: We emphasize that our bounds in Theorem 2 is in high probability form, which is more emergent and challenging, when compared to the previous results in expectation form (Chen et al., 2021; Fallah et al., 2021).
Revised: We emphasize that our bounds in Theorem 2 is in high probability form, which is more emergent and challenging, when compared to the previous results in expectation form (Chen et al., 2021; Fallah et al., 2021).

Original: The learning bounds in Theorem 2 are conducted for the empirical risk minimizer (cid:98) h .
Revised: The learning bounds in Theorem 2 are conducted for the empirical risk minimizer (cid:98) h .

Original: For the inexact minimizer of (cid:98) h , the proof technique and the final bounds only involve an extra optimization term.
Revised: For the inexact minimizer of (cid:98) h , the proof technique and the final bounds only involve an extra optimization term.

Original: For more details about the approximation error, we refer to (Wang et al., 2021;
Revised: For more details about the optimization error, we refer to (Wang et al., 2021;

Original: Su et al., 2021; Khaled et al., 2019).
Revised: Su et al., 2021; Khaled et al., 2019).

Original: 3. F AST L EARNING RATES FOR UNPARTICIPATING CLIENTS
Revised: F AST L EARNING RATES FOR UNPARTICIPATING CLIENTS

Original: In this subsection we provide fast learning rates for unparticipating clients in high probability.
Revised: In this subsection we provide fast learning rates for unparticipating clients in high probability.

Original: To the best of our knowledge, this is the first result derived for unparticipating clients in heterogeneous federated learning.
Revised: To the best of our knowledge, this is the first result derived for unparticipating clients in heterogeneous federated learning.

Original: To develope theoretical guarantees in our two-level framework, we extend the previous local Rademacher complexity as follows.

Original: Let F ∗ :=

Original: { f : ( X, Y ) (cid:55)→ ℓ ( h ( X ) , Y ) − ℓ ( h ∗ ( X ) , Y ) , h ∈ H} , where h ∗ is the population risk minimizer.

Original: We denote

Original: R m ( F ∗ , r ) by the local Rademacher complexity associated with meta-distribution P : R m ( F ∗ , r ) =

Original: E (cid:104) E

Original: [Equation]

Original: [sup

Original: [Equation]

Original: ( f )

Original: [Equation]

Original: m (cid:80) mi =

Original: [Equation]

Original: σ i f ( X i , Y i )] (cid:105) , where T m ( f ) =

Original: E D i ∼ P E Z i ∼ D

Original: i

Original: [ f ( X i )] 2 .

Original: Theorem 3.
Revised: Theorem 3.

Original: Let F be a family of functions bounded by b .
Revised: Let F be a family of functions bounded by b .

Original: Under assumptions 1, 3 and (b) of
Revised: Under assumptions 1, 3 and (b) of

Original: Assumption 2, when m ≥ cd log p ( m ) , for any δ > 0 , it follows that with probability at least 1 − δ ,
Revised: Assumption 2, when m ≥ cd log p ( m ) , for any δ > 0 , it follows that with probability at least 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: where c 

Revised: where c

Original: =
Revised: =

Original: KK − β ′′ , c 1 and c 2 are constants depending on γ, p, L, β ′′ and B 2 , b, β ′′ respectively.
Revised: KK − β ′′ , c 1 and c 2 are constants depending on γ, p, L, β ′′ and B 2 , b, β ′′ respectively.

Original: Remark 4.
Revised: Remark 4.

Original: Theorem 3 is developed across the two-level distribution framework, which brings extra challenges to the analysis.
Revised: Theorem 3 is developed across the two-level distribution framework, which brings extra challenges to the analysis.

Original: It is shown that the upper bound derived in 3 include semiempirical excess risk term L D ( (cid:98) h ) − L D ( (cid:98) h ∗ ) , which is an outcome of excess risk decomposition across two-level framework.
Revised: It is shown that the upper bound derived in 3 include semiempirical excess risk term L D ( (cid:98) h ) − L D ( (cid:98) h ∗ ) , which is an outcome of excess risk decomposition across two-level framework.

Original: Recall that β ′ and β ′′ are constants defined in Assumption 2.
Revised: Recall that β ′ and β ′′ are constants defined in Assumption 2.

Original: In the cases where β ′ = 1 and β ′′ = 1 , it can be shown that excess risk is of order O
Revised: In the cases where β ′ = 1 and β ′′ = 1 , it can be shown that excess risk is of order O

Original: ( 1 mn + 1 m ) with high probability.
Revised: ( 1 mn + 1 m ) with high probability.

Original: To present Theorem 3, we must construct a sub-root function that links the expected local Rademacher complexity associated with meta-distribution P and uniform entropy number.
Revised: To present Theorem 3, we must construct a sub-root function that links the expected local Rademacher complexity associated with meta-distribution P and uniform entropy number.

Original: However, the conventional Dudley’s integral bounds are build under empirical constraints (Boucheron et al., 2013).
Revised: However, the conventional Dudley’s integral bounds are build under empirical constraints (Boucheron et al., 2013).

Original: We tackle this challenge by extending the techniques developed in (Lei et al., 2016) to our two-level distribution framework.
Revised: We tackle this challenge by extending the techniques developed in (Lei et al., 2016) to our two-level distribution framework.

Original:  L EARNING R ATES FOR S UB -W EIBULL L OSSES
Revised:  L EARNING R ATES FOR S UB -W EIBULL L OSSES

Original: In this section, we provide generalization error bounds for unbounded losses in two-level distribution framework.
Revised: In this section, we provide generalization error bounds for unbounded losses in two-level distribution framework.

Original: In particular, we focus on loss functions satisfying sub-Weibull condition.
Revised: In particular, we focus on loss functions satisfying sub-Weibull condition.

Original: Definition 4 (Sub-Weibull random variables) .
Revised: Definition 4 (Sub-Weibull random variables) .

Original: A random variable X is said to be sub-Weibull if there is constant ∥ X ∥ ψ α < ∞ , such thatP ( | X | ≥ t ) ≤ 2 exp( − t α / ∥
Revised: A random variable X is said to be sub-Weibull if there is constant ∥ X ∥ ψ α < ∞ , such that

Original: X ∥ αψ α ) , for all t ≥ 0 .
Revised: [Equation]

Original: SubGaussian and sub-exponential random variables are two special cases of Sub-Weibull random variables, which corresponds to α = 2 and α = 1 , respectively.
Revised: Sub-Gaussian and sub-exponential random variables are two special cases of Sub-Weibull random variables, which corresponds to α = 2 and α = 1 , respectively.

Original: The learning rates derived in two-level framework for sub-exponential losses are deferred to the appendices.
Revised: The learning rates derived in two-level framework for sub-exponential losses are deferred to the appendices.

Original: In the following we use small-ball method to establish learning rates for more heavytailed losses, where two-side concentration inequalities may fail to hold.
Revised: In the following we use small-ball method to establish learning rates for more heavytailed losses, where two-side concentration inequalities may fail to hold.

Original: This subsection aims at establishing generalization bounds for unbounded losses that have heavier tails than sub-exponential distribution.
Revised: This subsection aims at establishing generalization bounds for unbounded losses that have heavier tails than sub-exponential distribution.

Original: Since the two side inequalities for empirical process fail to hold when the losses are heavy-tailed, the analysis of heavy-tailed losses require new method to relate empirical risk and population risk.
Revised: Since the two side inequalities for empirical process fail to hold when the losses are heavy-tailed, the analysis of heavy-tailed losses require new method to relate empirical risk and population risk.

Original: In this subsection, we establish generalization bounds for heterogeneous federated learning by extending the small-ball method from i.i.d setting to our twolevel distribution framework.
Revised: In this subsection, we establish generalization bounds for heterogeneous federated learning by extending the small-ball method from i.i.d setting to our twolevel distribution framework.

Original: We consider the quadratic loss function in this section.
Revised: We consider the quadratic loss function in this section.

Original: The extension to general losses can be achived by using the techniques presented in (Mendelson, 2018).
Revised: The extension to general losses can be achived by using the techniques presented in (Mendelson, 2018).

Original: In what follows, we denote by ∥
Revised: In what follows, we denote by ∥

Original: h ∥
Revised: h ∥

Original: L 2 ( µ ) for Banach spaces L 2 ( X , µ ) .
Revised: L 2 ( µ ) for Banach spaces L 2 ( X , µ ) .

Original: Recall that D is the semi-empirical distribution and P is meta-distribution.
Revised: Recall that D is the semi-empirical distribution and P is meta-distribution.

Original: In particular, we have ∥ h ∥
Revised: In particular, we have ∥ h ∥

Original: L 2 ( D ) =
Revised: L 2 ( D ) =

Original: ( 1 m (cid:80) m i =1 E X ∼ D
Revised: ( 1 m (cid:80) mi =1 E X ∼ D

Original: i
Revised: i

Original: [ h ( X )] 2 ) 1 / 2 and ∥
Revised: [ h ( X )] 2 ) 1 / 2 and ∥

Original: h ∥
Revised: h ∥

Original: L 2 ( P ) =
Revised: L 2 ( P ) =

Original: ( E D i ∼ P E X ∼ D
Revised: ( E D i ∼ P E X ∼ D

Original: i
Revised: i

Original: [ h ( X )] 2 ) 1 / 2 .
Revised: [ h ( X )] 2 ) 1 / 2 .

Original: For the sake of clear exposition, we first introduce the small-ball condition.
Revised: For the sake of clear exposition, we first introduce the small-ball condition.

Original: Assumption 4 (Small-ball condition) .
Revised: Assumption 4 (Small-ball condition) .

Original: Let H ⊂
Revised: Let H ⊂

Original: L 2 ( D ) be a closed and convex class of functions and H − H :
Revised: L 2 ( D ) be a closed and convex class of functions and H − H :

Original: = { h − h ′ :
Revised: = { h − h ′ :

Original: h, h ′ ∈ H} .
Revised: h, h ′ ∈ H} .

Original: (a) Let Q mn ( τ ) =
Revised: (a) Let Q mn ( τ ) =

Original: inf h ∈H−H P ( | h ( X 1 i )
Revised: inf h ∈H−H P ( | h ( X 1 i )

Original: | ≥ τ ∥
Revised: | ≥ τ ∥

Original: h ∥
Revised: h ∥

Original: L 2 ( D ) ) , where X 1 i represent the random sample at i -th participating client.
Revised: L 2 ( D ) ) , where X 1 i represent the random sample at i -th participating client.

Original: There is a τ ≥ 0 for which Q mn ( τ ) > 0 .
Revised: There is a τ ≥ 0 for which Q mn ( τ ) > 0 .

Original: (b) Let Q m ( τ ) =
Revised: (b) Let Q m ( τ ) =

Original: inf h ∈H−H P (cid:0) | E [ h ( X 1 i )]
Revised: inf h ∈H−H P (cid:0) | E [ h ( X 1 i )]

Original: | ≥ τ ∥ h ∥
Revised: | ≥ τ ∥ h ∥

Original: L 2 ( P ) (cid:1) , where X 1 i represent the random sample at i -th participating client.
Revised: L 2 ( P ) (cid:1) , where X 1 i represent the random sample at i -th participating client.

Original: There is a τ ≥ 0 for which Q m ( τ, P ) > 0 .
Revised: There is a τ ≥ 0 for which Q m ( τ, P ) > 0 .

Original: Assumption 4, small-ball condition, has been assumed for i.i.d and dependent data-generating process.
Revised: Assumption 4, small-ball condition, has been assumed for i.i.d and dependent data-generating process.

Original: To obtain high-probability theoretical guarantees, concentration techniques are widely used in the analysis of generalization error (Boucheron et al., 2013).
Revised: To obtain high-probability theoretical guarantees, concentration techniques are widely used in the analysis of generalization error (Boucheron et al., 2013).

Original: Intuitively, empirical risk will concentrate around population risk with high probability only when the loss function has well-behaved moments.
Revised: Intuitively, empirical risk will concentrate around population risk with high probability only when the loss function has well-behaved moments.

Original: However, this condition may fail to hold for heavy-tailed losses (Mendelson, 2015).
Revised: However, this condition may fail to hold for heavy-tailed losses (Mendelson, 2015).

Original: Assumption 4 appears first in the work of (Mendelson, 2015).
Revised: Assumption 4 appears first in the work of (Mendelson, 2015).

Original: Losses with any sort of moment equivalence satisfy small-ball condition, which is weaker than concentration condition and can be used to model heavy-tailedness.
Revised: Losses with any sort of moment equivalence satisfy small-ball condition, which is weaker than concentration condition and can be used to model heavy-tailedness.

Original: For example, even weak condition ∥
Revised: For example, even weak condition ∥

Original: h ∥
Revised: h ∥

Original: L 2 ( P ) ≤ c ∥ h ∥
Revised: L 2 ( P ) ≤ c ∥ h ∥

Original: L 1 ( P ) yields nontrivial small-ball estimate.
Revised: L 1 ( P ) yields nontrivial small-ball estimate.

Original: Moreover, the equivalence between higher-order moments and second-order moment such as ∥ h ∥
Revised: Moreover, the equivalence between higher-order moments and second-order moment such as ∥ h ∥

Original: L p ( P ) ≤ c ∥ h ∥
Revised: L p ( P ) ≤ c ∥ h ∥

Original: L 2 ( P ) also leads to small-ball condition (Lecu´e & Mendelson, 2016).
Revised: L 2 ( P ) also leads to small-ball condition (Lecu´e & Mendelson, 2016).

Original: Based on these observations, condition (b) of Assumption 4 is generally implied when we consider each local distribution D i as a random variable according to client distribution P .
Revised: Based on these observations, condition (b) of Assumption 4 is generally implied when we consider each local distribution D i as a random variable according to client distribution P .

Original: Let us discuss condition (a) of Assumption 4 in more detail.
Revised: Let us discuss condition (a) of Assumption 4 in more detail.

Original: Note that the establishment of this assumption 4 only requires H ⊂
Revised: Note that the establishment of this assumption 4 only requires H ⊂

Original: L 2 ( D ) with high probability, where D is the semi-empirical distribution.
Revised: L 2 ( D ) with high probability, where D is the semi-empirical distribution.

Original: This requirement is not too restritive since the elements of D are i.i.d sampled from P .
Revised: This requirement is not too restritive since the elements of D are i.i.d sampled from P .

Original: To our knowledge, this is the first time that small-ball condition is used under heterogeneous data generating assumption.
Revised: To our knowledge, this is the first time that small-ball condition is used under heterogeneous data generating assumption.

Original: 4. L EARNING RATES FOR PARTICIPATING CLIENTS WITH SMALL - BALL CONDITION
Revised: 4. L EARNING RATES FOR PARTICIPATING CLIENTS WITH SMALL - BALL CONDITION

Original: We first describe the basic idea of generalization analysis for participating clients.
Revised: We first describe the basic idea of generalization analysis for participating clients.

Original: Recall that (cid:98) h ∗ is the minimizer of semi-empirical risk L D ( h ) in H .
Revised: Recall that (cid:98) h ∗ is the minimizer of semi-empirical risk L D ( h ) in H .

Original: In this subsection we focus on the measure ∥ h − (cid:98) h ∗ ∥ 2 L 2 ( D ) , which represents the distance between h and (cid:98) h ∗ with respect to semi-empirical distribution D .
Revised: In this subsection we focus on the measure ∥ h − (cid:98) h ∗ ∥ 2 L 2 ( D ) , which represents the distance between h and (cid:98) h ∗ with respect to semi-empirical distribution D .

Original: For quadratic loss and every h ∈ H , we have
Revised: For quadratic loss and every h ∈ H , we have

Original: [Equation]
Revised: [Equation]

Original: n (cid:88) j = ξ ji ( h − (cid:98) h ∗ )( X ji ) , (2) where ξ ji = (cid:98) h ∗ ( X ji ) − Y ji .
Revised: where ξ ji = (cid:98) h ∗ ( X ji ) − Y ji .

Original: Since (cid:98) h is the minimizer of empirical risk L S ( h ) , we have L S ( (cid:98) h ) −
Revised: Since (cid:98) h is the minimizer of empirical risk L S ( h ) , we have L S ( (cid:98) h ) −

Original: L S ( (cid:98) h ∗ ) ≤ 0 .
Revised: L S ( (cid:98) h ∗ ) ≤ 0 .

Original: If on an event ∥ h − (cid:98) h ∗ ∥
Revised: If on an event ∥ h − (cid:98) h ∗ ∥

Original: L 2 ( D ) is large, then the summation of two terms in (2) is larger than 0 with high probability.
Revised: L 2 ( D ) is large, then the summation of two terms in (2) is larger than 0 with high probability.

Original: It follows that with high probability ∥ (cid:98) h − (cid:98) h ∗ ∥
Revised: It follows that with high probability ∥ (cid:98) h − (cid:98) h ∗ ∥

Original: L 2 ( D ) is small since
Revised: L 2 ( D ) is small since

Original: L S ( (cid:98) h ) − L S ( (cid:98) h ∗ ) ≤ 0 .
Revised: L S ( (cid:98) h ) − L S ( (cid:98) h ∗ ) ≤ 0 .

Original: Let { ( X ji , Y ji ) } ( m,n ) ( i,j )=(1 , 1) be global data samples whose elements { ( X ji , Y ji ) } nj =1 are i.i.d random pairs at i -th client.
Revised: Let { ( X ji , Y ji ) } ( m,n ) ( i,j )=(1 , 1) be global data samples whose elements { ( X ji , Y ji ) } nj =1 are i.i.d random pairs at i -th client.

Original: The analysis of the first term in (2) involves the following definition of
Revised: The analysis of the first term in (2) involves the following definition of

Original: Rademacher complexity.
Revised: Rademacher complexity.

Original: Definition 5.
Revised: Definition 5.

Original: We define H− (cid:98) h ∗ = { h −(cid:98) h ∗ : h  ∈ H} and denote by B m 2 the L 2 ( D ) unit ball entered at (cid:98) h ∗ , that is B m 2{+
+} ={ h ∈ H :
Revised: We define H − H = { h − h h, h ′ ∈ H} and denote by B m 2 the L 2 ( D ) unit ball entered at (cid:98) h ∗ , that is B m {+
+} =

Revised: h ∈ H :

Original: ∥ h − (cid:98) h ∗ ∥
Revised: ∥ h − (cid:98) h ∗ ∥

Original: L 2 ( D ) ≤ 1 } .
Revised: L 2 ( D ) ≤ 1 } .

Original: For every η > 0 , define
Revised: For every η > 0 , define

Original: [Equation]
Revised: [Equation]

Original: where σ j i are Rademacher random variables.
Revised: where σ ji are Rademacher random variables.

Original: The quantity ω mn ( η ) measures the Rademcher complexity of the localized function set { h − (cid:98) h ∗ : h ∈ H∩ sB m 2 } .
Revised: The quantity ω mn ( η ) measures the Rademcher complexity of the localized function set {H − H ∩ sB m 2 } .

Original: Note that ω mn ( η ) depends only on the hypothesis class H and global input samples drawn from semi-empirical distribution D .
Revised: Note that ω mn ( η ) depends only on the hypothesis class H and global input samples drawn from semi-empirical distribution D .

Original: Theorem 4.
Revised: Theorem 4.

Original: Fix τ > 0 for which Q m (2 τ ) > 0 and set η < τ 2 Q mn (2 τ ) / 32 .
Revised: Fix τ > 0 for which Q m (2 τ ) > 0 and set η < τ 2 Q mn (2 τ ) / 32 .

Original: If every random variable V ji = ξ ji h ( X ji ) − E [ ξ ji h ( X ji )] for all h ∈ H − (cid:98) h ∗ is Sub-Weibull.
Revised: If every random variable V ji = ξ ji h ( X ji ) − E [ ξ ji h ( X ji )] for all h ∈ H − (cid:98) h ∗ is Sub-Weibull.

Original: For every 0 < δ < 1 , 0 < ι < 14 and sufficiently large mn , with probability at least 1 − δ  − exp (cid:0) − mnQ 2 mn (2 τ ) / 2 (cid:1) one has
Revised: For every 0 < δ < 1 , 0 < ι < 14 and sufficiently large mn , with probability at least 1 − δ mn − exp (cid:0) − mnQ 2 mn (2 τ ) / 2 (cid:1) one has

Original: [Equation]
Revised: [Equation]

Original: where δ[-= exp {− ( c 1 η 2 ( mn ) 
 ι ∧ c 2 η α (-] mn ) α
Revised: where δ[-= exp {− ( c 1 η 2 ( mn ) 
 ι ∧ c 2 η α (-] mn =

Original: (1+4 ι )) }
Revised: [Equation]

Original: Remark 5.
Revised: Remark 5.

Original: To the best of our knowledge, Theorem 4 provide the first result on the generalization error of heterogeneous federated learning with heavy-tailed losses.
Revised: To the best of our knowledge, Theorem 4 provide the first result on the generalization error of heterogeneous federated learning with heavy-tailed losses.

Original: It suggests that both hypothesis ‘size’ and noise level play important roles in the generalization error of heterogeneous learning problems.
Revised: It suggests that both hypothesis ‘size’ and noise level play important roles in the generalization error of heterogeneous learning problems.

Original: Corollary 1.
Revised: In Theorem 4, the expression of δ mn dependents on the tail of sub-Weibull random variables V ji . Specifically, the heavier the tail of V ji , the larger δ mn will be.

Original: Under the same conditions of Theorem 4, for every 0 < δ < 1 , 0 < ι < 12 and sufficiently large mn , with probability at least 1 − δ mn − exp (cid:0) − mnQ 2 mn (2 τ ) / 2 (cid:1) one has
Revised: Note that in addition to ∥ V ji ∥ ψ α , δ mn also depends on ( mn ) 4 ι . That is, the larger mn and ι are, the smaller δ mn will be. To ensure that Theorem 4 holds with high probability, we must ensure that δ mn is small enough.

Original: [Equation]
Revised: Therefore, when mn is fixed, the heavier the tail of V ji , the larger ι should be. It can be seen from Theorem 4 that the larger ι is, the slower the convergence rate of ∥ (cid:98) h − (cid:98) h ∗ ∥

Original: where δ
Revised: L 2 ( D ) is.

Original: [Equation]
Revised: Corollary 1.

Original: =
Revised: Under the same conditions of Theorem 4, for convex function class H and sufficiently large mn , with probability at least 1 − δ mn − exp (cid:0) − mnQ 2 mn (2 τ ) / 2 (cid:1) one has

Original: [Equation]
Revised: [Equation]

Revised: where 0 < ι < 12 .

Original: Remark 6.
Revised: Remark 6.

Original: Corollary 1 provides the convergnece rate of semi-empirical excess risk for Sub-Weibull losses.
Revised: Corollary 1 provides the convergnece rate of semi-empirical excess risk for Sub-Weibull losses.

Original: Compared to Theorem 2, it shows that the convergnece rate of excess risk is slower than
Revised: Compared to Theorem 2, it shows that the convergnece rate of excess risk is slower than

Original: O ( 1 √ mn ) .
Revised: O ( 1 √ mn ) .

Original: 4. L EARNING RATES FOR UNPARTICIPATING CLIENTS WITH SMALL - BALL CONDITION
Revised: 4. L EARNING RATES FOR UNPARTICIPATING CLIENTS WITH SMALL - BALL CONDITION

Original: In the analysis of generalization error for unparticipating clients, we focus on the measure ∥
Revised: In the analysis of generalization error for unparticipating clients, we focus on the measure ∥

Original: h − h ∗ ∥ 2 L 2 ( P ) , which represents the distance between h and h ∗ with respect to meta-distribution P .
Revised: h − h ∗ ∥ 2 L 2 ( P ) , which represents the distance between h and h ∗ with respect to meta-distribution P .

Original: The analysis of generalization error for unparticipating clients follows a similar path to the previous analysis.
Revised: The analysis of generalization error for unparticipating clients follows a similar path to the previous analysis.

Original: Let { ( X i , Y i ) } mi =1 be dataset whose elements are sampled across the two-level framework, that is E
Revised: Let { ( X i , Y i ) } mi =1 be dataset whose elements are sampled across the two-level framework, that is E

Original: [( X i , Y i )]
Revised: [( X i , Y i )]

Original: =
Revised: =

Original: E D i ∼ P E
Revised: E D i ∼ P E

Original: ( X i ,Y i ) ∼ D
Revised: ( X i ,Y i ) ∼ D

Original: i
Revised: i

Original: ( X i , Y i )] .
Revised: ( X i , Y i )] .

Original: We present the different definitions of Rademacher complexity terms in the following.
Revised: We present the different definitions of Rademacher complexity terms in the following.

Original: Definition 6.
Revised: Definition 6.

Original: We define H − H
Revised: We define H − H

Original: = { h − h
Revised: = { h − h

Original: h, h ′ ∈ H} and denote by B 2 the L 2 ( P ) unit ball entered at h ∗ .
Revised: h, h ′ ∈ H} and denote by B 2 the L 2 ( P ) unit ball entered at h ∗ .

Original: For every η > 0 , define
Revised: For every η > 0 , define

Original: [Equation]

Original: [Equation]

Original: (cid:88)

Original: [Equation]

Original: [Equation]

Original: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:35)

Original: [Equation]
Revised: [Equation]

Original: (cid:41) where σ i are Rademacher random variables.
Revised: where σ i are Rademacher random variables.

Original: The quantity ω m ( η ) measures the localized complexity of { h − h ′ :
Revised: The quantity ω m ( η ) measures the localized complexity of { ( H − H )

Original: h, h ′ ∈ H ∩ sB 2 } .
Revised: ∩ sB 2 } .

Original: Theorem 5.
Revised: Theorem 5.

Original: Fix τ > 0 for which Q m (2 τ ) > 0 and set η < τ 2 Q m (2 τ ) / 32 .
Revised: Fix τ > 0 for which Q m (2 τ ) > 0 and set η < τ 2 Q m (2 τ ) / 32 .

Original: If every random variable V i =E [ ξ ji h
Revised: If for all h ∈ H − h ∗ the random variable V i =

Original:  ( X ji ) − E [ ξ i h ( X i )]] for all h ∈ H − h ∗ is Sub-Weibull.
Revised: E [ ξ 1 i h ( X 1 i )] − E [ ξ i h ( X i )] is Sub-Weibull.

Original: Forevery 0 < δ < 1 , 0 < ι < 14 and sufficiently large m , with probability at least 1 − δ m − exp (cid:0) − mQ 2 m (2 τ ) / 2 (cid:1) one has
Revised: For sufficiently large m , with probability at least 1 − δ m − exp (cid:0) − mQ 2 m (2 τ ) / 2 (cid:1) one has

Original: [Equation]
Revised: [Equation]

Original:  m 
Revised: where 0 < ι < 14 and δ m = exp {− (

Original: [Equation]
Revised: [Equation]

Original: where δ m = exp {− ( c 1 η 2 m 4 ι ∧ c 2 η

Original: α m α

Original: (1+4 ι )) } .

Original: Remark 7.
Revised: Remark 7.

Original: Theorem 5 provide the first result on the generalization error of unparticipating clients in heterogeneous federated learning with heavy-tailed losses.
Revised: Theorem 5 provide the first result on the generalization error of unparticipating clients in heterogeneous federated learning with heavy-tailed losses.

Original: Corollary 2.
Revised: Corollary 2.

Original: Assume every random variable V  i
Revised: Assume for all h ∈ H− (cid:98) h ∗ the random variable V ′ i

Original: =
Revised: =

Original: E [ h 2 ( X ji )] − E [ h 2 ( X i )] for all h ∈ H− (cid:98) h ∗ is SubWeibull.
Revised: E [ h 2 ( X ji )] − E [ h 2 ( X i )] is SubWeibull and the noise h ∗ ( X i ) − Y

Original:  Under the same conditions of Theorem 5, forevery 0 < δ < 1 , 0 < ι < 12 and sufficiently large mn , with probability at least 1 − δ  − exp (cid:0) − mnQ 2 mn (2 τ ) / 2 (cid:1) − exp (cid:0) − mQ 2 m (2 τ ) / 2 (cid:1) one hasL P ( (cid:98)
Revised: i is independent of X i . Under the same conditions of Theorem 5, for 0 < η < 1 and sufficiently large mn , with probability at least 1 − δ ′ − exp (cid:0) − mnQ 2 mn (2 τ ) / 2 (cid:1) − exp (cid:0) − mQ 2 m (2 τ ) / 2 (cid:1) one has

Original: [Equation]
Revised: [Equation]

Original: where δ = δ mn + δ m + exp {− ( c
Revised: − L P ( h

Original: [Equation]
Revised: [Equation]

Original: 2 m
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: m
Revised: max

Original: [Equation]
Revised: [Equation]

Original: [-mi =
 ∥ V i ∥ 2 ψα ∧-] c 
Revised: [-mi =
 ∥ V i ∥ 2 ψα ∧-]where c 0 = 21 − η , 0 < ι < 12 and δ ′

Original: [Equation]
Revised: =

Original: α m α [Equation] [Equation] max
Revised: δ mn + δ m + exp {− (

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Remark 8.
Revised: Remark 8.

Original: Corollary 2 provides the convergnece rate of excess risk for Sub-Weibull losses.
Revised: Corollary 2 provides the convergnece rate of excess risk for Sub-Weibull losses.

Original: Compared to Theorem 3, it shows that the convergnece rate of excess risk is slower than O ( 1 √ mn + 1 √ m ) .
Revised: Compared to Theorem 3, it shows that the convergnece rate of excess risk is slower than O ( 1 √ mn + 1 √ m ) .

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Generalization Error for Heterogeneous Federated Learning.
Revised: Generalization Error for Heterogeneous Federated Learning.

Original: Several attempts have been made in the analysis of generalization error for heterogeneous federated learning.
Revised: Several attempts have been made in the analysis of generalization error for heterogeneous federated learning.

Original: We compare our results with most related works in Table 1.
Revised: We compare our results with most related works in Table 1.

Original: Complexity-based bounds for participating clients are derived in the work of (Mohri et al., 2019), who present high probability slow rates of order O ( 1 √ mn ) for bounded losses.
Revised: Complexity-based bounds for participating clients are derived in the work of (Mohri et al., 2019), who present high probability slow rates of order O ( 1 √ mn ) [Table] [Table] [Table]

Original:  Fast rates bounds are obtained based on stability tools in (Chen et al., 2021;[Table]
Revised: [Table] for bounded losses. Fast rates bounds are obtained based on stability tools in (Chen et al., 2021;

Original: Fallah et al., 2021).
Revised: Fallah et al., 2021).

Original: However, their results are in expectation form.
Revised: However, their results are in expectation form.

Original: Among the existing theoretical work, different measurements are used to model the heterogeneity of local distributions.
Revised: Among the existing theoretical work, different measurements are used to model the heterogeneity of local distributions.

Original: These measurements include gradient dissimilarity and parameter dissimilarity of local optimal models.
Revised: These measurements include gradient dissimilarity and parameter dissimilarity of local optimal models.

Original: Here we argue that it is more natural to make an assumption from the perspective of the datagenerating process.
Revised: Here we argue that it is more natural to make an assumption from the perspective of the datagenerating process.

Original: Therefore, in this paper [Table] we assume that the local distributions are sampled from a higher meta-distribution.
Revised: Therefore, in this paper, we assume that the local distributions are sampled from a higher meta-distribution.

Original: A similar two-level distribution framework has been used in the analysis of meta-learning.
Revised: A similar two-level distribution framework has been used in the analysis of meta-learning.

Original: However, the learning scenarios and objectives of federated learning are different from that of meta-learning.
Revised: However, the learning scenarios and objectives of federated learning are different from that of meta-learning.

Original: The goal of meta-learning is to choose an optimal hypothesis space H from the hypothesis space family H .
Revised: The goal of meta-learning is to choose an optimal hypothesis space H from the hypothesis space family H .

Original: Ideally, the chosen hypothesis H should contain good hypothesis h ∈ H for each distribution
Revised: Ideally, the chosen hypothesis H should contain good hypothesis h ∈ H for each distribution

Original: D
Revised: D

Original: i sampled from the meta distribution P .
Revised: i sampled from the meta distribution P .

Original: In this paper, we focus on the performance of common model (cid:98) h trained by participating clients.
Revised: In this paper, we focus on the performance of common model (cid:98) h trained by participating clients.

Original: The performance of the common model is measured by the population risk with respect to meta distribution P .
Revised: The performance of the common model is measured by the population risk with respect to meta distribution P .

Original: Another line of research closely related to heterogeneous federated learning is domain adaptation/generalization.
Revised: Another line of research closely related to heterogeneous federated learning is domain adaptation/generalization.

Original: In this line, possibly the results in (Li et al., 2022) are most relevant to ours.
Revised: In this line, possibly the results in (Li et al., 2022) are most relevant to ours.

Original: Generalization error for Unbounded losses.
Revised: Generalization error for Unbounded losses.

Original: The unbounded assumption brings two major challenges to complexity-based generalization analysis.
Revised: The unbounded assumption brings two major challenges to complexity-based generalization analysis.

Original: One is that the two-side concentration inequalities do not hold when the losses are heavy-tailed.
Revised: One is that the two-side concentration inequalities do not hold when the losses are heavy-tailed.

Original: The other is that the standard techniques used to upper bound the complexity of hypothesis space are developed for bounded losses.
Revised: The other is that the standard techniques used to upper bound the complexity of hypothesis space are developed for bounded losses.

Original: The straightforward way to avoid these two challenges is to assume there exists an envelope function with respect to the underlying distribution and hypothesis class (Adamczak, 2008; Lecu ´ e & Mendelson, 2012).
Revised: The straightforward way to avoid these two challenges is to assume there exists an envelope function with respect to the underlying distribution and hypothesis class (Adamczak, 2008; Lecu ´ e & Mendelson, 2012).

Original: Small-ball method is first proposed to replace the concentration tools for empirical process in (Mendelson, 2015) and further developed in (Mendelson, 2018).
Revised: Small-ball method is first proposed to replace the concentration tools for empirical process in (Mendelson, 2015) and further developed in (Mendelson, 2018).

Original: Inspired by the small-ball method, Offset Rademacher complexity-based method provides another replacement for two-side concentration inequality (Liang et al., 2015).
Revised: Inspired by the small-ball method, Offset Rademacher complexity-based method provides another replacement for two-side concentration inequality (Liang et al., 2015).

Original: However, most existing generalization bounds for unbounded losses are derived in the i.i.d setting.
Revised: However, most existing generalization bounds for unbounded losses are derived in the i.i.d setting.

Original: Roy et al.
Revised: Roy et al.

Original: (2021) extend the small-ball method in the dependent data setting.
Revised: (2021) extend the small-ball method in the dependent data setting.

Original: In this paper, we focus on the heterogeneous federated learning scenario with unbounded losses, where the samples are independent but non-identically distributed.
Revised: In this paper, we focus on the heterogeneous federated learning scenario with unbounded losses, where the samples are independent but non-identically distributed.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We present a systematic generalization analysis of heterogeneous distributed learning.
Revised: We present a systematic generalization analysis of heterogeneous distributed learning.

Original: Our analysis captures the generalization performance of the learned model on both participating and unparticipating clients.
Revised: Our analysis captures the generalization performance of the learned model on both participating and unparticipating clients.

Original: To our knowledge, this is the first theoretical analysis under the assumption that the local distributions are sampled from a meta-distribution.
Revised: To our knowledge, this is the first theoretical analysis under the assumption that the local distributions are sampled from a meta-distribution.

Original: We recover the current state of art guarantees without using bounded assumptions.
Revised: We recover the current state of art guarantees without using bounded assumptions.

Original: Moreover, under the empirical risk minimization setting, we derive fast generalization rates in our two-level distribution setting.
Revised: Moreover, under the empirical risk minimization setting, we derive fast generalization rates in our two-level distribution setting.

Original: A CKNOWLEDGEMENTS
Revised: A CKNOWLEDGEMENTS

Original: We appreciate all the anonymous reviewers for their insightful and constructive comments, especially one reviewer’s suggestion to add more discussion and explanation.
Revised: We appreciate all the anonymous reviewers for their insightful and constructive comments, especially one reviewer’s suggestion to add more discussion and explanation.

Original: This work is supported by
Revised: This work is supported by

Original: National Natural Science Foundation of China NO. 62076234; the Beijing Natural Science Foundation No. 4222029; the Intelligent Social Governance Interdisciplinary Platform, Major Innovation & Planning Interdisciplinary Platform for the “Double-First Class” Initiative, Renmin University of China; the Beijing Outstanding Young Scientist Program NO.BJJWZYJH012019100020098;
Revised: National Natural Science Foundation of China NO. 62076234; the Beijing Natural Science Foundation No. 4222029; the Intelligent Social Governance Interdisciplinary Platform, Major Innovation & Planning Interdisciplinary Platform for the “Double-First Class” Initiative, Renmin University of China; the Beijing Outstanding Young Scientist Program NO.BJJWZYJH012019100020098;

Original: the
Revised: the

Original: Beijing Key Laboratory of Big Data Management and Analysis Methods, Gaoling School of Artificial Intelligence, Renmin University of China, Beijing 100872, China; the Public Computing Cloud,
Revised: Beijing Key Laboratory of Big Data Management and Analysis Methods, Gaoling School of Artificial Intelligence, Renmin University of China, Beijing 100872, China; the Public Computing Cloud,

Original: Renmin University of China; the Fundamental Research Funds for the Central Universities and the
Revised: Renmin University of China; the Fundamental Research Funds for the Central Universities and the

Original: Research Funds of Renmin University of China NO. 2021030199; the Huawei-Renmin University joint program on Information Retrieval; the Unicom Innovation Ecological Cooperation Plan; and the CCF-Huawei Populus Grove Fund.
Revised: Research Funds of Renmin University of China NO. 2021030199; the Huawei-Renmin University joint program on Information Retrieval; the Unicom Innovation Ecological Cooperation Plan; and the CCF-Huawei Populus Grove Fund.

Original: C ONTENTS 1 Introduction
Revised: C ONTENTS 1 Introduction

Original: 2 Two-level Distribution Framework 3 Fast Learning Rates in Two-level Distribution Framework 3.1 Fast Learning rates for participating clients . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: 2 Two-level Distribution Framework 3 Fast Learning Rates in Two-level Distribution Framework 3.1 Fast Learning rates for participating clients . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: 3.2 Fast Learning rates for unparticipating clients . .
Revised: 3.2 Fast Learning rates for unparticipating clients . .

Original: 4 Learning Rates for Sub-Weibull Losses 4.1 Learning rates for participating clients with small-ball condition . . . . . . . . . . . . . . . .
Revised: 4 Learning Rates for Sub-Weibull Losses 4.1 Learning rates for participating clients with small-ball condition . . . . . . . . . . . . . . . .

Original: 4.2 Learning rates for unparticipating clients with small-ball condition . . . . . . . . . . . . . . .
Revised: 4.2 Learning rates for unparticipating clients with small-ball condition . . . . . . . . . . . . . . .

Original: 5 Related Work 6 Conclusion
Revised: 5 Related Work 6 Conclusion

Original: A Setting, definitions
Revised: A Setting, definitions

Original: B Proof of Theorem
Revised: B Proof of Theorem

Original: B.1 Proof of Theorem 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: B.1 Proof of Theorem 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: C Proofs of fast rates with bounded losses
Revised: C Proofs of fast rates with bounded losses

Original: C.1 Examples Satisfying Assumption 2 . . .
Revised: C.1 Examples Satisfying Assumption 2 . . .

Original: C.1.1 Example 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: C.1.1 Example 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: C.1.2 Example 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: C.1.2 Example 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: C.2 Proof of Theorem 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: C.2 Proof of Theorem 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: C.3 Proof of Theorem 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .C .4 Proof of Theorem
Revised: C.3 Proof of Theorem 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: D Proofs of the results with sub-exponential losses
Revised: D Proofs of the results with sub-exponential losses

Original: D
Revised: D.

Original: .1 Learning Rates for Sub-exponential Losses
Revised: [Table] Rates [Table]

Original: E Proofs of Small-ball based Method
Revised: E Proofs of Small-ball based Method

Original: E.1 Proof of Theorem 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .E.1.
Revised: E.1 Proof of Theorem 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original:  of Corollary 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: E.1.1 Proof of Corollary 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: F Experimental Results
Revised: F Experimental Results

Revised: F

Revised: [Table]

Original: F.1 Convolutional Neural Networks for MNIST Task . . . . . . . . . . . . . . . . . . . . . . . .
Revised: [Table] Networks [Table] F

Revised: [Table]

Original: F.2 Federated Linear Regression with Synthetic Data . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Federated [Table] Regression with Synthetic [Table]

Original: A S ETTING , DEFINITIONS
Revised: A S ETTING , DEFINITIONS

Original: Let the hypothesis space H be a family of real-valued functions defined on X .

Original: The loss function ℓ : Y × Y → R

Original: + is a non-negative function.

Original: We denote the population risk L P ( h ) by
Revised: We denote the population risk L P ( h ) by

Original: [Equation]
Revised: [Equation]

Original: where h ∈ H represents the global hypothesis shared by all local clients. The population risk minimizer h ∗associated to population risk L P ( h ) is define as
Revised: where h ∈ H represents the global hypothesis shared by all local clients, ℓ : Y × Y → R + is the non-negative loss function. The population risk minimizer h ∗ is define as

Original: [Equation]
Revised: [Equation]

Original: However, it is impossible to minimize L P ( h ) directly because the exact meta distribution and client local distributions are unknown to us.

Original: We have access to only a finite number of clients and finite training data at each client.

Original: The global objective functiondefined as population risk is often optimized by the form of empirical risk minimization (ERM) objective function defined as:
Revised: In practice, the global objective function is often optimized by the form of empirical risk minimization (ERM) objective function, which is defined as:

Original: [Equation]
Revised: [Equation]

Original: where ( X ji , Y ji ) represents the j -th training data point at i -th participating client.
Revised: where ( X ji , Y ji ) represents the j -th training data point at i -th participating client.

Original: For simplicity, we denote Z ji =
Revised: We also use Z j i =

Original: ( X ji , Y ji )  the data point.
Revised: ( X j i , Y j i ) to denote the data point.

Original: Let S i
Revised: Let S i

Original: = { Z ji } nj =1 denotes the local training set at i -th participating client and S =
Revised: = { Z j i } n j =1 denotes the local training set at i -th participating client and S =

Original: S i (cid:83) · · · (cid:83) S m represent the global training set across all participating clients.
Revised: S i (cid:83) · · · (cid:83) S m be the global training set.

Original: The empirical risk minimizer (cid:98) hcondition on dataset S is define as
Revised: The empirical risk minimizer (cid:98) h is define as

Original: [Equation]
Revised: [Equation]

Original: To analyze the generalization in our two-level framework, we further define semi-empirical distribution D and the corresponding semi-empirical risk L D ( h ) by D = 1 m (cid:80) mi =1 D i and
Revised: The semi-empirical distribution D is defined as D = 1 m (cid:80) mi =1 D i . Moreover, the corresponding semi-empirical risk L D ( h ) is defined as:

Original: [Equation]
Revised: [Equation]

Original: Weextend the previous definitions and denote by (cid:98) h ∗ the semi-empirical risk minimizer(cid:98)
Revised: We denote by (cid:98) h ∗ the semi-empirical risk minimizer

Original: [Equation]
Revised: [Equation]

Original: The semi-excess risk for participating clients is defined as:L D ( (cid:98) h ) − L D ( (cid:98) h ∗ ) .
Revised: The semi-excess risk for participating clients is defined as:

Original:  Semi-excess risk indicates the performance of the learned model (cid:98) h on the unseen data associated with semi-empirical distribution D .
Revised: [Equation] Semi-excess risk indicates the performance of the learned model (cid:98) [Equation] on the participating clients.

Original: The excess risk for unparticipating clients is defined as:
Revised: The excess risk for unparticipating clients is defined as:

Original: [Equation]
Revised: [Equation]

Revised: Excess risk indicates the performance of the learned model (cid:98) h on the unparticipating clients.

Original: Throughout the paper we denote F by F = { z (cid:55)→ ℓ ( h, z ) : h ∈ H} .
Revised: In the following, we denote F by F = { z (cid:55)→ ℓ ( h, z ) : h ∈ H} .

Original: Moreover, we use Z i =
Revised: Moreover, we use Z i =

Original: ( X i , Y i ) to represent the random variables across two-level framework.
Revised: ( X i , Y i ) to represent the random variables across two-level framework.

Original: That is,E [ Z i ] = E D i ∼ P [ E Z i ∼ D
Revised: That is,

Original: i [ Z i ]] .
Revised: [Equation]

Original: B P ROOF OF T HEOREMB. P ROOF OF T HEOREM
Revised: B P ROOF OF T HEOREMB. P ROOF OF T HEOREM

Original: We first show that semi-excess risk L D ( (cid:98) h ) − L D ( h ∗ ) can be upper bounded by supremum of the empirical process indexed by H : 
Revised: We first show that semi-excess risk L D ( (cid:98) h ) − L D ( h ∗ ) can be upper bounded by supremum of the empirical process indexed by H : [Equation]

Original: [Equation]
Revised: [Equation]

Original: Recall that (cid:98) h is the empiricalminimizer and h ∗ is the population minimizer.
Revised: where the first inequality follows from the fact that (cid:98) h is the empirical minimizer.

Original: We decompose excess risk L P ( (cid:98) h ) − L P ( h ∗ ) in two-level framework:L P ( (cid:98) h ) − L P ( h ∗ ) ≤ 2 sup h ∈H |L P ( h ) − L D ( h ) | +
Revised: Next, we decompose excess risk L P ( (cid:98) h ) − L P ( h ∗ ) in two-level framework:

Original: 2 sup h ∈H |L D ( h ) − L S ( h ) | , where the inequality uses L S ( (cid:98) h ) − L S ( h ∗ ) ≤ 0 since (cid:98) h is the minimizer of L S ( h ) . Note that the first term sup h ∈H |L P ( h ) − L D ( h ) | in the upper bound of excess risk quantifies the participation gap. The second term sup h ∈H |L D ( h ) − L S ( h ) | quantifies the generalization error for participating clients.
Revised: [Equation]

Original: In the following Theorem, we present basic upper bound of excess risk by taking VC major class as an example. The goal of Theorem 1 is to present theoretical bounds of excess risk L P ( (cid:98) h ) −L P ( h ∗ ) , we first show that excess risk can be bounded by participation gap sup h ∈H {|L P ( h ) − L D ( h ) |} and participating error sup h ∈H {|L D ( h ) − L S ( h ) |} :
Revised: In details:

Original: [Equation]
Revised: [Equation]

Original: where the first inequality is due to L S ( (cid:98) h ) − L S ( h ∗ ) ≤ 0  .
Revised: where the first inequality uses L S ( (cid:98) h ) −L S ( h ∗ ) ≤ 0 since (cid:98) h is the minimizer of L S ( h ) .

Revised: Note that the first term sup h ∈H |L P ( h ) − L D ( h ) | in the upper bound of excess risk quantifies the participation gap.

Revised: The second term sup h ∈H |L D ( h ) − L S ( h ) | quantifies the generalization error for participating clients.

Original: Based on these observations, we first provide theoretical bounds on the term sup
Revised: Based on these observations, we first provide theoretical bounds on the term sup

Original: h ∈H |L D ( h ) − L S ( h ) | .
Revised: h ∈H |L D ( h ) − L S ( h ) | .

Original: Then we move to the participating gap sup[Equation] h [Equation]
Revised: Then we move to the participating gap sup h ∈H |L P ( h ) − L D ( h ) | .

Original: −

Original: [Equation] D [Equation] Let σ = { σ ji } i ∈ [ m ] ,j ∈ [ n ] be a collection of independent Rademacher variables, this is random variables taking values uniformly in { +1 , − 1 } .
Revised: Let σ = { σ ji } i ∈ [ m ] ,j ∈ [ n ] be a collection of independent Rademacher variables, this is random variables taking values uniformly in { +1 , − 1 } .

Original: We define the generalized rademacher complexity R mn ( F ) used in heterogeneous federated learning as is as follows:
Revised: We define the generalized rademacher complexity R mn ( F ) used in heterogeneous federated learning as is as follows:

Original: [Equation]
Revised: [Equation]

Original: Lemma 1 (Generalization error for participating clinets) .
Revised: Lemma 1 (Generalization error for participating clinets) .

Original: Let F be a family of functions related to hypothesis space H : F = { z (cid:55)→ ℓ ( h, z ) : h ∈ H} .
Revised: Let F be a family of functions related to hypothesis space H : F = { z (cid:55)→ ℓ ( h, z ) : h ∈ H} .

Original: Assume that loss function ℓ is bounded by M .
Revised: Assume that loss function ℓ is bounded by M .

Original: Then, for any δ ≥ 0 , with probability at least 1 − δ , we have
Revised: Then, for any δ ≥ 0 , with probability at least 1 − δ , we have

Original: [Equation]
Revised: [Equation]

Original: Lemma 1 provide high probability theoretical guarantees for participating clients under bounded assumption.
Revised: Lemma 1 provide high probability theoretical guarantees for participating clients under bounded assumption.

Original: Compared to the classical iid setting in learning theory, the results in Lemma 1 in under independent but non-identically distributed setting.
Revised: Compared to the classical i.i.d setting in learning theory, the results in Lemma 1 in under independent but non-identically distributed setting.

Original: Since each local distribution D i is sampled independently from P , we regard local population risk { E Z
Revised: Since each local distribution D i is sampled independently from P , we regard local population risk { E Z

Original: ∼ D i [ ℓ ( h, Z )] } i ∈ [ m ] as a collection of iid random variables.
Revised: ∼ D i [ ℓ ( h, Z )] } i ∈ [ m ] as a collection of iid random variables.

Original: Let σ = { σ i } i ∈ [ m ] be a collection of independent Rademacher variables, the rademacher complexity used in the analysis of participating gap is defined as:
Revised: Let σ = { σ i } i ∈ [ m ] be a collection of independent Rademacher variables, the rademacher complexity used in the analysis of participating gap is defined as:

Original: [Equation]
Revised: [Equation]

Original: Lemma 2 (Participation gap) .
Revised: Lemma 2 (Participation gap) .

Original: Let F be a family of functions related to hypothesis space H : F = { z (cid:55)→ ℓ ( h, z ) : h ∈ H} .
Revised: Let F be a family of functions related to hypothesis space H : F = { z (cid:55)→ ℓ ( h, z ) : h ∈ H} .

Original: Assume that loss function ℓ is bounded by b .
Revised: Assume that loss function ℓ is bounded by b .

Original: Then, for any δ ≥ 0 , with probability at least 1 − δ , we have
Revised: Then, for any δ ≥ 0 , with probability at least 1 − δ , we have

Original: [Equation]
Revised: [Equation]

Original: The participation gap in Lemma 2 quantifies the error caused by missing clients during training.
Revised: The participation gap in Lemma 2 quantifies the error caused by missing clients during training.

Original: The Rademacher complexity term R m ( F ) indicates the ’size’ of hypothesis class, which can be future bounded by convering number or VC dimension of the choosen hypothesis.
Revised: The Rademacher complexity term R m ( F ) indicates the ’size’ of hypothesis class, which can be future bounded by convering number or VC dimension of the choosen hypothesis.

Original: To show the explicit form of the rademacher complexity terms in Lemma 1 and Lemma 2, we take VC major class as an example.

Original: P ROOF OF L EMMA
Revised: P ROOF OF L EMMA

Original: Proof.
Revised: Proof.

Original: We first create sample set S ′ from S by changing one sample point.
Revised: We first create sample set S ′ from S by changing one sample point.

Original: Without loss generality, we assume the data point Z ks in S is changed into Z ′ ks .
Revised: Without loss generality, we assume the data point Z ks in S is changed into Z ′ ks .

Original: Next we define Φ( S ) by Φ( S )
Revised: Next we define Φ( S ) by Φ( S )

Original: = sup h ∈H L D ( h ) − L S ( h )  .
Revised: = sup h ∈H |L D [Equation] h ) − L S ( h ) | .

Original: By the definition of S ′ and S , we have
Revised: By the definition of S ′ and S , we have

Original: [Equation]
Revised: [Equation]

Original: By assuming that loss function ℓ is bounded by b
Revised: By assuming that loss function ℓ is bounded by b

Original: [Equation]
Revised: [Equation]

Original: then we have | Φ ( S ′ )
Revised: then we have | Φ ( S ′ )

Original: − Φ( S ) | ≤ bmn .
Revised: − Φ( S ) | ≤ bmn .

Original: We apply
Revised: We apply

Original: McDiarmid’s inequality and obtain, for any fixed δ ∈ (0 , 1) [Equation] it follows that with probability at least 
Revised: McDiarmid’s inequality and obtain, for any fixed δ ∈ (0 , 1) , it follows that with probability at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: Next we consider the expectation of Φ( S ) .
Revised: Next we consider the expectation of Φ( S ) .

Original: By symmetrization
Revised: By symmetrization

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where R mn ( F ) =
Revised: where R mn ( F ) =

Original: E σ (cid:104) sup h ∈H 1 mn (cid:80) ni
Revised: E σ (cid:104) sup h ∈H 1 mn (cid:80) ni

Original: =1 (cid:80) mj
Revised: =1 (cid:80) mj

Original: =1 σ ij ℓ ( h, Z ji ) (cid:105) .
Revised: =1 σ ij ℓ ( h, Z ji ) (cid:105) .

Original: Replacing one data point of S =
Revised: Replacing one data point of S =

Original: S 1 (cid:83) · · · (cid:83) S m make R mn ( F ) vary at most b mn , then applying McDiarmid’s inequality again, with probability at least 1 −
Revised: S 1 (cid:83) · · · (cid:83) S m make R mn ( F ) vary at most b mn , then applying McDiarmid’s inequality again, with probability at least 1 −

Original: δ we have
Revised: δ we have

Original: [Equation]
Revised: [Equation]

Original: P ROOF OF L EMMA
Revised: P ROOF OF L EMMA

Original: Proof of Lemma 2.
Revised: Proof of Lemma 2.

Original: We create distributions set D ′ from D by changing one client distribution.
Revised: We create distributions set D ′ from D by changing one client distribution.

Original: Without loss generality, we assume the k -th D k in D is changed into D ′
Revised: Without loss generality, we assume the k -th D k in D is changed into D ′

Original: k , then we define Φ( D ) by
Revised: k , then we define Φ( D ) by

Original: [Equation]
Revised: [Equation]

Original: From the definition of D ′ and D
Revised: From the definition of D ′ and D

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: By assuming that loss function f is bounded by b
Revised: By assuming that loss function f is bounded by b

Original: [Equation]
Revised: [Equation]

Original: then we have | Φ ( D ′ )
Revised: then we have | Φ ( D ′ )

Original: − Φ( D ) | ≤ bm .
Revised: − Φ( D ) | ≤ bm .

Original: According to McDiarmid’s inequality, for all δ ∈ (0 , 1) , with probability at least 1 − δ , we have
Revised: According to McDiarmid’s inequality, for all δ ∈ (0 , 1) , with probability at least 1 − δ , we have

Original: [Equation]
Revised: [Equation]

Original: Now we deal with the expectation of Φ( D ) .
Revised: Now we deal with the expectation of Φ( D ) .

Original: By symmetrization
Revised: By symmetrization

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where R m ( L ) =
Revised: where R m ( L ) =

Original: E σ (cid:2) sup h ∈H 1 m (cid:80) mi =1 σ i ℓ ( h, Z i ) (cid:3) .
Revised: E σ (cid:2) sup h ∈H 1 m (cid:80) mi =1 σ i ℓ ( h, Z i ) (cid:3) .

Original: Replacing one point of Z = { Z i } mi =1 make
Revised: Replacing one point of Z = { Z i } mi =1 make

Original: R m ( L ) vary at most Mm , then applying McDiarmid’s inequality again, with probability at least 1 −
Revised: R m ( L ) vary at most Mm , then applying McDiarmid’s inequality again, with probability at least 1 −

Original: δ we have
Revised: δ we have

Original: [Equation]
Revised: [Equation]

Revised: The goal of Theorem 1 is to present theoretical bounds of excess risk L P ( (cid:98) h ) − L P ( h ∗ ) , it has been shown that excess risk can be bounded by participation gap sup h ∈H {|L P ( h ) − L D ( h ) |} and participating error sup h ∈H {|L D ( h ) − L S ( h ) |} .

Revised: To show the explicit form of the rademacher complexity terms in Lemma 1 and Lemma 2, we take VC class as an example.

Original: P ROOF OF T HEOREM
Revised: P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: We use subgraph dimension(aka pseudo dimension) and Dudley’s theorem to bound the empirical rademacher complexity R mn ( H ) .
Revised: We use subgraph dimension(aka pseudo dimension) and Dudley’s theorem to bound the empirical rademacher complexity R mn ( H ) .

Original: According to Dudley (1978); Pollard (2012); Haussler (2018), the uniform matric entropy of real valued function class F with subgraph dimension
Revised: According to Dudley (1978); Pollard (2012); Haussler (2018), the uniform matric entropy of real valued function class F with subgraph dimension

Original: VC ( F ) =
Revised: VC ( F ) =

Original: d can be bounded
Revised: d can be bounded

Original: [Equation]
Revised: [Equation]

Original: where F is the envolop function of F .
Revised: where F is the envolop function of F .

Original: Then by Dudley’s theorem and van der Vaart & Wellner (1996), we have
Revised: Then by Dudley’s Theorem (Dudley, 1978; van der Vaart & Wellner, 1996), we have

Original: [Equation]
Revised: [Equation]

Original: Combining the results of Lemma 1 and 2 leads to the final results.
Revised: Combining the results of Lemma 1 and 2 leads to the final results.

Original: C P ROOFS OF FAST RATES WITH BOUNDED LOSSES
Revised: C P ROOFS OF FAST RATES WITH BOUNDED LOSSES

Original: Covering number can be used to give tighter estimate on the hypothesis size.
Revised: Covering number can be used to give tighter estimate on the hypothesis size.

Original: Here we provide the definition of convering number and uniform entropy number.
Revised: Here we provide the definition of convering number and uniform entropy number.

Original: Definition 7 (Convering number) .
Revised: Definition 7 (Convering number) .

Original: Let ( G , ρ ) be a metric space and F ⊆ G .
Revised: Let ( G , ρ ) be a metric space and F ⊆ G .

Original: For any ϵ ≥ 0 , F ϵ is an ϵ -cover of F with respect of ρ if for all f ∈ F , we can find f ′ ∈ F ϵ such that ρ ( f, f ′ )
Revised: For any ϵ ≥ 0 , F ϵ is an ϵ -cover of F with respect of ρ if for all f ∈ F , we can find f ′ ∈ F ϵ such that ρ ( f, f ′ )

Original: ≤
Revised: ≤

Original: ϵ .
Revised: ϵ .

Original: The covering number N ( ϵ, F , ρ ) is defined as the minimum size of an ϵ -cover:
Revised: The covering number N ( ϵ, F , ρ ) is defined as the minimum size of an ϵ -cover:

Original: N ( ϵ, F , ρ ) :
Revised: N ( ϵ, F , ρ ) :

Original: = min {|F ϵ | : F ϵ is an ϵ -cover of F w.r.t ρ } .
Revised: = min {|F ϵ | : F ϵ is an ϵ -cover of F w.r.t ρ } .

Original: Definition 8 (Uniform entropy number) .
Revised: Definition 8 (Uniform entropy number) .

Original: The entropy number is defined as the logarithm of the covering number.
Revised: The entropy number is defined as the logarithm of the covering number.

Original: Let ( G , ρ ) be a normed space with ρ ( f, f ′ )
Revised: Let ( G , ρ ) be a normed space with ρ ( f, f ′ )

Original: =
Revised: =

Original: ∥
Revised: ∥

Original: f − f ′ ∥ .
Revised: f − f ′ ∥ .

Original: Let F be an envelope function of F such that | f ( Z ) | ≤ F ( Z ) , for all Z and f .
Revised: Let F be an envelope function of F such that | f ( Z ) | ≤ F ( Z ) , for all Z and f .

Original: We further define uniform entropy number of F as: log N ( ϵ, F , ∥ · ∥ 2 ) =
Revised: We further define uniform entropy number of F as: log N ( ϵ, F , ∥ · ∥ 2 ) =

Original: sup Q log N ( ϵ, F , ∥ · ∥ L 2 ( Q ) ) , where Q is taken over all probability measures with 0 ≤ QF 2 ≤ ∞ .
Revised: sup Q log N ( ϵ, F , ∥ · ∥ L 2 ( Q ) ) , where Q is taken over all probability measures with 0 ≤ QF 2 ≤ ∞ .

Original: Definition 9.
Revised: Definition 9.

Original: A function φ ( r ) : [0 , ∞ ) (cid:55)→ [0 , ∞ ) is sub-root function if it is nondecreasing and r (cid:55)→ φ ( r ) / √ r is nonincreasing for r > 0 .
Revised: A function φ ( r ) : [0 , ∞ ) (cid:55)→ [0 , ∞ ) is sub-root function if it is nondecreasing and r (cid:55)→ φ ( r ) / √ r is nonincreasing for r > 0 .

Original: C. E XAMPLES S ATISFYING A SSUMPTIONC.1. E XAMPLE
Revised: C. E XAMPLES S ATISFYING A SSUMPTIONC.1. E XAMPLE

Original: In this subsection, we show that when the hypothesis class contains the (cid:98) h ∗ and h ∗ , Assumptionholds.
Revised: In this subsection, we show that when the hypothesis class contains the (cid:98) h ∗ and h ∗ , Assumptionholds.

Original: [Equation]
Revised: [Equation]

Original: Next we show that
Revised: Next we show that

Original: [Equation]
Revised: [Equation]

Original: Thus, if (cid:98) h ∗ ∈ H , we have
Revised: Thus, if (cid:98) h ∗ ∈ H , we have

Original: [Equation]
Revised: [Equation]

Original: Thus, the tuple (cid:16) D, ℓ, H , (cid:98)
Revised: Thus [Equation] the tuple (cid:16) D, ℓ, H , (cid:98)

Original: h ⋆ (cid:17) satisfies the Bernstein condition with parameter B 2 = 1 such that
Revised: h ⋆ (cid:17) satisfies the Bernstein condition with parameter B 2 = 1 such that

Original: [Equation]
Revised: [Equation]

Original: Similarly, we can prove that
Revised: Similarly [Equation] we can prove that

Original: [Equation]
Revised: [Equation]

Original: Thus
Revised: Y ) − [Equation] ( h [Equation]

Revised: ( X

Original: [Equation]
Revised: [Equation]

Original:  the tuple ( P, f, H , h ⋆ ) satisfies the Bernstein condition with parameter B 1 = 1 such that
Revised: , Y )] . Thus, the tuple ( P, f, H , h ⋆ ) satisfies the Bernstein condition with parameter B 1 = 1 such that

Original: [Equation]
Revised: [Equation]

Original: C.1. E XAMPLE
Revised: C.1. E XAMPLE

Original: In this example [Equation] we show that even when the hypothesis class does not include (cid:98) h ∗ and h ∗ , Assumption 2 still holds.
Revised: In this example, we show that even when the hypothesis class does not include (cid:98) h ∗ and h ∗ , Assumption 2 still holds.

Original: If F is a nonempty
Revised: If F is a nonempty, closed and convex subset of a Hilbert space with inner product

Original: [Equation]
Revised: [Equation]

Original: where (cid:98) h ∗ is the projection of Y in the space F .
Revised: where (cid:98) h ∗ is the projection of Y in the space F .

Original: By the definition we have ⟨ Y − (cid:98) h ∗ ( X) , h ( X ) − (cid:98) h ∗
Revised: By the definition we have ⟨ Y − (cid:98) h ∗ ( X

Original: [Equation]
Revised: [Equation]

Original:  X 
Revised: , h ( X ) −

Original: [Equation]
Revised: [Equation]

Original:  ⟩ ≤ 0 .
Revised: h ∗ ( X ) ⟩ ≤ 0 .

Original: [Equation]
Revised: [Equation]

Original: Thus, for regression problems with square loss function, where ( (cid:98) h ∗ / ∈ H ) , we have
Revised: Thus, for regression problems with square loss function, where ( (cid:98) h ∗ / ∈ H ) , we have

Revised: E ( X,Y ) ∼ D [ ℓ ( h ( X ) , Y

Original: [Equation]
Revised: [Equation]

Original: Similarly, we can prove that
Revised: Similarly [Equation] we can prove that

Original: [Equation]
Revised: [Equation]

Original: C. P ROOF OF T HEOREM
Revised: C. P ROOF OF T HEOREM

Original: Lemma 3 (Theorem 2.1 in (Bartlett et al., 2005)) .
Revised: Lemma 3 (Theorem 2.1 in (Bartlett et al., 2005)) .

Original: Let F be a family of functions satisfying ∀ f ∈
Revised: Let F be a family of functions satisfying ∀ f ∈

Original: F , ∀ Z ∈ Z , | f ( Z ) | ≤ b .
Revised: F , ∀ Z ∈ Z , | f ( Z ) | ≤ b .

Original: If sup f ∈F 1 m (cid:80) m i =1 E [ f ( Z i )] 2 ≤ r , then, for any δ ≥ 0 and any α ≥ 0 , with probability at least 1 − δ ,[Equation] m [Equation]
Revised: If sup f ∈F 1 m (cid:80) mi =1 E [ f ( Z i )] 2 ≤ r , then, for any δ ≥ 0 and any α ≥ 0 , with probability at least 1 − δ ,

Original: m

Original: [Equation]
Revised: [Equation]

Original: where R m ( F ) =
Revised: where R m ( F ) =

Original: E σ (cid:2) sup f ∈F 1 m (cid:80) mi =1 σ i f ( Z i ) (cid:3) .
Revised: E σ (cid:2) sup f ∈F 1 m (cid:80) mi =1 σ i f ( Z i ) (cid:3) .

Original: Lemma 4 (Theorem 1 in (Yousefi et al., 2018)) .
Revised: Lemma 4 (Theorem 1 in (Yousefi et al., 2018)) .

Original: Let F be a family of functions satisfying ∀ f ∈
Revised: Let F be a family of functions satisfying ∀ f ∈

Original: F , ∀ Z ∈ Z , | f ( Z ) | ≤ b . 
Revised: F , ∀ Z ∈ Z , | f ( Z ) | ≤ b . If sup f ∈F 1 mn [Equation] mi

Revised: [Equation]

Original: If sup f ∈F 1 mn (cid:80) mi =1 (cid:80) nj =1 E [ f ( Z ji )] 2 ≤ r , then, for any δ ≥ 0 and any α ≥ 0 , with probability at least 1 − δ ,
Revised: [Equation] nj [Equation] 1 E [ f ( Z ji )] 2 ≤ r , then, for any δ ≥ 0 and any α ≥ 0 , with probability at least 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original:  where R mn ( H ) =
Revised: mn , where R mn ( H ) =

Original: E σ (cid:104) sup f ∈F 1 mn (cid:80) mi =1 (cid:80) nj =

Revised: E σ (cid:104) sup f ∈F 1 mn (cid:80) mi =1 (cid:80) nj =

Original: σ ji f ( Z ji ) (cid:105) .
Revised: σ ji f ( Z ji ) (cid:105) .

Original: Lemma 5 (Lemma B.1 in (Yousefi et al., 2018)) .
Revised: Lemma 5 (Lemma B.1 in (Yousefi et al., 2018)) .

Original: Let c 1 , c 2 > 0 and s > q > 0 .
Revised: Let c 1 , c 2 > 0 and s > q > 0 .

Original: Then the equation x s − c 1 x q − c 2 = 0 has a unique positive solution x 0 satisfying
Revised: Then the equation x s − c 1 x q − c 2 = 0 has a unique positive solution x 0 satisfying

Original: [Equation]
Revised: [Equation]

Original: Moreover, for any x ≥ x 0 , we have x s ≥ c 1 x q + c 2 .
Revised: Moreover, for any x ≥ x 0 , we have x s ≥ c 1 x q + c 2 .

Original: Lemma 6 (Corollary 1 in Lei et al.
Revised: Lemma 6 (Corollary 1 in Lei et al.

Original: Let F be a function class with sup f ∈F ∥ f ∥ ∞ ≤ b .
Revised: Let F be a function class with sup f ∈F ∥ f ∥ ∞ ≤ b .

Original: Assume that there exist three positive numbers γ, d, p such that log N ( ϵ, F , ∥ · ∥ 2 ) ≤ d log p ( γ/ϵ ) for any 0 < ϵ ≤ γ , then for any 0 < r ≤ γ 2 and n ≥ γ − 2 there holds that
Revised: Assume that there exist three positive numbers γ, d, p such that log N ( ϵ, F , ∥ · ∥ 2 ) ≤ d log p ( γ/ϵ ) for any 0 < ϵ ≤ γ , then for any 0 < r ≤ γ 2 and n ≥ γ − 2 there holds that

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem 2.
Revised: Proof of Theorem 2.

Original: First we define (cid:98) F ∗ :=
Revised: First we define (cid:98) F ∗ :=

Original: { f : ( X, Y ) (cid:55)→ ℓ ( h ( X ) , Y ) − ℓ ( (cid:98) h ∗ ( X ) , Y ) , h ∈ H} .
Revised: { f : ( X, Y ) (cid:55)→ ℓ ( h ( X ) , Y ) − ℓ ( (cid:98) h ∗ ( X ) , Y ) , h ∈ H} .

Original: Step One: Combining Assumption 2 (a), for any f ∈ (cid:98) F ∗ we derive that 
Revised: Step One: Combining Assumption 2 (a), for any f ∈ (cid:98) F ∗ we derive that m [Equation] Y

Original: [Equation]
Revised: [Equation]

Revised: β .

Original: Let V ( f ) :
Revised: Let V ( f ) :

Original: =  L 
Revised: = 1 m (cid:80) mi =1 E (cid:104) ℓ ( h ( X 1 i ) , Y 1 i ) − ℓ ( (cid:98) h ∗ ( X 1 i ) , Y 1 i ) (cid:105) 2 , B = B ′ L 2 . Consider the function class G r associated with (cid:98) F ∗ and r ≥ 0 : G r := (cid:26) g :=

Revised: rf max

Original: [Equation]
Revised: [Equation]

Original: Consider the function class G r associated with (cid:98) F ∗ and r ≥ 0 :
Revised: [Equation] ( f )

Original: [Equation]
Revised: [Equation]

Revised: , f

Revised: [Equation]

Revised: (cid:98)

Revised: [Equation]

Revised: ∗ (cid:27) .

Original: We denote V + r by
Revised: We denote V + r by

Original: [Equation]
Revised: [Equation]

Original: Let K > 1 , 0 < β ≤ 1 and B ≥ 1 .
Revised: Let K > 1 , 0 < β ≤ 1 and B ≥ 1 .

Original: We first prove that, if V + r ≤ r 1 /β
Revised: We first prove that, if V + r ≤ r 1 /β

Original: BK , then
Revised: BK , then

Original: [Equation]
Revised: [Equation]

Revised: f ∈ F ,

Revised: [Equation]

Original: If V ( f ) ≤ r , then g = f .
Revised: If V ( f ) ≤ r , then g = f .

Original: It follows that with the assumption V + r ≤ r 1 /β
Revised: It follows that with the assumption V + r ≤ r 1 /β

Original: BK , we have
Revised: BK , we have

Original: [Equation]
Revised: [Equation]

Original: If V ( f ) ≥ r , then g = rf/V ( f ) .
Revised: If V ( f ) ≥ r , then g = rf/V ( f ) .

Original: It follows that with the assumption V + r ≤ r 1 /β
Revised: It follows that with the assumption V + r ≤ r 1 /β

Original: BK , we have
Revised: BK , we have

Original: [Equation]
Revised: [Equation]

Original: where the second inequality follows from Bernstein Condition and the third inequality follows from
Revised: where the second inequality follows from Bernstein Condition and the third inequality follows from

Original: Lemma5.
Revised: Lemma

Revised: [Equation]

Original: The obtained inequality can be rewritten as
Revised: The obtained inequality can be rewritten as

Original: [Equation]
Revised: [Equation]

Original: where the second inequality is due to b − c a − c ≤ ba for c < b ≤ a.
Revised: where the second inequality is due to b − c a − c ≤ ba for c < b ≤ a

Revised: [Equation]

Original: Step Two:
Revised: Step Two:

Original: By the construction of G r , it can be varified that
Revised: By the construction of G r , it can be varified that

Original: [Equation]
Revised: [Equation]

Original: In details:
Revised: In details:

Original: If V ( f ) ≤ r , we have g = f.
Revised: If V ( f ) ≤ r , we have g = f.

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: If V ( f ) > r [Equation] we have g = rf/V ( f ) .
Revised: If V ( f ) > r , we have g = rf/V ( f ) .

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: Combining the boundness of G r and Lemma 4, with probability at least 1 − δ, ∀ δ ≥ 0 , ∀ α ≥ 0 , we havesup g ∈G r
Revised: Combining the boundness of G r and Lemma 4, with probability at least 1 − δ, ∀ δ ≥ 0 , ∀ α ≥ 0 , we have

Original: (cid:34)mn m

Original: [Equation]
Revised: [Equation]

Original: mn
Revised: where

Original: [Equation]
Revised: [Equation]

Original: where R mn ( G r ) =

Original: E σ (cid:104) sup g ∈G r 1 mn (cid:80) mi =1 (cid:80) nj

Original: =1 σ ji g

Original: ( Z ji ) (cid:105) .

Original: Next we apply “peeling” technique to bound E
Revised: Next we apply “peeling” technique to bound E

Original: [ R mn ( G r )] .
Revised: [ R mn ( G r )] .

Original: Given λ > 1 , let (cid:98) F ∗ ( u, v ) :=
Revised: Given λ > 1 , let (cid:98) F ∗ ( u, v ) :=

Original: { f ∈ (cid:98) F ∗ : u ≤ V ( f ) ≤ v } and denote k as the smallest integer such that
Revised: { f ∈ (cid:98) F ∗ : u ≤ V ( f ) ≤ v } and denote k as the smallest integer such that

Original: rλ k +1 ≥ Bb β .
Revised: rλ k +1 ≥ Bb β .

Original: Then
Revised: Then

Original: E [

Original: [Table]

Original: [Equation]

Original: [Table]

Original: = n (cid:88)

Original: j =

Original: [Equation]

Original: [Equation]

Original: ji ) (cid:35)

Original: [Equation]
Revised: [Equation]

Original: mn m (cid:88) i = n (cid:88)

Original: j = [Equation] f ( [Equation]
Revised: [Table]

Original: ji ) (cid:35) + k (cid:88)

Original: j = λ − j E

Original: [Equation]

Original: ,σ (cid:34) sup f ∈ (cid:98) F

Original: [Equation]

Original: (0 ,rλ j +1 )

Original: mn m (cid:88) i = n (cid:88)

Original: j = σ ji f ( Z ji ) (cid:35)

Original: [Equation] [Equation] B . By the property of sub-root function it follows that we have ψ ( θr ) ≤ θ 12 ψ ( r ) for any θ ≥ 1 .
Revised: By the property of sub-root function it follows that we have ψ ( θr ) ≤ θ 12 ψ ( r ) for any θ ≥ 1 .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Taking λ = 4 it follows that E [ R mn ( G r )] ≤ 5 ψ ( r ) /B.
Revised: Taking λ = 4 it follows that E [ R mn ( G r )] ≤ 5 ψ ( r ) /B.

Original: Combining with the property ψ ( r ) ≤ (cid:112) r/r ∗ ψ ( r ∗ ) =
Revised: Combining with the property ψ ( r ) ≤ (cid:112) r/r ∗ ψ ( r ∗ ) =

Original: √
Revised: √

Original: rr ∗ .
Revised: rr ∗ .

Original: The following inequality can be obtained with probability at least 1 − δ, ∀ δ ≥ 0 ,
Revised: The following inequality can be obtained with probability at least 1 − δ, ∀ δ ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Original: where r ∗ is the fixed point of sub-root function ψ ( r ) .
Revised: where r ∗ is the fixed point of sub-root function ψ ( r ) .

Original: Step Three: Recall that the condition we get inequality (4) is
Revised: Step Three: Recall that the condition we get inequality (4) is

Original: [Equation]
Revised: [Equation]

Original: We denote A and C by
Revised: We denote A and C by

Original: [Equation]
Revised: [Equation]

Original:  Next, we need to solve A √ r + C ≤ r 1 /β
Revised: mn Next, we need to solve A √ r + C ≤ r 1 /β

Original: BK .
Revised: BK .

Original: Assume r 0 is the positive solution of A √ r + C =
Revised: Assume r 0 is the positive solution of A √ r + C =

Original: r 1 /β
Revised: r 1 /β

Original: BK , that is
Revised: BK , that is

Original: [Equation]
Revised: [Equation]

Original: Then by Lemma 5, we have
Revised: Then by Lemma 5, we have

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Original: where the second inequality follows from ( x + y ) p ≤ 2 p − 1 ( x p + y p ) for x, y ≥ 0 , p ≥ 1 .
Revised: where the second inequality follows from ( x + y ) p ≤ 2 p − 1 ( x p + y p ) for x, y ≥ 0 , p ≥ 1 .

Original: If r ∗ ≤ r 0 , we can take r = r 0 .
Revised: If r ∗ ≤ r 0 , we can take r = r 0 .

Original: Then we have V + r 0 ≤ A √ r 0 + C =
Revised: Then we have V + r 0 ≤ A √ r 0 + C =

Original: r 1 /β 

Revised: r 1 /β

Original: BK .
Revised: BK .

Original: Combining inequality 4, we get
Revised: Combining inequality 4, we get

Original: [Equation]
Revised: [Equation]

Original: If r ∗ > r 0 , we can take r = r ∗ .
Revised: If r ∗ > r 0 , we can take r = r ∗ .

Original: Then we have V + r ∗ ≤ A √ r ∗ + C =
Revised: Then we have V + r ∗ ≤ A √ r ∗ + C =

Original: ( r ∗ ) 1 /β BK .
Revised: ( r ∗ ) 1 /β BK .

Original: Combining inequality 4, we get
Revised: Combining inequality 4, we get

Original: [Equation]
Revised: [Equation]

Original: Step Four:
Revised: Step Four:

Revised: Note that ψ ( r ) is set to satisfy the following conditon,

Revised: [Equation]

Revised: where B = B 2 L 2 and

Original: Clearly G r ⊂ { f ∈ (cid:98) F ∗ : T mn ( f ) ≤ r } , where T mn ( f ) = 1 m (cid:80) mi =1 E Z ∼ D
Revised: Clearly G r ⊂ { f ∈ (cid:98) F ∗ : T mn ( f ) ≤ r } , where T mn ( f ) = 1 m (cid:80) mi =1 E Z ∼ D

Original: i
Revised: i

Original: [ f ( Z )] 2 .
Revised: [ f ( Z )] 2 .

Original: Thus,
Revised: Thus,

Original: E [ R mn ( G r )] =

Original: [Equation]

Original: Moreover, ψ ( r ) is set to satisfy the following conditon,

Original: [Equation]
Revised: [Equation]

Original: Lemma 6 implies that the sub-root function can be chosen as
Revised: Lemma 6 implies that the sub-root function can be chosen as

Original: [Equation]
Revised: [Equation]

Original: In a similar way to Lei et al.
Revised: In a similar way to Lei et al.

Original: [Equation] let r ∗ mn be its fixed point then we know that
Revised: (2016), let r ∗ mn be its fixed point then we know that

Original: [Equation]
Revised: [Equation]

Original: Solving this equality gives r ∗ mn ≤ cd ( mn ) − 1 log p ( mn ) .
Revised: Solving this equality gives r ∗ mn ≤ cd ( mn ) − 1 log p ( mn ) .

Original: Combining the fact that 1 mn (cid:80) mi =1 (cid:80) nj =1 (cid:104) ℓ ( (cid:98) h
Revised: Combining the fact that 1 mn (cid:80) mi =1 (cid:80) nj =1 (cid:104) ℓ ( (cid:98) h

Original: ( X ji ) , Y ji ) − ℓ ( (cid:98) h ∗ ( X ji ) , Y ji ) (cid:105) ≤ 0 and the result of step three [Equation] we get the following results
Revised: ( X ji ) , Y ji ) − ℓ ( (cid:98) h ∗ ( X ji ) , Y ji ) (cid:105) ≤ 0 and the result of step three, we get the following results

Original: [Equation]
Revised: [Equation]

Original: When mn ≥ cd log p ( mn ) , we have max(( r ∗ ) 12 − β , ( r ∗
Revised: When mn ≥ cd log p ( mn ) , we have max(( r ∗ ) 12 − β , ( r ∗

Original: [Equation]
Revised: [Equation]

Original: 1 β ) =
Revised: 1 β ) =

Original: [Equation]

Original:  r ∗ ) 12 − β .
Revised: ( r ∗ ) 12 − β .

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: C. P ROOF OF T HEOREM
Revised: C. P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: First we define F ∗ :=
Revised: First we define F ∗ :=

Original: { f : ( X, Y ) (cid:55)→ ℓ ( h ( X ) , Y ) − ℓ ( h ∗ ( X ) , Y ) , h ∈ H} .
Revised: { f : ( X, Y ) (cid:55)→ ℓ ( h ( X ) , Y ) − ℓ ( h ∗ ( X ) , Y ) , h ∈ H} .

Original: Step One: Combining Assumption 2 (b), for any f ∈ F ∗ , we have
Revised: Step One: Combining Assumption 2 (b), for any f ∈ F ∗ , we have

Original: [Equation]
Revised: [Equation]

Original: Let V ( f ) :
Revised: Let V ( f ) :=

Revised: E D i ∼ P E ( X,Y ) ∼ D

Original: = L 2 E [  h ( X 1 ) −  h ∗ ( X 1 )] 2 , B = B ′′ L 2 .
Revised: i [ ℓ ( h ( X ) , Y ) − ℓ ( h ∗ ( X ) , Y )] 2 , B = B ′′ L 2 .

Original: Consider the function class G r associated with F ∗ and r ≥ 0 :
Revised: Consider the function class G r associated with F ∗ and r ≥ 0 :

Original: G r := (cid:26)

Original: [Equation]
Revised: [Equation]

Original: We denote V + r by
Revised: We denote V + r by

Original: [Equation]
Revised: [Equation]

Original: Let K > 1 , 0 < β ≤ 1 and B ≥ 1 .
Revised: Let K > 1 , 0 < β ≤ 1 and B ≥ 1 .

Original: We first prove that, if V + r ≤ r 1 /β
Revised: We first prove that, if V + r ≤ r 1 /β

Original: BK , then
Revised: BK , then

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: If V ( f ) ≤ r , then g = f .
Revised: If V ( f ) ≤ r , then g = f .

Original: It follows that with the assumption V + r ≤ r 1 /β
Revised: It follows that with the assumption V + r ≤ r 1 /β

Original: BK , we have
Revised: BK , we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: If V ( f ) ≥ r , then g = rf/V ( f ) .
Revised: If V ( f ) ≥ r , then g = rf/V ( f ) .

Original: It follows that with the assumption V + r ≤ r 1 /β
Revised: It follows that with the assumption V + r ≤ r 1 /β

Original: BK , we have
Revised: BK , we have

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where the second inequality follows from Bernstein Condition and the third inequality follows from
Revised: where the second inequality follows from Bernstein Condition and the third inequality follows from

Original: Lemma 5.
Revised: Lemma 5.

Original: The obtained inequality can be rewritten as
Revised: The obtained inequality can be rewritten as

Original: [Equation]
Revised: [Equation]

Original: Step Two:
Revised: Step Two:

Original: By the construction of G r , it can be varified that
Revised: By the construction of G r , it can be varified that

Original: [Equation]
Revised: [Equation]

Original: In details:
Revised: In details:

Original: First, we have
Revised: First, we have

Original: [Equation]
Revised: [Equation]

Original: If V ( f ) ≤ r
Revised: If V ( f ) ≤ r

Original: [Equation]
Revised: [Equation]

Original: we have g = f.
Revised: we have g = f.

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: If V ( f ) > r
Revised: If V ( f ) > r

Original: [Equation]
Revised: [Equation]

Original: we have g = rf/V ( f ) .
Revised: we have g = rf/V ( f ) .

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: Combining the boundness of G r and Lemma
Revised: Combining the boundness of G r and Lemma

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − δ, ∀ δ > 0 , ∀ α > 0 , we have
Revised: with probability at least 1 − δ, ∀ δ > 0 , ∀ α > 0 , we have

Original: [Equation]
Revised: [Equation]

Original:  Thus, we getsup
Revised: m . Thus, we get

Original: [Equation]
Revised: [Equation]

Original: ln(
Revised: ln(

Original: [Equation]
Revised: [Equation]

Original: δ
Revised: δ

Original: [Equation]
Revised: [Equation]

Original: m .
Revised: m .

Original: where R m ( G r ) =
Revised: where R m ( G r ) =

Original: E σ (cid:2) sup g ∈G r 1 m (cid:80) mi =1 σ i g
Revised: E σ (cid:2) sup g ∈G r 1 m (cid:80) mi =1 σ i g

Original: ( Z i ) (cid:3) .
Revised: ( Z i ) (cid:3) .

Original: By “peeling” technique and similar following steps in the proof of Theorem 2, it follows with prob- ability at least 1 − δ, ∀ δ ≥
Revised: By “peeling” technique and similar following steps in the proof of Theorem 2, it follows with probability at least 1 − δ, ∀ δ ≥

Original: [Equation]
Revised: [Equation]

Original: where r ∗ is the fixed point of sub-root function ψ ( r ) .
Revised: where r ∗ is the fixed point of sub-root function ψ ( r ) .

Original: Then by Lemma 5, we have
Revised: Then by Lemma 5, we have

Original: [Equation]
Revised: [Equation]

Original: Clearly G r ⊂ { f ∈ (cid:98) F ∗ : T mn ( f ) ≤ r } , where T mn ( f ) = 1 m (cid:80) mi =1 E Z ∼ D

Original: i

Original: [ f ( Z )] 2 .

Original: Thus,

Original: E [ R mn ( G r )] =

Original: E S 1 , ··· ,S m ,σ   sup g ∈G r mn m (cid:88) i = n (cid:88)

Original: j = σ ji g ( Z ji ) 

Original:  ≤

Original: E S 1 , ··· ,S m ,σ 

Original:  sup T mn ( f ) ≤ r f ∈ (cid:98) F ∗ mn m (cid:88) i = n (cid:88)

Original: j = σ ji f ( X ji , Y ji ) 

Original:  =

Original: R mn ( (cid:98) F ∗ , r ) .

Original: C. P ROOF OF T HEOREM

Original: Proof.

Original: Let F be the loss function class.
Revised: Let F be the loss function class.

Original: We consider the functional T ( f ) :
Revised: We consider the functional T ( f ) :

Original: =
Revised: =

Original: Pf 2 here.
Revised: Pf 2 here.

Original: The structural result on covering numbers implies that
Revised: The structural result on covering numbers implies that

Original: [Equation]
Revised: [Equation]

Original: Lemma 6 implies that the sub-root function can be chosen as
Revised: Lemma 6 implies that the sub-root function can be chosen as

Original: [Equation]
Revised: [Equation]

Original: In a similar way to Lei et al.
Revised: In a similar way to Lei et al.

Original: (2016), let r ∗
Revised: (2016), let r ∗

Original: m
Revised: m

Original: be its fixed point then we know that
Revised: be its fixed point then we know that

Original: [Equation]
Revised: [Equation]

Original: Solving this equality gives r ∗ m ≤ cdm − [Equation] log p ( m ) .
Revised: Solving this equality gives r ∗ m ≤ cdm − 1 log p ( m ) .

Original: It can be directly checked that any f ∈ F also satisfies ∥ f ∥ ∞ ≤ 4 b 2 .

Original: D P ROOFS OF THE RESULTS WITH SUB - EXPONENTIAL LOSSES D. L EARNING R ATES FOR S UB - EXPONENTIAL L OSSES
Revised: D P ROOFS OF THE RESULTS WITH SUB - EXPONENTIAL LOSSES D. L EARNING R ATES FOR S UB - EXPONENTIAL L OSSES

Original: We state our results on the convergnece rate of generalization error for sub-exponential losses.
Revised: We state our results on the convergnece rate of generalization error for sub-exponential losses.

Original: First, we consider the participating clients.
Revised: First, we consider the participating clients.

Original: Let S
Revised: Let S

Original: =
Revised: =

Original: { Z ji } ( m,n ) ( i,j =1 , 1) be global data samples whose subsets
Revised: { Z ji } ( m,n ) ( i j =1 , 1) be global data samples whose subsets

Original: S i =
Revised: S i =

Original: { Z j i } nj =1 include i.i.d random variables at i -th client.
Revised: { Z ji } nj =1 include i.i.d random variables at i -th client.

Original: Theorem 6 (Participating error for sub-exponential losses) .
Revised: Theorem 6 (Participating error for sub-exponential losses) .

Original: Suppose Z ji take valued in a Banach space ( Z , ∥ · ∥ ) and each ∥
Revised: Suppose Z ji take valued in a Banach space ( Z , ∥ · ∥ ) and each ∥

Original: Z ji ∥ is sub-exponential distributed.
Revised: Z ji ∥ is sub-exponential distributed.

Original: We denote by F = { Z (cid:55)→ ℓ ( h, Z ) : h ∈ H} such that, ∀ f ∈ F and ∀ z, z ′ ∈ Z , | f ( z ) − f ( z ′ )
Revised: We denote by F = { Z (cid:55)→ ℓ ( h, Z ) : h ∈ H} such that, ∀ f ∈ F and ∀ z, z ′ ∈ Z , | f ( z ) − f ( z ′ )

Original: | ≤ L ∥
Revised: | ≤ L ∥

Original: z − z ′ ∥ .
Revised: z − z ′ ∥ .

Original: For any δ > 0 , if mn ≥ ln(1 /δ ) ≥ ln 2 , then with probability at least 1 − δ , we have
Revised: For any δ > 0 , if mn ≥ ln(1 /δ ) ≥ ln 2 , then with probability at least 1 − δ , we have

Original: [Equation]
Revised: [Equation]

Original: where R mn ( F ) =
Revised: where R mn ( F ) =

Original: E σ (cid:104) sup [Equation] ∈F 1 mn (cid:80) [Equation]{+mi =1 (cid:80) nj =
+}
Revised: E σ (cid:104) sup f ∈F 1 mn (cid:80) {+mi =1 (cid:80) nj =
+}

Original:  ji [Equation] Z [Equation]
Revised: σ ji f ( Z ji ) (cid:105) .

Original: Remark 9.
Revised: Remark 9.

Original: Theorem 6 can be used to bound semi-empirical excess by applying standard uniformly supremum of h ∈ H .
Revised: Theorem 6 can be used to bound semi-empirical excess by applying standard uniformly supremum of h ∈ H .

Original: It is worth emphasizing that the bounds derived in Theoreminclude Rademacher complexity term and ∥∥
Revised: It is worth emphasizing that the bounds derived in Theoreminclude Rademacher complexity term and ∥∥

Original: Z 

Revised: Z

Original: i ∥∥ ψ 1 measuring the tails of input data samples { Z ji } ( m,n ) ( i,j =1 , 1) .
Revised: i ∥∥ ψ 1 measuring the tails of input data samples { Z ji } ( m,n ) ( i,j =1 , 1) .

Original: Intuitively, in regression problems, as the noise added to the lables increases, it is expected that participating error increase as well.
Revised: Intuitively, in regression problems, as the noise added to the lables increases, it is expected that participating error increase as well.

Original: This phenomenon is ignored under the previous bounded assumption on losses.
Revised: This phenomenon is ignored under the previous bounded assumption on losses.

Original: Example 1 (Linear regression with unbounded loss) .
Revised: Example 1 (Linear regression with unbounded loss) .

Original: Let Z =
Revised: Let Z =

Original: ( X , R ) , where X is a Hilbert-space with norm ∥ · ∥ H .
Revised: ( X , R ) , where X is a Hilbert-space with norm ∥ · ∥ H .

Original: We denote by ( X ji , Y ji ) each sub-exponential random variables in H and R respectively.
Revised: We denote by ( X ji , Y ji ) each sub-exponential random variables in H and R respectively.

Original: Let loss function ℓ be a 1-Lipschitz function (absolute function or Huber loss) and
Revised: Let loss function ℓ be a 1-Lipschitz function (absolute function or Huber loss) and

Original: F = { ( x, y )
Revised: F = { ( x, y )

Original: → f ( x, y ) = ℓ ( ⟨ w, x ⟩ − y ) :
Revised: → f ( x, y ) = ℓ ( ⟨ w, x ⟩ − y ) :

Original: ∥ w ∥ H ≤ L } .
Revised: ∥ w ∥ H ≤ L } .

Original: If mn ≥ ln(1 /δ ) ≥ ln 2 , it follows with probability at least 1 − δ
Revised: If mn ≥ ln(1 /δ ) ≥ ln 2 , it follows with probability at least [Equation] δ

Original: [Equation]
Revised: [Equation]

Original: Theorem 7 (Participation gap with unbounded loss) .
Revised: Theorem 7 (Participation gap with unbounded loss) .

Original: Under the same conditons as Theorem 6 and
Revised: Under the same conditons as Theorem 6 and

Original: Example 1, we have
Revised: Example 1, we have

Original: [Equation]
Revised: [Equation]

Original: where X 1 is random vector with expectation across two-level distribution.
Revised: where X 1 is random vector with expectation across two-level distribution.

Original: This is, E [ ∥ X 1 ∥ ] =
Revised: This is, E [ ∥ X 1 ∥ ] =

Original: E
Revised: E

Original: D

Original: [Equation] P E [Equation] ∼ D i ∥ X 1 ∥ .
Revised: D i ∼ P E X 1 ∼ D i ∥ X 1 ∥ .

Original: Similarly, E [ ∥ Y 1 ∥ ] = E D [Equation] ∼ P E Y 1 ∼ D i ∥ Y 1 ∥ .
Revised: Similarly, E [ ∥ Y 1 ∥ ] = E D i ∼ P E Y 1 ∼ D i ∥ Y 1 ∥ .

Original: Remark 10.
Revised: Remark 10.

Original: Combining the results of Theorem 6 and Theorem 7, it can be shown that upper bound of excess risk is of order O ( 1 √ mn + 1 √ m ) .
Revised: Combining the results of Theorem 6 and Theorem 7, it can be shown that upper bound of excess risk is of order O ( 1 √ mn + 1 √ m ) .

Original: Though this bound is derived under the unbounded assumption, its order is comparable with basic results derived in Theorem 1.
Revised: Though this bound is derived under the unbounded assumption, its order is comparable with basic results derived in Theorem 1.

Original: Note that the upper bounds in Theorem 7 include terms such as ∥ X 1 ∥ and ∥ Y 1 ∥ , whose underlying distributions are across our two-level framework.
Revised: Note that the upper bounds in Theorem 7 include terms such as ∥ X 1 ∥ and ∥ Y 1 ∥ , whose underlying distributions are across our two-level framework.

Original: This reflects that the participation gap captures the generalization error caused by client sampling.
Revised: This reflects that the participation gap captures the generalization error caused by client sampling.

Original: Lemma 7 (Theorem 3.1 in Maurer & Pontil (2021)) .
Revised: Lemma 7 (Theorem 3.1 in Maurer & Pontil (2021)) .

Original: Let f : X n → R and X =
Revised: Let f : X n → R and X =

Original: ( X 1 , . . .
Revised: ( X 1 , . . .

Original: , X n ) be a random vector whose elements are independent and take values in a space X .
Revised: , X n ) be a random vector whose elements are independent and take values in a space X .

Original: Then for any t ≥P
Revised: Then for any t ≥

Original: [Equation]

Original:  

Original: [Equation]
Revised: [Equation]

Original:  
Revised:  

Original: P ROOF OF T HEOREM
Revised: P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: We first define a vector space
Revised: We first define a vector space

Original: [Equation]
Revised: [Equation]

Original: By definition, B is a normed space with norm ∥ g ∥ B = sup f ∈F | g
Revised: By definition, B is a normed space with norm ∥ g ∥ B = sup f ∈F | g

Original: ( f ) | .
Revised: ( f ) | .

Original: For each Z ji ∈ Z , we define ˆ Z ji ( f ) by ( mn ) − 1 ( f ( Z ji ) − E [ f ( Z ′ ji )]) .
Revised: For each Z ji ∈ Z , we define ˆ Z ji ( f ) by ( mn ) − 1 ( f ( Z ji ) − E [ f ( Z ′ ji )]) .

Original: Thus, E [ ˆ Z ji ] ≡ 0 , and (cid:13) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:88) i (cid:88) j
Revised: Thus, E [ ˆ Z ji ] ≡ 0 , and (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:88)

Original: [Equation]
Revised: [Equation]

Original: ji (cid:13) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
Revised: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) .

Original: B =

Original: [Equation] (cid:12) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
Revised: From Lemma 7, we have (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:88) i (cid:88) j

Original: [Equation]
Revised: [Equation]

Original: i (cid:88) j ( f ( Z ji ) − E [ f ( Z ′ ji )]) (cid:12) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) .

Original: From Lemma 7, we have (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:88) i (cid:88)

Original: j ˆ Z ji (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Original: B − E   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:88) i (cid:88)

Original: j ˆ Z ji (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) B   ≤ (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:88) i (cid:88)j ˆ Z ji − E  (cid:88) i (cid:88) j ˆ Z ji   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
Revised: (cid:88)

Original: B ≤ max i ∈ [ m ] 8 e √ mn ∥∥ ˆ Z 1 i ∥ B ∥ ψ 1 (cid:112) 2 ln(1 /δ ) .
Revised: [Equation]

Original: Observe that ∥∥ ˆ Zji
Revised: Observe that ∥∥ ˆ Z

Original: [Table]
Revised: [Equation]

Original: ∥

Original: [Table]
Revised: [Table]

Original: = 1 mn [Table] sup f [Table]
Revised: we get max i ∈ [ m ] ∥∥ ˆ Z

Original: E

Original: [Table]
Revised: [Table]

Original: f (
Revised: ∥

Original: [Table]
Revised: [Table]

Original: f
Revised: max

Original: [Table]
Revised: [Table]

Original: Therefore, we get max i ∈ [ m ]∥∥ ˆ Z 1 i ∥ B ∥ ψ 1 ≤ max i ∈ [ m ] 2 Lmn ∥∥ Z 1 i ∥∥ ψ 1 , and (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:88) i (cid:88) j
Revised: [ m ]

Original: [Table]
Revised: [Table]

Original: Z ji [Table] −
Revised: Lmn

Original: [Table]
Revised: [Table]

Original: By symmetrization, we have
Revised: By symmetrization, we have

Original: E 

Original: [Equation]

Original: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:88) i (cid:88)

Original: j

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: B

Original: [Table] [Equation] 
Revised: [Equation] [Table]

Original: F

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: mn

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation][Table]   
Revised: [Equation]

Original: 

Original: [Table]
Revised: [Table]

Original: P ROOF OF E XAMPLE
Revised: P ROOF OF E XAMPLE

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: Next, using Jensen’s inequality we can see that
Revised: Next, using Jensen’s inequality we can see that

Original: [Equation]
Revised: [Equation]

Revised:  

Original: 
Revised: 

Original: By the assumption that σ 1 , 1 , . . . , σ m,n are independent, then we have
Revised: By the assumption that σ 1 , 1 , . . . , σ m,n are independent, then we have

Original: Eσ (cid:20)(cid:13)(cid:13) (cid:13) (cid:13) n (cid:88) i = m (cid:88) j = σ ij X ji (cid:13)(cid:13) (cid:13) (cid:13) H (cid:21) = E σ   ( m,n ) (cid:88)
Revised: E

Original: ( i,j )=(1 , 1) ( m,n ) (cid:88) ( s,k )=(1 , 1) σ ij σ sk ⟨ X ji

Original: [Equation]
Revised: [Equation]

Original: ks ⟩   = E
Revised: n

Original: [Equation]
Revised: [Equation]

Original:     (cid:88) ( i,j ) ̸ =( s,k ) [Equation]
Revised: Thus, we have E σ   ∥ n (cid:88) i

Original: ( m[Equation] n ) (cid:88) ( i,j )=(1 , 1)
Revised: = m (cid:88) j =

Original: [Equation]
Revised: [Equation]

Original: ji ⟩   = E σ   ( m,n ) (cid:88) ( i,j )=(1 , 1) σ 2 ij⟨ X ji , X ji ⟩   = (cid:88) ( i,j ) ∥ X ji ∥ 2 H .
Revised: ij

Original: Thus, we have E σ   ∥ n (cid:88) i = m (cid:88)
Revised: [Equation]

Original: j = σ ij X ji ∥ H   ≤ (cid:115) (cid:88) ( i,j ) ∥
Revised: j i ∥ H  [Equation] (cid:115)(cid:88) ( i

Revised: [Equation]

Revised: j

Revised: [Equation]

Revised: ∥

Revised: [Equation]

Original: X ji ∥ 2 H .
Revised: j i ∥ 2 H .

Original: Similarly, we have
Revised: Similarly, we have

Original: E σ   n (cid:88) i = m (cid:88)
Revised: E σ   n (cid:88) i = m (cid:88)

Original: j = σ ij Y ji   ≤ (cid:115)(cid:88) ( i,j ) | Y ji |
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Since X 11 , · · · , X nm are independent and ∥ · ∥ 2 ≤ 2 ∥ · ∥ ψ 1 , we get
Revised: Since X 11 , · · · , X nm are independent and ∥ · ∥ 2 ≤ 2 ∥ · ∥ ψ 1 , we get

Original: [Equation]
Revised: [Equation]

Original: P ROOF OF T HEOREM
Revised: P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: We first define a vector space
Revised: We first define a vector space

Original: [Equation]
Revised: [Equation]

Original: By definition, B is a normed space with norm ∥ g ∥ B = sup f ∈F | g
Revised: By definition, B is a normed space with norm ∥ g ∥ B = sup f ∈F | g

Original: ( f ) | .
Revised: ( f ) | .

Original: Let f ( D i )
Revised: Let f ( D i )

Original: =
Revised: =

Original: E Z 1 i ∼ D i [ f ( Z 1 i )] and E [ f ( D i )] =
Revised: E Z 1 i ∼ D i [ f ( Z 1 i )] and E [ f ( D i )] =

Original: E D i ∼ P
Revised: E D i ∼ P

Original: [ E Z 1 i ∼ D i [ f ( Z 1 i )]] .
Revised: [ E Z 1 i ∼ D i [ f ( Z 1 i )]] .

Original: For each Z 1 i ∈ Z , we define ˆ Z i by (1 /m )
Revised: For each Z 1 i ∈ Z , we define ˆ Z i by (1 /m )

Original: ([ f ( D i )] −
Revised: ([ f ( D i )] −

Original: E [ f ( D i )]) .
Revised: E [ f ( D i )]) .

Original: Thus, E [ ˆ Z i ] ≡ 0 , and 
Revised: Thus, E [ ˆ Z i ] ≡ 0 , and (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:88) i [Equation] m

Original: [Equation]
Revised: [Equation]

Original: From Lemma 7, we have (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:88)
Revised: From Lemma 7, we have (cid:13)(cid:13) (cid:13)(cid:13)(cid:13) (cid:88)

Original: [Table]
Revised: i ˆ Z i (cid:13)(cid:13) (cid:13)(cid:13)(cid:13) B − E (cid:34)(cid:13)(cid:13) (cid:13)(cid:13)(cid:13) (cid:88) i ˆ Z i (cid:13)(cid:13) (cid:13)(cid:13)(cid:13) B (cid:35) ≤ (cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Revised: Observe that

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: that

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: ψ≤m (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) sup f ∈F | f ( D i ) − f ( D ′ i ) | (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ψ. E (cid:34)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) sup f ∈F | f 
Revised: f ( D i ) − f ( D ′ i ) | (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) p (cid:35) ≤ E (cid:34)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) sup f ∈F | f ( Z i )

Original: ( D i ) − f

Original: ( D ′ i ) | (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) p (cid:35) ≤ E (cid:34)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) sup f ∈F | f ( Z i )

Original: − f ( Z ′ i ) | (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) p (cid:35) ≤ E (cid:2)
Revised: − f ( Z ′ i ) | (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) p (cid:35) ≤ E (cid:2)

Original: | L ∥
Revised: | L ∥

Original: Z i − Z
Revised: Z i − Z

Original: i ∥| p (cid:3) .
Revised: i ∥| p (cid:3) .

Original: Therefore (cid:13) (cid:13) sup f ∈F | f
Revised: Therefore (cid:13)(cid:13) sup f ∈F | f

Original: ( D i ) − f
Revised: ( D i ) − f

Original: ( D ′ i ) | (cid:13)(cid:13) p ≤ ∥
Revised: ( D ′ i ) | (cid:13)(cid:13) p ≤ ∥

Original: L ∥
Revised: L ∥

Original: Z i − Z
Revised: Z i − Z

Original: i ∥∥ p and (cid:13) (cid:13) sup f ∈F | f
Revised: i ∥∥ p and (cid:13)(cid:13) sup f ∈F | f

Original: ( D i ) − f
Revised: ( D i ) − f

Original: ( D ′ i ) | (cid:13)(cid:13) ψ 1 ≤ ∥ L ∥
Revised: ( D ′ i ) | (cid:13)(cid:13) ψ 1 ≤ ∥ L ∥

Original: Z i − Z
Revised: Z i − Z

Original: ′ i ∥∥ ψ 1 .
Revised: ′ i ∥∥ ψ 1 .

Original: Then we get ∥∥ ˆ Z i
Revised: Then we get ∥∥ ˆ Z i

Original: ∥
Revised: ∥

Original: B ∥ ψ 1 ≤ 2 Lm
Revised: B ∥ ψ 1 ≤ 2 Lm

Original: ∥∥
Revised: ∥∥

Original: Z
Revised: Z

Original: i ∥∥ ψ 1 , and (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:88)
Revised: i ∥∥ ψ 1 , and (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:88)

Original: i ˆ Z i
Revised: i ˆ Z i

Original: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
Revised: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Original: B − E (cid:34)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:88) i ˆ Z 1 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
Revised: B − E (cid:34)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:88) i ˆ Z 1 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Original: B (cid:35) ≤ 16 eL
Revised: B (cid:35) ≤ 16 eL

Original: ∥∥
Revised: ∥∥

Original: Z i ∥∥ ψ(cid:114) 2 ln(1 /δ ) m .
Revised: Z i ∥∥ ψ(cid:114) 2 ln(1 /δ ) m .

Original: By symmetrization, we have
Revised: By symmetrization, we have

Original: E (cid:34)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:88) i 
Revised: E (cid:34)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:88) i ˆ Z

Original: ˆ Z i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
Revised: [Table]

Original: B (cid:35) ≤
Revised: (cid:34) [Table]

Original: E D 1 , ···
Revised: mn

Original: [Equation]
Revised: [Table]

Original:  Dm (cid:34)
Revised: [Table] D

Original: E σ (cid:34)  supf ∈F mn m (cid:88) i =
Revised: [Table] (cid:34) [Table] sup

Original: σ i f ( D i ) | D (cid:35)(cid:35) ≤ E Z 1 , ··· ,Z m (cid:34)
Revised: [Table] mn m [Table]

Original: E σ (cid:34) sup f ∈F mn m (cid:88) i = 
Revised: = [Table] f ( [Table]

Original: σ i f ( Z i ) | Z (cid:35)(cid:35) ≤ 
Revised: i ) | [Table] ≤ [Table]

Revised: E

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: E Z 1 , ··· ,Z m [ R m ( F )] ≤ 8 √ m ( L ∥∥ X 1 ∥∥ ψ 1 + ∥∥ Y 1 ∥∥ ψ 1 ) .
Revised: m ( L [Table] Y [Table]

Original: E P ROOFS OF S MALL - BALL BASED M ETHOD
Revised: E P ROOFS OF S MALL - BALL BASED M ETHOD

Original: Lemma 8 (Theorem 1 in (Zhang & Wei, 2022)) .
Revised: Definition 10. Let H ⊂ L 2 ( D ) be a closed and convex class of functions and H − H : = { h − h ′ :

Revised: h, h ′ ∈ H} .

Revised: Let Q H ( τ ) =

Revised: inf h ∈H P ( | h ( X 1 i )

Revised: | ≥ τ ∥

Revised: h ∥

Revised: L 2 ( D ) ) , where X 1 i represent the random sample at i -th participating client.

Revised: Let Q H ( τ, P ) =

Revised: inf h ∈H P (cid:0) | E [ h ( X 1 i )]

Revised: | ≥ τ ∥ h ∥

Revised: L 2 ( P ) (cid:1) , where X 1 i represent the random sample at i -th participating client.

Revised: Definition 11.

Revised: We denote by B m 2 the L 2 ( D ) unit ball entered at (cid:98) h ∗ , that is B m 2 = { h ∈ H :

Revised: ∥ h − (cid:98) h ∗ ∥

Revised: L 2 ( D ) ≤ 1 } .

Revised: For every η > 0 , define

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: where σ ji are Rademacher random variables.

Revised: Definition 12.

Revised: We denote by B 2 the L 2 ( P ) unit ball entered at h ∗ .

Revised: For every η > 0 , define

Revised: [Equation]

Revised: where σ i are Rademacher random variables.

Revised: Lemma 8 (Theorem 1 in (Zhang & Wei, 2022)) .

Original: If { X i } ni =1 are independent centralized random variables such that ∥ X i ∥
Revised: If { X i } ni =1 are independent centralized random variables such that ∥ X i ∥

Original: ψ α < ∞ for all 1 ≤ i ≤ n and some 1 > α > 0 , then for any weight vector w = ( w 1 , . .
Revised: ψ α < ∞ for all 1 ≤ i ≤ n and some 1 > α > 0 , then for any weight vector w = ( w 1 , . .

Original: . , w n ) ∈ R n , the following bounds holds true:
Revised: . , w n ) ∈ R n , the following bounds holds true:

Original: [Equation]
Revised: [Equation]

Original: E. P ROOF OF T HEOREM
Revised: E. P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: Here we use a different decomposition to measure the distance between (cid:98) h and (cid:98)

Original: h ∗ :
Revised: Step One:

Original: [Equation]
Revised: [Equation]

Original: Step One: The second term of the RHS of (27) is determined by the underline semi-empirical distribution D and the hypothesis space H , therefore we focus on the first term in the following.
Revised: The second term of the RHS of (27) is determined by the underline semi-empirical distribution D and the hypothesis space H , therefore we focus on the first term in the following.

Original: For any h ∈ H and u > 0 , 
Revised: For any h ∈ H and u > 0 , |{ ( i, j ) : h ( X ji ) > u }| = m (cid:88) [Equation] n (cid:88) j

Original: [Equation]
Revised: [Equation]

Original:  Also, 
Revised: {| [Equation] X ji ) |≥ u } . Also, mn m

Original: [Equation]
Revised: [Equation]

Original: Let ϕ u : R → [0 , 1] be the function
Revised: Let ϕ u : R → [0 , 1] be the function

Original: [Equation]
Revised: [Equation]

Original: Note that for every t ∈ R , ϕ u ( t ) ≥ 1 { t ≥ 2 u } and ϕ u ( t ) ≤ 1 { t ≥ u } .
Revised: Note that for every t ∈ R , ϕ u ( t ) ≥ 1 { t ≥ 2 u } and ϕ u ( t ) ≤ 1 { t ≥ u } .

Original: Thus,mn
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: [Table] Since function ϕ u ( t ) is bounded by 1 , using Mcdiarmid’s inequality, we get that, for every δ > 0 , with probability at least 1 − 2 exp( − 2 δ 2 ) ,
Revised: Since function ϕ u ( t ) is bounded by 1 , using Mcdiarmid’s inequality, we get that, for every δ > 0 , with probability at least 1 − 2 exp( − 2 δ 2 ) ,

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table] By the Lipschitz property of ϕ u ( | t | ) and the symmetrization theorem, we have
Revised: By the Lipschitz property of ϕ u ( | t | ) and the symmetrization theorem, we have

Original: E sup
Revised: E sup

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: ∈H

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: mn

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] Therefore, for every h ∈ H , it follows that with probability at least 1 − 2 exp( − 2 δ 2 ) , we have 
Revised: Therefore, for every h ∈ H , it follows that with probability at least 1 − 2 exp( − 2 δ 2 ) , we have mn

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: Step Two:
Revised: Step Two:

Original: The first term on the RHS can be bounded by small ball condtion.
Revised: The first term on the RHS can be bounded by small ball condtion.

Original: Let H ∗
Revised: Let H ∗

Original: =
Revised: =

Original: H − (cid:98) h ∗ .
Revised: H − (cid:98) h ∗ .

Original: We first prove that H ∗ is star-shaped around 0 .
Revised: We first prove that H ∗ is star-shaped around 0 .

Original: For every h − (cid:98) h ∗ ∈ H ∗ and 0 ≤ λ ≤ 1 , we have λ ( h − (cid:98) h ∗ )
Revised: For every h − (cid:98) h ∗ ∈ H ∗ and 0 ≤ λ ≤ 1 , we have λ ( h − (cid:98) h ∗ )

Original: = λh + (1 − λ ) (cid:98) h ∗ − (cid:98) h ∗ .
Revised: = λh + (1 − λ ) (cid:98) h ∗ − (cid:98) h ∗ .

Original: Since H is convex, it follows that λh + (1 − λ ) (cid:98) h ∗ ∈ H .
Revised: Since H is convex, it follows that λh + (1 − λ ) (cid:98) h ∗ ∈ H .

Original: Then the claim follows because λ ( h − (cid:98) h ∗ )
Revised: Then the claim follows because λ ( h − (cid:98) h ∗ )

Original: ∈ H − (cid:98) h ∗ .
Revised: ∈ H − (cid:98) h ∗ .

Original: Assume that these exsits τ > 0 for which Q H ∗ (2 τ ) > 0 .
Revised: Assume that these exsits τ > 0 for which Q H ∗ (2 τ ) > 0 .

Original: The for every s ≥ ω ( H ∗ , τQ H ∗ (2 τ ) / 16) , we have
Revised: The for every s ≥ ω ( H ∗ , τQ H ∗ (2 τ ) / 16) , we have

Original: [Equation]
Revised: [Equation]

Original: s.
Revised: s

Revised: [Equation]

Original: Let G be a function class associated with H ∗
Revised: Let G be a function class associated with H ∗

Original: [Equation]
Revised: [Equation]

Original: where B m 2 is the unit ball with respect with L 2 ( D ) and D is the semi-empirical distribution. 
Revised: where B m 2 is the unit ball with respect with L 2 ( D ) and D is the semi-empirical distribution. E sup

Original: [Equation]
Revised: [Equation]

Original: where the last inequality follows from Q G (2 τ ) ≥ Q H ∗ (2 τ ) 
Revised: where the last inequality follows from Q G (2 τ ) ≥ Q H ∗ (2 τ ) .

Original: [Equation]

Original: By equation (29) applied to the function class G , it follows that with probability at least 1 −[Equation] [Table] [Equation]
Revised: By equation (29) applied to the function class G , it follows that with probability at least 1 −

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: Now, setting
Revised: Now, setting

Original: [Equation]
Revised: [Equation]

Original: it follows that with probability at least 1 − 2 exp( − mnQ G (2 τ ) / 2)
Revised: it follows that with probability at least 1 − [Equation] − [Equation]

Original: [Equation]
Revised: [Equation]

Original: Using the condition Q G (2 τ ) ≥ Q H ∗ (2 τ ) , for every s ≥ ω ( H ∗ , τQ H ∗ (2 τ ) / 16) , it follows that with probability at least 1 − 2 exp( − mnQ H ∗ (2 τ ) / 2) , we get
Revised: Using the condition Q G (2 τ ) ≥ Q H ∗ (2 τ ) , for every s ≥ ω ( H ∗ , τQ H ∗ (2 τ ) / 16) , it follows that with probability at least 1 − 2 exp( − mnQ H ∗ (2 τ ) / 2) , we get

Original: [Equation]
Revised: [Equation]

Original: For every h − (cid:98) h ∗ ∈ H ∗ that satisfies ∥ h − (cid:98) h ∗ ∥
Revised: For every h − (cid:98) h ∗ ∈ H ∗ that satisfies ∥ h − (cid:98) h ∗ ∥

Original: L 2 ( D ) ≥ s , since H − (cid:98) h ∗ is star-shaped around 0 , we have (cid:16) s/ ∥ h − (cid:98) h ∗ ∥
Revised: L 2 ( D ) ≥ s , since H − (cid:98) h ∗ is star-shaped around 0 , we have (cid:16) s/ ∥ h − (cid:98) h ∗ ∥

Original: L 2 ( D ) (cid:17) ( h − (cid:98) h ∗ ) ∈
Revised: L 2 ( D ) (cid:17) ( h − (cid:98) h ∗ ) ∈

Original: H ∗ ∩ sB m 2 .
Revised: H ∗ ∩ sB m 2 .

Original: Thus, ( h − (cid:98) h ∗ ) /
Revised: Thus, ( h − (cid:98) h ∗ ) /

Original: ∥ h − (cid:98) h ∗ ∥
Revised: ∥ h − (cid:98) h ∗ ∥

Original: L 2 ( D ) ∈ G .
Revised: L 2 ( D ) ∈ G .

Original: Combining equation (30), if s ≥ ω ( H ∗ , τQ H ∗ (2 τ ) / 16) , then for every h ∈ H that satisfies ∥ h − (cid:98) h ∗ ∥
Revised: Combining equation (30), if s ≥ ω ( H ∗ , τQ H ∗ (2 τ ) / 16) , then for every h ∈ H that satisfies ∥ h − (cid:98) h ∗ ∥

Original: L 2 ( D ) ≥ s , if follows that with probability at least 1 − 2 exp( − mnQ H ∗ (2 τ ) / 2) , one has (cid:12) (cid:12) (cid:12) {
Revised: L 2 ( D ) ≥ s , if follows that with probability at least 1 − 2 exp( − mnQ H ∗ (2 τ ) / 2) , one has [Equation]

Original: [Equation]
Revised: [Equation]

Original: Therefore, on that event, we get
Revised: Therefore, on that event, we get

Original: [Equation]
Revised: [Equation]

Original:  Note that H ∗ ⊂ H − H , the same conclusion holds with Q H−H (2 τ ) replacing the larger Q H ∗ .
Revised: [Equation] Note that H ∗ ⊂ H − H , the same conclusion holds with Q H−H (2 τ ) replacing the larger Q H ∗ .

Original: That is, if s ≥ ω ( H ∗ , τQ H−H (2 τ ) / 16) , then for every h ∈ H that satisfies ∥ h − (cid:98) h ∗ ∥
Revised: That is, if s ≥ ω ( H ∗ , τQ H−H (2 τ ) / 16) , then for every h ∈ H that satisfies ∥ h − (cid:98) h ∗ ∥

Original: L 2 ( D ) ≥ s , if follows that with probability at least 1 − 2 exp( − mnQ H−H (2 τ ) / 2) , one has
Revised: L 2 ( D ) ≥ s , if follows that with probability at least 1 − 2 exp( − mnQ H−H (2 τ ) / 2) , one has

Original: [Equation]
Revised: [Equation]

Original: Step Three:
Revised: Step Three:

Original: Combining equation (2), with the same conditions we get
Revised: Combining equation (2), with the same conditions we get

Original: [Equation]
Revised: [Equation]

Original: Since 1 m (cid:80) mi =1 E (cid:104) ξ ( h − (cid:98) h ∗ )( X 1 i ) (cid:105) ≥ 0 , then we have mn m (cid:88) i = n 
Revised: Since 1 m (cid:80) mi =1 E (cid:104) ξ ( h − (cid:98) h ∗ )( X 1 i ) (cid:105) ≥ 0 , then we have mn m (cid:88) i = n (cid:88) j =

Original: [Equation]
Revised: [Equation]

Revised: mn m (cid:88) i = n (cid:88)

Revised: j =

Revised: [Equation]

Revised: [Equation]

Revised: (cid:98) h ∗

Revised: [Equation]

Revised: X ji

Revised: [Equation]

Revised: 1 m m (cid:88) i =

Revised: E X ∼ D

Revised: i (cid:104) ξ ( h − (cid:98) h ∗ )( X ) (cid:105) .

Original: According to Lemma
Revised: According to Lemma

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: P 

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: We denote C 1 , C 2 by
Revised: We denote C 1 , C 2 by

Revised: [Equation]

Revised: [Equation]

Revised: c

Original: [Equation]
Revised: [Equation]

Original: Then we have
Revised: Then we have

Revised: P 

Original: [Equation]
Revised: [Equation]

Original: To make sure that the probability tends to zero as mn increase
Revised:  

Original: [Equation]
Revised: [Equation]

Original:  we could chose ∥ h − (cid:98) h ∗ ∥ 
Revised: To make sure that the probability tends to zero as mn increase, we could chose ∥ h − (cid:98) [Equation] ∗ ∥ [Equation]

Original: L 2 ( D ) ≥ κ = ( mn ) − 14 + ι , where 0 < ι < 14 .
Revised: D [Equation] ≥ κ = ( mn ) − 14 + ι , where 0 < ι < 14 .

Original: Then with probability at least
Revised: Then with probability at least

Original: [Equation]
Revised: [Equation]

Original: we have [Equation]
Revised: we have (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)mn m (cid:88) i = n (cid:88)

Original: mn
Revised: j

Original: [Equation]
Revised: [Equation]

Original: Combining (2) and (33), if ∥ h − (cid:98) h ∗ ∥
Revised: Combining (2) and (33), if ∥ h − (cid:98) h ∗ ∥

Original: L 2 ( D ) ≥ max( κ, s ) , it follows that with probability at least 1 − δ − 2 exp( − mnQ H ∗ (2 τ ) / 2) ,
Revised: L 2 ( D ) ≥ max( κ, s ) , it follows that with probability at least 1 − δ [Equation] 2 exp( − [Equation]

Revised: H ∗ (2 τ ) / 2) ,

Original: [Equation]
Revised: [Equation]

Original: Consider η < τ 2 Q H−H (2 τ ) / 16 , we get
Revised: Consider η < τ 2 Q H−H (2 τ ) / 16 , we get

Revised: [Equation]

Revised: S

Original: [Equation]
Revised: [Equation]

Original: On the same event, we have
Revised: On the same event, we have

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: P ROOF OF C OROLLARY
Revised: P ROOF OF C OROLLARY

Original: Proof.
Revised: Proof.

Original: Let s = ω
Revised: Let s = ω

Original: ( H − H , τQ H−H (2 τ ) / 16) and κ = ( mn ) − 14 + ι , where 0 < ι < 14 .
Revised: ( H − H , τQ H−H (2 τ ) / 16) and κ = ( mn ) − 14 + ι , where 0 < ι < 14 .

Original: By theorem 4, we have 
Revised: By theorem 4, we have ∥

Original: [Equation]
Revised: [Equation]

Original: Therefore, it suffices to show that for quadratic loss we have
Revised: Therefore, it suffices to show that for quadratic loss we have

Original: [Equation]
Revised: [Equation]

Original: j =
Revised: Note that for h ∈ H either [Equation] or [Equation]

Revised: For the first case, we have

Original:  E (cid:104) ( h − (cid:98) h ∗ ) 2 ( X ji ) (cid:105) + 2 mn m (cid:88) i = n (cid:88)
Revised: [Equation] For the second case, [Equation] For convex function class H , we have E [ ξ ji ( h − (cid:98) h ∗ )( X ji )] ≥ 0 .

Original: j =
Revised: On the same condition that inequality (33) holds, we get [Equation]

Original: E (cid:104) ξ ji ( h − (cid:98) h ∗ )( X ji ) (cid:105) , (35)
Revised: ji ) (cid:105)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) [Equation] max [Equation]

Revised: κ

Original: If ξ ji = (cid:98) h ∗ ( X ji ) − Y ji is independent of X , then E (cid:104) ξ ji (h − (cid:98) h ∗)( X ji ) (cid:105)
Revised: [Equation] s [Equation] Since L S ( h ) − L S ( (cid:98) h ∗ ) ≤ 0 , it follows that ( L D ( h

Original: = 0 . Thus,
Revised: [Equation]

Original: L D ( (cid:98) h ) − L D ( (cid:98) h ∗ ) =
Revised: − L D ( (cid:98) h ∗ [Equation] ≤

Original: ∥ (cid:98) h − (cid:98) h ∗ ∥
Revised: [Equation] Q H−H ( [Equation]

Original: L 2 ( D ) ≤ max( κ 2 , s 2 ) .
Revised: max( κ 2 , s [Equation]

Original: P ROOF OF T HEOREM
Revised: P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Revised: For quadratic loss

Revised: [Equation]

Revised: we have

Revised: [Equation]

Original: Step One:
Revised: Step One:

Original: For any h ∈ H and u > 0 ,
Revised: For any h ∈ H and u > 0 ,

Original: [Equation]
Revised: [Equation]

Original: Also,
Revised: Also,

Original: [Equation]
Revised: [Equation]

Original: Let ϕ u : R → [0 , 1] be the function
Revised: Let ϕ u : R → [0 , 1] be the function

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Note that for every t ∈ R , ϕ u ( t ) ≥ 1 { t ≥ 2 u } and ϕ u ( t ) ≤ 1 { t ≥ u } .
Revised: Note that for every t ∈ R , ϕ u ( t ) ≥ 1 { t ≥ 2 u } and ϕ u ( t ) ≤ 1 { t ≥ u } .

Original: Thus,m
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: i = [Equation] Since function ϕ u ( t ) is bounded by 1 , using Mcdiarmid’s inequality, we get that, for every δ > 0 , with probability at least 1 − 2 exp( − 2 δ 2 ) ,
Revised: Since function ϕ u ( t ) is bounded by 1 , using Mcdiarmid’s inequality, we get that, for every δ > 0 , with probability at least 1 − 2 exp( − 2 δ 2 ) ,

Original: [Equation]
Revised: [Equation]

Original: By the Lipschitz property of ϕ u ( | t | ) and the symmetrization theorem, we have
Revised: By the Lipschitz property of ϕ u ( | t | ) and the symmetrization theorem, we have

Revised: [Equation]

Original: E sup
Revised: sup

Original: [Equation]
Revised: [Equation]

Original: where X i is the random sampled across two-level distribution framework.
Revised: where X i is the random sampled across two-level distribution framework.

Original: Therefore, for every h ∈ H , it follows that with probability at least 1 − 2 exp( − 2 δ 2 ) , we have
Revised: Therefore, for every h ∈ H , it follows that with probability at least 1 − 2 exp( − 2 δ 2 ) , we have

Original: [Equation]
Revised: [Equation]

Original: The first term on the RHS can be bounded by small ball condtion.
Revised: The first term on the RHS can be bounded by small ball condtion.

Original: Let H ∗
Revised: Let H ∗

Original: =
Revised: =

Original: H − h ∗ , we have proved that H ∗ is star-shaped around 0 .
Revised: H − h ∗ , we have proved that H ∗ is star-shaped around 0 .

Original: Assume that these exsits τ > 0 for which Q H ∗ (2 τ, P ) > 0 .
Revised: Assume that these exsits τ > 0 for which Q H ∗ (2 τ, P ) > 0 .

Original: Then for every s ≥ ω m ( H ∗ , τQ H ∗ (2 τ, P ) / 16) , we have
Revised: Then for every s ≥ ω m ( H ∗ , τQ H ∗ (2 τ, P ) / 16) , we have

Original: [Equation]
Revised: [Equation]

Original: Let G be a function class associated with H ∗G =
Revised: Let G be a function class associated with H ∗

Original: [Equation]
Revised: [Equation]

Original: where B m is the unit ball with respect with L 2 ( P ) and P is the population distribution.
Revised: where B m is the unit ball with respect with L 2 ( P ) and P is the population distribution

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Original: where the last inequality follows from Q G (2 τ, P ) ≥ Q H ∗ (2 τ, P ) .
Revised: where the last inequality follows from Q G (2 τ, P ) ≥ Q H ∗ (2 τ, P ) .

Original: By equation (37) applied to the function class G , it follows that with probability at least 1 − 2 exp( − 2 δ 2 )
Revised: By applying inequality (39) to the function class G , it follows that with probability at least 1 − 2 exp( − 2 δ 2 )

Original: [Equation]
Revised: [Equation]

Original: Now, setting
Revised: Now, setting

Original: [Equation]
Revised: [Equation]

Original: it follows that with probability at least 1 − 2 exp( − mQ G (2 τ, P ) / 2)
Revised: it follows that with probability at least 1 − 2 exp( − mQ G (2 τ

Original: [Equation]
Revised: [Equation]

Original:  Using the condition Q G (2 τ, P ) ≥ Q H ∗ (2 τ, P ) , for every s ≥ ω m ( H ∗ , τQ H ∗ (2 τ, P ) / 16) , it follows that with probability at least 1 − 2 exp( − mQ H ∗ (2 τ, P ) / 2) , we getinf
Revised: P [Equation] [Equation] Using the condition Q G (2 τ, P ) ≥ Q H ∗ (2 τ, P ) , for every s ≥ ω m ( H ∗ , τQ H ∗ (2 τ, P ) / 16) , it follows that with probability at least 1 − 2 exp( − mQ H ∗ (2 τ, P ) / 2) , we get

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: For every h − h ∗ ∈ H ∗ that satisfies ∥ h − h ∗ ∥ L 2 ( P ) ≥ s , since H − h ∗ is star-shaped around 0 , we have (cid:0) s/ ∥ h − h ∗ ∥
Revised: For every h − h ∗ ∈ H ∗ that satisfies ∥ h − h ∗ ∥ L 2 ( P ) ≥ s , since H − h ∗ is star-shaped around 0 , we have (cid:0) s/ ∥ h − h ∗ ∥

Original: L 2 ( P ) (cid:1) ( h − h ∗ ) ∈ H ∗ ∩ sB m .
Revised: L 2 ( P ) (cid:1) ( h − h ∗ ) ∈ H ∗ ∩ sB m .

Original: Thus, ( h − h ∗ ) /
Revised: Thus, ( h − h ∗ ) /

Original: ∥ h − h ∗ ∥
Revised: ∥ h − h ∗ ∥

Original: L 2 ( P ) ∈ G .
Revised: L 2 ( P ) ∈ G .

Original: Combining equation (38), if s ≥ ω m ( H ∗ , τQ H ∗ (2 τ, P ) / 16) , then for every h ∈ H that satisfies ∥ h − h ∗ ∥
Revised: Combining equation (40), if s ≥ ω m ( H ∗ , τQ H ∗ (2 τ, P ) / 16) , then for every h ∈ H that satisfies ∥ h − h ∗ ∥

Original: L 2 ( P ) ≥ s , if follows that with probability at least 1 − 2 exp( − mQ H ∗ (2 τ, P ) / 2) , one has 
Revised: L 2 ( P ) ≥ s , if follows that with probability at least 1 − 2 exp( − mQ H ∗ (2 τ, P ) / 2) , one has (cid:12) [Equation] { i : | E [( h [Equation]

Revised: h ∗ )

Original: [Equation]
Revised: [Equation]

Revised: X 1 i )] | > τ ∥

Revised: [Equation]

Original: Therefore, on that event, we get 
Revised: Therefore, on that event, we get m

Original: [Equation]
Revised: [Equation]

Original: Note that H ∗ ⊂ H − H , the same conclusion holds with Q H−H (2 τ, P ) replacing the larger
Revised: Note that H ∗ ⊂ H − H , the same conclusion holds with Q H−H (2 τ, P ) replacing the larger

Original: Q H ∗ (2 τ, P ) .
Revised: Q H ∗ (2 τ, P ) .

Original: That is, if s ≥ ω m ( H ∗ , τQ H−H (2 τ, P ) / 16) , then for every h ∈ H that satisfies ∥ h − h ∗ ∥
Revised: That is, if s ≥ ω m ( H ∗ , τQ H−H (2 τ, P ) / 16) , then for every h ∈ H that satisfies ∥ h − h ∗ ∥

Original: L 2 ( P ) ≥ s , if follows that with probability at least 1 − 2 exp( − mQ H−H (2 τ, P ) / 2) , one has
Revised: L 2 ( P ) ≥ s , if follows that with probability at least 1 − 2 exp( − mQ H−H (2 τ, P ) / 2) , one has

Original: [Equation]
Revised: [Equation]

Original: With the same conditions we get
Revised: With the same conditions we get

Original: [Equation]
Revised: [Equation]

Original:  Since E D i ∼ P E 
Revised: [Equation] Since E D i ∼ P E ( X 1 i ,Y 1 i ) ∼ D i (cid:2)

Original: ( X i ,Y i ) ∼ D i [ ξ  ( h − h ∗ )( X  i )] ≥ 0 , then we have 
Revised: ξ 1 i ( h − h ∗ )( X 1 i ) (cid:3) ≥ 0 , then we have m

Original: [Equation]
Revised: [Equation]

Original: According to Lemma 7, we have
Revised: According to Lemma 7, we have

Original: [Equation]
Revised: [Equation]

Original: m

Original: [Equation]

Original: , where
Revised: , where

Original: [Equation]
Revised: [Equation]

Original:  = 
Revised: i = E

Original: [Equation]
Revised: [Equation]

Original: We denote C 1 , C 2 by
Revised: We denote C 1 , C 2 by

Original: [Equation]
Revised: [Equation]

Revised: 1 =

Revised: [Equation]

Original: Then we have
Revised: Then we have

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: To make sure that the probability tends to zero as m increase, we could chose ∥ h − h ∗ ∥
Revised: To make sure that the probability tends to zero as m increase, we could chose ∥ h − h ∗ ∥

Original: L 2 ( P ) ≥ κ = m − 14 + ι , where 0 <
Revised: L 2 ( P ) ≥ κ = m − 14 + ι , where 0 <

Original: ι < 14 .
Revised: ι < 14 .

Original: If ∥ h − h ∗ ∥ L 2 ( P ) ≥ max( κ, s ) , it follows that with probability at least 1 − δ  − 2 exp( − mQ H ∗ (2 τ, P ) / 2) , 
Revised: If ∥ h − h ∗ ∥ L 2 ( P ) ≥ max( κ, s ) , it follows that with probability at least 1 − δ m − 2 exp( − mQ H ∗ (2 τ, P ) / 2) , [Equation] , where δ m = exp {− (

Original: [Equation]
Revised: [Equation]

Original:  Consider η < τ 2 Q H−H (2 τ, P ) / 16 , we get
Revised: ) } Consider η < τ 2 Q H−H (2 τ, P ) / 16 , we get

Original: L D ( h ) − L D ( h ∗ ) ≥ 0 .
Revised: L [Equation]

Original: On the same event, we have
Revised: On the same event, we have

Original: [Equation]
Revised: [Equation]

Original: E.1. P ROOF OF C OROLLARY
Revised: E.1. P ROOF OF C OROLLARY

Original: Proof.
Revised: Proof.

Original: Since excess risk is defined across our two-level framework, the steps in the proof of Corollary 1 can not be applied directly to derive Corollary 2.
Revised: Since excess risk is defined across our two-level framework [Equation] the steps in the proof of Corollary 1 can not be applied directly to derive Corollary 2.

Original: The key step to derive Corollary 2 is to bound ∥ (cid:98) h − h ∗ ∥ 2 L 2 ( P ) .
Revised: The key step to derive Corollary 2 is to bound ∥ (cid:98) h − h ∗ ∥ 2 L 2 ( P ) .

Original: First, this term can be decomposed as
Revised: First, this term can be decomposed as

Original: [Equation]
Revised: [Equation]

Original: Note that ∥ (cid:98) h ∗ − h ∗ ∥ 2 L 2 ( P ) has been bounded by Theorem 5.
Revised: Note that ∥ (cid:98) h ∗ − h ∗ ∥ 2 L 2 ( P ) has been bounded by Theorem 5.

Original: To bound ∥ (cid:98) h − (cid:98) h ∗ ∥ 2 L 2 ( P ) , we use the following decomposition:
Revised: To bound ∥ (cid:98) h − (cid:98) h ∗ ∥ 2 L 2 ( P ) [Equation] we use the following decomposition:

Original: [Equation]
Revised: [Equation]

Original: Note that ∥ (cid:98) h − (cid:98) h ∗ ∥
Revised: Note that ∥ (cid:98) h − (cid:98) h ∗ ∥

Original: L 2 ( D ) has been bounded by Theorem 4.
Revised: L 2 ( D ) has been bounded by Theorem 4.

Original: According to Lemma 7, we have
Revised: According to Lemma [Equation] we have

Original: [Equation]
Revised: [Equation]

Original: , where
Revised: , where

Original: V i =

Original: E
Revised: V

Original: [Equation]
Revised: [Equation]

Original: We denote C 1 , C 2 by
Revised: We denote C 1 , C 2 by

Original: [Equation]
Revised: [Equation]

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: To make sure that the probability tends to zero as m increase [Equation] we could chose ∥ h − (cid:98) h ∗ ∥
Revised: To make sure that the probability tends to zero as m increase, we could chose ∥ h − (cid:98) h ∗ ∥

Original: L 2 ( P ) ≥ m − 14 + ι , where 0 < ι < 14 .
Revised: L 2 ( P ) ≥ m − 1 4 + ι , where 0 < ι < 14 .

Original: By following the similar steps of proof of Theorem [Equation] it can be proved that with probability at least
Revised: By following the similar steps of proof of Theorem 5, it can be proved that with probability at least

Original: [Equation]
Revised: [Equation]

Original:  one has
Revised: , one has

Original: [Equation]
Revised: [Equation]

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: Moreover,
Revised: Moreover,

Original: [Equation]
Revised: [Equation]

Original: The remaining steps are similar to those of Corollary
Revised: If ξ i = h ∗ ( X i ) − Y i is independent of X , then E [ ξ i ( h − h ∗ ) ( X i )] = 0 .

Revised: [Equation]

Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: F E XPERIMENTAL R ESULTS F. C ONVOLUTIONAL N EURAL N ETWORKS FOR MNIST T ASK
Revised: F E XPERIMENTAL R ESULTS F. C ONVOLUTIONAL N EURAL N ETWORKS FOR MNIST T ASK

Original: To check the validity of our theory for over-parameterized models [Equation] we train convolutional neural network for MINIST task (Cohen et al., 2017).
Revised: To check the validity of our theory for over-parameterized models, we train convolutional neural network for EMINIST task (Cohen et al., 2017).

Original: In particular, we use FedAdam (Reddi et al., 2020) with server momentum = 0 .
Revised: In particular, we use FedAdam (Reddi et al., 2020) with server momentum = 0 .

Original: The participating and unparticipating clients are split based on the[Figure] methods proposed in (Yuan et al., 2021).
Revised: The participating and unparticipating clients are split based on the methods proposed in (Yuan et al., 2021).

Original: We set the unparticipating rate as 0.2.
Revised: We set the unparticipating rate as 0.2.

Original: Our experiments are based on Tensorflow Federated (TFF).
Revised: Our experiments are based on Tensorflow Federated (TFF).

Original: In Figure

Original: [Figure]
Revised: [Figure]

Original:  we show how generalization errors for unparticipating clients and participating clients convergence when we increase the number of participating clients m .
Revised: In Figure 2, we show how generalization errors for unparticipating clients and participating clients convergence when we increase the number of participating clients m .

Original: Here we fix n = 100 .
Revised: Here we fix n = 100 .

Original: It can be seen that the convergence rate of participating error is slower than that of unparticipating error.
Revised: It can be seen that the convergence rate of participating error is slower than that of unparticipating error.

Original: This phenomenon matches our theretical results in Theorem 1 well.
Revised: This phenomenon matches our theretical results in Theorem 1 well.

Original: In our results, the convergence rate of unparticipating error is of order O
Revised: In our results, the convergence rate of unparticipating error is of order O

Original: ( 1 √ mn + 1 √ m ) .
Revised: ( 1 √ mn + 1 √ m ) .

Original: Compared to the convergence rate of participating error, which is of order O ( 1 √ mn ) , unparticipating error is expected to have faster convergence rate [Figure]
Revised: Compared to the convergence rate of participating error, which is of order O ( 1 √ mn ) , unparticipating error is expected to have faster convergence rate.

Original: In Figure [Figure] we show how generalization errors for unparticipating clients and unparticipating clients convergence when we increase n .
Revised: In Figure 3, we show how generalization errors for unparticipating clients and unparticipating clients convergence when we increase n .

Original: Here we fix m = 20 .
Revised: Here we fix m = 20 .

Original: It can be seen that the convergence rates of participating error and unparticipating error are similar.
Revised: It can be seen that the convergence rates of participating error and unparticipating error are similar.

Original: This phenomenon matches our theretical results in theorem 1 as well.
Revised: This phenomenon matches our theretical results in theorem 1 as well.

Original: When m is fixed, the order of O ( 1 √ mn + 1 √ m ) is same as the order of
Revised: When m is fixed, the order of O ( 1 √ mn + 1 √ m ) is same as the order of

Original: O ( 1 √ mn ) .
Revised: O ( 1 √ mn ) .

Revised: [Figure]

Original: F. F EDERATED L INEAR R EGRESSION WITH S YNTHETIC D ATA
Revised: F. F EDERATED L INEAR R EGRESSION WITH S YNTHETIC D ATA

Original: In Figure 4, we show the numerical experiments results based on the linear regression model.
Revised: In Figure [Figure] we show the numerical experiments results based on the linear regression model.

Original: We first describe our linear regression setting as follows.
Revised: We first describe our linear regression setting as follows.

Original: For client i ∈ [ m ] , the dataset is given as
Revised: For client i ∈ [ m ] , the dataset is given as

Original: S i = {
Revised: S i [Equation]

Original:  X j i , Y j i } with n samples.
Revised: { X ji , Y ji } with n samples.

Original: Let d be the dimensionality of the input space.
Revised: Let d be the dimensionality of the input space.

Original: We focus on the setting:
Revised: We focus on the setting:

Original: [Equation]
Revised: [Equation]

Original: where σ 2 k is a noise parameter.
Revised: where σ 2 k is a noise parameter.

Original: In our experiments, we set σ i = 0 .
Revised: In our experiments [Figure] we set σ i = 0 .

Original: For excess risk, we fix n = 20 .
Revised: For excess risk, we fix n = 20 .

Original: For semi-excess risk we fix m = 40 .
Revised: For semi-excess risk we fix m = 40 .

Original: [Figure]
Revised: [Figure]