Original: S HARPER B OUNDS FOR U NIFORMLY S TABLE A LGORITHMS WITH S TATIONARYφ - MIXING P ROCESS
Revised: S HARPER B OUNDS FOR U NIFORMLY S TABLE A LGORITHMS WITH S TATIONARY MIXING P ROCESSES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Generalization analysis of learning algorithms often builds on a critical assumption that training examples are independently and identically distributed (i i d.), which is often violated in practical problems such as time series prediction.
Revised: Generalization analysis of learning algorithms often builds on a critical assumption that training examples are independently and identically distributed, which is often violated in practical problems such as time series prediction.

Original: In this paper, we use algorithmic stability to study the generalization performance of learning algorithms with φ -mixing data, where the dependency between observations weakens over time.
Revised: In this paper, we use algorithmic stability to study the generalization performance of learning algorithms with ψ -mixing data, where the dependency between observations weakens over time.

Original: We show uniformly stable algorithms guarantee high-probability generalization bounds, which significantly improve the state of the art by a factor of √ n with n being the sample size.
Revised: We show uniformly stable algorithms guarantee high-probability generalization bounds of the order O (1 / √ n ) (within a logarithmic factor), where n is the sample size.

Original: We apply our general result to specific algorithms including regularization schemes, stochastic gradient descent and localized iterative regularization and develop excess population risk bounds for learning with φ -mixing data.
Revised: We apply our general result to speciﬁc algorithms including regularization schemes, stochastic gradient descent and localized iterative regularization, and develop excess population risk bounds for learning with ψ -mixing data.

Original: Our analysis builds on a novel moment bound for weakly-dependent random variables on a mixing sequence and a novel error decomposition of generalization error
Revised: Our analysis builds on a novel moment bound for weakly-dependent random variables on a ϕ -mixing sequence and a novel error decomposition of generalization error

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Generalization gap refers to the discrepancy between training and testing, which is a quantity of central importance in statistical learning theory (SLT) (Shalev-Shwartz & Ben-David, 2014).
Revised: Generalization gap refers to the discrepancy between training and testing, which is a quantity of central importance in statistical learning theory (SLT) (Shalev-Shwartz & Ben-David, 2014).

Original: A popular approach to controlling the generalization gap is to bound it by the uniform convergence between training and testing errors over a function space (Bartlett & Mendelson, 2002), which leads to bounds depending on the complexity of function spaces, such as VC dimension (Vapnik, 2013), covering number (Cucker & Zhou, 2007) and Rademacher complexity (Bartlett & Mendelson, 2002).
Revised: A popular approach to controlling the generalization gap is to bound it by the uniform convergence between training and testing errors over a function space (Bartlett & Mendelson, 2002), which leads to bounds depending on the complexity of function spaces, such as VC dimension (Vapnik, 2013), covering number (Cucker & Zhou, 2007) and Rademacher complexity (Bartlett & Mendelson, 2002).

Original: These complexity-based bounds do not exploit the property of a learning algorithm and would generally admit a square-root dependency on the dimension (Feldman, 2016), which are not favorable for large-scale problems.
Revised: These complexity-based bounds do not exploit the property of a learning algorithm and would generally admit a square-root dependency on the dimension (Feldman, 2016), which are not favorable for large-scale problems.

Original: To incorporate the property of a learning algorithm and remove the dependency on dimension, a concept of algorithmic stability has been introduced into
Revised: To incorporate the property of a learning algorithm and remove the dependency on dimension, a concept of algorithmic stability has been introduced into

Original: SLT (Bousquet & Elisseeff, 2002).
Revised: SLT (Bousquet & Elisseeff, 2002).

Original: Intuitively speaking, algorithmic stability measures how a small perturbation of the training dataset would affect the output model of a learning algorithm, which has close connection to several key properties such as learnability (Shalev-Shwartz et al., 2010), robustness and privacy (Bassily et al., 2020).
Revised: Intuitively speaking, algorithmic stability measures how a small perturbation of the training dataset would affect the output model of a learning algorithm, which has close connection to several key properties such as learnability (Shalev-Shwartz et al., 2010), robustness and privacy (Bassily et al., 2020).

Original: Recent research has witnessed an increasing interest in leveraging stability to study the generalization behavior of various algorithms, such as stochastic gradient descent (Hardt et al., 2016), structured prediction (London et al., 2016), meta learning (Maurer, 2005) and transfer learning (Kuzborskij & Lampert, 2018).
Revised: Recent research has witnessed an increasing interest in leveraging stability to study the generalization behavior of various algorithms, such as stochastic gradient descent (Hardt et al., 2016), structured prediction (London et al., 2016), meta learning (Maurer, 2005) and transfer learning (Kuzborskij & Lampert, 2018).

Original: Most of these discussions are based on a critical assumption that the training examples are independently and identically distributed (i.i.d.).
Revised: Most of these discussions are based on a critical assumption that the training examples are independently and identically distributed (i.i.d.).

Original: This assumption is often violated in practical applications.
Revised: This assumption is often violated in practical applications.

Original: For example, the i.i.d.
Revised: For example, the i.i.d.

Original: assumption is too restrictive in time series prediction (Vidyasagar, 2013).
Revised: assumption is too restrictive in time series prediction (Vidyasagar, 2013).

Original: The prices of the same stock on different days may have temporal dependence.
Revised: The prices of the same stock on different days may have temporal dependence.

Original: These phenomena motivate several analysis to derive meaningful bounds for learning problems with observations drawn from a non-i.i.d.
Revised: These phenomena motivate several analyses to derive meaningful bounds for learning problems with observations drawn from a non-i.i.d.

Original: process (Yu, 1994; Vidyasagar, 2013).
Revised: process (Yu, 1994; Vidyasagar, 2013).

Original: A widely used relaxation of the i.i.d.
Revised: A widely used relaxation of the i.i.d.

Original: assumption is to assume the observations are drawn from a mixing process (Yu, 1994;Kuznetsov & Mohri, 2017; Meir, 2000; Lozano et al., 2005; Vidyasagar, 2013), where the dependency between two observations is quantified by a mixing coefficient as a function of the discrepancy of the associated two indices.
Revised: assumption is to assume the observations are drawn from a mixing process (Yu, 1994; Meir, 2000; Lozano et al., 2005; Vidyasagar, 2013), where the dependency between two observations is quantiﬁed by a mixing coefﬁcient as a function of the discrepancy of the associated two indices.

Original: These mixing coefficients decay either as a polynomial function or an exponential function of the discrepancy (Vidyasagar, 2013).
Revised: These mixing coefﬁcients decay either as a polynomial function or an exponential function of the discrepancy (Vidyasagar, 2013).

Original: Several mixing processes have been introduced into the literature, including theα -mixing, β  -mixing and φ -mixing processes (Yu, 1994; Meir, 2000; Lozano et al., 2005).
Revised: Several mixing processes have been introduced into the literature, including the β -mixing, ϕ -mixing and ψ -mixing processes (Yu, 1994; Meir, 2000; Lozano et al., 2005).

Original: Within this formulation, various generalization bounds have been developed to show how the dependency among observations would affect the learning process.
Revised: Within this formulation, various generalization bounds have been developed to show how the dependency among observations would affect the learning process.

Original: Interestingly, these discussions imply a concept called “effective size” which plays a similar role of the sample size in the i.i.d.
Revised: Interestingly, these discussions imply a concept called “effective size” which plays a similar role of the sample size in the i.i.d.

Original: scenario (Yu, 1994; Kuznetsov & Mohri, 2017).
Revised: scenario (Yu, 1994; Kuznetsov & Mohri, 2017).

Original: As in the i.i.d.
Revised: As in the i.i.d.

Original: case, most generalization analysis in the non-i.i.d.
Revised: case, most generalization analyses in the non-i.i.d.

Original: case focus on complexity-based bounds (Meir, 2000; Yu, 1994; Kuznetsov & Mohri, 2017).
Revised: case focus on complexity-based bounds (Meir, 2000; Yu, 1994; Kuznetsov & Mohri, 2017).

Original: There is little stability analysis of learning algorithms in the non-i.i.d.
Revised: There are few stability analyses of learning algorithms in the non-i.i.d.

Original: cases.
Revised: cases.

Original: An exception is the work in Mohri & Rostamizadeh (2010), which, to our knowledge, gives the first systematic analysis on the stability and generalization in a non-i.i.d.
Revised: An exception is the work in Mohri & Rostamizadeh (2010), which, to our knowledge, gives the ﬁrst systematic analysis on the stability and generalization in a non-i.i.d.

Original: case.
Revised: case.

Original: The authors developed high-probability generalization bounds for learning with stationary φ -mixing and β -mixing sequences, which are then applied to general kernel regularization-based bounds.
Revised: The authors developed high-probability generalization bounds for learning with stationary ϕ -mixing and β -mixing sequences, which are then applied to general kernel regularizationbased bounds.

Original: Due to the algorithm-specific nature, these bounds are preferable to complexity-based bounds if the associated hypothesis space has a very large complexity.
Revised: Due to the algorithm-speciﬁc nature, these bounds are preferable to complexity-based bounds if the associated hypothesis space has a very large complexity.

Original: However, the stability analysis (Mohri & Rostamizadeh, 2010)can only imply sub-optimal generalization bounds.
Revised: However, the stability analysis (Mohri & Rostamizadeh, 2010) only implies sub-optimal generalization bounds.

Original: In more details, for β -uniformly stable algorithms, the high-probability bounds in 
Revised: Indeed, for β -uniformly stable algorithms, the high-probability bounds in Mohri &

Original: Mohri & Rostamizadeh (2010) are of the order of O ( √ nβ + ∆ n / √ n ) , where n is the sample size and ∆ n is a term depending on the decay rate of mixing coefficients.
Revised: Rostamizadeh (2010) are of the order of O ( √ nβ + ∆ n / √ n ) , where n is the sample size and ∆ n is a term depending on the decay rate of mixing coefﬁcients.

Original: For learning with λ -strongly convex problems, the uniform stability parameter is of the order O
Revised: For learning with λ -strongly convex problems, the uniform stability parameter is of the order O

Original: (1 / ( nλ ))
Revised: (1 / ( nλ ))

Original: (Bousquet & Elisseeff, 2002) and therefore the bounds in Mohri & Rostamizadeh (2010) become O
Revised: (Bousquet & Elisseeff, 2002) and therefore the bounds in Mohri & Rostamizadeh (2010) become O

Original: (1 / ( √ nλ )
Revised: (1 / ( √ nλ )

Original: +∆ n / √ n ) .
Revised: + ∆ n / √ n ) .

Original: A typical choice of λ is λ ≈ n − α for α > 

Revised: A typical choice of λ is λ ≈ n − α for α >

Original:  and then the bounds further become O ( n α − 12 + ∆ n / √ n ) , which cannot imply the optimal bounds O
Revised: (Shalev-Shwartz & Ben-David, 2014) and then the bounds further become O ( n α − 12 + ∆ n / √ n ) , which cannot imply the optimal bounds O

Original: (1 / √ n ) even if ∆ n = O (1) .
Revised: (1 / √ n ) even if ∆ n = O (1) .

Original: For learning with i.i.d.
Revised: For learning with i.i.d.

Original: data, recent breakthrough (Feldman & Vondrak, 2019; Bousquet et al., 2020; Klochkov & Zhivotovskiy, 2021) in stability analysis shows that β -uniformly stable algorithms enjoy generalization bounds of the order (cid:101) O
Revised: data, recent breakthroughs (Feldman & Vondrak, 2019; Bousquet et al., 2020) in stability analysis show that β -uniformly stable algorithms enjoy generalization bounds of the order (cid:101) O

Original: (1 / √ n ) 1 .
Revised: (1 / √ n ) 1 .

Original: This motivates a natural question: can we develop generalization bounds of the order (cid:101)
Revised: This motivates a natural question: can we develop generalization bounds of the order (cid:101)

Original: O (1 / √ n ) for uniformly stable algorithms applied to mixing process?
Revised: O (1 / √ n ) for uniformly stable algorithms applied to mixing process?

Original: In this paper, we provide an affirmative answer to the above question.
Revised: This paper provides an afﬁrmative answer to the above question.

Original: Our contributions are summarized as follows.
Revised: Our contributions are listed below.

Original: We develop a moment bound for weakly dependent random variables defined on a mixing sequence.
Revised: We develop a moment bound for weakly dependent random variables deﬁned on a ϕ -mixing sequence.

Original: We show our bound matches the existing moment bounds for i.i.d.
Revised: We show our bound matches the existing moment bounds for i.i

Original:  random variables up to a logarithmic factor.
Revised: d. random variables up to a logarithmic factor.

Original: As a byproduct, we develop a Marcinkiewicz-Zygmund inequality for dependent random variables, which may be interesting on its own right.
Revised: As a byproduct, we develop a Marcinkiewicz-Zygmund inequality for a ϕ -mixing sequence, which may be interesting in its own right.

Original: We develop high-probability bounds  for uniformly stable algorithms for learning with φ -mixing sequences, which significantly improve the existing bound in Mohri & Rostamizadeh (2010) by a factor of √ n (up to a logarithmic factor).
Revised: We develop high-probability bounds of order (cid:101) O (1 / √ n ) for uniformly stable algorithms for learning with ψ -mixing sequences (our results actually require assumptions on ϕ (cid:48) -mixing coefﬁcients which are weaker than assumptions on ψ -mixing coefﬁcients).

Original: We achieve this by introducing a different decomposition of generalization errors to make sure we get weakly-dependent and mean-zero random variables, which is more challenging than the i.i
Revised: We achieve this by introducing a different decomposition of generalization errors to make sure we get weakly-dependent and mean-zero random variables, which is more challenging than the i i

Original: d. case.
Revised: d. case.

Original: If the training examples are i.i.d., our results recover the existing bounds within a constant factor.
Revised: Our results recover the existing bounds within a constant factor in the i.i d. case.

Original: We apply our general bound to some specific algorithms to show the effectiveness of our results, including kernel regularization schemes, stochastic gradient descent (SGD) and iterative localization.
Revised: We apply our general bound to some speciﬁc algorithms to show the effectiveness of our results, including kernel regularization schemes, stochastic gradient descent (SGD) and iterative localization.

Original: The paper is organized as follows.
Revised: The paper is organized as follows.

Original: We present the related work in Section 2.
Revised: We present the related work in Section 2.

Original: We develop concentration inequalities for φ -mixing data in Section 3 and present general stability-based bounds in Section 4.
Revised: We develop concentration inequalities for ϕ -mixing data in Section 3 and present general stability-based bounds in Section 4.

Original: We apply our general result to specific algorithms in Section 5.
Revised: We apply our general result to speciﬁc algorithms in Section 5.

Original: We conclude the paper in Section 6.
Revised: We conclude the paper in Section 6.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: In this section, we discuss the related work.
Revised: In this section, we discuss the related work.

Original: We first discuss the related work on algorithmic stability and then the related work on learning with dependent data.
Revised: We ﬁrst discuss the related work on algorithmic stability and then the related work on learning with dependent data.

Original: Algorithmic stability .
Revised: Algorithmic stability .

Original: Algorithmic stability measures how the replacement/removal of a single (or a few) example would affect the output model, which is an important concept in SLT (Bousquet &
Revised: Algorithmic stability measures how the replacement/removal of a single (or a few) example would affect the output model, which is an important concept in SLT (Bousquet &

Original: Elisseeff, 2002).
Revised: Elisseeff, 2002).

Original: A nice property of algorithmic stability is that it only considers the behavior of the output model and therefore can imply capacity-independent generalization bounds.
Revised: A nice property of algorithmic stability is that it only considers the behavior of the output model and therefore can imply capacity-independent generalization bounds.

Original: An important stability measure called uniform stability was introduced in an influential work (Bousquet & Elisseeff, 2002), which was used to study the generalization behavior of regularization schemes.
Revised: An important stability measure called uniform stability was introduced in an inﬂuential work (Bousquet & Elisseeff, 2002), which was used to study the generalization behavior of regularization schemes.

Original: This uniform stability was extended to the setting of randomized algorithms (Elisseeff et al., 2005), which was further used to study the generalization guarantee of SGD (Hardt et al., 2016).
Revised: This uniform stability was extended to the setting of randomized algorithms (Elisseeff et al., 2005), which was further used to study the generalization guarantee of SGD (Hardt et al., 2016).

Original: To better exploit the training examples for better generalization bounds, a relaxation of uniform stability called on-average stability has been introduced (Shalev-Shwartz et al., 2010).
Revised: To better exploit the training examples for better generalization bounds, a relaxation of uniform stability called on-average stability has been introduced (Shalev-Shwartz et al., 2010).

Original: In particular, the on-average stability was shown to be equivalent to learnability (Shalev-Shwartz et al., 2010) and was used to derive data-dependent error bounds (Kuzborskij & Lampert, 2018;
Revised: In particular, the on-average stability was shown to be equivalent to learnability (Shalev-Shwartz et al., 2010) and was used to derive data-dependent error bounds (Kuzborskij & Lampert, 2018;

Original: Lei & Ying, 2020; Zhou et al., 2021; Li et al., 2020; Nikolakakis et al., 2022).
Revised: Lei & Ying, 2020; Zhou et al., 2021; Li et al., 2020; Nikolakakis et al., 2022).

Original: The smoothness assumption for stability analysis of SGD was removed in the papers (Lei & Ying, 2020; Bassily et al., 2020).
Revised: The smoothness assumption for stability analysis of SGD was removed in the papers (Lei & Ying, 2020; Bassily et al., 2020).

Original: For nonconvex problems, stability and generalization of learning algorithms that converge to global optima were studied for gradientdominated problems (Charles & Papailiopoulos, 2018;
Revised: For nonconvex problems, stability and generalization of learning algorithms that converge to global optima were studied for gradient-dominated problems (Charles & Papailiopoulos, 2018;

Original: Lei & Ying, 2021).
Revised: Lei & Ying, 2021).

Original: While most discussions focus on upper bounds on the stability, recent work also develops lower bounds on the stability of SGD (Bassily et al., 2020; Amir et al., 2021).
Revised: While most discussions focus on upper bounds on the stability, recent work also develops lower bounds on the stability of SGD (Bassily et al., 2020; Amir et al., 2021).

Original: While most stability analysis implies optimal bounds in expectation, the recent study shows that uniform stability can imply almost optimal bounds with high probability (Feldman & Vondrak, 2019; Bousquet et al., 2020; Klochkov & Zhivotovskiy, 2021; Yuan & Li, 2022; Li & Liu, 2022).
Revised: While most stability analyses imply optimal bounds in expectation, the recent study shows that uniform stability can imply almost optimal bounds with high probability (Feldman & Vondrak, 2019; Bousquet et al., 2020; Klochkov & Zhivotovskiy, 2021; Yuan & Li, 2022; Li & Liu, 2022).

Original: Algorithmic stability has found wide applications in various learning problems, including transfer learning (Kuzborskij & Lampert, 2018), meta-learning (Maurer, 2005), structured prediction (London et al., 2016), hyperparameter optimization (Bao et al., 2021), neural networks (Richards & Kuzborskij, 2021) and adversarial training (Xing et al., 2021).
Revised: Algorithmic stability has found wide applications in various learning problems, including transfer learning (Kuzborskij & Lampert, 2018), meta-learning (Maurer, 2005), structured prediction (London et al., 2016), hyperparameter optimization (Bao et al., 2021), neural networks (Richards & Kuzborskij, 2021) and adversarial training (Xing et al., 2021).

Original: Learning with dependent data .
Revised: Learning with dependent data .

Original: For learning with dependent data, one generally assumes that the data are drawn froma stationary and mixing sequence with the dependence between observations diminishing appropriately over time (Doukhan, 1994).
Revised: For learning with dependent data, one generally assumes that the data are drawn from stationary and mixing sequences with the dependence between observations diminishing appropriately over time (Doukhan, 1994).

Original: Initially, generalization bounds were established via a uniform convergence approach based on complexity measures of function classes, such as VC dimension (Yu, 1994), covering numbers (Meir, 2000) and Rademacher complexity (Mohri & Rostamizadeh, 2008).
Revised: Initially, generalization bounds were established via a uniform convergence approach based on complexity measures of function classes, such as VC dimension (Yu, 1994), covering numbers (Meir, 2000) and Rademacher complexity (Mohri & Rostamizadeh, 2008).

Original: Based on a localization idea and self-bounding loss functions, Steinwart & Christmann (2009) developed fast learning rates for regularized algorithms with geometrically α -mixing data.
Revised: Based on a localization idea and self-bounding loss functions, Steinwart & Christmann (2009) developed fast learning rates for regularized algorithms with geometrically α -mixing data.

Original: Ralaivola et al.
Revised: Ralaivola et al.

Original: (2009) and Alquier et al.
Revised: (2009) and Alquier et al.

Original: (2013) established convergence rates under the assumption of stationary and weak dependence.
Revised: (2013) established convergence rates under the assumption of stationary and weak dependence.

Original: While most discussions focus on stationary sequence, Kuznetsov & Mohri (2017) used Rademacher complexity to study learning bounds with non-stationary φ -mixing and β -mixing sequence.
Revised: While most discussions focus on stationary sequence, Kuznetsov & Mohri (2017) used Rademacher complexity to study learning bounds with non-stationary ϕ -mixing and β -mixing sequences.

Original: The first stability analysis of learning with mixing sequences was given in the paper (Mohri & Rostamizadeh, 2010).
Revised: The ﬁrst stability analysis of learning with mixing sequences was given in the paper (Mohri & Rostamizadeh, 2010).

Original: The stability approach was also used to study online learning with dependent data (Agarwal & Duchi, 2013) and learning with graph-dependent data (Zhang et al., 2019).
Revised: The stability approach was also used to study online learning with dependent data (Agarwal & Duchi, 2013) and learning with graph-dependent data (Zhang et al., 2019).

Original:  C ONCENTRATION I NEQUALITIES FOR φ - MIXING S EQUENCES
Revised:  C ONCENTRATION I NEQUALITIES FOR ϕ - MIXING S EQUENCES

Original: We consider learning problems with a sequence of dependent observations.
Revised: We consider learning problems with a sequence of dependent observations.

Original: We assume the dependency between two observations decays with their gap.
Revised: We assume the dependency between two observations decays with their gap.

Original: There are several concepts to quantify the dependency relationship within a stationary sequence such as β  -mixing and φ -mixing (Mohri & Rostamizadeh, 2010; Yu, 1994).
Revised: There are several concepts to quantify the dependency relationship within a stationary sequence such as β -mixing, ϕ -mixing and ψ -mixing (Mohri & Rostamizadeh, 2010; Yu, 1994).

Original: We focus on the φ -mixing and ψ -mixing sequence in this paper.
Revised: We focus on the ϕ -mixing and ψ -mixing sequences in this paper.

Original: Definition 1 ( φ -Mixing Sequence) .

Original: Let Z = { Z t } ∞ t =
Revised: Let Z = { Z t } ∞ t =

Original: −∞ be a stationary sequence of random variables.
Revised: −∞ be a stationary sequence of random variables.

Original: For any i, j ∈ N , let σ ji denote the σ -algebra generated by the random variables
Revised: For any i, j ∈ N , let σ ji denote the σ -algebra generated by the random variables

Original: Z k , i ≤ k ≤ j .
Revised: Z k , i ≤ k ≤ j .

Original: Then for any k ∈ N , the φ -mixing coefficient ofthe stochastic process Z is defined as
Revised: Deﬁnition 1 ( ϕ -Mixing Sequence) . For any k ∈ N , the ϕ -mixing coefﬁcient of Z is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Z is said to be φ -mixing if φ ( k ) → 0 as k → ∞ .
Revised: Z is said to be ϕ -mixing if ϕ ( k ) → 0 as k → ∞ .

Original: It is said to be algebraically φ -mixing
Revised: It is said to be algebraically ϕ

Original: with degree r > 0 ) if there exists a real number φ 0 > 0 such that φ ( k ) ≤ φ 0 /k r for all k , exponentially mixing (with degree r ) if there exist real numbers φ 0 , φ 1 such that φ ( k ) ≤ φ 0 exp( − φ 1 k r ) for all k .
Revised: -mixing (with degree r > 0 ) if there exists a real number ϕ 0 > 0 such that ϕ ( k ) ≤ ϕ 0 /k r for all k , exponentially mixing (with degree r ) if there exist real numbers ϕ 0 , ϕ 1 such that ϕ ( k ) ≤ ϕ 0 exp( − ϕ 1 k r ) for all k .

Original: Definition 2 ( ψ -Mixing Sequence (Bradley, 2007)) .
Revised: Deﬁnition 2 ( ψ -Mixing Sequence (Bradley, 2007)) .

Original: Let Z = { Z t } ∞ t =

Original: −∞ be a stationary sequence of random variables.

Original: For any i, j ∈ N , let σ ji denote the σ -algebra generated by the random variables

Original: Z k , i ≤ k ≤ j .

Original: Then for any k ∈ N , the ψ -mixing coefficient of the stochastic process Z is defined as
Revised: For any k ∈ N , the ψ -mixing coefﬁcient of the stochastic process Z is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Z is said to be ψ -mixing if ψ ( k ) → 0 as k → ∞ .
Revised: Z is said to be ψ -mixing if ψ ( k ) → 0 as k → ∞ .

Original: It is said to be algebraically ψ -mixing
Revised: It is said to be algebraically ψ -mixing

Original: with degree r > 0 ) if there exists a real number ψ 0 > 0 such that ψ ( k ) ≤ ψ 0 /k r for all k , exponentially mixing (with degree r ) if there exist real numbers ψ 0 , ψ 1 such that ψ ( k ) ≤ ψ 0 exp( − ψ 1 k r ) for all k . 
Revised: with degree r > 0 ) if there exists a real number ψ 0 > 0 such that ψ ( k ) ≤ ψ 0 /k r for all k , exponentially mixing (with degree r ) if there exist real numbers ψ 0 , ψ 1 such that ψ ( k ) ≤ ψ 0 exp( − ψ 1 k r ) for all k . We will use ψ -mixing to give a bound on the stability analysis version of ϕ (cid:48)

Original:  Intuitively speaking, φ ( k ) and ψ ( k ) measure the dependency of an event on those happened k units of time ahead.
Revised: -mixing in Lemma 3. Intuitively speaking, ϕ ( k ) and ψ ( k ) measure the dependency of an event on those happened k units of time ahead.

Original: By the definition, we know that ψ -mixing is stronger than the φ -mixing.
Revised: By the deﬁnition, we know that ψ -mixing is stronger than the ϕ -mixing.

Revised: Below we provide examples of ϕ -mixing and ψ -mixing sequences in Kesten & O’Brien (1976).

Original: To develop error bounds for learning with φ -mixing sequence, we first develop concentration inequalities for φ -mixing sequences.
Revised: We ﬁrst consider random variables V n , U n and S n for n ∈ Z deﬁned on the probability space (Ω , F , P ) , which are independent of each other and have the following distributions: for all n ∈ Z , P ( V n = i ) = β i , i = 0 , 1 and 0 < β 0 < β 0 + β 1 = 1 ; P ( U n = k ) =

Revised: p k ≥ 0 , k = 0 , 1 , · · · and (cid:80) ∞

Revised: k =0 p k = 1 ; P ( S n = j ) = γ j , j = 0 , 1 , 2 and 0 < γ 0 < γ 0 + γ 1 < γ 0 + γ 1 + γ 2 = 1 .

Revised: Example 1 (Example of ϕ -mixing sequence) .

Revised: Let { f k , k ≥ 1 } be a non-increasing sequence such that f 1 ≤ 1 , f k → 0 as k → ∞ and 2 log(1 − f k +1 ) ≥ log(1 − f k )+log(1 − f k +2 ) for { k : f k < 1 } .

Revised: For any 0 < (cid:15) < 12 we deﬁne β 0 = (cid:15) , β 1 = 1 − (cid:15) .

Revised: Deﬁne { p n } by

Revised: [Equation]

Revised: if f

Revised: [Equation]

Revised: if f

Revised: [Equation]

Revised: [Equation]

Revised: Note that p k =

Revised: (1 − f k +1 )(1 − f k +2 ) − 1 − (1 − f k )(1 − f k +1 )

Revised: − 1 ≥ 0 for { k : f k < 1 } since 2 log(1 − f k +1 ) ≥ log(1 − f k )

Revised: + log(1 − f k +2 ) .

Revised: Let X n = U n + 12 V n + 14 W n where W n = V n − U n .

Revised: Then { X n , n ∈ Z } is a ϕ -mixing sequence with (1 − (cid:15) )

Revised: f k ≤ ϕ ( k ) ≤ f k .

Revised: Next we provide an example of ψ -mixing sequence.

Revised: Example 2 (Example of ψ -mixing sequence) .

Revised: Let { g k , k ≥ 1 } be a sequence such that g 1 − g 2 = 1 , g k

Revised: → 0 as k → ∞ and 2 g k +1 ≤ g k + g k +2 , k

Revised: = 1 , 2 , · · · .

Original: {+For any (cid:15) ∈ (0 , 1) , Let γ 2 = (cid:15) , γ 
+}
Revised: {+For any (cid:15) ∈ (0 , 1) , Let γ 2 = (cid:15) , γ 
+}

Revised: = γ 1 = 1 2 (1 − (cid:15) ) , β 0 = β 1 = 1 2 .

Revised: Deﬁne { p n } by

Revised: [Equation]

Revised: Note that p k ≥ 0 for k = 0 , 1 , · · · .

Revised: Let U n , V n and S n be as before.

Revised: Let Z n = S n I [ S n =0 or 1] +

Revised: V n − U n I [ S n =2] , where I is the indicator function.

Revised: Finally we deﬁne X n = V n + 2 Z n .

Revised: Then { X n , n ∈

Revised: Z } is a ψ -mixing sequence with (cid:15)

Revised: (1 + (cid:15) ) − 1 g k ≤ ψ ( k ) ≤ exp (cid:2) (cid:15) (1 − (cid:15) )

Revised: − 1 g k (cid:3) − 1 .

Revised: To develop error bounds for learning with ϕ -mixing sequences, we ﬁrst develop concentration inequalities for ϕ -mixing sequences.

Original: In the following theorem to be proved in Section A, we derive a tail bound for the summation of dependent random variables in terms of the tail behavior of each individual random variable.
Revised: In the following theorem to be proved in Section A, we derive a tail bound for the summation of dependent random variables in terms of the tail behavior of each individual random variable.

Original: Let ∆ n = 1 + 2 (cid:80) nk =1 φ ( k ) .
Revised: Let ∆ n = 1 + 2 (cid:80) nk =1 ϕ ( k ) .

Original: The L p -norm of a real-valued random variable Z is denoted by ∥ Z ∥ p :=
Revised: The L p -norm of a real-valued random variable Z is denoted by (cid:107) Z (cid:107) p :

Original:  (cid:0) E [ | Z | p ] (cid:1) 1 p , p ≥ 1 .
Revised: = (cid:0) E [ | Z | p ] (cid:1) 1 p , p ≥ 1 .

Original: Theorem 1.
Revised: Theorem 1.

Original: Let X 1 , . . .
Revised: Let X 1 , . . .

Original: , X n be random variables distributed according to φ -mixing distributions.
Revised: , X n be a ﬁnite contiguous subsequence from a ϕ -mixing sequence.

Original: Let Z i be a function of X i with E [ Z i ] = 

Revised: Let Z i be a function of X i with E [ Z i ] =

Original: and Pr {| Z i | > ˜ ϵ } ≤ 2 exp( − ˜ ϵ 2 /b ) .
Revised: and Pr {| Z i | > ˜ (cid:15) } ≤ 2 exp( − ˜ (cid:15) 2 /b ) .

Original: Then for any p ≥we have
Revised: Then for any p ≥ 1 we have

Original: [Equation]
Revised: [Equation]

Original: Remark 1.
Revised: Remark 1.

Original: Theorem 1 is an extension of Marcinkiewicz-Zygmund inequality for independent random variables to φ -mixing sequence.
Revised: Theorem 1 is an extension of Marcinkiewicz-Zygmund inequality for independent random variables to ϕ -mixing sequences.

Original: Indeed, if Z i are i.i.d.
Revised: Indeed, if Z i are i.i.d.

Revised: , it was shown (cid:13)(cid:13) (cid:80) ni

Original: , the following inequality was developed (Ren & Liang, 2001) [Equation]
Revised: =1 Z i (cid:13)(cid:13) p = O ( p √ nb ) (Ren & Liang, 2001).

Original: We show how the mixing behavior would affect the concentration by including ∆ n in our bound.
Revised: We show how the mixing behavior would affect the concentration by including ∆ n in our bound.

Original: In particular, if Z i are independent then ∆ n = 1 and in this case, our result matches the MarcinkiewiczZygmund inequality up to a logarithmic factor. 
Revised: In particular, if Z i are independent then ∆ n = 1 and in this case, our result matches the Marcinkiewicz-Zygmund inequality up to a logarithmic factor. Note ∆ n = O (1) for algebraical ϕ

Original: Note ∆ n = O (1) for algebraical φ -mixing  and exponential φ -mixing sequences (Mohri & Rostamizadeh, 2010). {+Under the assumption (cid:80) ∞ k =1 ϕ 12 ( k ) < ∞ , it was shown (cid:13)(cid:13)(cid:13) (cid:80) ni =
+}
Revised: -mixing with r > 1 and exponential ϕ -mixing sequences (Mohri & Rostamizadeh, 2010). {+Under the assumption (cid:80) ∞ k =1 ϕ 12 ( k ) < ∞ , it was shown (cid:13)(cid:13)(cid:13) (cid:80) ni =
+}

Original: Based on Theorem 1, we develop a moment bound for Lipschitz functions (w.r.t.
Revised: Z i (cid:13)(cid:13)(cid:13) p ≤ C p (cid:16)(cid:16) (cid:80) ni =1 (cid:107) Z i (cid:107) pp (cid:17) 1 p + (cid:16) (cid:80) ni =1 (cid:107)

Revised: Z i (cid:107) 22 (cid:17) 12 (cid:17) (Xuejun et al., 2010).

Revised: This bound requires an assumption involving (cid:80) ∞ k

Revised: =1 ϕ 12 ( k ) , which is larger than (cid:80) nk =1 ϕ ( k ) in ∆ n since ϕ 12 ( k ) ≥ ϕ ( k ) .

Revised: For example, if ϕ ( k ) =

Revised: O ( k − 1 ) then (cid:80) nk =1 ϕ

Revised: ( k )

Revised: =

Revised: O (log n ) while (cid:80) nk

Revised: =1 ϕ 12 ( k ) =

Revised: O ( √ n ) .

Revised: Moreover, the bound in Xuejun et al.

Revised: (2010) involves C p which is not explicitly stated.

Revised: As a comparison, our bound involves all explicit constants.

Revised: Remark 2.

Revised: Our basic idea to prove Theorem 1 is to apply a McDiarmid inequality (Lemma A.1) to a Lipschitz function deﬁned on a ϕ -mixing sequence.

Revised: If we deﬁne Φ (cid:48) ( X 1 , . . .

Revised: , X n )

Revised: = (cid:80) ni

Revised: =1 Z i , one cannot guarantee the Lipschitz continuity of Φ due to the unboundedness of Z i .

Revised: Our novelty is to deﬁne (cid:101)

Revised: Z i

Revised: =

Revised: Z i I

Revised: | Z i |≤ (cid:15) where I [ · ] is an indicator function and (cid:15)

Revised: =

Revised: O ( (cid:112) b log(1 /δ )) .

Revised: The boundedness of (cid:101)

Revised: Z i implies the (2 (cid:15) ) -Lipschitz

Revised: continuity of Φ and therefore we can apply the McDiarmid inequality to study its decay rate of Φ .

Revised: Furthermore, the assumption Pr {| Z i | > ˜ (cid:15) } ≤ 2 exp( − ˜ (cid:15) 2 /b ) shows that Φ and Φ (cid:48) are equal with a high probability.

Revised: We then combine these two observations together to derive a high-probability bound for Φ (cid:48) , which further leads to a bound on the L p -norm of Φ (cid:48) by the equivalence between high-probability bound and the L p -norm bound.

Revised: Based on Theorem 1, we develop a moment bound for Lipschitz functions (w.r.t.

Original: the Hamming distance) defined on mixing sequences.
Revised: the Hamming distance) deﬁned on mixing sequences.

Original: The following theorem is an extension of a result in the i.i.d.
Revised: The following theorem is an extension of a result in the i.i.d.

Original: case (Bousquet et al., 2020) to the case with mixing sequences.
Revised: case (Bousquet et al., 2020) to the case with mixing sequences.

Original: This result plays a major role in developing our generalization bounds for learning with mixing sequences.
Revised: This result plays a major role in developing our generalization bounds for learning with mixing sequences.

Revised: The ﬁrst assumption is a conditional boundedness assumption which is standard for concentration inequalities.

Revised: The second assumption implies that the g i is of mean zero conditioned on any ﬁxed Z [ n ] \ [ i ] , which is stronger than E [ g i ] = 0 since the conditional expectation holds for any ﬁxed Z [ n ] \ [ i ] .

Revised: The last assumption implies that g i is insensitive to the change of any single example, which implies that g i is concentrated around its expectation by McDiarmid’s inequality.

Original: Our proof follows from the framework in Bousquet et al.
Revised: Our proof follows from the framework in Bousquet et al.

Original: (2020), which is given in Section B.
Revised: (2020), which is given in Section B.

Original: Theorem 2 (Concentration Inequality for φ -Mixing Sequence) .
Revised: Theorem 2 (Concentration Inequality for ϕ -Mixing Sequence) .

Original: Let Z= ( Z 1 , . . .
Revised: Let Z 1 , . . .

Original: , Z n) be a vector of random variables drawn from a φ -mixing sequence. 
Revised: , Z n be a ﬁnite contiguous subsequence from a ϕ -mixing sequence. Denote Z = { Z 1 , . . . , Z n } .

Original: Let g 1 , . .
Revised: Let g 1 , . .

Original: . , g n be some functions g
Revised: . , g n be some functions g

Original: i : Z n (cid:55)→ R such that the following holds for any i ∈ [ n ]
Revised: i : Z n (cid:55)→ R such that the following holds for any i ∈ [ n ]

Original: • (cid:12)(cid:12) E 
Revised: • (cid:12)(cid:12) E Z [ n ] \

Original: [  g i ( Z ) | Z i ] (cid:12)(cid:12) ≤ M almost surely, • E  [ g i  | Z [ n ] \ [ i ] ] = 

Revised: [ i ] [ g i ( Z ) | Z i ] (cid:12)(cid:12) ≤ M almost surely, • E Z i [ g i ( Z ) | Z [ n ] \ [ i ] ] =

Original: a.s., •
Revised: a.s., •

Original: g
Revised: g

Original: i is β -Lipschitz w.r.t.
Revised: i is β -Lipschitz w.r.t.

Original: the Hamming distance.
Revised: the Hamming distance.

Original: Then for any p ≥ 1 we have 
Revised: Then for any p ≥ 1 we have ( k = (cid:100) log 2 n (cid:101) ) (cid:13)(cid:13)(cid:13) n (cid:88) i = g i (cid:13)(cid:13)(cid:13) p ≤ 3 M ∆

Original: [Equation]
Revised: [Equation]

Original: where k = ⌈ log 2 n ⌉ .

Original: Remark 2.
Revised: Remark 3.

Original: If the sequence is i.i.d., the following bound was developed (Bousquet et al., 2020)
Revised: If the sequence is i.i.d., the following bound was developed (Bousquet et al., 2020)

Original: [Equation]
Revised: [Equation]

Original: We now show that our bound recovers the existing result in the i.i.d. case.
Revised: Our bound recovers the existing result in the i.i.d. case.

Original: Note ∆ n = 1 for any n and then Theorem 2 implies [Equation]
Revised: Note ∆ n = 1 for any n and then Theoremimplies (cid:13)(cid:13)(cid:13) (cid:80) ni

Original: This{+=
 g i (cid:13)(cid:13)(cid:13) p ≤ 3 M √ 2 pn + npβ (5+log 2 (cid:100) n (cid:101) ) 2 , which+} matches Eq.
Revised: {+=
 g i (cid:13)(cid:13)(cid:13) p ≤ 3 M √ 2 pn + npβ (5+log 2 (cid:100) n (cid:101) ) 2 , which+} matches Eq.

Original: (3.1) up to a logarithmic factor.
Revised: (3.1) up to a logarithmic factor.

Revised: This is the ﬁrst extension of the result in Bousquet et al.

Revised: (2020) to a ϕ -mixing setting.

Original: S TABILITY AND G ENERALIZATION
Revised: We follow the framework in Bousquet et al. (2020) to prove Theorem 2. The difference is to replace the Marcinkiewicz-Zygmund inequality for i.i.d.

Revised: random variables by Theorem 1 for ϕ -mixing random variables.

Revised: The basic idea is to use the representation (cid:80) ni

Original: {+=
+}
Revised: {+=
+}

Revised: g i =

Revised: (cid:80) ni

Revised: =1 E [ g i | Z i ] + (cid:80)

Revised: k − 1 l =0 (cid:80) ni

Revised: =1 (cid:0) g li − g l +1 i (cid:1) , where g li is the expectation of g i conditioned on some random variables and k is an integer depending on n .

Revised: We then use a McDiarmid inequality and the conditional boundedness of E [ g i |

Revised: Z i ] to control (cid:80)

Revised: ni =1 E [ g i |

Revised: Z i ] , and use Theorem 1 to control (cid:80) k − 1 l =0 (cid:80) ni

Revised: =1 (cid:0)

Revised: [Equation]

Revised: li

Revised: [Equation]

Revised: g

Revised: [Equation]

Revised: +

Revised: [Equation]

Revised: (cid:1) .

Revised:  S TABILITY AND G ENERALIZATION

Original: Let Z = X
Revised: Let Z = X

Original: × Y be a sample space, where X ⊆ R d is an input space and Y is an output space.
Revised: × Y be a sample space, where X ⊆ R d is an input space and Y is an output space.

Original: We consider supervised learning problems wherea sequence of input-output pairs S 
Revised: We consider supervised learning problems where S =

Original: = { z 1 , . ..
Revised: z 1 , . .

Original: , z n } = (cid:8) ( x 1 , y 1 ) , . . .
Revised: z n } = (cid:8) ( x 1 , y 1 ) , . . .

Original: , ( x n , y n ) (cid:9) is drawn from a φ -mixing sequence.
Revised: , ( x n , y n ) (cid:9) is a contiguous subsequence from a ψ -mixing sequence.

Original: Based on S , we wish to find a model h : X (cid:55)→ Y .
Revised: Based on S , we wish to ﬁnd a model h : X (cid:55)→ Y .

Original: We consider parametric models where the model is determined by a parameter w in a parameter space W .
Revised: We consider parametric models where the model is determined by a parameter w in a parameter space W .

Original: The performance of a model w on a single example z can be measured by a loss function f ( w ; z ) .
Revised: The performance of a model w on a single example z can be measured by a loss function f ( w ; z ) .

Original: The empirical risk and population risk are then defined by
Revised: The empirical risk and population risk are then deﬁned by

Original: [Equation]
Revised: [Equation]

Original: which measure the behavior of w on training examples and test examples, respectively.
Revised: which measure the behavior of w on training examples and test examples, respectively.

Original: Here the test point z is assumed to be dependent of S , which is  the most realistic setting considered in Mohri &
Revised: Here the test point z is assumed to be dependent on S (i.e., z is assumed to follow immediately after the sample

Original:  Rostamizadeh (2010).
Revised: S ), which is the most realistic setting considered in Mohri & Rostamizadeh (2010).

Original: We refer to the discrepancy between training and testing as the generalization gap F ( w ) − F S ( w ) .
Revised: We refer to the discrepancy between training and testing as the generalization gap F ( w ) − F S ( w ) .

Original: In machine learning, we often apply an algorithm to get a model with a small training error.
Revised: In machine learning, we often apply an algorithm to get a model with a small training error.

Original: Meanwhile, we wish the output model also admits a small generalization gap to enjoy good generalization to test data.
Revised: Meanwhile, we wish the output model also admits a small generalization gap to enjoy good generalization to test data.

Original: In this paper, we are interested in developing generalization error bounds that decay to zero as the sample size goes to infinity.
Revised: In this paper, we are interested in developing generalization error bounds that decay to zero as the sample size goes to inﬁnity.

Original: Our basic tool is the algorithmic stability, which measures the sensitivity of the output up to the perturbation of a single example.
Revised: Our basic tool is the algorithmic stability, which measures the sensitivity of the output up to the perturbation of a single example.

Original: Various concepts of stability have been introduced in the literature.
Revised: Various concepts of stability have been introduced in the literature.

Original: In this paper, we focus on the uniform stability which is arguably the most widely used algorithmic stability.
Revised: In this paper, we focus on the uniform stability which is arguably the most widely used algorithmic stability.

Original: We use w S to mean the output model if we apply an algorithm A to the dataset S .
Revised: We use w S to mean the output model if we apply an algorithm A to the dataset S .

Original: Note we omit the dependency of the notation on A , which should be clear from the context.
Revised: Note we omit the dependency of the notation on A , which should be clear from the context.

Revised: We say two sets are neighboring datasets if they differ by one example.

Original: Definition 3 (Uniform Stability (Bousquet & Elisseeff, 2002)) .
Revised: Deﬁnition 3 (Uniform Stability (Bousquet & Elisseeff, 2002)) .

Original: A randomized algorithm A is ϵuniformly stable if for all training datasets S, S 
Revised: A randomized algorithm A is (cid:15) - uniformly stable if for all neighboring datasets S, S (cid:48) ∈

Original: ′ ∈ Z nthat differ by at most one example, we have [Equation]
Revised: Z n we have sup z (cid:2) f ( w S ; z ) − f ( w S (cid:48) ; z ) (cid:3) ≤ (cid:15).

Original: Our stability analysis requires a different mixing coefficient defined as follows
Revised: Our stability analysis requires a different mixing coefﬁcient deﬁned as follows

Original: [Equation]
Revised: [Equation]

Original: It is clear that φ ′ ( k ) ≥ φ ( k ) .
Revised: It is clear that ϕ (cid:48) ( k ) ≥ ϕ ( k ) .

Original: The following lemma controls φ ′ ( k ) in terms of the ψ -mixing coefficients.
Revised: The following lemma controls ϕ (cid:48) ( k ) in terms of the ψ -mixing coefﬁcients.

Original: According to the following lemma, one can show that if ψ ( k ) =
Revised: According to the following lemma, one can show that if ψ ( k ) =

Original: O ( k − r ) then φ ′ ( k ) =
Revised: O ( k − r ) then ϕ (cid:48) ( k ) =

Original: O ( k − r ) .
Revised: O ( k − r ) .

Original: If ψ ( k ) =
Revised: If ψ ( k ) =

Original: O (exp( − ψ 1 k r )) 
Revised: O (exp( − ψ 1 k r )) then ϕ (cid:48)

Original: then φ ′ ( k ) =
Revised: ( k ) =

Original: O (exp( − ψ 1 k r )) .
Revised: O (exp( − ψ 1 k r )) .

Original: Lemma 3.
Revised: Lemma 3.

Original: Let Z = { Z t } ∞ t =
Revised: Let Z = { Z t } ∞ t =

Original: −∞ be  a ψ -mixing sequence and assume ψ ( k ) < 1 .
Revised: −∞ be drawn from a ψ -mixing distribution and assume ψ ( k ) < 1 .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: To apply Theorem 2 to learning with mixing sequence, we need to introduce a sequence of functions
Revised: To apply Theorem 2 to learning with mixing sequences, we need to introduce a sequence of functions

Original: g i satisfying the conditions in Theorem 2 and relate them to the generalization gap.
Revised: g i satisfying the conditions in Theorem 2 and relate them to the generalization gap.

Original: Let z 
Revised: Let z (cid:48) i (resp.

Original: i ( z ′′ i ) be drawn from the same distribution of z i , i.e., the conditional distribution of z ′
Revised: z (cid:48)(cid:48) i ) be drawn from the same distribution of z i , i.e., the conditional distribution of z (cid:48)

Original: i ( z ′′ i ) given z 1 , . . .
Revised: i (resp. z (cid:48)(cid:48) i ) given z 1 , . . .

Original: z i − 1 , z i +1 , . . .
Revised: z i − 1 , z i +1 , . . .

Original: , z n is the same as that of z
Revised: , z n is the same as that of z

Original: i given z 1 , . . .
Revised: i given z 1 , . . .

Original: z i − 1 , z i +1 , . . .
Revised: z i − 1 , z i +1 , . . .

Original: , z n .
Revised: , z n .

Original: Let
Revised: Let

Original: S i,b = { z 1 , . . .
Revised: S i,b = { z 1 , . . .

Original: z i − b − 1 , z i , z
Revised: z i − b − 1 , z i , z

Original: i + b +1 , . . .
Revised: i + b +1 , . . .

Original: , z n − b } , i.e., we remove 2 b points around z i .
Revised: , z n − b } , i.e., we remove 2 b points around z i .

Original: For any i ∈ [ n ] , let S ii,b
Revised: For any i ∈ [ n ] , let S ii,b

Original: = { z 1 , . . .
Revised: = { z 1 , . . .

Original: z i − b − 1 , z ′
Revised: z i − b − 1 , z (cid:48)

Original: i , z i + b +1 , . . .
Revised: i , z i + b +1 , . . .

Original: , z n − b } .
Revised: , z n − b } .

Original: We then define the followingsequence of random variables.
Revised: We then deﬁne the following random variables

Original: For any i ∈ [ n ] , introduce

Original: [Equation]
Revised: [Equation]

Original: The following lemma to be proved in Section C gives generalization bounds in terms of stability and (cid:80) n i
Revised: The following lemma to be proved in Section C gives generalization bounds in terms of stability and (cid:80) ni

Original: =1{+=
+} g i .
Revised: {+=
+} g i .

Revised: We will use ϕ (cid:48)

Revised: ( b ) in Theorem 5 and all corollaries in Section 5.

Revised: The underlying reason is that we need to remove 2 b points around z

Revised: i to get S i,b for the application of Theorem 2.

Revised: An upper bound | F ( w S ) − E z (cid:48)(cid:48) i [ f ( w S i,b ; z (cid:48)(cid:48) i )] | requires to use ϕ (cid:48)

Revised: ( b ) .

Original: Lemma 4.
Revised: Lemma 4.

Original: Let S be drawn from a φ -mixing sequence.
Revised: Let S be drawn from a ψ -mixing distribution.

Original: Let b ∈ { 0 , . .
Revised: Let b ∈ { 0 , . .

Original: . , n } denote the number of last  removed in S , and let w S denote the hypothesis trained on S .
Revised: . , n } denote the number of last points removed in S , i.e., S b = { z 1 , . . , z n − b } . Let w S denote the hypothesis trained on S .

Original: If the algorithm A is β -uniformly stable and the loss function is bounded by M > 0 , then the following inequality holds with g i defined
Revised: If the algorithm A is β -uniformly stable and the loss function is bounded by M > 0 , then the following inequality holds with g i deﬁned in Eq.

Original: [Equation]
Revised: [Equation]

Original: We can apply Theorem 2 to control the term (cid:80) ni
Revised: We can apply Theorem 2 to control the term (cid:80) ni

Original: =

Revised: =

Original: g i and derive the following generalization bounds in terms of mixing coefficients.
Revised: g i and derive the following generalization bounds in terms of mixing coefﬁcients.

Original: Theorem 5 (General Mixing Stability Bound) .
Revised: Theorem 5 (General Mixing Stability Bound) .

Original: Let w S denote the hypothesis returned by a βuniformly stable algorithm trained on a sample S drawn from a φ -mixing stationary distribution.
Revised: Let w S denote the hypothesis returned by a βuniformly stable algorithm trained on a sample S drawn from a ψ -mixing stationary distribution.

Original: Let
Revised: Let

Original: M denote the uniform bound of the loss function.
Revised: M denote the uniform bound of the loss function.

Original: Then for any b ∈ { 0 , . . . , n } and any δ ∈ (0 , 1) , the following inequality holds with probability at least 1 − δ ( k = ⌈ log 2 n ⌉ ) 
Revised: Then for any b ∈ { 0 , . . . , n } and any δ ∈ (0 , 1) , the following inequality holds with probability at least 1 − δ ( k = (cid:100) log 2 n (cid:101) ) [Equation] F

Original: [Equation]
Revised: [Equation]

Original: We denote B ≍ (cid:101) B if there are absolute constants c 1 and c 2 such that c 1 B ≤ (cid:101) B ≤ c 2 B .
Revised: w S ) − F S ( w S ) (cid:12)(cid:12) ≤ 2(3 b + 1) β + M ( ϕ ( b ) + ϕ (cid:48) ( b )) + 3 eM ∆ n (cid:114)

Revised: [Equation]

Original: Remark 3.
Revised: Remark 4.

Original: Under the same assumptions, the following high-probability bounds were developedin (Mohri & Rostamizadeh, 2010)
Revised: For ϕ -mixing sequences, the following high-probability bounds were developed (Mohri & Rostamizadeh, 2010)

Original: [Equation]
Revised: [Equation]

Original: As a comparison [Equation] our generalization bound in Theorem 5 becomes
Revised: As a comparison, our generalization bound in Theorem 5 becomes

Original: [Equation]
Revised: [Equation]

Original: /δ

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: Eq.
Revised: Eq.

Original: (4.4) improves Eq.
Revised: (4.5) improves Eq.

Original: (4.3) in the following ways: (1) we replace √ nφ ( b ) with φ ′ ( b ) ; (2) we replace √ nbβ ∆ n with β ( b + ∆ 2 n log 2 n ) .
Revised: (4.4) as follows: (1) we replace √ nϕ ( b ) with ϕ (cid:48) ( b ) ; (2) we replace √ nbβ ∆ n with β ( b + ∆ 2 n log 2 n ) .

Original: The above two terms save a factor of √ n .
Revised: The above two terms save a factor of √ n .

Revised: Meanwhile it should be also mentioned that our bounds involve ϕ (cid:48) , while the bounds in Mohri & Rostamizadeh (2010) involve ϕ .

Original: We now compare these two bounds  in three cases.
Revised: We now compare these two bounds under assumptions of ϕ (cid:48) in three cases (results in Mohri &

Revised: Rostamizadeh (2010) hold for ϕ -mixing coefﬁcients and also hold for ϕ (cid:48) -mixing coefﬁcients).

Original: In thespecial i.i.d. case, Eq. (4.4) becomes 
Revised: In the i.i.d. case, (4.5) becomes (cid:12)(cid:12) F ( w S )

Original: [Equation]
Revised: − F S ( w S ) (cid:12)(cid:12) = O (cid:0)(cid:112) n − 1 log(1 /δ ) + β log 2 n log(1 /δ )

Original:  which matchesthe existing stability-based bounds (Bousquet et al., 2020) up to a logarithmic factor.
Revised: (cid:1) , which matches existing stability-based bounds (Bousquet et al., 2020) up to a logarithmic factor.

Original: As a comparison, Eq. (4.3) becomes 
Revised: As a comparison, Eq. (4.4) becomes (cid:12)(cid:12) F ( w S ) − F S ( w S ) (cid:12)(cid:12) =

Original: [Equation]
Revised: O (cid:0)(cid:112) log(1 /δ ) ( √ nβ + 1 / √ n ) (cid:1) .

Original: We now considerthe algebraically mixing sequence, i.e., φ ′ ( k ) ≤ φ 0 k − r with r > 1 .
Revised: We now consider algebraically mixing sequences, i.e., ϕ (cid:48) ( k ) ≤ ϕ 0 k − r with r > 1 .

Original: In this case, we know ∆ n = O (1) (Mohri & Rostamizadeh, 2010).
Revised: In this case, we know ∆ n = O (1) (Mohri & Rostamizadeh, 2010).

Original: If we choose b ≍ β − 1 r +1 in Eq.
Revised: If we choose b (cid:16) β − 1 r +1 in Eq.

Original: (4.4), we get βb ≍ b − r ≍ β rr +

Revised: (4.5), we get βb (cid:16) b − r (cid:16) β rr +

Original: and therefore 
Revised: and therefore (we denote B (cid:16) (cid:101) B if there are absolute constants c 1 and c 2 such that c 1 B ≤ (cid:101) B ≤ c 2 B .)

Original: [Equation]
Revised: [Equation]

Original: As a comparison, the optimal choice b ≍ β − 1 r +1 in Eq.
Revised: As a comparison, the optimal choice b (cid:16) β − 1 r +1 in Eq.

Original: (4.3) implies
Revised: (4.4) implies

Original: [Equation]
Revised: [Equation]

Original: It is clearthat our bound replaces √ nβ rr +1 in Eq.
Revised: It is clear our bound replaces √ nβ rr +1 in Eq.

Original: (4.6) by β rr +1 + β log 2 n log(1 /δ ) , which issignificantly smaller.
Revised: (4.7) by β rr +1 + β log 2 n log(1 /δ ) , which is smaller.

Original: Finally, we considerthe exponential mixing sequence, i.e., φ ′ ( k ) ≤ φ 0 exp ( − φ 1 k r ) . In this case, we know ∆ n = O (1) .
Revised: Finally, we consider exponential mixing sequences, i.e., ϕ (cid:48) ( k ) ≤ ϕ 0 exp ( − ϕ 1 k r ) , which imply ∆ n = O (1) .

Original: If we choose b = ⌈ log 1 r (1 /β ) ⌉ , we know exp( − b r ) ≤ bβ =
Revised: If we ﬁx b = (cid:100) log 1 r (1 /β ) (cid:101) , we know exp( − b r ) ≤ bβ =

Original: O ( β log 1 r (1 /β )) . Our bound then implies
Revised: O ( β log 1 r (1 /β )) and (cid:12)(cid:12) F [Equation]

Original: [Equation]
Revised: [Equation]

Original: As a comparison, Eq. (4.3) implies 
Revised: As a comparison, Eq. (4.4) implies (cid:12)(cid:12) F ( w S ) − F S ( w S ) (cid:12)(cid:12) =

Original: [Equation]
Revised: O (cid:0)(cid:112) log(1 /δ ) (cid:0) √ nβ log 1 r (1 /β ) + √ nβ + 1 / √ n (cid:1)(cid:1) .

Original: It is clear that our analysis removes a factor of √ n in front of the stability parameter β .
Revised: It is clear that our analysis removes a factor of √ n in front of the stability parameter β .

Original: Remark 4.
Revised: Remark 5.

Original: The analysis in the mixing case is more challenging than that in the i.i.d. case.
Revised: The analysis in the mixing case is more challenging than that in the i.i.d. case.

Original: The analysis in Bousquet et al.
Revised: The analysis in Bousquet et al.

Original: (2020) introduces ˜ g
Revised: (2020) introduces ˜ g

Original: i =
Revised: i =

Original: E z ′
Revised: E z (cid:48)

Original: i E z i [ f ( w s i ; z ) − f ( w S i ; z i )] , where z is independently drawn from the stationary distribution.
Revised: i E z i [ f ( w S i ; z ) − f ( w S i ; z i )] , where z is independently drawn from the stationary distribution and S i = { z 1 , . .

Revised: , z i − 1 , z (cid:48)

Revised: i , z i +1 , . . .

Revised: , z n } .

Original: While E  [˜ g i ] = 0 in the i.i.d. case, we cannot guarantee E  [˜ g i ]  in the mixing case due to the dependency between z i and z j ( j ̸ = i ).
Revised: While E z i [˜ g i ] = 0 in the i.i.d. case, we cannot guarantee E z i [˜ g i ] = 0 in the mixing case due to the dependency between z i and z j ( j (cid:54) = i ).

Original: In this way, one cannot apply Theorem 1 to ˜ g
Revised: In this way, one cannot apply Theorem 2 to ˜ g

Original: i in the mixing case.
Revised: i in the mixing case.

Original: We use a much more complicated decomposition of the generalization error in terms of g i in Eq.
Revised: We use a much more complicated decomposition of the generalization error in terms of g i in Eq.

Original: (4.2), which is  mean zero in the mixing case.
Revised: (4.3), which is of mean zero in the mixing case.

Original: In this process, we introduce concepts such as S i,b , S ii,b to fully exploit the stability and mixing property.
Revised: In this process, we introduce concepts such as S i,b , S ii,b to fully exploit the stability and mixing property.

Revised: It should be mentioned that S i,b , S ii,b have been introduced in Mohri & Rostamizadeh (2010).

Original: A PPLICATIONS
Revised: However, the aim is different. These concepts are used in Mohri & Rostamizadeh (2010) to get bounds in expectation for Φ( S ) : = F ( w S ) − F S ( w S ) via a lemma in Yu (1994) for β -mixing sequence, where the concentration of Φ( S ) around its expectation can be directly studied via the McDiarmid inequality for ϕ -mixing sequence.

Revised: As a comparison, our aim is to replace the sequence ˜ g

Revised: i =

Revised: E z (cid:48)

Revised: i E z i [ f ( w S i ; z ) − f ( w S i ; z i )] (with non-zero conditional mean) by the sequence g i in Eq.

Revised: (4.3) with zero conditional mean, which is then controlled by our new concentration inequality for ϕ -mixing sequences (Theorem 2).

Original: We now present applications of Theorem 5 to several algorithms, including the kernel regularization algorithm, SGD and localized iterative regularization.
Revised: We now present applications of Theorem 5 to several algorithms, including the kernel regularization algorithm, SGD and localized iterative regularization.

Original: Let A be an algorithm which outputs a model
Revised: Let A be an algorithm which outputs a model

Original: A ( S ) after observing the dataset S .
Revised: A ( S ) after observing the dataset S .

Original: We are interested in the excess population risk of a model
Revised: We are interested in the excess population risk of a model

Original: A ( S ) defined by F ( A ( S ))
Revised: A ( S ) deﬁned by F ( A ( S ))

Original: − F ( w ∗ ) , which measures the performance of the output model
Revised: − F ( w ∗ ) , which measures the performance of the output model

Original: A ( S ) as compared to the best model w ∗ =
Revised: A ( S ) as compared to the best model w ∗ =

Original: arg min w F ( w ) .
Revised: arg min w F ( w ) .

Original: An efficient approach to this aim is based on the following error decomposition (Bousquet & Bottou, 2008)
Revised: An efﬁcient approach to this aim is based on the following error decomposition (Bousquet & Bottou, 2008)

Original: [Equation]
Revised: [Equation]

Original: where we refer to F ( A ( S ))
Revised: where we refer to F ( A ( S ))

Original: − F S ( A ( S )) as the generalization gap and F S ( A ( S ))
Revised: − F S ( A ( S )) as the generalization gap and F S ( A ( S ))

Original: − F S ( w ∗ ) as the optimization error.
Revised: − F S ( w ∗ ) as the optimization error.

Original: The last term F S ( w ∗ ) − F ( w ∗ ) is easy to control since w ∗ is independent of S .
Revised: The last term F S ( w ∗ ) − F ( w ∗ ) is easy to control since w ∗ is independent of S .

Original: We will apply stability analysis to control the generalization gap, and tools in optimization theory to control the optimization errors.
Revised: We will apply stability analysis to control the generalization gap, and tools in optimization theory to control the optimization error.

Original: To this aim, we first introduce some necessary definitions.
Revised: To this aim, we give some necessary deﬁnitions.

Original: The Lipschitz condition means the gradient is bounded, and the smoothness means the gradient is Lipschitz continuous.
Revised: The Lipschitz condition means the gradient is bounded, and the smoothness means the gradient is Lipschitz continuous.

Original: Examples of Lipschitz loss functions include the hinge loss,the logistic loss andthe absolute loss.
Revised: Examples of Lipschitz loss functions include the hinge loss, logistic loss and absolute loss.

Original: Examples of Lipschitz and smooth loss functions include the logistic loss and Huber loss.
Revised: Examples of Lipschitz and smooth loss functions include the logistic loss and Huber loss.

Original: Definition 4 (Lipschitz) .
Revised: Deﬁnition 4 (Lipschitz, smoothness and convexity) .

Original: Let L > 0  .
Revised: Let L, γ > 0 and µ ≥ 0 .

Original: We say f : W × Z (cid:55)→ R is L -Lipschitz continuous if [Equation] w
Revised: Let f : W × Z (cid:55)→ R .

Original: [Equation] w [Equation] Definition 5 (Smoothness) .
Revised: • We say f is L -Lipschitz continuous if | f ( w ; z ) − f ( w (cid:48) ; z ) | ≤ L (cid:107) w − w (cid:48) (cid:107) for any w , w (cid:48) ,

Original: Let γ > 0 . 
Revised: z . •

Original: We say f: W × Z (cid:55)→ R is γ -smooth if [Equation] Definition 6 (Strong convexity) .
Revised: We say f is γ -smooth if (cid:107)∇ f ( w ; z ) − ∇ f ( w (cid:48) ; z ) (cid:107) ≤ γ (cid:107) w − w (cid:48) (cid:107) for any w , w (cid:48) , z .

Original: Let µ ≥ 0 . We say f: W × Z (cid:55)→ R is µ -strongly convex if f ( w ; z ) − f ( w ′ ; z ) − ⟨ [Equation]
Revised: We say f is µ -strongly convex if f ( w ; z ) − f ( w (cid:48) ; z ) −(cid:104)

Original: [Equation] [Equation]
Revised: w − w (cid:48) , ∇ f ( w (cid:48) ; z ) (cid:105) ≥ µ 2 (cid:107) w − w (cid:48) (cid:107)for any w , w (cid:48) , z .

Original: We say f is convex if it is µ -strongly convex with µ = 0 .
Revised: We say f is convex if it is µ -strongly convex with µ = 0 .

Original: 5. K ERNEL R EGULARIZATION S CHEMES
Revised: 5. K ERNEL R EGULARIZATION S CHEMES

Original: We first consider kernel regularization schemes with convex and Lipschitz loss functions.
Revised: We ﬁrst consider kernel regularization schemes with convex and Lipschitz loss functions.

Original: Let
Revised: Let

Original: K : X × X (cid:55)→ R be a Mercer kernel (i.e., K is symmetric and positive definite) and W be the associated reproducing kernel Hilbert space with the norm ∥ ·∥ K .
Revised: K : X × X (cid:55)→ R be a Mercer kernel (i.e., K is symmetric and positive deﬁnite) and W be the associated reproducing kernel Hilbert space with the norm (cid:107) ·

Original: We consider the following kernel regularization scheme

Original: [Equation]
Revised: [Equation]

Revised: K .

Revised: We consider the following model

Original: [Equation]
Revised: [Equation]

Original: where λ > 0 is a regularization parameter to tradeoff the data-fitting term F S and the regularizer ∥ w ∥ 2 K .
Revised: where λ > 0 is a regularization parameter to tradeoff the data-ﬁtting term F S and the regularizer (cid:107) w (cid:107) 2 K .

Original: The following corollary gives high-probability excess population risk bounds on kernel regularization.
Revised: The following corollary gives high-probability excess population risk bounds on kernel regularization.

Original: The proof is given in Section D.1.
Revised: The proof is given in Section D.1.

Original: Corollary 6.
Revised: Corollary 6.

Original: Let w S,λ denote the hypothesis returned by Eq.
Revised: Let w S,λ denote the hypothesis returned by Eq.

Original: (5.2) when trained on a sample S drawn from a φ -mixing stationary distribution.
Revised: (5.2) when trained on a sample S drawn from a ψ -mixing stationary distribution.

Original: Assume f is  L -Lipschitz.
Revised: Assume f is convex, L -Lipschitz and bounded by

Original:  Then, with probability at least 1 − δ , the following generalization bound holds ( k = ⌈ log 2 n ⌉ )
Revised: M > 0 . Then, with probability at least 1 − δ , the following excess risk bound holds ( k = (cid:100) log 2 n (cid:101) )

Original: [Equation]
Revised: [Equation]

Original: Remark 5.
Revised: Remark 6.

Original: We now instantiate the above bounds under special mixing sequences.
Revised: We now instantiate the above bounds under special mixing sequences.

Original: We first consider the algebraically mixing sequence, i [Equation]
Revised: We ﬁrst consider the algebraically mixing sequence, i.e., ϕ (cid:48)

Original: e., φ ′ ( k ) ≤ φ 0 k − r with r > 1 .
Revised: ( k ) ≤ ϕ 0 k − r with r > 1 .

Original: In this case, analysis similar to
Revised: In this case, analysis similar to

Original: Remark 3 implies the following bound with an appropriate choice of b 
Revised: Remark 4 implies the following bound with an appropriate choice of b [Equation] λ

Original: [Equation]
Revised: [Equation]

Original:  If ∥ w ∗ ∥ K = O (1) , then we choose λ ≍ 1 / √ n and get
Revised: w ∗ (cid:107) [Equation] If (cid:107) w ∗ (cid:107) K = O (1) , then we choose λ (cid:16) 1 / √ n and get

Original: [Equation]
Revised: [Equation]

Original: We now consider the exponential mixing case, i.e., φ ′ ( k ) ≤ φ 0 exp ( − φ 1 k r ) .
Revised: We now consider the exponential mixing case, i.e., ϕ (cid:48) ( k ) ≤ ϕ 0 exp ( − ϕ 1 k r ) .

Original: In this case, analysis similar to Remark 3 implies
Revised: In this case, analysis similar to Remark 4 implies

Original: [Equation]
Revised: [Equation]

Original:  If ∥ w ∗ ∥ K = O (1) , then we can choose λ ≍ 1 / √ n to derive
Revised: w ∗ (cid:107) [Equation] If (cid:107) w ∗ (cid:107) K = O (1) , then we can choose λ (cid:16) 1 / √ n to derive

Original: [Equation]
Revised: [Equation]

Original: 5. S TOCHASTIC G RADIENT D ESCENT
Revised: 5. S TOCHASTIC G RADIENT D ESCENT

Original: We apply our generalization bounds to SGD with convex and smooth loss functions, which has wide applications in training complex models in the big-data era due to its simplicity and efficiency.
Revised: We apply our generalization bounds to SGD with convex and smooth loss functions, which has wide applications in training complex models in the big-data era due to its simplicity and efﬁciency.

Original: Definition 7 (Stochastic Gradient Descent) .
Revised: Deﬁnition 5 (Stochastic Gradient Descent) .

Original: Let w 

Revised: Let w

Original: =
Revised: =

Original: ∈
Revised: ∈

Original: R
Revised: R

Original: d be an initial point and { η t } t be a sequence of positive step sizes.
Revised: d be an initial point and { η t } t be a sequence of positive step sizes.

Original: SGD updates models by 
Revised: SGD updates models by w t

Original: [Equation]
Revised: +1 =

Original:  where ∇ f ( w t , z i t ) denotes a gradient of f w.r.t.
Revised: w t − η t ∇ f ( w t ; z i t ) , where ∇ f ( w t , z i t ) denotes a gradient of f w.r.t.

Original: the first argument
Revised: the ﬁrst argument

Original: and i t is independently drawn from the uniform distribution over [ n ] = { 1 , . . . , n } .
Revised: and i t is independently drawn from the uniform distribution over [ n ] = { 1 , . . . , n } .

Original: We assume the algorithm produces w S = 1 T (cid:80)
Revised: We assume the algorithm produces w S = 1 T (cid:80)

Original: Tt =
Revised: Tt =

Original: 1 w t , which is an average of SGD iterates.
Revised: 1 w t , which is an average of SGD iterates.

Original: We first present the generalization error bounds.
Revised: We ﬁrst present the generalization error bounds.

Original: The proofsof Corollary 7 and Corollary 8 are given in Section D.2.
Revised: The proofs are given in Section D.2.

Original: Corollary 7 (Generalization bound) .
Revised: Corollary 7 (Generalization bound) .

Original: Assume that the loss function f ( · ; z ) is γ -smooth, convex and
Revised: Assume that the loss function f ( · ; z ) is γ -smooth, convex,

Original: L -Lipschitz  for every z .
Revised: L -Lipschitz and bounded by M > 0 for every z .

Original: Suppose that we run SGD with step sizes η t ≤ min(2 /γ, η ) for T ≍ n steps on a sample S drawn from a φ -mixing stationary distribution.
Revised: Suppose that we run SGD with step sizes η t ≤ min(2 /γ, η ) for T (cid:16) n steps on a sample S drawn from a ψ -mixing stationary distribution.

Original: Then, with probability at least 1 − δ we have ( k = ⌈ log 2 n ⌉ )
Revised: Then, with probability at least 1 − δ we have ( k = (cid:100) log 2 n (cid:101) )

Original: [Equation]
Revised: [Equation]

Original: We can combine the above generalization bounds and existing optimization error bounds together to derive the following excess risk bounds.
Revised: As a corollary, we develop the following excess risk bounds.

Original: Corollary 8 (Excess risk bound) .
Revised: Corollary 8 (Excess risk bound) .

Original: Assume that the loss function f ( · ; z ) is γ -smooth, convex and L -Lipschitz for every z .
Revised: Assume that the loss function f ( · ; z ) is γ -smooth, convex, LLipschitz and bounded by M > 0 for every z .

Original: Suppose that we run SGD with step sizes η t = η ≍ 1 √ T for T ≍ n steps on a sample S drawn from a φ -mixing stationary distribution.
Revised: Suppose that we run SGD with step sizes η t = η (cid:16) 1 √ T for T (cid:16) n steps on a sample S drawn from a ψ -mixing stationary distribution.

Original: Then, with probability at least 1 − δ we have ( k = ⌈ log 2 n ⌉ )[Equation]
Revised: Then, with probability at least 1 − δ we have ( k = (cid:100) log 2 n (cid:101) )

Original: Remark 6.

Original: We first consider the algebraically mixing sequence,

Original: i

Original: [Equation]

Original: e., φ ′ ( k ) ≤ φ 0 k − r with r > 1 .

Original: In this case, analysis similar to Remark 3 implies

Original: [Equation]
Revised: [Equation]

Original: We now consider the exponential mixing case,

Original: i

Original: [Equation] e., φ ′ ( k ) ≤ φ 0 exp ( − φ 1 k r )  .In this case, analysis similar to Remark 3 implies [Equation]
Revised: ) √ n (cid:19) .

Original: 5. I TERATIVE L OCALIZED A LGORITHM
Revised: 5. I TERATIVE L OCALIZED A LGORITHM

Original: We now turn to convex and non-smooth problems.
Revised: We now turn to convex and non-smooth problems.

Original: In this case, SGD requires a very small step size to enjoy good stability, which however would affect the optimization process.
Revised: In this case, SGD requires a very small step size to enjoy good stability, which however would affect the optimization process.

Original: Indeed, a tradeoff between generalization and optimization requires to run SGD with O ( n 2 ) iterations, which is not computationally efficient (Lei & Ying, 2020; Bassily et al., 2020).
Revised: Indeed, a tradeoff between generalization and optimization requires running SGD with O ( n 2 ) iterations, which is not computationally efﬁcient (Lei & Ying, 2020; Bassily et al., 2020).

Original: To speed up the learning process, we consider an iterative localization scheme (Algorithm 1), which was introduced in the work (Feldman et al., 2020).
Revised: To speed up the algorithm, we consider an iterative localization scheme (Algorithm 1 is deferred to Section D.3), which was introduced in Feldman et al.

Original: The basic idea of Algorithm 1 is to implement the optimization in epochs.
Revised: The basic idea of Algorithm 1 is to implement the optimization in epochs.

Original: At each epoch, Algorithm 1 builds an objective function with a regularizer depending on the output of the previous epoch, which is solved by SGD with T i iterations and learning rates { η t } .

Original: Input: initial point w 0 = 0 , parameter γ > 0 , m = (cid:6) 12 log 2 n (cid:7) 1 for i = 1 , 2 , . .

Original: . , m do set T i ≍ n, γ i =

Original: γ 2 i , η t = γ i n t +1 , t ∈ N draw a sample S i ofsize n from the mixing distribution apply SGD with T i iterations and step size η t to minimize the following problem and get w i (cid:101) [Equation] S i
Revised: At each epoch, Algorithm 1 builds an objective function with a regularizer depending on the output of the previous epoch, which is solved by SGD with T i iterations and learning rates { η t } .

Original: [Equation]

Original: w ) :=

Original: [Equation]

Original: (cid:88) [Equation] 5 end The following corollary gives the generalization bounds and excess risk bounds for Algorithm 1.
Revised: The following corollary gives error bounds for Algorithm 1.

Original: The proof is given in Section D.3.
Revised: The proof is given in Section D.3.

Original: Corollary 9.
Revised: Corollary 9.

Original: Assume that the loss function f ( · ; z ) is convex and L -Lipschitz  for every z .
Revised: Assume that the loss function f ( · ; z ) is convex, L -Lipschitz and bounded by M > 0 for every z .

Original: We run Algorithm 1 on sample S i , i ∈ [ m ] drawn from a φ -mixing stationary distribution.
Revised: We run Algorithm 1 on sample S i , i ∈ [ m ] drawn from a ψ -mixing stationary distribution.

Original: If we choose γ ≍ n − 12 , then, with probability at least 1 − δ
Revised: If we choose γ (cid:16) n − 12 , then, with probability at least 1 − δ

Original: F ( w m ) − F ( w ∗ ) =
Revised: F ( w m ) − F ( w ∗ ) =

Original: O (cid:16) n − 12 log n ∆ n log 12 ( n/δ ) + bn − 12 +  log nφ ′ b [Equation] k
Revised: O (cid:16) n − 12 log n ∆ n log 12 ( n/δ )+ bn − 12 + M log nϕ (cid:48)

Original: [Equation]

Original: [Equation]

Original: l

Original: [Equation]

Original: l

Original: [Equation]

Original: log( n/

Original: [Equation] (cid:17) [Equation] where k = ⌈ log 2 n ⌉ .
Revised: ( b )+ n −k − 1 (cid:88) l = l ∆ 22 l log( n/δ ) (cid:17) , where k = (cid:100) log 2 n (cid:101) .

Original: Moreover, Algorithm 1 requires only O ( n log n ) gradient computations to achieve this generalization bound.
Revised: Moreover, Algorithm 1 requires only O ( n log n ) gradient computations to achieve this generalization bound.

Original: Remark 7.

Original: We first consider the algebraically mixing sequence, i.e., φ ′ ( k ) ≤ φ 0 k − r with r > 1 .

Original: In this case, analysis similar to Remark 3 implies

Original: [Equation]

Original: Then, we consider the exponential mixing case, i.e., φ ′ ( k ) ≤ φ 0 exp ( − φ 1 k r ) .

Original: Similarly we have

Original: [Equation]

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this paper, we study the stability and generalization  of uniformly stable algorithms for learning with a φ -mixing sequence, which improves the existing bounds (Mohri & Rostamizadeh, 2010) by a factor of √ n .
Revised: With high probability, we develop the ﬁrst stability-based generalization bounds of the order (cid:101) O (1 / √ n ) for learning with a mixing sequence.

Original: We apply our results to several specific algorithms such as regularization schemes, SGD and localized iterative regularization.
Revised: We apply our results to several speciﬁc algorithms such as regularization schemes, SGD and localized iterative regularization.

Original: Our analysis relies on a new moment bound for weakly-dependent random variables defined on a mixing sequence.
Revised: Our analysis relies on a new moment bound for weakly-dependent random variables deﬁned on a mixing sequence.

Original: In this paper we only consider learning with φ -mixing variables. It would be  interesting to extend the analysis here to other mixing sequences of random variables such as β -mixing data. A key step in this direction is to develop a Mcdiarmid inequality for β -mixing data.
Revised: In this paper our generalization bounds involve ϕ (cid:48) -mixing coefﬁcients, which are larger than the ϕ -mixing coefﬁcients. It would be very interesting to investigate whether these ϕ (cid:48) -mixing coefﬁcients can be replaced by ϕ -mixing coefﬁcients.

Original: It would be also interesting to apply our general stability-based bounds to other learning tasks such as structured prediction and meta-learning.
Revised: We guess ϕ (cid:48) -mixing would be more similar to ϕ -mixing than ψ -mixing since both ϕ - and ϕ (cid:48) -coefﬁcients measure the difference between a conditional probability and a probability (i.e., of the form | Pr( A | B ) − Pr( A ) | ). As a comparison, ψ -mixing considers the difference between 1 and the ratio of probabilities (i.e. of the form | 1 − Pr( A ∩ B ) / Pr( A )Pr( B ) | ).

Original: A P ROOF OF T HEOREM
Revised: A P ROOF OF T HEOREM

Original: In this section we prove Theorem 1.
Revised: In this section we prove Theorem 1.

Original: To this aim we introduce several lemmas.
Revised: To this aim we introduce several lemmas.

Original: The following lemma is a Mcdiarmid inequality for stable functions defined on mixing sequences (Kontorovich &
Revised: The following lemma is a McDiarmid inequality for stable functions deﬁned on mixing sequences (Kontorovich &

Original: Ramanan, 2008).
Revised: Ramanan, 2008).

Original: Lemma A.1.
Revised: Lemma A.1.

Original: Let Φ : Z n (cid:55)→ R be a measurable function that is c -Lipschitz w.r t. the Hamming distance for some c > and let Z 1 , . . .
Revised: Let Φ : Z n (cid:55)→ R be a measurable function that is c -Lipschitz w.r.t. the Hamming distance for some c > and let Z 1 , . . .

Original: , Z n be random variables distributed according to φ -mixing distributions.
Revised: , Z n be random variables distributed according to a ϕ -mixing distribution.

Original: Then for any ϵ > 0 the following inequality holds
Revised: Then for any (cid:15) > 0 the following inequality holds

Original: [Equation]
Revised: [Equation]

Original: Furthermore, for any δ ∈ (0 , 1) the following inequality holds with probability at least 1 − δ
Revised: Furthermore, for any δ ∈ (0 , 1) the following inequality holds with probability at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: The following lemma shows the equivalence of tails and moments (Bousquet et al., 2020).
Revised: The following lemma shows the equivalence of tails and moments (Bousquet et al., 2020).

Original: Lemma A.2.
Revised: Lemma A.2.

Original: Let Y be a random variable.
Revised: Let Y be a random variable.

Original: If for any δ ∈ (0 , 1) , with probability at least 1 − δ
Revised: If for any δ ∈ (0 , 1) , with probability at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: then for any p ≥ 1 it holds that
Revised: then for any p ≥ 1 it holds that

Original: [Equation]
Revised: [Equation]

Original: If ∥ Y ∥ p ≤ √ pa + pb for any p ≥ 1 , then for any δ ∈ (0 , 1) we have with probability at least 1 − δ
Revised: If (cid:107) Y (cid:107) p ≤ √ pa + pb for any p ≥ 1 , then for any δ ∈ (0 , 1) we have with probability at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem 1.
Revised: Proof of Theorem 1.

Original: Let ϵ > 0 be a number to be fixed later and define
Revised: Let (cid:15) > 0 be a number to be ﬁxed later and deﬁne

Original: [Equation]
Revised: [Equation]

Original: where I [ · ] is the indicator function ( 1 if the argument is true and 0 otherwise).
Revised: where I [ · ] is the indicator function ( 1 if the argument is true and 0 otherwise).

Original: Define Φ( X 1 , . . . , X n ) 
Revised: Deﬁne Φ( X 1 , . . . , X n ) =

Original: = (cid:80) ni =1 (cid:101)
Revised: (cid:80) ni =1 (cid:101)

Original: Z i .
Revised: Z i .

Original: First we show the Lipschitz continuity of Φ w.r.t.
Revised: First we show the Lipschitz continuity of Φ w.r.t.

Original: the Hamming distance.
Revised: the Hamming distance.

Original: Suppose we change X 1 by X ′ 1 and keep other X i .
Revised: Suppose we change X 1 by X (cid:48) 1 and keep other X i .

Original: Define Z ′ 1 , . . .
Revised: Deﬁne Z (cid:48) 1 , . . .

Original: , Z ′ n similarly to Z 1 , . . .
Revised: , Z (cid:48) n similarly to Z 1 , . . .

Original: , Z n but as functions of X ′ 1 , X 2 , . . .
Revised: , Z n but as functions of X (cid:48) 1 , X 2 , . . .

Original: , X n .
Revised: , X n .

Original: Since Z j is a function of X j (i.e., once we know X j we know
Revised: Since Z j is a function of X j (i.e., once we know X j we know

Original: Z j ), we know (cid:101) Z j = (cid:101) 
Revised: Z j ), we know (cid:101) Z j = (cid:101) Z (cid:48)

Original: Z ′ j for all j ̸ = 1 .
Revised: j for all j (cid:54) = 1 .

Original: Then 
Revised: Then (cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original:  According to Lemma A.1 with c = 2 ϵ we derive the following inequality with probability at least 1 −( [Equation] δ
Revised: [Equation] According to Lemma A.1 with c = 2 (cid:15) we derive the following inequality with probability at least 1 −

Original: [Equation]
Revised: [Equation]

Original: According to the assumption Pr {| Z i | > ˜ ϵ } ≤ 2 exp( − ˜ ϵ 2 /b ) , we know with probability at least 1 − 2 δ/ ( en )
Revised: According to the assumption Pr {| Z i | > ˜ (cid:15) } ≤ 2 exp( − ˜ (cid:15) 2 /b ) , we know with probability at least 1 − 2 δ/ ( en )

Original: [Equation]
Revised: [Equation]

Original: We fix ϵ = (cid:112) b log( en/δ ) .
Revised: We ﬁx (cid:15) = (cid:112) b log( en/δ ) .

Original: We now assume that Eq.
Revised: We now assume that Eq.

Original: (A.1) and Eq.
Revised: (A.1) and Eq.

Original: (A.2) hold for all i ∈ [ n ] , which happen with probability at least 1 − δ .
Revised: (A.2) hold for all i ∈ [ n ] , which happen with probability at least 1 − δ .

Original: Under this event, we have Z i =
Revised: Under this event, we have Z i =

Original: (cid:101)
Revised: (cid:101)

Original: Z i and (for simplicity we assume n ≥ 2 / ( e − 2) )
Revised: Z i and (for simplicity we assume n ≥ 2 / ( e − 2) )

Original: [Equation]
Revised: [Equation]

Original: The following inequality then holds with probability at least 1 − δ
Revised: The following inequality then holds with probability at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: According to Lemma A.2, for any p ≥ 1 it holds that
Revised: According to Lemma A.2, for any p ≥ 1 it holds that

Original: [Equation]
Revised: [Equation]

Original: The stated bound follows directly.
Revised: The stated bound follows directly.

Original: The proof is completed.
Revised: The proof is completed.

Original: B P ROOF OF T HEOREM
Revised: B P ROOF OF T HEOREM

Original: We follow the framework in Bousquet et al.
Revised: We follow the framework in Bousquet et al.

Original: (2020) to prove Theorem 2.
Revised: (2020) to prove Theorem 2.

Original: Proof of Theorem 2.
Revised: Proof of Theorem 2.

Original: Without loss of generality, we assume n = 2 k .
Revised: Without loss of generality, we assume n = 2 k .

Original: Consider a sequence of partitions
Revised: Consider a sequence of partitions

Original: B 0 , . . .
Revised: B 0 , . . .

Original: , B k with B k = { 1 , 2 , . . .
Revised: , B k with B k = { 1 , 2 , . . .

Original: , 2 k } .
Revised: , 2 k } .

Original: We then obtain B l from B l +1 by splitting each subset in B l + into two equal parts.
Revised: We then obtain B l from B l +1 by splitting each subset in B l + into two equal parts.

Original: In this way, we get
Revised: In this way, we get

Original: [Equation]
Revised: [Equation]

Original:  { [ n ] } .
Revised: [Equation] , { 2 k }} , B 1 = {{ 1 , 2 } , { 3 , 4 } , . . . , { 2 k − 1 , 2 k }} , . . . , B k = { [ n ] } .

Original: For each i ∈ [ n ] and l = 0 , 1 , . . .
Revised: For each i ∈ [ n ] and l = 0 , 1 , . . .

Original: , k , denote by B l ( i )
Revised: , k , denote by B l ( i )

Original: ∈
Revised: ∈

Original: B l the only set from B l that contains i .
Revised: B l the only set from B l that contains i .

Original: In particular, B 0 ( i ) =
Revised: In particular, B 0 ( i ) =

Original: { i } and B k ( i ) =
Revised: { i } and B k ( i ) =

Original: [ n ] .
Revised: [ n ] .

Original: For each i ∈ [ n ] and each l = 0 , 1 , . . .
Revised: For each i ∈ [ n ] and each l = 0 , 1 , . . .

Original: , k , introduce the random variables 
Revised: , k , introduce the random variables g li =

Original: [Equation]
Revised: g li ( Z i , Z [ n ] \ B l ( i ) ) = E [ g i | Z i , Z [ n ] \ B l ( i ) ] .

Original: That is, we condition on Z i and all the variables that are not in the same set as Z i in the partition B l .
Revised: That is, we condition on Z i and all the variables that are not in the same set as Z i in the partition B l .

Original: One can check that g 

Revised: One can check that g

Original: i =
Revised: i =

Original: g i and g ki
Revised: g i and g ki

Original: = E [ g i |
Revised: = E [ g i |

Original: Z i ] .
Revised: Z i ] .

Original: We can write a telescopic sum for each i ∈ [ n ] 
Revised: We can write a telescopic sum for each i ∈ [ n ] g i =

Original: [Equation]
Revised: E [ g i | Z i ] + k − 1 (cid:88) l = ( g li − g l +1 i ) .

Original: It then follows from the triangle inequality that
Revised: It then follows from the triangle inequality that

Original: [Equation]
Revised: [Equation]

Revised: n (cid:88) i =

Original: B
Revised: E

Original: [Equation]
Revised: [Equation]

Revised: k

Original: [Equation]
Revised: [Equation]

Revised: l

Revised: [Equation]

Revised: (cid:13)(cid:13)(cid:13) n (cid:88) i =

Revised: [Equation]

Revised: [Equation]

Revised: g

Original:  Since | E [ g i | Z i ] | ≤ M , one can check that Φ( Z 1 , . . .
Revised: [Equation] i ) (cid:13)(cid:13)(cid:13) p . (B.1) Since | E [ g i | Z i ] | ≤ M , one can check that Φ( Z 1 , . . .

Original: , Z n )
Revised: , Z n )

Original: = (cid:80) ni
Revised: = (cid:80) ni

Original: =1 E [ g i |
Revised: =1 E [ g i |

Original: Z i ] is 2 M -Lipschitz w.r.t.
Revised: Z i ] is 2 M -Lipschitz w.r.t.

Original: the Hamming distance.
Revised: the Hamming distance.

Original: Furthermore, we have E [ E [ g i |
Revised: Furthermore, we have E [ E [ g i |

Original: Z i ]]
Revised: Z i ]]

Original: = 0 .
Revised: = 0 .

Original: Now we can apply Lemma A.1 with c = 2 M to derive the following inequality with at least 1 − δ
Revised: Now we can apply Lemma A.1 with c = 2 M to derive the following inequality with at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: It then follows from Lemma A.2 that
Revised: It then follows from Lemma A.2 that

Original: [Equation]
Revised: [Equation]

Original: According to the definition of g li , one can see that
Revised: According to the deﬁnition of g li , one can see that

Original: [Equation]
Revised: [Equation]

Original: Furthermore, according to our assumption we know g li as a function of Z j , j ∈ B l +1 ( i ) \ B l ( i ) satisfies the β -Lipschitz continuity w.r.t.
Revised: Furthermore, according to our assumption we know g li as a function of Z j , j ∈ B l +1 ( i ) \ B l ( i ) satisﬁes the β -Lipschitz continuity w.r.t.

Original: the Hamming distance.
Revised: the Hamming distance.

Original: Therefore, one can apply Lemma
Revised: Therefore, one can apply Lemma

Original: A.1 with c = β and Φ = g li to derive the following inequality with (there are 2 l random variables)
Revised: A.1 with c = β and Φ = g li to derive the following inequality with (there are 2 l random variables)

Original: [Equation]
Revised: [Equation]

Original: where the probability is w.r.t.
Revised: where the probability is w.r.t.

Original: Z B l +1 ( i ) \ B l ( i ) .
Revised: Z B l +1 ( i ) \ B l ( i ) .

Original: Let us consider the sum (cid:80) i ∈
Revised: Let us consider the sum (cid:80) i ∈

Original: B l ( g li − g l +1 i ) for any B l ∈ B l .
Revised: B l ( g li − g l +1 i ) for any B l ∈ B l .

Original: Note Z 
Revised: Note Z (cid:48)

Original: ′ i :=
Revised: i :=

Original: g li − g l +1 i is a function of
Revised: g li − g l +1 i is a function of

Original: Z i , Z [ n ] \ B l .
Revised: Z i , Z [ n ] \ B l .

Original: We now condition on Z [ n ] \ B l and then Z ′
Revised: We now condition on Z [ n ] \ B l and then Z (cid:48)

Original: i is a function of Z i .
Revised: i is a function of Z i .

Original: According to Eq.
Revised: According to Eq.

Original: (B.3), we can apply Theorem 1 with b = 2 l − 1 β 2 ∆ 22 l to derive the following inequality
Revised: (B.3), we can apply Theorem 1 with b = 2 l − 1 β 2 ∆ 22 l to derive the following inequality

Original: [Equation]
Revised: [Equation]

Original: We now take integration w.r.t.
Revised: We now take integration w.r.t.

Original: Z [ n ] \ B l and get
Revised: Z [ n ] \ B l and get

Original: [Equation]
Revised: [Equation]

Original:  (cid:88) i 
Revised: According to the triangle inequality, we further get (cid:13)(cid:13) (cid:88) i ∈ [ n ]

Original: [Equation]
Revised: [Equation]

Original: B l [Equation] g li − g [Equation]
Revised: where we have used the fact that |B l | = 2 k − l .

Original: ∆ [Equation] According to the triangle inequality, we further get (cid:13)(cid:13) (cid:88) i ∈ [ n ] ( g li − g l +1 i ) (cid:13)(cid:13) p ≤ (cid:88)
Revised: It follows that k − 1 (cid:88)

Original: [Equation]
Revised: [Equation]

Original: ∈B

Original: [Equation]
Revised: [Equation]

Original:  (cid:88) i ∈
Revised: n (cid:88) i =

Original: B l [Equation] g li − g l +1 i ) (cid:13)(cid:13)(cid:13) p ≤ 2 k− l · (9 + l ) p 2 l β ∆ 22 l = (9 + l )2 k pβ ∆ 22 l . where we have used the fact that |B l | = 2 k − l . It follows that k −
Revised: ( g li − g l +1 i ) (cid:13)(cid:13)(cid:13) p ≤ 2 k

Original: [Equation] (cid:88) l= (cid:13)(cid:13)(cid:13) n (cid:88) i = (
Revised: pβ k − 1 (cid:88) l

Original: [Equation]
Revised: = (9 + l )∆ 22 l .

Original: We can plug Eq.
Revised: We can plug Eq.

Original: (B.2) and the above inequality back into Eq.
Revised: (B.2) and the above inequality back into Eq.

Original: (B.1) to derive 
Revised: (B.1) to derive (cid:13)(cid:13)(cid:13) n (cid:88) i = [Equation]

Original: [Equation]
Revised: [Equation]

Original: The proof is completed.
Revised: The proof is completed.

Original: C P ROOF OF T HEOREM
Revised: C P ROOF OF T HEOREM

Original: To prove Theorem 5, we require the following lemma to control the difference between two test error: one with the test example drawn from the mixing sequence and one with the test example drawn from the independent stationary distribution.
Revised: To prove Theorem 5, we require the following lemma to control the difference between two test errors: one with the test example drawn from the mixing sequence and one with the test example drawn from the independent stationary distribution.

Original: Let S b
Revised: Let S b

Original: = { z 1 , . . .
Revised: = { z 1 , . . .

Original: , z n − b } be the sequence by removing the last b points of S .
Revised: , z n − b } be the sequence by removing the last b points of S .

Original: Lemma C.1 (Mohri & Rostamizadeh 2010) .
Revised: Lemma C.1 (Mohri & Rostamizadeh 2010) .

Original: Let F ( w S ) =
Revised: Let F ( w S ) =

Original: E z
Revised: E z

Original: [ f ( w S ; z ) | S ] denote the expectation in the dependent case (i.e., z depends on S ) and ˜ F ( w S b )
Revised: [ f ( w S ; z ) | S ] denote the expectation in the dependent case (i.e., z depends on S ) and ˜ F ( w S b )

Original: =
Revised: =

Original: E (cid:101) z
Revised: E (cid:101) z

Original: [ f ( w S b ; (cid:101) z )] denote the expectation where the test points are assumed independent of the training  (i.e., ˜ z is independent of S ).
Revised: [ f ( w S b ; (cid:101) z )] denote the expectation where the test points are assumed independent of the training data (i.e., ˜ z is independent of S ).

Original: If A is β -uniformly stable and f ( w ; z ) ∈ [0 , M ] , then the following inequality holds for any b >
Revised: If A is β -uniformly stable and f ( w ; z ) ∈ [0 , M ] , then the following inequality holds for any b >

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma 4.
Revised: Proof of Lemma 4.

Original: Let z 
Revised: Let z (cid:48)

Original: ′ i ( z ′′ i ) be drawn from the same distribution of z i , i.e., the conditional distribution of z ′
Revised: i (resp. z (cid:48)(cid:48) i ) be drawn from the same distribution of z i , i.e., the conditional distribution of z (cid:48)

Original: i ( z ′′ i ) given z 1 , . . .
Revised: i (resp. z (cid:48)(cid:48) i ) given z 1 , . . .

Original: z i − 1 , z i +1 , . . .
Revised: z i − 1 , z i +1 , . . .

Original: , z n is the same as that of z
Revised: , z n is the same as that of z

Original: i given z 1 , . . .
Revised: i given z 1 , . . .

Original: z i − 1 , z i +1 , . . .
Revised: z i − 1 , z i +1 , . . .

Original: , z n .
Revised: , z n .

Original: Let S i,b
Revised: Let S i,b

Original: = { z 1 , . . .
Revised: = { z 1 , . . .

Original: z i − b − 1 , z i , z
Revised: z i − b − 1 , z i , z

Original: i + b +1 , . . .
Revised: i + b +1 , . . .

Original: , z n − b } .
Revised: , z n − b } .

Original: For any i ∈ [ n ] , let S ii,b = { z 1 , . . .
Revised: For any i ∈ [ n ] , let S ii,b = { z 1 , . . .

Original: z i − b − 1 , z ′
Revised: z i − b − 1 , z (cid:48)

Original: i , z i + b +1 , . . .
Revised: i , z i + b +1 , . . .

Original: , z n − b } .
Revised: , z n − b } .

Original: We have the following decomposition
Revised: We have the following decomposition

Original: [Equation]
Revised: [Equation]

Original: According to the definition of β -uniform stability we know
Revised: According to the deﬁnition of β -uniform stability we know

Original: [Equation]
Revised: [Equation]

Original: For any i ∈ [ n ] [Equation] introduce
Revised: For any i ∈ [ n ] , introduce

Original: [Equation]
Revised: [Equation]

Original: Then [Equation] we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: According to Lemma C.1 we know
Revised: According to Lemma C.1 we know

Original: [Equation]
Revised: [Equation]

Original: By the definition of φ ′ [Equation] we know
Revised: By the deﬁnition of ϕ (cid:48) , we know

Original: [Equation]
Revised: [Equation]

Original: Furthermore, the definition of stability implies
Revised: Furthermore, the deﬁnition of stability implies

Original: [Equation]
Revised: [Equation]

Original: We combine the above three inequalities together and derive n (cid:88) 
Revised: We combine the above three inequalities together and derive n (cid:88) [Equation]

Revised: ,b ; z (cid:48)(cid:48)

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Revised: ] | + n

Revised: [Equation]

Revised: i =

Revised: [Equation]

Revised: Mϕ

Revised: [Equation]

Revised: b )

Revised: [Equation]

Revised: n.

Original: Combining the above inequality and Eq.
Revised: Combining the above inequality and Eq.

Original: (C.1)
Revised: (C.1), we obtain (cid:12)(cid:12) n ( F ( w S ) − F S ( w S )) (cid:12)(cid:12) ≤ (cid:12) (cid:12)(cid:12) n (cid:88) i = (cid:0) E

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: + n (cid:88) i [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original:  i = | F ( w S ) − E z ′′ i [ f ( w Si,b ; z ′′ i )] | + n (cid:88)
Revised: + n (cid:88) i = (cid:12) (cid:12) f ( w S

Revised: [Equation]

Revised: z i

Original: i = (cid:12) (cid:12) f ( w S i,b ; z i ) − f ( w S ; z i ) (cid:12) (cid:12) ≤ (6 b + 2) nβ + n ( Mφ ( b ) + Mφ ′ ( b )) + (cid:12)(cid:12) n (cid:88) i=
Revised: [Equation] − f ( w S [Equation] z i ) (cid:12) (cid:12) ≤ (6 b + 2) nβ + n ( Mϕ ( b ) + Mϕ (cid:48) ( b )) + (cid:12)(cid:12) n (cid:88) i

Original:  g i (cid:12)(cid:12) .
Revised: = g i (cid:12)(cid:12) .

Original: The proof is completed.
Revised: The proof is completed.

Original: Proof of Theorem 5.
Revised: Proof of Theorem 5.

Original: Recall the definition of g i ,
Revised: Recall the deﬁnition of g i ,

Original: [Equation]
Revised: [Equation]

Original: Since z i and z ′′ i follow from the same distribution [Equation] we know
Revised: Since z i and z (cid:48)(cid:48) i follow from the same distribution, we know

Original: [Equation]
Revised: [Equation]

Original: One can check other assumptions in Theorem 2 also hold.
Revised: One can check other assumptions in Theorem 2 also hold.

Original: Therefore, one can apply Theorem 2 to derive the following inequality with probability at least 1 − δ
Revised: Therefore, one can apply Theorem 2 to derive the following inequality with probability at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: where k = ⌈ log 2 n ⌉ .
Revised: where k = (cid:100) log 2 n (cid:101) .

Original: According to Lemma A.2 we further get the following inequality with probability at least 1 − δ n (cid:88)
Revised: According to Lemma A.2 we further get the following inequality with probability at least 1 − δ n (cid:88)

Original: [Equation]
Revised: [Equation]

Original: According to Lemma
Revised: According to Lemma

Original: [Equation]
Revised: [Equation]

Original: we know
Revised: we know

Original: [Equation]
Revised: [Equation]

Original: We can combine the above inequality with Eq.
Revised: We can combine the above inequality with Eq.

Original: (C.3) to derive the following inequality with probability at least 1 − δ
Revised: (C.3) to derive the following inequality with probability at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: ∆ n (cid:112) 2 n log( e/δ ) + 2 k +1 β k − 1 (cid:88) l

Original: = (9 + l )∆ 22 l log( e/δ ) (cid:17) + (6 b + 2) nβ + n ( Mφ ( b ) + Mφ ′ ( b )) .

Original: The proof is completed.
Revised: The proof is completed.

Original: D P ROOF OF A PPLICATIONS D. P ROOF OF C OROLLARY
Revised: D P ROOF OF A PPLICATIONS D. P ROOF OF C OROLLARY

Original: To prove Corollary 6, we require the following lemma on the uniform stability of kernel regularization (Bousquet & Elisseeff, 2002).
Revised: To prove Corollary 6, we require the following lemma on the uniform stability of kernel regularization (Bousquet & Elisseeff, 2002).

Original: Lemma D.1 (Bousquet & Elisseeff 2002) .
Revised: Lemma D.1 (Bousquet & Elisseeff 2002) .

Original: Let the loss function f be L -Lipschitz and convex.
Revised: Let the loss function f be L -Lipschitz and convex.

Original: Let the algorithm A be defined in (5.2) .
Revised: Let the algorithm A be deﬁned in (5.2) .

Original: Then A is β -uniformly
Revised: Then A is β -uniformly

Original: stable
Revised: stable

Original: with β ≤ L 2 λn .
Revised: with β ≤ L 2 λn .

Original: Proof of Corollary 6.
Revised: Proof of Corollary 6.

Original: Let A be the algorithm which returns w S,λ .
Revised: Let A be the algorithm which returns w S,λ .

Original: By Lemma D.1, we know A is β -uniformly
Revised: By Lemma D.1, we know A is β -uniformly

Original: stable
Revised: stable

Original: where β ≤ L 2 nλ .
Revised: where β ≤ L 2 nλ .

Original: Plugging the above inequality into Theorem 5, we derive the following inequality with probability at least 1 − δ
Revised: Plugging the above inequality into Theorem 5, we derive the following inequality with probability at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: By Lemma A.1, we have the following inequality with probability at least 1 − δ
Revised: By Lemma A.1, we have the following inequality with probability at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: Furthermore, we have the following error decomposition
Revised: Furthermore, we have the following error decomposition

Original: [Equation]
Revised: [Equation]

Original: where we have used the definition of w S,λ .
Revised: where we have used the deﬁnition of w S,λ .

Original: We can combine the above three inequalities together to derive the stated bound.
Revised: We can combine the above three inequalities together to derive the stated bound.

Original: The proof is completed.
Revised: The proof is completed.

Original: D. P ROOF OF C OROLLARY 7 AND C OROLLARY
Revised: D. P ROOF OF C OROLLARY 7 AND C OROLLARY

Original: First, we prove the stability bound for convex loss minimization via SGD.
Revised: First, we prove the stability bound for convex loss minimization via SGD.

Original: Then, we apply the stability bound and Theorem 5 to the generalization bound.
Revised: Then, we apply the stability bound and Theorem 5 to the generalization bound.

Original: To develop high-probability bounds, we need to introduce a concentration inequality (Wainwright, 2019).
Revised: To develop high-probability bounds, we need to introduce a concentration inequality (Wainwright, 2019).

Original: Lemma D.2 (Chernoff’s Bound) .
Revised: Lemma D.2 (Chernoff’s Bound) .

Original: Let X 1 , . . .
Revised: Let X 1 , . . .

Original: , X t be independent random variables taking values in { 0 , 1 } .
Revised: , X t be independent random variables taking values in { 0 , 1 } .

Original: Let X
Revised: Let X

Original: =
Revised: =

Original: (cid:80) tj =1 X j and µ = E
Revised: (cid:80) tj =1 X j and µ = E

Original: [ X ] .
Revised: [ X ] .

Original: Then for any ˜ δ > 0 with probability at least 1 − exp (cid:16) − µ ˜ δ 2 / (2 + ˜ δ ) (cid:17) we have X ≤ (1 + ˜ δ ) µ .
Revised: Then for any ˜ δ > 0 with probability at least 1 − exp (cid:0) − µ ˜ δ 2 / (2 + ˜ δ ) (cid:1) we have X ≤ (1 + ˜ δ ) µ .

Original: Furthermore, for any δ ∈ (0 , 1) with probability at least 1 − δ we have
Revised: Furthermore, for any δ ∈ (0 , 1) with probability at least 1 − δ we have

Original: [Equation]
Revised: [Equation]

Original: Proof of Corollary 7.
Revised: Proof of Corollary 7.

Original: Let S and S ′ be two samples of size n differing in only a single example.
Revised: Let S and S (cid:48) be two samples of size n differing in only a single example.

Original: Consider the gradient updates w 1 , . .
Revised: Consider the gradient updates w 1 , . .

Original: . , w T and w ′ 1 , . .
Revised: . , w T and w (cid:48) 1 , . .

Original: . , w ′ T induced by running SGD on sample S and S ′ .
Revised: w (cid:48) T induced by running SGD on sample S and S (cid:48) .

Original: We now suppose S and S ′ differ by the first example and apply the Lipschitz condition on f ( · ; z ) to get
Revised: We now suppose S and S (cid:48) differ by the ﬁrst example and apply the Lipschitz condition on f ( · ; z ) to get

Original: [Equation]
Revised: [Equation]

Original: where δ T = ∥ w T − w ′ T ∥ .
Revised: where δ T = (cid:107) w T − w (cid:48) T (cid:107) .

Original: Observe that at step t , with probability 1 − 1 /n , the example selected by
Revised: Observe that at step t , with probability 1 − 1 /n , the example selected by

Original: SGD is the same in both S and S ′ .
Revised: SGD is the same in both S and S (cid:48) .

Original: The convexity and γ -smoothness imply that (Hardt et al., 2016)
Revised: The convexity and γ -smoothness imply that (Hardt et al., 2016)

Original: [Equation]
Revised: [Equation]

Original: If i t ̸ = 1 , by η t ≤ 2 /γ
Revised: If i t (cid:54) = 1 , by η t ≤ 2 /γ

Original: we know(cid:13)(cid:13) w t
Revised: we know

Original: [Equation]
Revised: [Equation]

Original: With probability 1 /n
Revised: With probability 1 /n

Original: [Equation]
Revised: [Equation]

Original: the example selected is different
Revised: the example selected is different

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: e. i t = 1 .
Revised: e. i t = 1 .

Original: Then, by the triangle equality and
Revised: Then, by the triangle equality and

Original: Eq.
Revised: Eq.

Original: D.3,
Revised: D.3,

Original: [Equation]
Revised: [Equation]

Original: Combining the above two cases [Equation] we can conclude that for every t ,
Revised: Combining the above two cases, we can conclude that for every t ,

Original: [Equation]
Revised: [Equation]

Original: where I denotes the indicator function.
Revised: where I denotes the indicator function.

Original: Solving recursive inequality gives,
Revised: Solving recursive inequality gives,

Original: [Equation]
Revised: [Equation]

Original: We can apply Lemma D.2 with X k
Revised: We can apply Lemma D.2 with X k

Original: = I
Revised: = I

Original: [ i k =1] , µ = t/n to get the following inequality with probability at least 1 − δ t 
Revised: [ i k =1] , µ = t/n to get the following inequality with probability at least 1 − δ t (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Therefore, with probability at least 1 − δ , there holds
Revised: Therefore, with probability at least 1 − δ [Equation] there holds

Original: [Equation]
Revised: [Equation]

Original: By the convexity of the norm ∥ · ∥ , we get the following inequality with probability at least 1 − δ
Revised: By the convexity of the norm (cid:107) · (cid:107) [Equation] we get the following inequality with probability at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: Plugging the inequality back into Eq.
Revised: Plugging the inequality back into Eq.

Original: (D.1), we obtain that, with probability at least 1 − δ 
Revised: (D.1), we obtain that, with probability at least 1 − δ |

Original: [Equation]
Revised: [Equation]

Revised: D.7)

Original: We can combine Eq.
Revised: We can combine Eq.

Original: D.7 and Theorem 5 to obtain the following inequality with probability at least 
Revised: (D.7) and Theorem 5 to obtain the following inequality with probability at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: By the choice of T ≍ n , we can get
Revised: By the choice of T (cid:16) n , we can get

Original: [Equation]
Revised: [Equation]

Original: n

Original: [Equation]

Original: The proof is completed.
Revised: The proof is completed.

Original: To prove excess risk bounds, we require the following high-probability bound on optimization error.
Revised: To prove excess risk bounds, we require the following high-probability bound on optimization error.

Original: Notice that optimization error analysis does not depend on the mixing property of the dataset since the randomness is taken with respect to the random indices.
Revised: Notice that optimization error analysis does not depend on the mixing property of the dataset since the randomness is taken with respect to the random indices.

Original: Lemma D.3 (Optimization Error (Lei & Tang, 2018)) .
Revised: Lemma D.3 (Optimization Error (Lei & Tang, 2018)) .

Original: Assume that the loss function f ( · ; z ) is convex and L -Lipschitz for every z .
Revised: Assume that the loss function f ( · ; z ) is convex and L -Lipschitz for every z .

Original: Suppose that we run SGD with step sizes η t = η ≍ 1 √ T then with probability at least 1 − δ we have
Revised: Suppose that we run SGD with step sizes η t = η (cid:16) 1 √ T then with probability at least 1 − δ we have

Original: [Equation]
Revised: [Equation]

Original: Proof of Corollary 8.
Revised: Proof of Corollary 8.

Original: By Corollary 7, we know with probability at least 1 − δ that
Revised: By Corollary 7, we know with probability at least 1 − δ that

Original: [Equation]
Revised: [Equation]

Original: By Lemma A.1, we have the following inequality with probability at least 1 − 
Revised: By Lemma A. [Equation] we have the following inequality with probability at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: Lemma D.3 shows the following inequality with probability at least 1 − δ
Revised: Lemma D.3 shows the following inequality with probability at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: We plug the above three inequalities back into Eq.
Revised: We plug the above three inequalities back into Eq.

Original: (5.1), and derive the following inequality with probability at least 1 − 3 δ
Revised: (5.1), and derive the following inequality with probability at least 1 − 3 δ

Original: [Equation]
Revised: [Equation]

Original: The proof is completed.
Revised: The proof is completed.

Original: D. P ROOF OF C OROLLARY
Revised: D. P ROOF OF C OROLLARY

Original: In this section, we present the proofrelated to stability based on the  iterative localization technique. 
Revised: In this section, we present the proof on the stability of the iterative localization technique. To this aim, we ﬁrst present Algorithm 1. Algorithm

Revised: Input: initial point w 0 = 0 , parameter γ > 0 , m = (cid:6) 12 log 2 n

Revised: [Equation]

Revised: 1 for i = 1 , 2 , . . .

Original:  Then we move to generalization bound for Corollary 9.
Revised: , m do set T i (cid:16) n, γ i = γ 2 i , η t = γ i n t +1 , t ∈ N draw a sample S i of size n from the mixing distribution apply SGD with T i iterations and step size η t to minimize the following problem and get w i [Equation] 5 end Then we move to generalization bound for Corollary 9.

Original: To this aim, We need to introduce some definitions for our proof.
Revised: To this aim, We need to introduce some deﬁnitions for our proof.

Original: For any i , let
Revised: For any i , let

Original: [Equation]
Revised: [Equation]

Original: where (cid:101) F S i is defined in Algorithm 1.
Revised: where (cid:101) F S i is deﬁned in Algorithm 1.

Original: Lemma D.4 (Optimization Error Bound) .
Revised: Lemma D.4 (Optimization Error Bound) .

Original: Suppose that the function w (cid:55)→ f ( w ; z ) is µ -strongly convex (with respect to ∥ · ∥ ) and L -Lipschitz.
Revised: Suppose that the function w (cid:55)→ f ( w ; z ) is µ -strongly convex (with respect to (cid:107) · (cid:107) ) and L -Lipschitz.

Original: Let { w t } t be produced by SGD on sample S and step size η t = 2 / ( 
Revised: Let { w t } t be produced by SGD on sample S and step size η t = 2 / ( µ ( t + 1)) to minimize F S ( w ) = 1 n (cid:80) ni

Original: [Equation]
Revised: =1 f ( w ; z i ) .

Original: to
Revised: Set ¯ w (cid:48) t

Original: [Equation]
Revised: = (cid:16)(cid:80) tj =1 j w j (cid:17) / (cid:80) tj =1 j .

Original: Then, for any δ ∈ (0 , 1) , with probability at least 1 − δ ,
Revised: Then, for any δ ∈ (0 , 1) , with probability at least 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: The proof of Lemma D.4 can be found in Harvey et al.
Revised: The proof of Lemma D.4 can be found in Harvey et al.

Original: According to Algorithm 1, w i is the output by SGD with η t = γ i n/ ( t + 1) to minimize (cid:101) F S i ( w ) , with the iterates weighted as Lemma
Revised: According to Algorithm 1, w i is the output by SGD with η t = γ i n/ ( t + 1) to minimize (cid:101) F S i ( w ) , with the iterates weighted as Lemma

Original: D.4.
Revised: D.4.

Original: The following lemma establishes the bound of Euclidean distance of w i and ˆ w i .
Revised: The following lemma establishes the bound of Euclidean distance of w i and ˆ w i .

Original: Lemma D.5.
Revised: Lemma D.5.

Original: Suppose that the function w (cid:55)−→ f ( w ; z ) is L -Lipschitz and µ -strongly convex.
Revised: Suppose that the function w (cid:55)−→ f ( w ; z ) is L -Lipschitz and µ -strongly convex.

Original: For any δ ∈ (0 , 1) , the following inequality holds with probability at least 1 − δ
Revised: For any δ ∈ (0 , 1) , the following inequality holds with probability at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: From Algorithm 1, we know that (cid:101) F S i is λ
Revised: From Algorithm 1, we know that (cid:101) F S i is λ

Original: i := 2 / ( γ i n ) -strongly convex.
Revised: i := 2 / ( γ i n ) -strongly convex.

Original: According to
Revised: According to

Original: Lemma D.4, the following inequality holds with probability at least 1 − δ
Revised: Lemma D.4, the following inequality holds with probability at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: It then follows from the definition of ˆ w i and the strong convexity that
Revised: It then follows from the deﬁnition of ˆ w i and the strong convexity that

Original: [Equation]
Revised: [Equation]

Original: and therefore
Revised: and therefore

Original: [Equation]
Revised: [Equation]

Original: The proof is completed.
Revised: The proof is completed.

Original: Lemma D.6 (Bousquet & Elisseeff 2002) .
Revised: Lemma D.6 (Bousquet & Elisseeff 2002) .

Original: Suppose the function f : W × Z (cid:55)→ R takes a structure f = ℓ + r , where ℓ : W × Z (cid:55)→ R and r : W (cid:55)→ R .
Revised: Suppose the function f : W × Z (cid:55)→ R takes a structure f = (cid:96) + r , where (cid:96) : W × Z (cid:55)→ R and r : W (cid:55)→ R .

Original: Assume for all z , we have ∥∇ ℓ ( w ; z ) ∥ ≤ L .
Revised: Assume for all z , we have (cid:107)∇ (cid:96) ( w ; z ) (cid:107) ≤ L .

Original: Suppose F S = 1 n (cid:80) ni
Revised: Suppose F S = 1 n (cid:80) ni

Original: =1 f ( w ; z i ) is µ -strongly convex and define A as A ( S ) =
Revised: =1 f ( w ; z i ) is µ -strongly convex and deﬁne A as A ( S ) =

Original: arg min w ∈W F S ( w ) .
Revised: arg min w ∈W F S ( w ) .

Original: Then A is 4 L 2 nµ -uniformly stable.
Revised: Then A is 4 L 2 nµ -uniformly stable.

Original: Lemma D.7.
Revised: Lemma D.7.

Original: Assume for any z, w (cid:55)→ f ( w ; z ) is L -Lipschitz.
Revised: Assume for any z, w (cid:55)→ f ( w ; z ) is L -Lipschitz.

Original: Let ˆ w i be defined in Eq.
Revised: Let ˆ w i be deﬁned in Eq.

Original: (D.9) .
Revised: (D.9) .

Original: With probability at least 1 − δ/ (2 m ) we have the following inequality uniformly for any w
Revised: With probability at least 1 − δ/ (2 m ) we have the following inequality uniformly for any w

Original: [Equation]
Revised: [Equation]

Original: where k = ⌈ log 2 n ⌉ .
Revised: where k = (cid:100) log 2 n (cid:101) .

Original: Proof.
Revised: Proof.

Original: For any i , define
Revised: For any i , deﬁne

Original: [Equation]
Revised: [Equation]

Original: and w ∗ i =
Revised: and w ∗ i =

Original: arg min w (cid:101) F i ( w ) , where we assume z is independently drawn from the stationary distribution of mixing sequence.
Revised: arg min w (cid:101) F i ( w ) , where we assume z is independently drawn from the stationary distribution of mixing sequence.

Original: Let A i be the algorithm outputting the minimizer of (cid:101) F S
Revised: Let A i be the algorithm outputting the minimizer of (cid:101) F S

Original: i .
Revised: i .

Original: We know (cid:101) F S i is λ i =
Revised: We know (cid:101) F S i is λ i =

Original: 2 / ( γ i n ) -strongly convex.
Revised: 2 / ( γ i n ) -strongly convex.

Original: Then analysis similar to Corollary 6 implies the stated inequality with probability at least 1 − δ/ (2 m ) .
Revised: Then analysis similar to Corollary 6 implies the stated inequality with probability at least 1 − δ/ (2 m ) .

Original: The proof is completed.
Revised: The proof is completed.

Original: Based on the above lemmas, we now turn to proving Corollary 9.
Revised: Based on the above lemmas, we now turn to proving Corollary 9.

Original: Proof of Corollary 9.
Revised: Proof of Corollary 9.

Original: Let ˆ w 0 = w ∗ and ˆ w i be defined by Eq.
Revised: Let ˆ w 0 = w ∗ and ˆ w i be deﬁned by Eq.

Original: (D.9).
Revised: (D.9).

Original: We can decompose F ( w m ) −
Revised: We can decompose F ( w m ) −

Original: F ( w ∗ ) by
Revised: F ( w ∗ ) by

Original: [Equation]
Revised: [Equation]

Original: Since f is L lipschitz [Equation] Lemma D.5 implies the following inequality with probability at least 1 − δ/ (2 m )
Revised: Since f is L lipschitz, Lemma D.5 implies the following inequality with probability at least 1 − δ/ (2 m )

Original: [Equation]
Revised: [Equation]

Original: Furthermore [Equation] we can apply Lemma D.7 with w = ˆ w i − 1 to derive the following inequality with probability at least 1 − δ/ (2 m )
Revised: Furthermore, we can apply Lemma D.7 with w = ˆ w i − 1 to derive the following inequality with probability at least 1 − δ/ (2 m )

Original: [Equation]
Revised: [Equation]

Original: The following inequality then holds with probability at least 1 − δ/m (cid:88)
Revised: The following inequality then holds with probability at least 1 − δ/m (cid:88)

Original: [Equation]
Revised: [Equation]

Original: By Lemma D.5, we further get the following inequality with probability at least 1 − δ m (cid:88) 
Revised: By Lemma D.5, we further get the following inequality with probability at least 1 − δ m (cid:88) i = [Equation] = [Equation]

Revised: i =

Original: [Equation]
Revised: [Equation]

Original: where in the last two steps we have used γ i
Revised: where in the last two steps we have used γ i

Original: =
Revised: =

Original: γ/ 2 i .
Revised: γ/ 2 i .

Original: We can plug the above inequality and Eq.
Revised: We can plug the above inequality and Eq.

Original: (D.11) back into Eq.
Revised: (D.11) back into Eq.

Original: (D.10), and derive the following inequality with probability 1 − δF
Revised: (D.10), and derive the following inequality with probability 1 − δ

Original: [Equation]
Revised: [Equation]

Original: This gives the stated bound.
Revised: This gives the stated bound.

Original: The proof is completed.
Revised: The proof is completed.

Original: E P ROOF OF L EMMA
Revised: E P ROOF OF L EMMA

Original: Proof of Lemma 3.
Revised: Proof of Lemma 3.

Original: For simplicity, we only consider discrete random variables.
Revised: For simplicity, we only consider discrete random variables.

Original: Let A ∈ σ n − k
Revised: Let A ∈ σ n − k

Original: −∞ , B ∈ σ ∞ n + k and Z n , z ∈ σ nn .
Revised: −∞ , B ∈ σ ∞ n + k and Z n , z ∈ σ nn .

Original: According to the definition of mixing sequence, we know
Revised: According to the deﬁnition of mixing sequence, we know

Original: [Equation]
Revised: [Equation]

Original: By Eq.
Revised: By Eq.

Original: (E.1) we know
Revised: (E.1) we know

Original: [Equation]
Revised: [Equation]

Original: It then follows that
Revised: It then follows that

Original: [Equation]
Revised: [Equation]

Original: In a similar way [Equation] one can show
Revised: In a similar way, one can show

Original: [Equation]
Revised: [Equation]

Original: The proof is completed by combining the above two inequalities together
Revised: The proof is completed by combining the above two inequalities together

Original: [Equation]
Revised: [Equation]