Original: S HARPER B OUNDS FOR U NIFORMLY S TABLE A LGORITHMS WITH S TATIONARY MIXING P ROCESSES
Revised: S HARPER B OUNDS FOR U NIFORMLY S TABLE A LGORITHMS WITH S TATIONARY M IXING P ROCESSES

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Generalization analysis of learning algorithms often builds on a critical assumption that training examples are independently and identically distributed, which is often violated in practical problems such as time series prediction.
Revised: Generalization analysis of learning algorithms often builds on a critical assumption that training examples are independently and identically distributed, which is often violated in practical problems such as time series prediction.

Original: In this paper, we use algorithmic stability to study the generalization performance of learning algorithms with ψ -mixing data, where the dependency between observations weakens over time.
Revised: In this paper, we use algorithmic stability to study the generalization performance of learning algorithms with ψ -mixing data, where the dependency between observations weakens over time.

Original: We show uniformly stable algorithms guarantee high-probability generalization bounds of the order O (1 / √ n ) (within a logarithmic factor), where n is the sample size.
Revised: We show uniformly stable algorithms guarantee high-probability generalization bounds of the order O (1 / √ n ) (within a logarithmic factor), where n is the sample size.

Original: We apply our general result to speciﬁc algorithms including regularization schemes, stochastic gradient descent and localized iterative regularization, and develop excess population risk bounds for learning with ψ -mixing data.
Revised: We apply our general result to specific algorithms including regularization schemes, stochastic gradient descent and localized iterative regularization, and develop excess population risk bounds for learning with ψ -mixing data.

Original: Our analysis builds on a novel moment bound for weakly-dependent random variables on a ϕ -mixing sequence and a novel error decomposition of generalization error
Revised: Our analysis builds on a novel moment bound for weakly-dependent random variables on a φ -mixing sequence and a novel error decomposition of generalization error.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Generalization gap refers to the discrepancy between training and testing, which is a quantity of central importance in statistical learning theory (SLT) (Shalev-Shwartz & Ben-David, 2014).
Revised: Generalization gap refers to the discrepancy between training and testing, which is a quantity of central importance in statistical learning theory (SLT) (Shalev-Shwartz & Ben-David, 2014).

Original: A popular approach to controlling the generalization gap is to bound it by the uniform convergence between training and testing errors over a function space (Bartlett & Mendelson, 2002), which leads to bounds depending on the complexity of function spaces, such as VC dimension (Vapnik, 2013), covering number (Cucker & Zhou, 2007) and Rademacher complexity (Bartlett & Mendelson, 2002).
Revised: A popular approach to controlling the generalization gap is to bound it by the uniform convergence between training and testing errors over a function space (Bartlett & Mendelson, 2002), which leads to bounds depending on the complexity of function spaces, such as VC dimension (Vapnik, 2013), covering number (Cucker & Zhou, 2007) and Rademacher complexity (Bartlett & Mendelson, 2002).

Original: These complexity-based bounds do not exploit the property of a learning algorithm and would generally admit a square-root dependency on the dimension (Feldman, 2016), which are not favorable for large-scale problems.
Revised: These complexity-based bounds do not exploit the property of a learning algorithm and would generally admit a square-root dependency on the dimension (Feldman, 2016), which are not favorable for large-scale problems.

Original: To incorporate the property of a learning algorithm and remove the dependency on dimension, a concept of algorithmic stability has been introduced into
Revised: To incorporate the property of a learning algorithm and remove the dependency on dimension, a concept of algorithmic stability has been introduced into

Original: SLT (Bousquet & Elisseeff, 2002).
Revised: SLT (Bousquet & Elisseeff, 2002).

Original: Intuitively speaking, algorithmic stability measures how a small perturbation of the training dataset would affect the output model of a learning algorithm, which has close connection to several key properties such as learnability (Shalev-Shwartz et al., 2010), robustness and privacy (Bassily et al., 2020).
Revised: Intuitively speaking, algorithmic stability measures how a small perturbation of the training dataset would affect the output model of a learning algorithm, which has close connection to several key properties such as learnability (Shalev-Shwartz et al., 2010), robustness and privacy (Bassily et al., 2020).

Original: Recent research has witnessed an increasing interest in leveraging stability to study the generalization behavior of various algorithms, such as stochastic gradient descent (Hardt et al., 2016), structured prediction (London et al., 2016), meta learning (Maurer, 2005) and transfer learning (Kuzborskij & Lampert, 2018).
Revised: Recent research has witnessed an increasing interest in leveraging stability to study the generalization behavior of various algorithms, such as stochastic gradient descent (Hardt et al., 2016), structured prediction (London et al., 2016), meta learning (Maurer, 2005) and transfer learning (Kuzborskij & Lampert, 2018).

Original: Most of these discussions are based on a critical assumption that the training examples are independently and identically distributed (i.i.d.).
Revised: Most of these discussions are based on a critical assumption that the training examples are independently and identically distributed (i.i.d.).

Original: This assumption is often violated in practical applications.
Revised: This assumption is often violated in practical applications.

Original: For example, the i.i.d.
Revised: For example, the i.i.d.

Original: assumption is too restrictive in time series prediction (Vidyasagar, 2013).
Revised: assumption is too restrictive in time series prediction (Vidyasagar, 2013).

Original: The prices of the same stock on different days may have temporal dependence.
Revised: The prices of the same stock on different days may have temporal dependence.

Original: These phenomena motivate several analyses to derive meaningful bounds for learning problems with observations drawn from a non-i.i.d.
Revised: These phenomena motivate several analyses to derive meaningful bounds for learning problems with observations drawn from a non-i.i.d.

Original: process (Yu, 1994; Vidyasagar, 2013).
Revised: process (Yu, 1994; Vidyasagar, 2013).

Original: A widely used relaxation of the i.i.d.
Revised: A widely used relaxation of the i.i.d.

Original: assumption is to assume the observations are drawn from a mixing process (Yu, 1994; Meir, 2000; Lozano et al., 2005; Vidyasagar, 2013), where the dependency between two observations is quantiﬁed by a mixing coefﬁcient as a function of the discrepancy of the associated two indices.
Revised: assumption is to assume the observations are drawn from a mixing process (Yu, 1994; Meir, 2000; Lozano et al., 2005; Vidyasagar, 2013), where the dependency between two observations is quantified by a mixing coefficient as a function of the discrepancy of the associated two indices.

Original: These mixing coefﬁcients decay either as a polynomial function or an exponential function of the discrepancy (Vidyasagar, 2013).
Revised: These mixing coefficients decay either as a polynomial function or an exponential function of the discrepancy (Vidyasagar, 2013).

Original: Several mixing processes have been introduced into the literature, including the β -mixing, ϕ -mixing and ψ -mixing processes (Yu, 1994; Meir, 2000; Lozano et al., 2005).
Revised: Several mixing processes have been introduced into the literature, including the β -mixing, φ -mixing and ψ -mixing processes (Yu, 1994; Meir, 2000; Lozano et al., 2005).

Original: Within this formulation, various generalization bounds have been developed to show how the dependency among observations would affect the learning process.
Revised: Within this formulation, various generalization bounds have been developed to show how the dependency among observations would affect the learning process.

Original: Interestingly, these discussions imply a concept called “effective size” which plays a similar role of the sample size in the i.i.d.
Revised: Interestingly, these discussions imply a concept called “effective size” which plays a similar role of the sample size in the i.i.d.

Original: scenario (Yu, 1994; Kuznetsov & Mohri, 2017).
Revised: scenario (Yu, 1994; Kuznetsov & Mohri, 2017).

Original: As in the i.i.d.
Revised: As in the i.i.d.

Original: case, most generalization analyses in the non-i.i.d.
Revised: case, most generalization analyses in the non-i.i.d.

Original: case focus on complexity-based bounds (Meir, 2000; Yu, 1994; Kuznetsov & Mohri, 2017).
Revised: case focus on complexity-based bounds (Meir, 2000; Yu, 1994; Kuznetsov & Mohri, 2017).

Original: There are few stability analyses of learning algorithms in the non-i.i.d.
Revised: There are few stability analyses of learning algorithms in the non-i.i.d.

Original: cases.
Revised: cases.

Original: An exception is the work in Mohri & Rostamizadeh (2010), which, to our knowledge, gives the ﬁrst systematic analysis on the stability and generalization in a non-i.i.d.
Revised: An exception is the work in Mohri & Rostamizadeh (2010), which, to our knowledge, gives the first systematic analysis on the stability and generalization in a non-i.i.d.

Original: case.
Revised: case.

Original: The authors developed high-probability generalization bounds for learning with stationary ϕ -mixing and β -mixing sequences, which are then applied to general kernel regularizationbased bounds.
Revised: The authors developed high-probability generalization bounds for learning with stationary φ -mixing and β -mixing sequences, which are then applied to general kernel regularizationbased bounds.

Original: Due to the algorithm-speciﬁc nature, these bounds are preferable to complexity-based bounds if the associated hypothesis space has a very large complexity.
Revised: Due to the algorithm-specific nature, these bounds are preferable to complexity-based bounds if the associated hypothesis space has a very large complexity.

Original: However, the stability analysis (Mohri & Rostamizadeh, 2010) only implies sub-optimal generalization bounds.
Revised: However, the stability analysis (Mohri & Rostamizadeh, 2010) only implies sub-optimal generalization bounds.

Original: Indeed, for β -uniformly stable algorithms, the high-probability bounds in Mohri &
Revised: Indeed, for β -uniformly stable algorithms, the high-probability bounds in Mohri &

Original: Rostamizadeh (2010) are of the order of O ( √ nβ + ∆ n / √ n ) , where n is the sample size and ∆ n is a term depending on the decay rate of mixing coefﬁcients.
Revised: Rostamizadeh (2010) are of the order of O ( √ nβ + ∆ n / √ n ) , where n is the sample size and ∆ n is a term depending on the decay rate of mixing coefficients.

Original: For learning with λ -strongly convex problems, the uniform stability parameter is of the order O
Revised: For learning with λ -strongly convex problems, the uniform stability parameter is of the order O

Original: (1 / ( nλ ))
Revised: (1 / ( nλ ))

Original: (Bousquet & Elisseeff, 2002) and therefore the bounds in Mohri & Rostamizadeh (2010) become O
Revised: (Bousquet & Elisseeff, 2002) and therefore the bounds in Mohri & Rostamizadeh (2010) become O

Original: (1 / ( √ nλ )
Revised: (1 / ( √ nλ )

Original: + ∆ n / √ n ) .
Revised: + ∆ n / √ n ) .

Original: A typical choice of λ is λ ≈ n − α for α > 

Revised: A typical choice of λ is λ ≈ n − α for α >

Original: (Shalev-Shwartz & Ben-David, 2014) and then the bounds further become O ( n α − 12 + ∆ n / √ n ) , which cannot imply the optimal bounds O
Revised: (Shalev-Shwartz & Ben-David, 2014) and then the bounds further become O ( n α − 12 + ∆ n / √ n ) , which cannot imply the optimal bounds O

Original: (1 / √ n ) even if ∆ n = O (1) .
Revised: (1 / √ n ) even if ∆ n = O (1) .

Original: For learning with i.i.d.
Revised: For learning with i.i.d.

Original: data, recent breakthroughs (Feldman & Vondrak, 2019; Bousquet et al., 2020) in stability analysis show that β -uniformly stable algorithms enjoy generalization bounds of the order (cid:101) O
Revised: data, recent breakthroughs (Feldman & Vondrak, 2019; Bousquet et al., 2020) in stability analysis show that β -uniformly stable algorithms enjoy generalization bounds of the order (cid:101) O

Original: (1 / √ n ) 1 .
Revised: (1 / √ n ) 1 .

Original: This motivates a natural question: can we develop generalization bounds of the order (cid:101)
Revised: This motivates a natural question: can we develop generalization bounds of the order (cid:101)

Original: O (1 / √ n ) for uniformly stable algorithms applied to mixing process?
Revised: O (1 / √ n ) for uniformly stable algorithms applied to mixing process?

Original: This paper provides an afﬁrmative answer to the above question.
Revised: This paper provides an affirmative answer to the above question.

Original: Our contributions are listed below.
Revised: Our contributions are listed below.

Original: We develop a moment bound for weakly dependent random variables deﬁned on a ϕ -mixing sequence.
Revised: We develop a moment bound for weakly dependent random variables defined on a φ -mixing sequence.

Original: We show our bound matches the existing moment bounds for i.i
Revised: We show our bound matches the existing moment bounds for i.i.d.

Original: d. random variables up to a logarithmic factor.
Revised: random variables up to a logarithmic factor.

Original: As a byproduct, we develop a Marcinkiewicz-Zygmund inequality for a ϕ -mixing sequence, which may be interesting in its own right.
Revised: As a byproduct, we develop a Marcinkiewicz-Zygmund inequality for a φ -mixing sequence, which may be interesting in its own right.

Original: We develop high-probability bounds of order (cid:101)
Revised: We develop high-probability bounds of order (cid:101)

Original: O (1 / √ n )
Revised: O (1 / √ n )

Original: for uniformly stable algorithms for learning with ψ -mixing sequences (our results actually require assumptions on ϕ (cid:48) -mixing coefﬁcients which are weaker than assumptions on ψ -mixing coefﬁcients).
Revised: for uniformly stable algorithms for learning with ψ -mixing sequences (our results actually require assumptions on φ ′ -mixing coefficients which are weaker than assumptions on ψ -mixing coefficients).

Original: We achieve this by introducing a different decomposition of generalization errors to make sure we get weakly-dependent and mean-zero random variables, which is more challenging than the i
Revised: We achieve this by introducing a different decomposition of generalization errors to make sure we get weakly-dependent and mean-zero random variables, which is more challenging than the i

Original: i
Revised: i

Original: d. case.
Revised: d. case.

Original: Our results recover the existing bounds within a constant factor in the i.i d. case.
Revised: Our results recover the existing bounds within a constant factor in the i.i.d. case.

Original: We apply our general bound to some speciﬁc algorithms to show the effectiveness of our results, including kernel regularization schemes, stochastic gradient descent (SGD) and iterative localization.
Revised: We apply our general bound to some specific algorithms to show the effectiveness of our results, including kernel regularization schemes, stochastic gradient descent (SGD) and iterative localization.

Original: The paper is organized as follows.
Revised: The paper is organized as follows.

Original: We present the related work in Section 2.
Revised: We present the related work in Section 2.

Original: We develop concentration inequalities for ϕ -mixing data in Section 3 and present general stability-based bounds in Section4.
Revised: We develop concentration inequalities for φ -mixing data in Section 3 and present general stability based bounds in Section

Original: We apply our general result to speciﬁc algorithms in Section5.
Revised: We apply our general result to specific algorithms in Section

Original: We conclude the paper in Section 6.
Revised: We conclude the paper in Section 6.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: In this section, we discuss the related work.
Revised: In this section, we discuss the related work.

Original: We ﬁrst discuss the related work on algorithmic stability and then the related work on learning with dependent data.
Revised: We first discuss the related work on algorithmic stability and then the related work on learning with dependent data.

Original: Algorithmic stability .
Revised: Algorithmic stability .

Original: Algorithmic stability measures how the replacement/removal of a single (or a few) example would affect the output model, which is an important concept in SLT (Bousquet &
Revised: Algorithmic stability measures how the replacement/removal of a single (or a few) example would affect the output model, which is an important concept in SLT (Bousquet &

Original: Elisseeff, 2002).
Revised: Elisseeff, 2002).

Original: A nice property of algorithmic stability is that it only considers the behavior of the output model and therefore can imply capacity-independent generalization bounds.
Revised: A nice property of algorithmic stability is that it only considers the behavior of the output model and therefore can imply capacity-independent generalization bounds.

Original: An important stability measure called uniform stability was introduced in an inﬂuential work (Bousquet & Elisseeff, 2002), which was used to study the generalization behavior of regularization schemes.
Revised: An important stability measure called uniform stability was introduced in an influential work (Bousquet & Elisseeff, 2002), which was used to study the generalization behavior of regularization schemes.

Original: This uniform stability was extended to the setting of randomized algorithms (Elisseeff et al., 2005), which was further used to study the generalization guarantee of SGD (Hardt et al., 2016).
Revised: This uniform stability was extended to the setting of randomized algorithms (Elisseeff et al., 2005), which was further used to study the generalization guarantee of SGD (Hardt et al., 2016).

Original: To better exploit the training examples for better generalization bounds, a relaxation of uniform stability called on-average stability has been introduced (Shalev-Shwartz et al., 2010).
Revised: To better exploit the training examples for better generalization bounds, a relaxation of uniform stability called on-average stability has been introduced (Shalev-Shwartz et al., 2010).

Original: In particular, the on-average stability was shown to be equivalent to learnability (Shalev-Shwartz et al., 2010) and was used to derive data-dependent error bounds (Kuzborskij & Lampert, 2018;
Revised: In particular, the on-average stability was shown to be equivalent to learnability (Shalev-Shwartz et al., 2010) and was used to derive data-dependent error bounds (Kuzborskij & Lampert, 2018;

Original: Lei & Ying, 2020; Zhou et al., 2021;
Revised: Lei & Ying, 2020; Zhou et al., 2021;

Original: Li et al., 2020; Nikolakakis et al., 2022).
Revised: Li et al., 2020; Nikolakakis et al., 2022).

Original: The smoothness assumption for stability analysis of SGD was removed in the papers (Lei & Ying, 2020; Bassily et al., 2020).
Revised: The smoothness assumption for stability analysis of SGD was removed in the papers (Lei & Ying, 2020; Bassily et al., 2020).

Original: For nonconvex problems, stability and generalization of learning algorithms that converge to global optima were studied for gradient-dominated problems (Charles & Papailiopoulos, 2018;
Revised: For nonconvex problems, stability and generalization of learning algorithms that converge to global optima were studied for gradient-dominated problems (Charles & Papailiopoulos, 2018;

Original: Lei & Ying, 2021).
Revised: Lei & Ying, 2021).

Original: While most discussions focus on upper bounds on the stability, recent work also develops lower bounds on the stability of SGD (Bassily et al., 2020; Amir et al., 2021).
Revised: While most discussions focus on upper bounds on the stability, recent work also develops lower bounds on the stability of SGD (Bassily et al., 2020; Amir et al., 2021).

Original: While most stability analyses imply optimal bounds in expectation, the recent study shows that uniform stability can imply almost optimal bounds with high probability (Feldman & Vondrak, 2019; Bousquet et al., 2020; Klochkov & Zhivotovskiy, 2021; Yuan & Li, 2022; Li & Liu, 2022).
Revised: While most stability analyses imply optimal bounds in expectation, the recent study shows that uniform stability can imply almost optimal bounds with high probability (Feldman & Vondrak, 2019; Bousquet et al., 2020; Klochkov & Zhivotovskiy, 2021; Yuan & Li, 2022; Li & Liu, 2022).

Original: Algorithmic stability has found wide applications in various learning problems, including transfer learning (Kuzborskij & Lampert, 2018), meta-learning (Maurer, 2005), structured prediction (London et al., 2016), hyperparameter optimization (Bao et al., 2021), neural networks (Richards & Kuzborskij, 2021) and adversarial training (Xing et al., 2021).
Revised: Algorithmic stability has found wide applications in various learning problems, including transfer learning (Kuzborskij & Lampert, 2018), meta-learning (Maurer, 2005), structured prediction (London et al., 2016), hyperparameter optimization (Bao et al., 2021), neural networks (Richards & Kuzborskij, 2021) and adversarial training (Xing et al., 2021).

Original: Learning with dependent data .
Revised: Learning with dependent data .

Original: For learning with dependent data, one generally assumes that the data are drawn from stationary and mixing sequences with the dependence between observations diminishing appropriately over time (Doukhan, 1994).
Revised: For learning with dependent data, one generally assumes that the data are drawn from stationary and mixing sequences with the dependence between observations diminishing appropriately over time (Doukhan, 1994; Smale & Zhou, 2009).

Original: Initially, generalization bounds were established via a uniform convergence approach based on complexity measures of function classes, such as VC dimension (Yu, 1994), covering numbers (Meir, 2000) and Rademacher complexity (Mohri & Rostamizadeh, 2008).
Revised: Initially, generalization bounds were established via a uniform convergence approach based on complexity measures of function classes, such as VC dimension (Yu, 1994), covering numbers (Meir, 2000) and Rademacher complexity (Mohri & Rostamizadeh, 2008).

Original: Based on a localization idea and self-bounding loss functions, Steinwart & Christmann (2009) developed fast learning rates for regularized algorithms with geometrically α -mixing data.
Revised: Based on a localization idea and self-bounding loss functions, Steinwart & Christmann (2009) developed fast learning rates for regularized algorithms with geometrically α -mixing data.

Original: Ralaivola et al.
Revised: Ralaivola et al.

Original: (2009) and Alquier et al.
Revised: (2009) and Alquier et al.

Original: (2013) established convergence rates under the assumption of stationary and weak dependence.
Revised: (2013) established convergence rates under the assumption of stationary and weak dependence.

Original: While most discussions focus on stationary sequence, Kuznetsov & Mohri (2017) used Rademacher complexity to study learning bounds with non-stationary ϕ -mixing and β -mixing sequences.
Revised: While most discussions focus on stationary sequence, Kuznetsov & Mohri (2017) used Rademacher complexity to study learning bounds with non-stationary φ -mixing and β -mixing sequences.

Original: The ﬁrst stability analysis of learning with mixing sequences was given in the paper (Mohri & Rostamizadeh, 2010).
Revised: The first stability analysis of learning with mixing sequences was given in the paper (Mohri & Rostamizadeh, 2010).

Original: The stability approach was also used to study online learning with dependent data (Agarwal & Duchi, 2013) and learning with graph-dependent data (Zhang et al., 2019).
Revised: The stability approach was also used to study online learning with dependent data (Agarwal & Duchi, 2013) and learning with graph-dependent data (Zhang et al., 2019).

Revised: SGD with Markov sampling has also been recently studied (Sun et al., 2018; Wang et al., 2022).

Original:  C ONCENTRATION I NEQUALITIES FOR ϕ - MIXING S EQUENCES
Revised:  C ONCENTRATION I NEQUALITIES FOR φ - MIXING S EQUENCES

Original: We consider learning problems with a sequence of dependent observations.
Revised: We consider learning problems with a sequence of dependent observations.

Original: We assume the dependency between two observations decays with their gap.
Revised: We assume the dependency between two observations decays with their gap.

Original: There are several concepts to quantify the dependency relationship within a stationary sequence such as β -mixing, ϕ -mixing and ψ -mixing (Mohri & Rostamizadeh, 2010; Yu, 1994).
Revised: There are several concepts to quantify the dependency relationship within a stationary sequence such as β -mixing, φ -mixing and ψ -mixing (Mohri & Rostamizadeh, 2010; Yu, 1994).

Original: We focus on the ϕ -mixing and ψ -mixing sequences in this paper.
Revised: We focus on the φ -mixing and ψ -mixing sequences in this paper.

Original: Let Z = { Z t } ∞ t =
Revised: Let Z = { Z t } ∞ t =

Original: −∞ be a stationary sequence of random variables.
Revised: −∞ be a stationary sequence of random variables.

Original: For any i, j ∈ N , let σ ji denote the σ -algebra generated by the random variables
Revised: For any i, j ∈ N , let σ ji denote the σ -algebra generated by the random variables

Original: Z k , i ≤ k ≤ j .
Revised: Z k , i ≤ k ≤ j .

Original: Deﬁnition 1 ( ϕ -Mixing Sequence) .
Revised: Definition 1 ( φ -Mixing Sequence) .

Original: For any k ∈ N , the ϕ -mixing coefﬁcient of Z is deﬁned as
Revised: For any k ∈ N , the φ -mixing coefficient of Z is defined as

Original: [Equation]
Revised: [Equation]

Original: Z is said to be ϕ -mixing if ϕ ( k ) → 0 as k → ∞ .
Revised: Z is said to be φ -mixing if φ ( k ) → 0 as k → ∞ .

Original: It is said to be algebraically ϕ
Revised: It is said to be algebraically φ -mixing

Original: -mixing (with degree r > 0 ) if there exists a real number ϕ 0 > 0 such that ϕ ( k ) ≤ ϕ 0 /k r for all k , exponentially mixing (with degree r ) if there exist real numbers ϕ 0 , ϕ 1 such that ϕ ( k ) ≤ ϕ 0 exp( − ϕ 1 k r ) for all k .
Revised: with degree r > 0 ) if there exists a real number φ 0 > 0 such that φ ( k ) ≤ φ 0 /k r for all k , exponentially mixing (with degree r ) if there exist real numbers φ 0 , φ 1 such that φ ( k ) ≤ φ 0 exp( − φ 1 k r ) for all k .

Original: Deﬁnition 2 ( ψ -Mixing Sequence (Bradley, 2007)) .
Revised: Definition 2 ( ψ -Mixing Sequence (Bradley, 2007)) .

Original: For any k ∈ N , the ψ -mixing coefﬁcient of the stochastic process Z is deﬁned as
Revised: For any k ∈ N , the ψ -mixing coefficient of the stochastic process Z is defined as

Original: [Equation]
Revised: [Equation]

Original: Z is said to be ψ -mixing if ψ ( k ) → 0 as k → ∞ .
Revised: Z is said to be ψ -mixing if ψ ( k ) → 0 as k → ∞ .

Original: It is said to be algebraically ψ -mixing
Revised: It is said to be algebraically ψ -mixing

Original: with degree r > 0 ) if there exists a real number ψ 0 > 0 such that ψ ( k ) ≤ ψ 0 /k r for all k , exponentially mixing (with degree r ) if there exist real numbers ψ 0 , ψ 1 such that ψ ( k ) ≤ ψ 0 exp( − ψ 1 k r ) for all k .
Revised: with degree r > 0 ) if there exists a real number ψ 0 > 0 such that ψ ( k ) ≤ ψ 0 /k r for all k , exponentially mixing (with degree r ) if there exist real numbers ψ 0 , ψ 1 such that ψ ( k ) ≤ ψ 0 exp( − ψ 1 k r ) for all k .

Original: We will use ψ -mixing to give a bound on the stability analysis version of ϕ (cid:48) -mixing in Lemma 3.
Revised: We will use ψ -mixing to give a bound on the stability analysis version of φ ′ -mixing in Lemma 3.

Original: Intuitively speaking, ϕ ( k ) and ψ ( k ) measure the dependency of an event on those happened k units of time ahead.
Revised: Intuitively speaking, φ ( k ) and ψ ( k ) measure the dependency of an event on those happened k units of time ahead.

Original: By the deﬁnition, we know that ψ -mixing is stronger than the ϕ -mixing.
Revised: By the definition, we know that ψ -mixing is stronger than the φ -mixing.

Original: Below we provide examples of ϕ -mixing and ψ -mixing sequences in Kesten & O’Brien (1976).
Revised: Below we provide examples of φ -mixing and ψ -mixing sequences in Kesten & O’Brien (1976).

Original: We ﬁrst consider random variables V n , U n and S n for n ∈ Z deﬁned on the probability space (Ω , F , P ) , which are independent of each other and have the following distributions: for all n ∈ Z ,
Revised: We first consider random variables V n , U n and S n for n ∈ Z defined on the probability space (Ω , F , P ) , which are independent of each other and have the following distributions: for all n ∈ Z ,

Original: P ( V n = i )
Revised: P ( V n = i )

Original: =
Revised: =

Original: β i , i = 0 , 1 and 0 < β 0 < β 0 + β 1 = 1 ; P ( U n = k ) =
Revised: β i , i = 0 , 1 and 0 < β 0 < β 0 + β 1 = 1 ; P ( U n = k ) =

Original: p k ≥ 0 , k = 0 , 1 , · · · and (cid:80) ∞
Revised: p k ≥ 0 , k = 0 , 1 , · · · and (cid:80) ∞

Original: k =0 p k = 1 ; P ( S n = j ) = γ j , j = 0 , 1 , 2 and 0 < γ 0 < γ 0 + γ 1 < γ 0 + γ 1 + γ 2 = 1 .
Revised: k =0 p k = 1 ; P ( S n = j ) = γ j , j = 0 , 1 , 2 and 0 < γ 0 < γ 0 + γ 1 < γ 0 + γ 1 + γ 2 = 1 .

Original: Example 1 (Example of ϕ -mixing sequence) .
Revised: Example 1 (Example of φ -mixing sequence) .

Original: Let { f k , k ≥ 1 } be a non-increasing sequence such that f 1 ≤ 1 , f k → 0 as k → ∞ and 2 log(1 − f k +1 ) ≥ log(1 − f k )+log(1 − f k +2 ) for { k : f k < 1 } .
Revised: Let { f k , k ≥ 1 } be a non-increasing sequence such that f 1 ≤ 1 , f k → 0 as k → ∞ and 2 log(1 − f k +1 ) ≥ log(1 − f k )+log(1 − f k +2 ) for { k : f k < 1 } .

Original: For any 0 < (cid:15) < 12 we deﬁne β 0{+
+} = (cid:15) , β 1 = 1 − (cid:15) .
Revised: For any 0 < ϵ < 12 we define β {+
+} = ϵ , β 1 = 1 − ϵ .

Original: Deﬁne { p n } by
Revised: Define { p n } by

Original: [Equation]
Revised: [Equation]

Original: iff
Revised: if

Original: [Equation]
Revised: [Equation]

Original: iff
Revised: if

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Note that p k =
Revised: Note that p k =

Original: (1 − f k +1 )(1 − f k +2 ) − 1 − (1 − f k )(1 − f k +1 )
Revised: (1 − f k +1 )(1 − f k +2 ) − 1 − (1 − f k )(1 − f k +1 )

Original: − 1 ≥ 0 for { k : f k < 1 } since 2 log(1 − f k +1 ) ≥ log(1 − f k )
Revised: − 1 ≥ 0 for { k : f k < 1 } since 2 log(1 − f k +1 ) ≥ log(1 − f k )

Original: + log(1 − f k +2 ) .
Revised: + log(1 − f k +2 ) .

Original: Let X n = U n + 12 V n + 14 W n where W n = V n − U n .
Revised: Let X n = U n + 12 V n + 14 W n where W n = V n − U n .

Original: Then { X n , n ∈ Z } is a ϕ -mixing sequence with (1 − (cid:15) )
Revised: Then { X n , n ∈ Z } is a φ -mixing sequence with (1 − ϵ )

Original: f k ≤ ϕ ( k ) ≤ f k .
Revised: f k ≤ φ ( k ) ≤ f k .

Original: Next we provide an example of ψ -mixing sequence.
Revised: Next we provide an example of ψ -mixing sequence.

Original: Example 2 (Example of ψ -mixing sequence) .
Revised: Example 2 (Example of ψ -mixing sequence) .

Original: Let { g k , k ≥ 1 } be a sequence such that g 1 − g 2 = 1 , g k
Revised: Let { g k , k ≥ 1 } be a sequence such that g 1 − g 2 = 1 , g k

Original: → 0 as k → ∞ and 2 g k +1 ≤ g k + g k +2 , k
Revised: → 0 as k → ∞ and 2 g k +1 ≤ g k + g k +2 , k

Original: = 1 , 2 , · · · .
Revised: = 1 , 2 , · · · .

Original: For any (cid:15) ∈ (0 , 1) , Let γ2 = (cid:15) , γ 
 
Revised: For any ϵ ∈ (0 , 1) , Let γ 
 =

Original: {+ϵ , γ 
+} = γ 1 = 1 2 (1 − (cid:15) ) , β 0 = β 1 = 1 2 .
Revised: {+ϵ , γ 
+} = γ 1 = 12 (1 − ϵ ) , β 0 = β 1 = 12 .

Original: Deﬁne { p n } by
Revised: Define { p n } by

Original: [Equation]
Revised: [Equation]

Original: Note that p k ≥ 0 for k = 0 , 1 , · · · .
Revised: Note that p k ≥ 0 for k = 0 , 1 , · · · .

Original: Let U n , V n and S n be as before.
Revised: Let U n , V n and S n be as before.

Original: Let Z n = S n I [ S n =0 or 1] +
Revised: Let Z n = S n I [ S n =0 or 1] +

Original: V n − U n I [ S n =2] , where I is the indicator function.
Revised: V n − U n I [ S n =2] , where I is the indicator function.

Original: Finally we deﬁne X n = V n + 2 Z n .
Revised: Finally we define X n = V n + 2 Z n .

Original: Then { X n , n ∈
Revised: Then { X n , n ∈

Original: Z } is a ψ -mixing sequence with (cid:15) (1 + (cid:15) ) − 1 g k ≤ ψ ( k ) ≤ exp (cid:2) (cid:15) (1 − (cid:15) )
Revised: Z } is a ψ -mixing sequence with ϵ (1 + ϵ ) − 1 g k ≤ ψ ( k ) ≤ exp (cid:2) ϵ (1 − ϵ )

Original: − 1 g k (cid:3) − 1 .
Revised: − 1 g k (cid:3) − 1 .

Original: To develop error bounds for learning with ϕ -mixing sequences, we ﬁrst develop concentration inequalities for ϕ -mixing sequences.
Revised: To develop error bounds for learning with φ -mixing sequences, we first develop concentration inequalities for φ -mixing sequences.

Original: In the following theorem to be proved in Section A, we derive a tail bound for the summation of dependent random variables in terms of the tail behavior of each individual random variable.
Revised: In the following theorem to be proved in Section A, we derive a tail bound for the summation of dependent random variables in terms of the tail behavior of each individual random variable.

Original: Let ∆ n = 1 + 2 (cid:80) nk =1 ϕ ( k ) .
Revised: Let ∆ n = 1 + 2 (cid:80) nk =1 φ ( k ) .

Original: The L p -norm of a real-valued random variable Z is denoted by (cid:107) Z (cid:107) p :
Revised: The L p -norm of a real-valued random variable Z is denoted by ∥ Z ∥ p :=

Original: = (cid:0) E [ | Z | p ] (cid:1) 1 p , p ≥ 1 .
Revised: (cid:0) E [ | Z | p ] (cid:1) 1 p , p ≥ 1 .

Original: Theorem 1.
Revised: Theorem 1.

Original: Let X 1 , . . .
Revised: Let X 1 , . . .

Original: , X n be a ﬁnite contiguous subsequence from a ϕ -mixing sequence.
Revised: , X n be a finite contiguous subsequence from a φ -mixing sequence.

Original: Let Z i be a function of X i with E [ Z i ] = 

Revised: Let Z i be a function of X i with E [ Z i ] =

Original: and Pr {| Z i | > ˜ (cid:15) } ≤ 2 exp( − ˜ (cid:15) 2 /b ) .
Revised: and Pr {| Z i | > ˜ ϵ } ≤ 2 exp( − ˜ ϵ 2 /b ) .

Original: Then for any p ≥ 1 we have
Revised: Then for any p ≥ 1 we have

Original: [Equation]
Revised: [Equation]

Original: Remark 1.
Revised: Remark 1.

Original: Theorem 1 is an extension of Marcinkiewicz-Zygmund inequality for independent random variables to ϕ -mixing sequences.
Revised: Theorem 1 is an extension of Marcinkiewicz-Zygmund inequality for independent random variables to φ -mixing sequences.

Original: Indeed, if Z i are i.i.d.
Revised: Indeed, if Z i are i.i.d.

Original: , it was shown (cid:13)(cid:13) (cid:80) ni
Revised: , it was shown (cid:13)(cid:13) (cid:80) ni

Original: =1 Z i (cid:13)(cid:13) p
Revised: =1 Z i (cid:13)(cid:13) p

Original: =
Revised: =

Original: O ( p √ nb ) (Ren & Liang, 2001).
Revised: O ( p √ nb ) (Ren & Liang, 2001).

Original: We show how the mixing behavior would affect the concentration by including ∆ n in our bound.
Revised: We show how the mixing behavior would affect the concentration by including ∆ n in our bound.

Original: In particular, if Z i are independent then ∆ n = 1 and in this case, our result matches the Marcinkiewicz-Zygmund inequality up to a logarithmic factor.
Revised: In particular, if Z i are independent then ∆ n = 1 and in this case, our result matches the Marcinkiewicz-Zygmund inequality up to a logarithmic factor.

Original: Note ∆ n = O (1) for algebraical ϕ -mixing with r > 1 and exponential ϕ -mixing sequences (Mohri & Rostamizadeh, 2010).
Revised: Note ∆ n = O (1) for algebraical φ -mixing with r > 1 and exponential φ -mixing sequences (Mohri & Rostamizadeh, 2010).

Original: Under the assumption (cid:80) ∞ k
Revised: Under the assumption (cid:80) ∞ k

Original: =1 ϕ 12 ( k ) < ∞ , it was shown (cid:13)(cid:13)(cid:13) (cid:80) ni
Revised: =1 φ 12 ( k ) < ∞ , it was shown (cid:13)(cid:13)(cid:13) (cid:80) ni

Original: =

Revised: =

Original: Z i (cid:13)(cid:13)(cid:13) p ≤ C p (cid:16)(cid:16) (cid:80) ni=1 (cid:107)
Revised: Z i (cid:13)(cid:13)(cid:13) p ≤ C p (cid:16)(cid:16) (cid:80) ni

Revised: =1 ∥

Original: Z i (cid:107) pp (cid:17) 1 p + (cid:16) (cid:80) ni
Revised: Z i ∥ pp (cid:17) 1 p + (cid:16) (cid:80) ni

Original: =1 (cid:107){+=
 ∥+}
Revised: {+=
 ∥+}

Original: Z i (cid:107) 22 (cid:17) 12 (cid:17) (Xuejun et al., 2010).
Revised: Z i ∥ 22 (cid:17) 12 (cid:17) (Xuejun et al., 2010).

Original: This bound requires an assumption involving (cid:80) ∞ k
Revised: This bound requires an assumption involving (cid:80) ∞ k

Original: =1 ϕ 12 ( k ) , which is larger than (cid:80) nk =1 ϕ ( k ) in ∆ n since ϕ 12 ( k ) ≥ ϕ ( k ) .
Revised: =1 φ 12 ( k ) , which is larger than (cid:80) nk =1 φ ( k ) in ∆ n since φ 1 2 ( k ) ≥ φ ( k ) .

Original: For example, if ϕ ( k ) =
Revised: For example, if φ ( k ) =

Original: O ( k − 1 ) then (cid:80) nk=1 ϕ
Revised: O ( k − 1 ) then (cid:80) nk

Original:  ( k ) =
Revised: =1 φ ( k ) =

Original: O (log n ) while (cid:80) nk
Revised: O (log n ) while (cid:80) nk

Original: =1 ϕ 12 ( k ) =
Revised: =1 φ 12 ( k ) =

Original: O ( √ n ) .
Revised: O ( √ n ) .

Original: Moreover, the bound in Xuejun et al.
Revised: Moreover, the bound in Xuejun et al.

Original: (2010) involves C p which is not explicitly stated.
Revised: (2010) involves C p which is not explicitly stated.

Original: As a comparison, our bound involves all explicit constants.
Revised: As a comparison, our bound involves all explicit constants.

Original: Remark 2.
Revised: Remark 2.

Original: Our basic idea to prove Theorem 1 is to apply a McDiarmid inequality (Lemma A.1) to a Lipschitz function deﬁned on a ϕ -mixing sequence.
Revised: Our basic idea to prove Theorem 1 is to apply a McDiarmid inequality (Lemma A.1) to a Lipschitz function defined on a φ -mixing sequence.

Original: If we deﬁne Φ (cid:48) ( X 1 , . . .
Revised: If we define Φ ′ ( X 1 , . . .

Original: , X n )
Revised: , X n )

Original: = (cid:80) ni
Revised: = (cid:80) ni

Original: =1 Z i , one cannot guarantee the Lipschitz continuity of Φ due to the unboundedness of Z i .
Revised: =1 Z i , one cannot guarantee the Lipschitz continuity of Φ due to the unboundedness of Z i .

Original: Our novelty is to deﬁne (cid:101)
Revised: Our novelty is to define (cid:101)

Original: Z i
Revised: Z i

Original: =
Revised: =

Original: Z i I
Revised: Z i I

Original: | Z i |≤ (cid:15) where I [ · ] is an indicator function and (cid:15) =
Revised: | Z i |≤ ϵ where I [ · ] is an indicator function and ϵ =

Original: O ( (cid:112) b log(1 /δ )) .
Revised: O ( (cid:112) b log(1 /δ )) .

Original: The boundedness of (cid:101)
Revised: The boundedness of (cid:101)

Original: Z i implies the (2 (cid:15) ) -Lipschitz
Revised: Z i implies the (2 ϵ ) -Lipschitz

Original: continuity of Φ and therefore we can apply the McDiarmid inequality to study its decay rate of Φ .
Revised: continuity of Φ and therefore we can apply the McDiarmid inequality to study its decay rate of Φ .

Original: Furthermore, the assumption Pr {| Z i | > ˜ (cid:15) } ≤ 2 exp( − ˜ (cid:15) 2 /b ) shows that Φ and Φ (cid:48) are equal with a high probability.
Revised: Furthermore, the assumption Pr {| Z i | > ˜ ϵ } ≤ 2 exp( − ˜ ϵ 2 /b ) shows that Φ and Φ ′ are equal with a high probability.

Original: We then combine these two observations together to derive a high-probability bound for Φ (cid:48) , which further leads to a bound on the L p -norm of Φ (cid:48) by the equivalence between high-probability bound and the L p -norm bound.
Revised: We then combine these two observations together to derive a high-probability bound for Φ ′ , which further leads to a bound on the L p -norm of Φ ′ by the equivalence between high-probability bound and the L p -norm bound.

Original: Based on Theorem 1, we develop a moment bound for Lipschitz functions (w.r.t.
Revised: Based on Theorem 1, we develop a moment bound for Lipschitz functions (w.r.t.

Original: the Hamming distance) deﬁned on mixing sequences.
Revised: the Hamming distance) defined on mixing sequences.

Original: The following theorem is an extension of a result in the i.i.d.
Revised: The following theorem is an extension of a result in the i.i.d.

Original: case (Bousquet et al., 2020) to the case with mixing sequences.
Revised: case (Bousquet et al., 2020) to the case with mixing sequences.

Original: This result plays a major role in developing our generalization bounds for learning with mixing sequences.
Revised: This result plays a major role in developing our generalization bounds for learning with mixing sequences.

Original: The ﬁrst assumption is a conditional boundedness assumption which is standard for concentration inequalities.
Revised: The first assumption is a conditional boundedness assumption which is standard for concentration inequalities.

Original: The second assumption implies that the g i is of mean zero conditioned on any ﬁxed Z [ n ] \ [ i ] , which is stronger than E [ g i ] = 0 since the conditional expectation holds for any ﬁxed Z [ n ] \ [ i ] .
Revised: The second assumption implies that the g i is of mean zero conditioned on any fixed Z [ n ] \ [ i ] , which is stronger than E [ g i ] = 0 since the conditional expectation holds for any fixed Z [ n ] \ [ i ] .

Original: The last assumption implies that g i is insensitive to the change of any single example, which implies that g i is concentrated around its expectation by McDiarmid’s inequality.
Revised: The last assumption implies that g i is insensitive to the change of any single example, which implies that g i is concentrated around its expectation by McDiarmid’s inequality.

Original: Our proof follows from the framework in Bousquet et al.
Revised: Our proof follows from the framework in Bousquet et al.

Original: (2020), which is given in Section B.
Revised: (2020), which is given in Section B.

Original: Theorem 2 (Concentration Inequality for ϕ -Mixing Sequence) .
Revised: Theorem 2 (Concentration Inequality for φ -Mixing Sequence) .

Original: Let Z 1 , . . .
Revised: Let Z 1 , . . .

Original: , Z n be a ﬁnite contiguous subsequence from a ϕ -mixing sequence.
Revised: , Z n be a finite contiguous subsequence from a φ -mixing sequence.

Original: Denote Z = { Z 1 , . . .
Revised: Denote Z = { Z 1 , . . .

Original: , Z n } .
Revised: , Z n } .

Original: Let g 1 , . .
Revised: Let g 1 , . .

Original: . , g n be some functions g
Revised: . , g n be some functions g

Original: i : Z n (cid:55)→ R such that the following holds for any i ∈[ n ]
Revised: i : Z n (cid:55)→ R such that the following holds for any i ∈

Original:  • (cid:12)(cid:12) E Z [ n ] \ 
Revised: [ n ] • (cid:12)(cid:12) E Z [ n ] \ [ i ] [ g i ( Z )

Original: [ i ] [ g i ( Z ) | Z i ] (cid:12)(cid:12) ≤ M almost surely, • E Z i [ g i ( Z ) | Z [ n ] \ [ i ] ] = 

Revised: | Z i ] (cid:12)(cid:12) ≤ M almost surely, • E Z i [ g i ( Z ) | Z [ n ] \ [ i ] ] =

Original: a.s., •
Revised: a.s., •

Original: g
Revised: g

Original: i is β -Lipschitz w.r.t.
Revised: i is β -Lipschitz w.r.t.

Original: the Hamming distance.
Revised: the Hamming distance.

Original: Then for any p ≥ 1 we have ( k = (cid:100) log 2 n (cid:101) )
Revised: Then for any p ≥ 1 we have ( k = ⌈ log 2 n ⌉ )

Original: (cid:13)(cid:13)(cid:13) n (cid:88)i =
Revised: [Equation] n (cid:88)

Original: g i (cid:13)(cid:13)(cid:13) p ≤ 3 M ∆

Original: [Equation]
Revised: [Equation]

Original: Remark 3.
Revised: Remark 3.

Original: If the sequence is i.i.d., the following bound was developed (Bousquet et al., 2020)
Revised: If the sequence is i.i.d., the following bound was developed (Bousquet et al., 2020)

Original: [Equation]
Revised: [Equation]

Original: Our bound recovers the existing result in the i.i.d. case.
Revised: Our bound recovers the existing result in the i.i.d. case.

Original: Note ∆ n = 1 for any n and then Theoremimplies (cid:13)(cid:13)(cid:13) (cid:80) ni
Revised: Note ∆ n = 1 for any n and then Theoremimplies (cid:13)(cid:13)(cid:13) (cid:80) ni

Original: =

Revised: =

Original: g i (cid:13)(cid:13)(cid:13) p ≤ 3 M √ 2 pn
Revised: g i (cid:13)(cid:13)(cid:13) p ≤ 3 M √ 2 pn

Original: + npβ (5+log 2 (cid:100) n (cid:101) ) 2 , which matches Eq.
Revised: + npβ (5+log 2 ⌈ n ⌉ ) 2 , which matches Eq.

Original: (3.1) up to a logarithmic factor.
Revised: (3.1) up to a logarithmic factor.

Original: This is the ﬁrst extension of the result in Bousquet et al.
Revised: This is the first extension of the result in Bousquet et al.

Original: (2020) to a ϕ -mixing setting.
Revised: to a φ -mixing setting.

Original: We follow the framework in Bousquet et al.
Revised: We follow the framework in Bousquet et al.

Original: (2020) to prove Theorem 2.
Revised: (2020) to prove Theorem 2.

Original: The difference is to replace the
Revised: The difference is to replace the

Original: Marcinkiewicz-Zygmund inequality for i.i.d.
Revised: Marcinkiewicz-Zygmund inequality for i.i.d.

Original: random variables by Theorem 1 for ϕ -mixing random variables.
Revised: random variables by Theorem 1 for φ -mixing random variables.

Original: The basic idea is to use the representation (cid:80) ni
Revised: The basic idea is to use the representation (cid:80) ni

Original: =

Revised: =

Original: g i =
Revised: g i =

Original: (cid:80) ni
Revised: (cid:80) ni

Original: =1 E [ g i | Z i ] + (cid:80)
Revised: =1 E [ g i | Z i ] + (cid:80)

Original: k − 1 l =0 (cid:80) ni
Revised: k − 1 l =0 (cid:80) ni

Original: =1 (cid:0) g li − g l +1 i (cid:1) , where g li is the expectation of g i conditioned on some random variables and k is an integer depending on n .
Revised: =1 (cid:0) g li − g l +1 i (cid:1) , where g li is the expectation of g i conditioned on some random variables and k is an integer depending on n .

Original: We then use a McDiarmid inequality and the conditional boundedness of E [ g i |
Revised: We then use a McDiarmid inequality and the conditional boundedness of E [ g i |

Original: Z i ] to control (cid:80)
Revised: Z i ] to control (cid:80)

Original: ni =1 E [ g i |
Revised: ni =1 E [ g i |

Original: Z i ] , and use Theorem 1 to control (cid:80) k − 1 l =0 (cid:80) ni=1 (cid:0) [Equation] li
Revised: Z i ] , and use Theorem 1 to control (cid:80) k − 1 l =0 (cid:80) ni

Original: [Equation]

Original: g

Original: [Equation] + [Equation] (cid:1) .
Revised: =1 (cid:0) g li − g l +1 i (cid:1) .

Original:  S TABILITY AND G ENERALIZATION
Revised:  S TABILITY AND G ENERALIZATION

Original: Let Z = X
Revised: Let Z = X

Original: × Y be a sample space, where X ⊆ R d is an input space and Y is an output space.
Revised: × Y be a sample space, where X ⊆ R d is an input space and Y is an output space.

Original: We consider supervised learning problems where
Revised: We consider supervised learning problems where

Original: S =
Revised: S =

Original: z 1 , . .
Revised: z 1 , . .

Original: z n } = (cid:8) ( x 1 , y 1 ) , . . .
Revised: z n } = (cid:8) ( x 1 , y 1 ) , . . .

Original: , ( x n , y n ) (cid:9) is a contiguous subsequence from a ψ -mixing sequence.
Revised: , ( x n , y n ) (cid:9) is a contiguous subsequence from a ψ -mixing sequence.

Original: Based on S , we wish to ﬁnd a model h : X (cid:55)→ Y .
Revised: Based on S , we wish to find a model h : X (cid:55)→ Y .

Original: We consider parametric models where the model is determined by a parameter w in a parameter space W .
Revised: We consider parametric models where the model is determined by a parameter w in a parameter space W .

Original: The performance of a model w on a single example z can be measured by a loss function f ( w ; z ) .
Revised: The performance of a model w on a single example z can be measured by a loss function f ( w ; z ) .

Original: The empirical risk and population risk are then deﬁned by
Revised: The empirical risk and population risk are then defined by

Original: [Equation]
Revised: [Equation]

Revised: f

Revised: [Equation]

Revised: z

Revised: [Equation]

Revised: and F ( w

Revised: [Equation]

Original:  which measure the behavior of w on training examples and test examples, respectively.
Revised: = E z [ f ( w ; z )] , (4.1) which measure the behavior of w on training examples and test examples, respectively.

Original: Here the test point z is assumed to be dependent on S (i.e., z is assumed to follow immediately after the sample
Revised: Here the test point z is assumed to be dependent on S (i.e., z is assumed to follow immediately after the sample

Original: S ), which is the most realistic setting considered in Mohri & Rostamizadeh (2010).
Revised: S ), which is the most realistic setting considered in Mohri & Rostamizadeh (2010).

Original: We refer to the discrepancy between training and testing as the generalization gap F ( w ) − F S ( w ) .
Revised: We refer to the discrepancy between training and testing as the generalization gap F ( w ) − F S ( w ) .

Original: In machine learning, we often apply an algorithm to get a model with a small training error.
Revised: In machine learning, we often apply an algorithm to get a model with a small training error.

Original: Meanwhile, we wish the output model also admits a small generalization gap to enjoy good generalization to test data.
Revised: Meanwhile, we wish the output model also admits a small generalization gap to enjoy good generalization to test data.

Original: In this paper, we are interested in developing generalization error bounds that decay to zero as the sample size goes to inﬁnity.
Revised: In this paper, we are interested in developing generalization error bounds that decay to zero as the sample size goes to infinity.

Original: Our basic tool is the algorithmic stability, which measures the sensitivity of the output up to the perturbation of a single example.
Revised: Our basic tool is the algorithmic stability, which measures the sensitivity of the output up to the perturbation of a single example.

Original: Various concepts of stability have been introduced in the literature.
Revised: Various concepts of stability have been introduced in the literature.

Original: In this paper, we focus on the uniform stability which is arguably the most widely used algorithmic stability.
Revised: In this paper, we focus on the uniform stability which is arguably the most widely used algorithmic stability.

Original: We use w S to mean the output model if we apply an algorithm A to the dataset S .
Revised: We use w S to mean the output model if we apply an algorithm A to the dataset S .

Original: Note we omit the dependency of the notation on A , which should be clear from the context.
Revised: Note we omit the dependency of the notation on A , which should be clear from the context.

Original: We say two sets are neighboring datasets if they differ by one example.
Revised: We say two sets are neighboring datasets if they differ by one example.

Original: Deﬁnition 3 (Uniform Stability (Bousquet & Elisseeff, 2002)) .
Revised: Definition 3 (Uniform Stability (Bousquet & Elisseeff, 2002)) .

Original: A randomized algorithm A is (cid:15) - uniformly stable if for all neighboring datasets S, S(cid:48) ∈
Revised: A randomized algorithm A is ϵuniformly stable if for all neighboring datasets S, S

Original:  Z n we have sup z (cid:2) f ( w S ; z ) − f ( w S (cid:48) ; z ) (cid:3) ≤ (cid:15).
Revised: ′ ∈ Z n we have sup z (cid:2) f ( w S ; z ) − f ( w S ′ ; z ) (cid:3) ≤ ϵ.

Original: Our stability analysis requires a different mixing coefﬁcient deﬁned as follows
Revised: Our stability analysis requires a different mixing coefficient defined as follows

Original: [Equation]
Revised: [Equation]

Original: It is clear that ϕ (cid:48) ( k ) ≥ ϕ ( k ) .
Revised: It is clear that φ ′ ( k ) ≥ φ ( k ) .

Original: The following lemma controls ϕ (cid:48) ( k ) in terms of the ψ -mixing coefﬁcients.
Revised: The following lemma controls φ ′ ( k ) in terms of the ψ -mixing coefficients.

Original: According to the following lemma, one can show that if ψ ( k ) =
Revised: According to the following lemma, one can show that if ψ ( k ) =

Original: O ( k − r ) then ϕ (cid:48) ( k ) =
Revised: O ( k − r ) then φ ′ ( k ) =

Original: O ( k − r ) .
Revised: O ( k − r ) .

Original: If ψ ( k ) =
Revised: If ψ ( k ) =

Original: O (exp( − ψ 1 k r ))then ϕ (cid:48)
Revised: O (exp( − ψ 1 k r ))

Original:  ( k ) =
Revised: then φ ′ ( k ) =

Original: O (exp( − ψ 1 k r )) .
Revised: O (exp( − ψ 1 k r )) .

Original: Lemma 3.
Revised: Lemma 3.

Original: Let Z = { Z t } ∞ t =
Revised: Let Z = { Z t } ∞ t =

Original: −∞ be drawn from a ψ -mixing distribution and assume ψ ( k ) < 1 .
Revised: −∞ be drawn from a ψ -mixing distribution and assume ψ ( k ) < 1 .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: To apply Theorem 2 to learning with mixing sequences, we need to introduce a sequence of functions
Revised: To apply Theorem 2 to learning with mixing sequences, we need to introduce a sequence of functions

Original: g i satisfying the conditions in Theorem 2 and relate them to the generalization gap.
Revised: g i satisfying the conditions in Theorem 2 and relate them to the generalization gap.

Original: Let z (cid:48) i (resp.
Revised: Let z ′ i (resp.

Original: z (cid:48)(cid:48) i ) be drawn from the same distribution of z i , i.e., the conditional distribution of z (cid:48)
Revised: z ′′ i ) be drawn from the same distribution of z i , i.e., the conditional distribution of z ′

Original: i (resp.
Revised: i (resp.

Original: z (cid:48)(cid:48) i ) given z 1 , . ..
Revised: z ′′ i ) given z 1 , . .

Original: z i − 1 , z i +1 , . . .
Revised: z i − 1 , z i +1 , . . .

Original: , z n is the same as that of z
Revised: , z n is the same as that of z

Original: i given z 1 , . . .
Revised: i given z 1 , . . .

Original: z i − 1 , z i +1 , . . .
Revised: z i − 1 , z i +1 , . . .

Original: , z n .
Revised: , z n .

Original: Let
Revised: Let

Original: S i,b = { z 1 , . . .
Revised: S i,b = { z 1 , . . .

Original: z i − b − 1 , z i , z
Revised: z i − b − 1 , z i , z

Original: i + b +1 , . . .
Revised: i + b +1 , . . .

Original: , z n − b } , i.e., we remove 2 b points around z i .
Revised: , z n − b } , i.e., we remove 2 b points around z i .

Original: For any i ∈ [ n ] , let S ii,b
Revised: For any i ∈ [ n ] , let S ii,b

Original: = { z 1 , . . .
Revised: = { z 1 , . . .

Original: z i − b − 1 , z (cid:48)
Revised: z i − b − 1 , z ′

Original: i , z i + b +1 , . . .
Revised: i , z i + b +1 , . . .

Original: , z n − b } .
Revised: , z n − b } .

Original: We then deﬁne the following random variables
Revised: We then define the following random variables

Original: [Equation]
Revised: [Equation]

Original: The following lemma to be proved in Section C gives generalization bounds in terms of stability and (cid:80)ni
Revised: The following lemma to be proved in Section C gives generalization bounds in terms of stability and (cid:80)

Original: [-=
-]
Revised: [-=
-][Equation]

Original:  g i .
Revised: 1 g i .

Original: We will use ϕ (cid:48) ( b ) in Theorem 5 and all corollaries in Section 5.
Revised: We will use φ ′ ( b ) in Theorem 5 and all corollaries in Section 5.

Original: The underlying reason is that we need to remove 2 b points around z
Revised: The underlying reason is that we need to remove 2 b points around z

Original: i to get S i,b for the application of Theorem 2.
Revised: i to get S i,b for the application of Theorem 2.

Original: An upper bound | F ( w S ) − E z (cid:48)(cid:48) i [ f ( w S i,b ; z (cid:48)(cid:48) i )] | requires to use ϕ (cid:48) ( b ) .
Revised: An upper bound | F ( w S ) − E z ′′ i [ f ( w S i,b ; z ′′ i )] | requires to use φ ′ ( b ) .

Original: Lemma 4.
Revised: Lemma 4.

Original: Let S be drawn from a ψ -mixing distribution.
Revised: Let S be drawn from a ψ -mixing distribution.

Original: Let b ∈ { 0 , . .
Revised: Let b ∈ { 0 , . .

Original: . , n } denote the number of last points removed in S , i.e., S b = { z 1 , . .
Revised: . , n } denote the number of last points removed in S , i.e., S b = { z 1 , . .

Original: , z n − b } .
Revised: , z n − b } .

Original: Let w S denote the hypothesis trained on S .
Revised: Let w S denote the hypothesis trained on S .

Original: If the algorithm A is β -uniformly stable and the loss function is bounded by M > 0 , then the following inequality holds with g i deﬁned in Eq.
Revised: If the algorithm A is β -uniformly stable and the loss function is bounded by M > 0 , then the following inequality holds with g i defined in Eq.

Original: [Equation]
Revised: [Equation]

Original: We can apply Theorem 2 to control the term (cid:80) ni
Revised: We can apply Theorem 2 to control the term (cid:80) ni

Original: =

Revised: =

Original: g i and derive the following generalization bounds in terms of mixing coefﬁcients.
Revised: g i and derive the following generalization bounds in terms of mixing coefficients.

Original: Theorem 5 (General Mixing Stability Bound) .
Revised: Theorem 5 (General Mixing Stability Bound) .

Original: Let w S denote the hypothesis returned by a βuniformly stable algorithm trained on a sample S drawn from a ψ -mixing stationary distribution.
Revised: Let w S denote the hypothesis returned by a βuniformly stable algorithm trained on a sample S drawn from a ψ -mixing stationary distribution.

Original: Let
Revised: Let

Original: M denote the uniform bound of the loss function.
Revised: M denote the uniform bound of the loss function.

Original: Then for any b ∈ { 0 , . . . , n } and any δ ∈ (0 , 1) , the following inequality holds with probability at least 1 − δ ( k = (cid:100) log 2 n (cid:101) )
Revised: Then for any b ∈ { 0 , . . . , n } and any δ ∈ (0 , 1) , the following inequality holds with probability at least 1 − δ ( k = ⌈ log 2 n ⌉ )

Original: [Equation]
Revised: [Equation]

Original: F
Revised: F

Original: [Equation]
Revised: [Equation]

Original: w S)
Revised: S

Original: − F S ( w S ) (cid:12)(cid:12) ≤2(3 b + 1) β
Revised: [Equation] F S [Equation] (cid:12)(cid:12) ≤

Original: + M ( ϕ ( b ) + ϕ (cid:48) ( b)) +3 eM ∆ n (cid:114)
Revised: [Equation] b +

Original: [Equation]
Revised: [Equation]

Revised: β

Original: Remark 4.
Revised: [Equation] M ( φ [Equation] + φ

Revised: [Equation]

Revised: Remark 4.

Original: For ϕ -mixing sequences, the following high-probability bounds were developed (Mohri & Rostamizadeh, 2010)
Revised: For φ -mixing sequences, the following high-probability bounds were developed (Mohri & Rostamizadeh, 2010)

Original: [Equation]
Revised: [Equation]

Original: As a comparison, our generalization bound in Theorem 5 becomes
Revised: As a comparison, our generalization bound in Theorem 5 becomes

Original: [Equation]
Revised: [Equation]

Original: Eq.
Revised: Eq.

Original: (4.5) improves Eq.
Revised: (4.5) improves Eq.

Original: (4.4) as follows: (1) we replace √ nϕ ( b ) with ϕ (cid:48) ( b ) ; (2) we replace √ nbβ ∆ n with β ( b + ∆ 2 n log 2 n ) .
Revised: (4.4) as follows: (1) we replace √ nφ ( b ) with φ ′ ( b ) ; (2) we replace √ nbβ ∆ n with β ( b + ∆ 2 n log 2 n ) .

Original: The above two terms save a factor of √ n .
Revised: The above two terms save a factor of √ n .

Original: Meanwhile it should be also mentioned that our bounds involve ϕ (cid:48) , while the bounds in Mohri & Rostamizadeh (2010) involve ϕ .
Revised: Meanwhile it should be also mentioned that our bounds involve φ ′ , while the bounds in Mohri & Rostamizadeh (2010) involve φ .

Original: We now compare these two bounds under assumptions of ϕ (cid:48) in three cases (results in Mohri &
Revised: We now compare these two bounds under assumptions of φ ′ in three cases (results in Mohri &

Original: Rostamizadeh (2010) hold for ϕ -mixing coefﬁcients and also hold for ϕ (cid:48) -mixing coefﬁcients).
Revised: Rostamizadeh (2010) hold for φ -mixing coefficients and also hold for φ ′ -mixing coefficients).

Original: In the i.i.d. case, (4.5) becomes (cid:12)(cid:12) F ( w S )
Revised: In the i.i.d. case, (4.5) becomes (cid:12)(cid:12) F ( w S )

Original: − F S ( w S ) (cid:12)(cid:12) =
Revised: − F S ( w S ) (cid:12)(cid:12) =

Original: O (cid:0)(cid:112) n − 1 log(1 /δ )
Revised: O (cid:0)(cid:112) n − 1 log(1 /δ )

Original: + β log 2 n log(1 /δ )
Revised: + β log 2 n log(1 /δ )

Original: (cid:1) , which matches existing stability-based bounds (Bousquet et al., 2020)
Revised: (cid:1) , which matches existing stability-based bounds (Bousquet et al., 2020)

Original: up to a logarithmic factor.
Revised: up to a logarithmic factor.

Original: As a comparison, Eq. (4.4) becomes (cid:12)(cid:12) F ( w S ) − F S ( w S ) (cid:12)(cid:12)
Revised: As a comparison, Eq. (4.4) becomes (cid:12)(cid:12) F [Equation]

Original: =
Revised: w

Original: O (cid:0)(cid:112) log(1 /δ ) ( √ nβ + 1 / √ n ) (cid:1) .
Revised: [Equation]

Original: We now consider algebraically mixing sequences, i.e., ϕ (cid:48) ( k ) ≤ ϕ 0 k − r with r > 1 .
Revised: We now consider algebraically mixing sequences, i.e., φ ′ ( k ) ≤ φ 0 k − r with r > 1 .

Original: In this case, we know ∆ n = O (1) (Mohri & Rostamizadeh, 2010).
Revised: In this case, we know ∆ n = O (1) (Mohri & Rostamizadeh, 2010).

Original: If we choose b (cid:16) β − 1 r +1 in Eq.
Revised: If we choose b ≍ β − 1 r +1 in Eq.

Original: (4.5), we get βb (cid:16) b − r (cid:16) β rr +

Revised: (4.5), we get βb ≍ b − r ≍ β rr +

Original: and therefore (we denote B (cid:16) (cid:101) B if there are absolute constants c 1 and c 2 such that c 1 B ≤ (cid:101) B ≤ c 2 B .)
Revised: and therefore (we denote B ≍ (cid:101) B if there are absolute constants c 1 and c 2 such that c 1 B ≤ (cid:101) B ≤ c 2 B .)

Original: [Equation]
Revised: [Equation]

Original: As a comparison, the optimal choice b (cid:16) β − 1 r +1 in Eq.
Revised: As a comparison, the optimal choice b ≍ β − 1 r +1 in Eq.

Original: (4.4) implies
Revised: (4.4) implies

Original: [Equation]
Revised: [Equation]

Original: It is clear our bound replaces √ nβ rr +1 in Eq.
Revised: It is clear our bound replaces √ nβ rr +1 in Eq.

Original: (4.7) by β rr +1 + β log 2 n log(1 /δ ) , which is smaller.
Revised: (4.7) by β rr +1 + β log 2 n log(1 /δ ) , which is smaller.

Original: Finally, we consider exponential mixing sequences, i.e., ϕ (cid:48) ( k ) ≤ ϕ 0 exp ( − ϕ 1 k r ) , which imply ∆ n = O (1) .
Revised: Finally, we consider exponential mixing sequences, i.e., φ ′ ( k ) ≤ φ 0 exp ( − φ 1 k r ) , which imply ∆ n = O (1) .

Original: If we ﬁx b = (cid:100) log 1 r (1 /β ) (cid:101) , we know exp( − b r ) ≤ bβ =
Revised: If we fix b = ⌈ log 1 r (1 /β ) ⌉ , we know exp( − b r ) ≤ bβ =

Original: O ( β log 1 r (1 /β )) and (cid:12)(cid:12) F 
Revised: O ( β log 1 r (1 /β )) and (cid:12)(cid:12) F ( w S )

Original: [Equation]
Revised: − F S ( w S ) (cid:12)(cid:12) = O (cid:16)(cid:112) n − 1 log(1 /δ ) + β log 2 n log(1 /δ ) + β log 1 r (1 /β )

Original: [Equation]
Revised: (cid:17) .

Original: As a comparison, Eq. (4.4) implies (cid:12)(cid:12) F ( w S ) − F S ( w S ) (cid:12)(cid:12)
Revised: As a comparison, Eq. (4.4) implies (cid:12)(cid:12) F ( w S ) − F S ( w S ) (cid:12)(cid:12)

Original: =
Revised: =

Original: O (cid:0)(cid:112) log(1 /δ ) (cid:0) √ nβ log 1 r (1 /β )
Revised: O (cid:0)(cid:112) log(1 /δ ) (cid:0) √ nβ log 1 r (1 /β )

Original: + √ nβ + 1 / √ n (cid:1)(cid:1) .
Revised: + √ nβ + 1 / √ n (cid:1)(cid:1) .

Original: It is clear that our analysis removes a factor of √ n in front of the stability parameter β .
Revised: It is clear that our analysis removes a factor of √ n in front of the stability parameter β .

Original: Remark 5.
Revised: Remark 5.

Original: The analysis in the mixing case is more challenging than that in the i.i.d. case.
Revised: The analysis in the mixing case is more challenging than that in the i.i.d. case.

Original: The analysis in Bousquet et al.
Revised: The analysis in Bousquet et al.

Original: (2020) introduces ˜ g
Revised: (2020) introduces ˜ g

Original: i =
Revised: i =

Original: E z (cid:48)
Revised: E z ′ i

Original: i E z i [ f ( w S i ; z ) − f ( w S i ; z i )] , where z is independently drawn from the stationary distribution and S i = { z 1 , . . 
Revised: E z i [ f ( w S i ; z ) − f ( w S i ; z i )] , where z is independently drawn from the stationary distribution and S i = { z 1 , . . .

Original: , z i − 1 , z (cid:48)
Revised: , z i − 1 , z ′

Original: i , z i +1 , . . .
Revised: i , z i +1 , . . .

Original: , z n } .
Revised: , z n } .

Original: While E z i [˜ g i ] = 0 in the i.i.d.
Revised: While E z i [˜ g i ] = 0 in the i.i.d.

Original: case, we cannot guarantee E z i
Revised: case, we cannot guarantee E z i

Original: [˜ g i ]
Revised: [˜ g i ]

Original: = 0 in the mixing case due to the dependency between z i and z j ( j (cid:54) = i ).
Revised: = 0 in the mixing case due to the dependency between z i and z j ( j ̸ = i ).

Original: In this way, one cannot apply Theorem 2 to ˜ g
Revised: In this way, one cannot apply Theorem 2 to ˜ g

Original: i in the mixing case.
Revised: i in the mixing case.

Original: We use a much more complicated decomposition of the generalization error in terms of g i in
Revised: We use a much more complicated decomposition of the generalization error in terms of g i in

Original: Eq.
Revised: Eq.

Original: (4.3), which is of mean zero in the mixing case.
Revised: (4.3), which is of mean zero in the mixing case.

Original: In this process, we introduce concepts such as
Revised: In this process, we introduce concepts such as

Original: S i,b , S ii,b to fully exploit the stability and mixing property.
Revised: S i,b , S ii,b to fully exploit the stability and mixing property.

Original: It should be mentioned that S i,b , S ii,b have been introduced in Mohri & Rostamizadeh (2010).
Revised: It should be mentioned that S i,b , S ii,b have been introduced in Mohri & Rostamizadeh (2010).

Original: However, the aim is different.
Revised: However, the aim is different.

Original: These concepts are used in Mohri & Rostamizadeh (2010) to get bounds in expectation for Φ( S ) :
Revised: These concepts are used in Mohri & Rostamizadeh (2010) to get bounds in expectation for Φ( S ) :

Original: =
Revised: =

Original: F ( w S ) − F S ( w S ) via a lemma in Yu (1994) for β -mixing sequence, where the concentration of Φ( S ) around its expectation can be directly studied via the McDiarmid inequality for ϕ -mixing sequence.
Revised: F ( w S ) − F S ( w S ) via a lemma in Yu (1994) for β -mixing sequence, where the concentration of Φ( S ) around its expectation can be directly studied via the McDiarmid inequality for φ -mixing sequence.

Original: As a comparison, our aim is to replace the sequence ˜ g
Revised: As a comparison, our aim is to replace the sequence ˜ g

Original: i =
Revised: i =

Original: E z (cid:48)
Revised: E z ′

Original: i E z i [ f ( w S i ; z ) − f ( w S i ; z i )] (with non-zero conditional mean) by the sequence g i in Eq.
Revised: i E z i [ f ( w S i ; z ) − f ( w S i ; z i )] (with non-zero conditional mean) by the sequence g i in Eq.

Original: (4.3) with zero conditional mean, which is then controlled by our new concentration inequality for ϕ -mixing sequences (Theorem 2).
Revised: (4.3) with zero conditional mean, which is then controlled by our new concentration inequality for φ -mixing sequences (Theorem 2).

Original:  A PPLICATIONS
Revised:  A PPLICATIONS

Original: We now present applications of Theorem 5 to several algorithms, including the kernel regularization algorithm, SGD and localized iterative regularization.
Revised: We now present applications of Theorem 5 to several algorithms, including the kernel regularization algorithm, SGD and localized iterative regularization.

Original: Let A be an algorithm which outputs a model
Revised: Let A be an algorithm which outputs a model

Original: A ( S ) after observing the dataset S .
Revised: A ( S ) after observing the dataset S .

Original: We are interested in the excess population risk of a model
Revised: We are interested in the excess population risk of a model

Original: A ( S ) deﬁned by F ( A ( S ))
Revised: A ( S ) defined by F ( A ( S ))

Original: − F ( w ∗ ) , which measures the performance of the output model
Revised: − F ( w ∗ ) , which measures the performance of the output model

Original: A ( S ) as compared to the best model w ∗ =
Revised: A ( S ) as compared to the best model w ∗ =

Original: arg min w F ( w ) .
Revised: arg min w F ( w ) .

Original: An efﬁcient approach to this aim is based on the following error decomposition (Bousquet & Bottou, 2008)
Revised: An efficient approach to this aim is based on the following error decomposition (Bousquet & Bottou, 2008)

Original: [Equation]
Revised: [Equation]

Original: where we refer to F ( A ( S ))
Revised: where we refer to F ( A ( S ))

Original: − F S ( A ( S )) as the generalization gap and F S ( A ( S ))
Revised: − F S ( A ( S )) as the generalization gap and F S ( A ( S ))

Original: − F S ( w ∗ ) as the optimization error.
Revised: − F S ( w ∗ ) as the optimization error.

Original: The last term F S ( w ∗ ) − F ( w ∗ ) is easy to control since w ∗ is independent of S .
Revised: The last term F S ( w ∗ ) − F ( w ∗ ) is easy to control since w ∗ is independent of S .

Original: We will apply stability analysis to control the generalization gap, and tools in optimization theory to control the optimization error.
Revised: We will apply stability analysis to control the generalization gap, and tools in optimization theory to control the optimization error.

Original: To this aim, we give some necessary deﬁnitions.
Revised: To this aim, we give some necessary definitions.

Original: The Lipschitz condition means the gradient is bounded, and the smoothness means the gradient is Lipschitz continuous.
Revised: The Lipschitz condition means the gradient is bounded, and the smoothness means the gradient is Lipschitz continuous.

Original: Examples of Lipschitz loss functions include the hinge loss, logistic loss and absolute loss.
Revised: Examples of Lipschitz loss functions include the hinge loss, logistic loss and absolute loss.

Original: Examples of Lipschitz and smooth loss functions include the logistic loss and Huber loss.
Revised: Examples of Lipschitz and smooth loss functions include the logistic loss and Huber loss.

Original: Deﬁnition 4 (Lipschitz, smoothness and convexity) .
Revised: Definition 4 (Lipschitz, smoothness and convexity) .

Original: Let L, γ > 0 and µ ≥ 0 .
Revised: Let L, γ > 0 and µ ≥ 0 .

Original: Let f : W × Z
Revised: Let f : W × Z

Original: (cid:55)→ R .
Revised: (cid:55)→ R .

Original: • We say f is L -Lipschitz continuous if | f ( w ; z ) − f ( w (cid:48) ; z ) | ≤ L (cid:107) w − w (cid:48) (cid:107) for any w , w (cid:48) , z .•
Revised: • We say f is L -Lipschitz continuous if | f ( w ; z ) − f ( w ′ ; z ) | ≤ L ∥ w − w ′ ∥ for any w , w ′ , z .

Original: We say f is γ -smooth if (cid:107)∇ f ( w ; z ) − ∇ f ( w (cid:48) ; z ) (cid:107) ≤ γ (cid:107) w − w (cid:48) (cid:107) for any w , w (cid:48) , z .
Revised: We say f is γ -smooth if ∥∇ f ( w ; z ) − ∇ f ( w ′ ; z ) ∥ ≤ γ ∥ w − w ′ ∥ for any w , w ′ , z .

Original: We say f is µ -strongly convex if f ( w ; z ) − f ( w (cid:48) ; z ) −(cid:104)
Revised: We say f is µ -strongly convex if f ( w ; z ) − f ( w ′ ; z ) −⟨ w − w ′ , ∇ f ( w ′ ; z ) ⟩ ≥ µ 2 ∥ w − w ′ ∥

Original: w − w (cid:48) , ∇ f ( w (cid:48) ; z ) (cid:105) ≥ µ 2 (cid:107) w − w (cid:48) (cid:107)for any w , w (cid:48) , z .
Revised: for any w , w ′ , z .

Original: We say f is convex if it is µ -strongly convex with µ = 0 .
Revised: We say f is convex if it is µ -strongly convex with µ = 0 .

Original: 5. K ERNEL R EGULARIZATION S CHEMES
Revised: 5. K ERNEL R EGULARIZATION S CHEMES

Original: We ﬁrst consider kernel regularization schemes with convex and Lipschitz loss functions.
Revised: We first consider kernel regularization schemes with convex and Lipschitz loss functions.

Original: Let
Revised: Let

Original: K : X × X (cid:55)→ R be a Mercer kernel (i.e., K is symmetric and positive deﬁnite) and W be the associated reproducing kernel Hilbert space with the norm (cid:107) · [Equation] K .
Revised: K : X × X (cid:55)→ R be a Mercer kernel (i.e., K is symmetric and positive definite) and W be the associated reproducing kernel Hilbert space with the norm ∥ · ∥ K .

Original: We consider the following model
Revised: We consider the following model

Original: [Equation]
Revised: [Equation]

Original: where λ > 0 is a regularization parameter to tradeoff the data-ﬁtting term F S and the regularizer (cid:107) w (cid:107) 2 K .
Revised: where λ > 0 is a regularization parameter to tradeoff the data-fitting term F S and the regularizer ∥ w ∥ 2 K .

Original: The following corollary gives high-probability excess population risk bounds on kernel regularization.
Revised: The following corollary gives high-probability excess population risk bounds on kernel regularization.

Original: The proof is given in Section D.1.
Revised: The proof is given in Section D.1.

Original: Corollary 6.
Revised: Corollary 6.

Original: Let w S,λ denote the hypothesis returned by Eq.
Revised: Let w S,λ denote the hypothesis returned by Eq.

Original: (5.2) when trained on a sample S drawn from a ψ -mixing stationary distribution.
Revised: (5.2) when trained on a sample S drawn from a ψ -mixing stationary distribution.

Original: Assume f is convex, L -Lipschitz and bounded by
Revised: Assume f is convex, L -Lipschitz and bounded by

Original: M > 0 .
Revised: M > 0 .

Original: Then, with probability at least 1 − δ , the following excess risk bound holds ( k = (cid:100) log 2 n (cid:101) )
Revised: Then, with probability at least 1 − δ , the following excess risk bound holds ( k = ⌈ log 2 n ⌉ )

Original: [Equation]
Revised: [Equation]

Original: Remark 6.
Revised: Remark 6.

Original: We now instantiate the above bounds under special mixing sequences.
Revised: We now instantiate the above bounds under special mixing sequences.

Original: We ﬁrst consider the algebraically mixing sequence, i.e., ϕ (cid:48) ( k ) ≤ ϕ 0 k − r with r > 1 .
Revised: We first consider the algebraically mixing sequence, i.e., φ ′ ( k ) ≤ φ 0 k − r with r > 1 .

Original: In this case, analysis similar to
Revised: In this case, analysis similar to

Original: Remark 4 implies the following bound with an appropriate choice of b
Revised: Remark 4 implies the following bound with an appropriate choice of b

Original: [Equation]
Revised: [Equation]

Original: λ [Equation] w ∗ (cid:107)
Revised: If ∥ w ∗ ∥ K = O (1) , then we choose λ ≍ 1 / √ n and get

Original: [Equation]
Revised: [Equation]

Original: If (cid:107) w ∗ (cid:107) K = O (1) , then we choose λ (cid:16) 1 / √ n and get
Revised: We now consider the exponential mixing case, i

Original: [Equation]
Revised: [Equation]

Original: We now consider the exponential mixing case, i.e., ϕ (cid:48)

Original:  ( k ) ≤ ϕ 0 exp ( − ϕ 1 k r ) .
Revised: e., φ ′ ( k ) ≤ φ 0 exp ( − φ 1 k r ) .

Original: In this case, analysis similar to Remark 4 implies
Revised: In this case, analysis similar to Remark 4 implies

Original: [Equation]
Revised: [Equation]

Original:  w ∗ (cid:107)
Revised: If ∥ w ∗ ∥ K = O (1)

Original: [Equation]
Revised: [Equation]

Original: If (cid:107) w ∗ (cid:107) K

Original: = O (1) , then we can choose λ (cid:16) 1 / √ n to derive
Revised: then we can choose λ ≍ 1 / √ n to derive

Original: [Equation]
Revised: [Equation]

Original: 5. S TOCHASTIC G RADIENT D ESCENT
Revised: 5. S TOCHASTIC G RADIENT D ESCENT

Original: We apply our generalization bounds to SGD with convex and smooth loss functions, which has wide applications in training complex models in the big-data era due to its simplicity and efﬁciency.
Revised: We apply our generalization bounds to SGD with convex and smooth loss functions, which has wide applications in training complex models in the big-data era due to its simplicity and efficiency.

Original: Deﬁnition 5 (Stochastic Gradient Descent) .
Revised: Definition 5 (Stochastic Gradient Descent) .

Original: Let w 

Revised: Let w

Original: =
Revised: =

Original: ∈
Revised: ∈

Original: R
Revised: R

Original: d be an initial point and { η t } t be a sequence of positive step sizes.
Revised: d be an initial point and { η t } t be a sequence of positive step sizes.

Original: SGD updates models by w t
Revised: SGD updates models by w t

Original: +1 =
Revised: +1 =

Original: w t − η t ∇ f ( w t ; z i t ) , where ∇ f ( w t , z i t ) denotes a gradient of f w.r.t.
Revised: w t − η t ∇ f ( w t ; z i t ) , where ∇ f ( w t , z i t ) denotes a gradient of f w.r.t.

Original: the ﬁrst argument
Revised: the first argument

Original: and i t is independently drawn from the uniform distribution over [ n ] = { 1 , . . . , n } .
Revised: and i t is independently drawn from the uniform distribution over [ n ] = { 1 , . . . , n } .

Original: We assume the algorithm produces w S = 1 T (cid:80)
Revised: We assume the algorithm produces w S = 1 T (cid:80)

Original: Tt =
Revised: Tt =

Original: 1 w t , which is an average of SGD iterates.
Revised: 1 w t , which is an average of SGD iterates.

Original: We ﬁrst present the generalization error bounds.
Revised: We first present the generalization error bounds.

Original: The proofs are given in Section D.2.
Revised: The proofs are given in Section D.2.

Original: Corollary 7 (Generalization bound) .
Revised: Corollary 7 (Generalization bound) .

Original: Assume that the loss function f ( · ; z ) is γ -smooth, convex,
Revised: Assume that the loss function f ( · ; z ) is γ -smooth, convex,

Original: L -Lipschitz and bounded by M > 0 for every z .
Revised: L -Lipschitz and bounded by M > 0 for every z .

Original: Suppose that we run SGD with step sizes η t ≤ min(2 /γ, η ) for T (cid:16) n steps on a sample S drawn from a ψ -mixing stationary distribution.
Revised: Suppose that we run SGD with step sizes η t ≤ min(2 /γ, η ) for T ≍ n steps on a sample S drawn from a ψ -mixing stationary distribution.

Original: Then, with probability at least 1 − δ we have ( k = (cid:100) log 2 n (cid:101) )
Revised: Then, with probability at least 1 − δ we have ( k = ⌈ log 2 n ⌉ )

Original: [Equation]
Revised: [Equation]

Original: As a corollary, we develop the following excess risk bounds.
Revised: As a corollary, we develop the following excess risk bounds.

Original: Corollary 8 (Excess risk bound) .
Revised: Corollary 8 (Excess risk bound) .

Original: Assume that the loss function f ( · ; z ) is γ -smooth, convex, LLipschitz and bounded by M > 0 for every z .
Revised: Assume that the loss function f ( · ; z ) is γ -smooth, convex, LLipschitz and bounded by M > 0 for every z .

Original: Suppose that we run SGD with step sizes η t = η (cid:16) 1  √ T for T (cid:16) n steps on a sample S drawn from a ψ -mixing stationary distribution.
Revised: Suppose that we run SGD with step sizes η t = η ≍ 1 / √ T for T ≍ n steps on a sample S drawn from a ψ -mixing stationary distribution.

Original: Then, with probability at least 1 − δ we have ( k = (cid:100) log 2 n (cid:101) )
Revised: Then, with probability at least 1 − δ we have ( k = ⌈ log 2 n ⌉ )

Original: [Equation]
Revised: [Equation]

Original: ) √ n (cid:19) .

Original: 5. I TERATIVE L OCALIZED A LGORITHM
Revised: 5. I TERATIVE L OCALIZED A LGORITHM

Original: We now turn to convex and non-smooth problems.
Revised: We now turn to convex and non-smooth problems.

Original: In this case, SGD requires a very small step size to enjoy good stability, which however would affect the optimization process.
Revised: In this case, SGD requires a very small step size to enjoy good stability, which however would affect the optimization process.

Original: Indeed, a tradeoff between generalization and optimization requires running SGD with O ( n 2 ) iterations, which is not computationally efﬁcient (Lei & Ying, 2020;
Revised: Indeed, a tradeoff between generalization and optimization requires running SGD with O ( n 2 ) iterations, which is not computationally efficient (Lei & Ying, 2020;

Original: Bassily et al., 2020).
Revised: Bassily et al., 2020).

Original: To speed up the algorithm, we consider an iterative localization scheme (Algorithm 1 is deferred to Section D.3), which was introduced in Feldman et al.
Revised: To speed up the algorithm, we consider an iterative localization scheme (Algorithm 1 is deferred to Section D.3), which was introduced in Feldman et al.

Original: The basic idea of Algorithm 1 is to implement the optimization in epochs.
Revised: The basic idea of Algorithm 1 is to implement the optimization in epochs.

Original: At each epoch, Algorithm 1 builds an objective function with a regularizer depending on the output of the previous epoch, which is solved by SGD with T i iterations and learning rates { η t } .
Revised: At each epoch, Algorithm 1 builds an objective function with a regularizer depending on the output of the previous epoch, which is solved by SGD with T i iterations and learning rates { η t } .

Original: The following corollary gives error bounds for Algorithm 1.
Revised: The following corollary gives error bounds for Algorithm 1.

Original: The proof is given in Section D.3.
Revised: The proof is given in Section D.3.

Original: Corollary 9.
Revised: Corollary 9.

Original: Assume that the loss function f ( · ; z ) is convex, L -Lipschitz and bounded by M > 0 for every z .
Revised: Assume that the loss function f ( · ; z ) is convex, L -Lipschitz and bounded by M > 0 for every z .

Original: We run Algorithm 1 on sample S i , i ∈ [ m ] drawn from a ψ -mixing stationary distribution.
Revised: We run Algorithm 1 on sample S i , i ∈ [ m ] drawn from a ψ -mixing stationary distribution.

Original: If we choose γ (cid:16) n − 12 , then, with probability at least 1 − δ
Revised: If we choose γ ≍ n − 12 , then, with probability at least 1 − δ

Original: F ( w m ) − F ( w ∗ ) =
Revised: F [Equation]

Original: O (cid:16) n − 12 log n ∆ n log 12 ( n/δ )+ bn − 12 + M log nϕ (cid:48) ( b )+ n −k − 1 (cid:88) l = l ∆ 22 l log( n/δ ) (cid:17) , where k = (cid:100) log 2 n (cid:101) .
Revised: where k = ⌈ log 2 n ⌉ .

Original: Moreover, Algorithm 1 requires only O ( n log n ) gradient computations to achieve this generalization bound.
Revised: Moreover, Algorithm 1 requires only O ( n log n ) gradient computations to achieve this generalization bound.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: With high probability, we develop the ﬁrst stability-based generalization bounds of the order (cid:101)
Revised: With high probability, we develop the first stability-based generalization bounds of the order (cid:101)

Original: O (1 / √ n ) for learning with a mixing sequence.
Revised: O (1 / √ n ) for learning with a mixing sequence.

Original: We apply our results to several speciﬁc algorithms such as regularization schemes, SGD and localized iterative regularization.
Revised: We apply our results to several specific algorithms such as regularization schemes, SGD and localized iterative regularization.

Original: Our analysis relies on a new moment bound for weakly-dependent random variables deﬁned on a mixing sequence.
Revised: Our analysis relies on a new moment bound for weakly-dependent random variables defined on a mixing sequence.

Original: In this paper our generalization bounds involve ϕ (cid:48) -mixing coefﬁcients, which are larger than the ϕ -mixing coefﬁcients.
Revised: Our generalization bounds involve φ ′ -mixing coefficients, which are larger than the φ -mixing coefficients.

Original: It would be very interesting to investigate whether these ϕ (cid:48) -mixing coefﬁcients can be replaced by ϕ -mixing coefﬁcients.
Revised: It would be very interesting to investigate whether these φ ′ -mixing coefficients can be replaced by φ -mixing coefficients.

Original: We guess ϕ (cid:48) -mixing would be more similar to ϕ -mixing than ψ -mixing since both ϕ - and ϕ (cid:48) -coefﬁcients measure the difference between a conditional probability and a probability (i.e., of the form | Pr( A | B ) − Pr( A ) | ).
Revised: We guess φ ′ -mixing would be more similar to φ -mixing than ψ -mixing since both φ - and φ ′ -coefficients measure the difference between a conditional probability and a probability (i.e., of the form | Pr( A | B ) − Pr( A ) | ).

Original: As a comparison, ψ -mixing considers the difference between 1 and the ratio of probabilities (i.e. of the form | 1 − Pr( A ∩ B ) /
Revised: As a comparison, ψ -mixing considers the difference between 1 and the ratio of probabilities (i.e. of the form | 1 − Pr( A ∩ B ) /

Original: Pr( A )Pr( B ) | ).
Revised: Pr( A )Pr( B ) | ).

Revised: A CKNOWLEDGEMENT

Revised: We appreciate the Area Chair and Reviewers for constructive comments.

Revised: The work was done when

Revised: Shi was an intern at JD Explore Academy, and Yunwen was at the School of Computer Science,

Revised: University of Birmingham.

Original: A P ROOF OF T HEOREM
Revised: A P ROOF OF T HEOREM

Original: In this section we prove Theorem 1.
Revised: In this section, we prove Theorem 1.

Original: To this aim we introduce several lemmas.
Revised: To this aim, we introduce several lemmas.

Original: The following lemma is a McDiarmid inequality for stable functions deﬁned on mixing sequences (Kontorovich &
Revised: The following lemma is a McDiarmid inequality for stable functions defined on mixing sequences (Kontorovich &

Original: Ramanan, 2008).
Revised: Ramanan, 2008).

Original: Lemma A.1.
Revised: Lemma A.1.

Original: Let Φ : Z n (cid:55)→ R be a measurable function that is c -Lipschitz w.r.t.
Revised: Let Φ : Z n (cid:55)→ R be a measurable function that is c -Lipschitz w.r.t.

Original: the Hamming distance for some c >  and let Z 1 , . . .
Revised: the Hamming distance for some c > 0 , and let Z 1 , . . .

Original: , Z n be random variables distributed according to a ϕ -mixing distribution.
Revised: , Z n be random variables distributed according to a φ -mixing distribution.

Original: Then for any (cid:15) > 0 the following inequality holds
Revised: Then for any ϵ > 0 the following inequality holds

Original: [Equation]
Revised: [Equation]

Original: Furthermore, for any δ ∈ (0 , 1) the following inequality holds with probability at least 1 − δ
Revised: Furthermore, for any δ ∈ (0 , 1) the following inequality holds with probability at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: The following lemma shows the equivalence of tails and moments (Bousquet et al., 2020).
Revised: The following lemma shows the equivalence of tails and moments (Bousquet et al., 2020).

Original: Lemma A.2.
Revised: Lemma A.2.

Original: Let Y be a random variable.
Revised: Let Y be a random variable.

Original: If for any δ ∈ (0 , 1) , with probability at least 1 − δ
Revised: If for any δ ∈ (0 , 1) , with probability at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: then for any p ≥ 1 it holds that
Revised: then for any p ≥ 1 it holds that

Original: [Equation]
Revised: [Equation]

Original: If (cid:107) Y (cid:107) p ≤ √ pa + pb for any p ≥ 1 , then for any δ ∈ (0 , 1) we have with probability at least 1 − δ
Revised: If ∥ Y ∥ p ≤ √ pa + pb for any p ≥ 1 , then for any δ ∈ (0 , 1) we have with probability at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem 1.
Revised: Proof of Theorem 1.

Original: Let (cid:15) > 0 be a number to be ﬁxed later and deﬁne
Revised: Let ϵ > 0 be a number to be fixed later and define

Original: [Equation]
Revised: [Equation]

Original: where I [ · ] is the indicator function ( 1 if the argument is true and 0 otherwise).
Revised: where I [ · ] is the indicator function ( 1 if the argument is true and 0 otherwise).

Original: Deﬁne Φ( X 1 , . . . , X n )=
Revised: Define Φ( X 1 , . . . , X n )

Original:  (cid:80) ni =1 (cid:101)
Revised: = (cid:80) ni =1 (cid:101)

Original: Z i .
Revised: Z i .

Original: First we show the Lipschitz continuity of Φ w.r.t.
Revised: First we show the Lipschitz continuity of Φ w.r.t.

Original: the Hamming distance.
Revised: the Hamming distance.

Original: Suppose we change X 1 by X (cid:48) 1 and keep other X i .
Revised: Suppose we change X 1 by X ′ 1 and keep other X i .

Original: Deﬁne Z (cid:48) 1 , . . .
Revised: Define Z ′ 1 , . . .

Original: , Z (cid:48) n similarly to Z 1 , . . .
Revised: , Z ′ n similarly to Z 1 , . . .

Original: , Z n but as functions of X (cid:48) 1 , X 2 , . . .
Revised: , Z n but as functions of X ′ 1 , X 2 , . . .

Original: , X n .
Revised: , X n .

Original: Since Z j is a function of X j (i.e., once we know X j we know
Revised: Since Z j is a function of X j (i.e., once we know X j we know

Original: Z j ), we know (cid:101) Z j = (cid:101)Z (cid:48)
Revised: Z j ), we know (cid:101) Z j = (cid:101)

Original:  j for all j (cid:54) = 1 .
Revised: Z ′ j for all j ̸ = 1 .

Original: Then(cid:12)(cid:12) [Equation]
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: According to Lemma A.1 with c = 2 (cid:15) we derive the following inequality with probability at least 1 − 
Revised: According to Lemma A.1 with c = 2 ϵ we derive the following inequality with probability at least 1 − ( e

Original: [Equation]
Revised: [Equation]

Original:  According to the assumption Pr {| Z i | > ˜ (cid:15) } ≤ 2 exp( − ˜ (cid:15) 2 /b ) , we know with probability at least 1 − 2 δ/ ( en )
Revised: ) δ/e [Equation] According to the assumption Pr {| Z i | > ˜ ϵ } ≤ 2 exp( − ˜ ϵ 2 /b ) , we know with probability at least 1 − 2 δ/ ( en )

Original: [Equation]
Revised: [Equation]

Original: We ﬁx (cid:15) = (cid:112) b log( en/δ ) .
Revised: We fix ϵ = (cid:112) b log( en/δ ) .

Original: We now assume that Eq.
Revised: We now assume that Eq.

Original: (A.1) and Eq.
Revised: (A.1) and Eq.

Original: (A.2) hold for all i ∈ [ n ] , which happen with probability at least 1 − δ .
Revised: (A.2) hold for all i ∈ [ n ] , which happen with probability at least 1 − δ .

Original: Under this event, we have Z i =
Revised: Under this event, we have Z i =

Original: (cid:101)
Revised: (cid:101)

Original: Z i and (for simplicity we assume n ≥ 2 / ( e − 2) )
Revised: Z i and (for simplicity we assume n ≥ 2 / ( e − 2) )

Original: [Equation]
Revised: [Equation]

Original: The following inequality then holds with probability at least 1 − δ
Revised: The following inequality then holds with probability at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: According to Lemma A.2, for any p ≥ 1 it holds that
Revised: According to Lemma A.2, for any p ≥ 1 it holds that

Original: [Equation]
Revised: [Equation]

Original: The stated bound follows directly.
Revised: The stated bound follows directly.

Original: The proof is completed.
Revised: The proof is completed.

Original: B P ROOF OF T HEOREM
Revised: B P ROOF OF T HEOREM

Original: We follow the framework in Bousquet et al.
Revised: We follow the framework in Bousquet et al.

Original: (2020) to prove Theorem 2.
Revised: (2020) to prove Theorem 2.

Original: Proof of Theorem 2.
Revised: Proof of Theorem 2.

Original: Without loss of generality, we assume n = 2 k .
Revised: Without loss of generality, we assume n = 2 k .

Original: Consider a sequence of partitions
Revised: Consider a sequence of partitions

Original: B 0 , . . .
Revised: B 0 , . . .

Original: , B k with B k = { 1 , 2 , . . .
Revised: , B k with B k = { 1 , 2 , . . .

Original: , 2 k } .
Revised: , 2 k } .

Original: We then obtain B l from B l +1 by splitting each subset in B l + into two equal parts.
Revised: We then obtain B l from B l +1 by splitting each subset in B l + into two equal parts.

Original: In this way, we get
Revised: In this way, we get

Original: [Equation] [Equation] , { 2 k }} , B 1 = {{ 1 , 2 } , { 3 , 4 } , . . .
Revised: B 0 = {{ 1 } , { 2 } , . . , { 2 k }} , B 1 = {{ 1 , 2 } , { 3 , 4 } , . . .

Original: , { 2 k − 1 , 2 k }} , . . .
Revised: , { 2 k − 1 , 2 k }} , . . .

Original: , B k = { [ n ] } .
Revised: , B k = { [ n ] } .

Original: For each i ∈ [ n ] and l = 0 , 1 , . . .
Revised: For each i ∈ [ n ] and l = 0 , 1 , . . .

Original: , k , denote by B l ( i )
Revised: , k , denote by B l ( i )

Original: ∈
Revised: ∈

Original: B l the only set from B l that contains i .
Revised: B l the only set from B l that contains i .

Original: In particular, B 0 ( i ) =
Revised: In particular, B 0 ( i ) =

Original: { i } and B k ( i ) =
Revised: { i } and B k ( i ) =

Original: [ n ] .
Revised: [ n ] .

Original: For each i ∈ [ n ] and each l = 0 , 1 , . . .
Revised: For each i ∈ [ n ] and each l = 0 , 1 , . . .

Original: , k , introduce the random variablesg li =
Revised: , k , introduce the random variables

Original: g li ( Z i , Z [ n ] \ B l ( i ) ) = E [ g i | Z i , Z [ n ] \ B l ( i ) ] .
Revised: [Equation]

Original: That is, we condition on Z i and all the variables that are not in the same set as Z i in the partition B l .
Revised: That is, we condition on Z i and all the variables that are not in the same set as Z i in the partition B l .

Original: One can check that g 

Revised: One can check that g

Original: i =
Revised: i =

Original: g i and g ki
Revised: g i and g ki

Original: = E [ g i |
Revised: = E [ g i |

Original: Z i ] .
Revised: Z i ] .

Original: We can write a telescopic sum for each i ∈ [ n ]g i =
Revised: We can write a telescopic sum for each i ∈ [ n ]

Original: E [ g i | Z i ] + k − 1 (cid:88) l

Original: = ( g li − g l +1 i ) .

Original: It then follows from the triangle inequality that

Original: [Equation]
Revised: [Equation]

Original: n (cid:88) i =

Original: E

Original: [Equation]

Original: k

Original: [Equation]

Original: l

Original: [Equation] (cid:13)(cid:13)(cid:13) n (cid:88) i = [Equation] [Equation]
Revised: It then follows from the triangle inequality that

Original: g

Original: [Equation]
Revised: [Equation]

Original: i ) (cid:13)(cid:13)(cid:13) p .

Original: (B.1) Since | E [ g i | Z i ] | ≤ M , one can check that Φ( Z 1 , . . .
Revised: Since | E [ g i | Z i ] | ≤ M , one can check that Φ( Z 1 , . . .

Original: , Z n )
Revised: , Z n )

Original: = (cid:80) ni
Revised: = (cid:80) ni

Original: =1 E [ g i |
Revised: =1 E [ g i |

Original: Z i ] is 2 M -Lipschitz w.r.t.
Revised: Z i ] is 2 M -Lipschitz w.r.t.

Original: the Hamming distance.
Revised: the Hamming distance.

Original: Furthermore, we have E [ E [ g i |
Revised: Furthermore, we have E [ E [ g i |

Original: Z i ]]
Revised: Z i ]]

Original: = 0 .
Revised: = 0 .

Original: Now we can apply Lemma A.1 with c = 2 M to derive the following inequality with at least 1 − δ
Revised: Now we can apply Lemma A.1 with c = 2 M to derive the following inequality with at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: It then follows from Lemma A.2 that
Revised: It then follows from Lemma A.2 that

Original: [Equation]
Revised: [Equation]

Original: According to the deﬁnition of g li , one can see that
Revised: According to the definition of g li , one can see that

Original: [Equation]
Revised: [Equation]

Original: Furthermore, according to our assumption we know g li as a function of Z j , j ∈ B l +1 ( i ) \ B l ( i ) satisﬁes the β -Lipschitz continuity w.r.t.
Revised: Furthermore, according to our assumption we know g li as a function of Z j , j ∈ B l +1 ( i ) \ B l ( i ) satisfies the β -Lipschitz continuity w.r.t.

Original: the Hamming distance.
Revised: the Hamming distance.

Original: Therefore, one can apply Lemma
Revised: Therefore, one can apply Lemma

Original: A.1 with c = β and Φ = g li to derive the following inequality with (there are 2 l random variables)
Revised: A.1 with c = β and Φ = g li to derive the following inequality with (there are 2 l random variables)

Original: [Equation]
Revised: [Equation]

Original: where the probability is w.r.t.
Revised: where the probability is w.r.t.

Original: Z B l +1 ( i ) \ B l ( i ) .
Revised: Z B l +1 ( i ) \ B l ( i ) .

Original: Let us consider the sum (cid:80) i ∈
Revised: Let us consider the sum (cid:80) i ∈

Original: B l ( g li − g l +1 i ) for any B l ∈ B l .
Revised: B l ( g li − g l +1 i ) for any B l ∈ B l .

Original: Note Z(cid:48)
Revised: Note Z

Original:  i :=
Revised: ′ i :=

Original: g li − g l +1 i is a function of
Revised: g li − g l +1 i is a function of

Original: Z i , Z [ n ] \ B l .
Revised: Z i , Z [ n ] \ B l .

Original: We now condition on Z [ n ] \ B l and then Z (cid:48)
Revised: We now condition on Z [ n ] \ B l and then Z ′

Original: i is a function of Z i .
Revised: i is a function of Z i .

Original: According to Eq.
Revised: According to Eq.

Original: (B.3), we can apply Theorem 1 with b = 2 l − 1 β 2 ∆ 22 l to derive the following inequality
Revised: (B.3), we can apply Theorem 1 with b = 2 l − 1 β 2 ∆ 22 l to derive the following inequality

Original: [Equation]
Revised: [Equation]

Original: We now take integration w.r.t.
Revised: We now take integration w.r.t.

Original: Z [ n ] \ B l and get 
Revised: Z [ n ] \ B l and get (cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: According to the triangle inequality, we further get (cid:13)(cid:13) (cid:88) i ∈ [ n ]
Revised: B l

Original: [Equation]
Revised: [Equation]

Original: where we have used the fact that |B l | = 2 k − l . It follows that k − 1 (cid:88)
Revised: g [Equation] i

Original: [Equation]
Revised: [Equation]

Revised: ∆ 22 l .

Revised: According to the triangle inequality, we further get

Original: [Equation]
Revised: [Equation]

Original: n (cid:88) i = ( g li − g l +1 i ) (cid:13)(cid:13)(cid:13) p ≤ 2 kpβ k −1 (cid:88) l 
Revised: where we have used the fact that |B l | = 2 k − l . It follows that

Original: = (9 + l )∆ 22 l .
Revised: [Equation]

Original: We can plug Eq.
Revised: We can plug Eq.

Original: (B.2) and the above inequality back into Eq.
Revised: (B.2) and the above inequality back into Eq.

Original: (B.1) to derive (cid:13)(cid:13)(cid:13)n (cid:88) i =
Revised: (B.1) to derive (cid:13)(cid:13)(cid:13)

Original: [Equation]

Original: [Equation]
Revised: [Table]

Original: The proof is completed.
Revised: The proof is completed

Revised: [Table]

Original: C P ROOF OF T HEOREM
Revised: C P ROOF OF T HEOREM

Original: To prove Theorem 5, we require the following lemma to control the difference between two test errors: one with the test example drawn from the mixing sequence and one with the test example drawn from the independent stationary distribution.
Revised: To prove Theorem 5, we require the following lemma to control the difference between two test errors: one with the test example drawn from the mixing sequence and one with the test example drawn from the independent stationary distribution.

Original: Let S b
Revised: Let S b

Original: = { z 1 , . . .
Revised: = { z 1 , . . .

Original: , z n − b } be the sequence by removing the last b points of S .
Revised: , z n − b } be the sequence by removing the last b points of S .

Original: Lemma C.1 (Mohri & Rostamizadeh 2010) .
Revised: Lemma C.1 (Mohri & Rostamizadeh 2010) .

Original: Let F ( w S ) =
Revised: Let F ( w S ) =

Original: E z
Revised: E z

Original: [ f ( w S ; z ) | S ] denote the expectation in the dependent case (i.e., z depends on S ) and ˜ F ( w S b )
Revised: [ f ( w S ; z ) | S ] denote the expectation in the dependent case (i.e., z depends on S ) and ˜ F ( w S b )

Original: =
Revised: =

Original: E (cid:101) z
Revised: E (cid:101) z

Original: [ f ( w S b ; (cid:101) z )] denote the expectation where the test points are assumed independent of the training data (i.e., ˜ z is independent of S ).
Revised: [ f ( w S b ; (cid:101) z )] denote the expectation where the test points are assumed independent of the training data (i.e., ˜ z is independent of S ).

Original: If A is β -uniformly stable and f ( w ; z ) ∈ [0 , M ] , then the following inequality holds for any b >
Revised: If A is β -uniformly stable and f ( w ; z ) ∈ [0 , M ] , then the following inequality holds for any b >

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma 4.
Revised: Proof of Lemma 4.

Original: Let z(cid:48)
Revised: Let z

Original:  i (resp.
Revised: ′ i (resp.

Original: z (cid:48)(cid:48) i ) be drawn from the same distribution of z i , i.e., the conditional distribution of z (cid:48)
Revised: z ′′ i ) be drawn from the same distribution of z i , i.e., the conditional distribution of z ′

Original: i (resp.
Revised: i (resp.

Original: z (cid:48)(cid:48) i ) given z 1 , . ..
Revised: z ′′ i ) given z 1 , . .

Original: z i − 1 , z i +1 , . . .
Revised: z i − 1 , z i +1 , . . .

Original: , z n is the same as that of z
Revised: , z n is the same as that of z

Original: i given z 1 , . . .
Revised: i given z 1 , . . .

Original: z i − 1 , z i +1 , . . .
Revised: z i − 1 , z i +1 , . . .

Original: , z n .
Revised: , z n .

Original: Let S i,b
Revised: Let S i,b

Original: = { z 1 , . . .
Revised: = { z 1 , . . .

Original: z i − b − 1 , z i , z
Revised: z i − b − 1 , z i , z

Original: i + b +1 , . . .
Revised: i + b +1 , . . .

Original: , z n − b } .
Revised: , z n − b } .

Original: For any i ∈ [ n ] , let
Revised: For any i ∈ [ n ] , let

Original: S ii,b = { z 1 , . . .
Revised: S ii,b = { z 1 , . . .

Original: z i − b − 1 , z (cid:48)
Revised: z i − b − 1 , z ′

Original: i , z i + b +1 , . . .
Revised: i , z i + b +1 , . . .

Original: , z n − b } .
Revised: , z n − b } .

Original: We have the following decomposition 
Revised: We have the following decomposition n (cid:88)

Original: [Equation]
Revised: [Equation]

Original: According to the deﬁnition of β -uniform stability we know
Revised: According to the definition of β -uniform stability we know

Original: [Equation]
Revised: [Equation]

Original: For any i ∈ [ n ] , introduce
Revised: For any i ∈ [ n ] [Equation] introduce

Original: [Equation]
Revised: [Equation]

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: According to Lemma C.1 we know
Revised: According to Lemma C.1 we know

Original: [Equation]
Revised: [Equation]

Original: By the deﬁnition of ϕ (cid:48) , we know
Revised: By the definition of φ ′ , we know

Original: [Equation]
Revised: [Equation]

Original: Furthermore, the deﬁnition of stability implies
Revised: Furthermore, the definition of stability implies

Original: [Equation]
Revised: [Equation]

Original: We combine the above three inequalities together and derive n (cid:88)
Revised: We combine the above three inequalities together and derive n (cid:88)

Original: [Equation]
Revised: [Equation]

Original: ,b ; z (cid:48)(cid:48)

Original: i [Equation] ] | + n [Equation]
Revised: Combining the above inequality and Eq.

Original: i =

Original: [Equation] Mϕ [Equation] b )
Revised: (C.1)

Original: [Equation]
Revised: [Equation]

Original: n. Combining the above inequality and Eq. (C.1), we obtain(cid:12)(cid:12) n ( F ( w S ) − F S ( w S )) (cid:12)(cid:12) ≤ (cid:12) (cid:12)(cid:12) n (cid:88)
Revised: we obtain

Original: i = (cid:0) E

Original: [Equation]
Revised: [Equation]

Original: + n (cid:88) i [Equation] [Equation]
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: + n(cid:88) i = (cid:12) (cid:12) f ( w S
Revised: n

Original: [Equation]
Revised: [Equation]

Original: z i 
Revised: i =

Original: [Equation]

Original: − f ( w S

Original: [Equation]
Revised: [Equation]

Original: z i ) (cid:12) (cid:12) ≤ (6 b + 2) nβ + n ( Mϕ ( b ) + Mϕ (cid:48) ( b )) + (cid:12)(cid:12) n (cid:88) i

Original: = g i (cid:12)(cid:12) .

Original: The proof is completed.
Revised: The proof is completed.

Original: Proof of Theorem 5.
Revised: Proof of Theorem 5.

Original: Recall the deﬁnition of g i ,
Revised: Recall the definition of g i ,

Original: [Equation]
Revised: [Equation]

Original: Since z i and z (cid:48)(cid:48) i follow from the same distribution, we know
Revised: Since z i and z ′′ i follow from the same distribution [Equation] we know

Original: [Equation]
Revised: [Equation]

Original: One can check other assumptions in Theorem 2 also hold.
Revised: One can check other assumptions in Theorem 2 also hold.

Original: Therefore, one can apply Theorem 2 to derive the following inequality with probability at least 1 − δ
Revised: Therefore, one can apply Theorem 2 to derive the following inequality with probability at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: where k = (cid:100) log 2 n (cid:101) .
Revised: where k = ⌈ log 2 n ⌉ .

Original: According to Lemma A.2 we further get the following inequality with probability at least 1 − δ n(cid:88)
Revised: According to Lemma A.2 we further get the following inequality with probability at least 1 − δ n

Original: [Equation]
Revised: [Equation]

Original: According to Lemma
Revised: According to Lemma

Original: [Equation]
Revised: [Equation]

Original: we know
Revised: we know

Original: [Equation]
Revised: [Equation]

Original: We can combine the above inequality with Eq.
Revised: We can combine the above inequality with Eq.

Original: (C.3) to derive the following inequality with probability at least 1 − δ
Revised: (C.3) to derive the following inequality with probability at least 1 − δ

Original: [Equation]
Revised: [Equation]

Revised: n (cid:112) 2 n log( e/δ )

Revised: + 2 k

Revised: +1 β

Revised: k − 1 (cid:88)

Revised: l = (9 + l )∆ 22 l log( e/δ ) (cid:17) + (6 b + 2) nβ + n ( Mφ ( b ) + Mφ ′ ( b )) .

Original: The proof is completed.
Revised: The proof is completed.

Original: D P ROOF OF A PPLICATIONS D. P ROOF OF C OROLLARY
Revised: D P ROOF OF A PPLICATIONS D. P ROOF OF C OROLLARY

Original: To prove Corollary 6, we require the following lemma on the uniform stability of kernel regularization (Bousquet & Elisseeff, 2002).
Revised: To prove Corollary 6, we require the following lemma on the uniform stability of kernel regularization (Bousquet & Elisseeff, 2002).

Original: Lemma D.1 (Bousquet & Elisseeff 2002) .
Revised: Lemma D.1 (Bousquet & Elisseeff 2002) .

Original: Let the loss function f be L -Lipschitz and convex.
Revised: Let the loss function f be L -Lipschitz and convex.

Original: Let the algorithm A be deﬁned in (5.2) .
Revised: Let the algorithm A be defined in (5.2) .

Original: Then A is β -uniformly
Revised: Then A is β -uniformly

Original: stable
Revised: stable

Original: with β ≤ L 2 λn .
Revised: with β ≤ L 2 λn .

Original: Proof of Corollary 6.
Revised: Proof of Corollary 6.

Original: Let A be the algorithm which returns w S,λ .
Revised: Let A be the algorithm which returns w S,λ .

Original: By Lemma D.1, we know A is β -uniformly
Revised: By Lemma D.1, we know A is β -uniformly

Original: stable
Revised: stable

Original: where β ≤ L 2 nλ .
Revised: where β ≤ L 2 nλ .

Original: Plugging the above inequality into Theorem 5, we derive the following inequality with probability at least 1 − δ
Revised: Plugging the above inequality into Theorem 5, we derive the following inequality with probability at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: By Lemma A.1, we have the following inequality with probability at least 1 − δ
Revised: By Lemma A.1, we have the following inequality with probability at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: Furthermore, we have the following error decomposition
Revised: Furthermore, we have the following error decomposition

Original: [Equation]
Revised: [Equation]

Original: where we have used the deﬁnition of w S,λ .
Revised: where we have used the definition of w S,λ .

Original: We can combine the above three inequalities together to derive the stated bound.
Revised: We can combine the above three inequalities together to derive the stated bound.

Original: The proof is completed.
Revised: The proof is completed.

Original: D. P ROOF OF C OROLLARY 7 AND C OROLLARY
Revised: D. P ROOF OF C OROLLARY 7 AND C OROLLARY

Original: First, we prove the stability bound for convex loss minimization via SGD.
Revised: First, we prove the stability bound for convex loss minimization via SGD.

Original: Then, we apply the stability bound and Theorem 5 to the generalization bound.
Revised: Then, we apply the stability bound and Theorem 5 to the generalization bound.

Original: To develop high-probability bounds, we need to introduce a concentration inequality (Wainwright, 2019).
Revised: To develop high-probability bounds, we need to introduce a concentration inequality (Wainwright, 2019).

Original: Lemma D.2 (Chernoff’s Bound) .
Revised: Lemma D.2 (Chernoff’s Bound) .

Original: Let X 1 , . . .
Revised: Let X 1 , . . .

Original: , X t be independent random variables taking values in { 0 , 1 } .
Revised: , X t be independent random variables taking values in { 0 , 1 } .

Original: Let X
Revised: Let X

Original: =
Revised: =

Original: (cid:80) tj =1 X j and µ = E
Revised: (cid:80) tj =1 X j and µ = E

Original: [ X ] .
Revised: [ X ] .

Original: Then for any ˜ δ > 0 with probability at least 1 − exp (cid:0) − µ ˜ δ 2 / (2 + ˜ δ ) (cid:1) we have X ≤ (1 + ˜ δ ) µ .
Revised: Then for any ˜ δ > 0 with probability at least 1 − exp (cid:0) − µ ˜ δ 2 / (2 + ˜ δ ) (cid:1) we have X ≤ (1 + ˜ δ ) µ .

Original: Furthermore, for any δ ∈ (0 , 1) with probability at least 1 − δ we have
Revised: Furthermore, for any δ ∈ (0 , 1) with probability at least 1 − δ we have

Original: [Equation]
Revised: [Equation]

Original: Proof of Corollary 7.
Revised: Proof of Corollary 7.

Original: Let S and S (cid:48) be two samples of size n differing in only a single example.
Revised: Let S and S ′ be two samples of size n differing in only a single example.

Original: Consider the gradient updates w 1 , . .
Revised: Consider the gradient updates w 1 , . .

Original: . , w T and w (cid:48) 1 , . .
Revised: . , w T and w ′ 1 , . .

Original:  w (cid:48) T induced by running SGD on sample S and S (cid:48) .
Revised: . , w ′ T induced by running SGD on sample S and S ′ .

Original: We now suppose S and S (cid:48) differ by the ﬁrst example and apply the Lipschitz condition on f ( · ; z ) to get
Revised: We now suppose S and S ′ differ by the first example and apply the Lipschitz condition on f ( · ; z ) to get

Original: [Equation]
Revised: [Equation]

Original: where δ T = (cid:107) w T − w (cid:48) T (cid:107) .
Revised: where δ T = ∥ w T − w ′ T ∥ .

Original: Observe that at step t , with probability 1 − 1 /n , the example selected by
Revised: Observe that at step t , with probability 1 − 1 /n , the example selected by

Original: SGD is the same in both S and S (cid:48) .
Revised: SGD is the same in both S and S ′ .

Original: The convexity and γ -smoothness imply that (Hardt et al., 2016)
Revised: The convexity and γ -smoothness imply that (Hardt et al., 2016)

Original: [Equation]
Revised: [Equation]

Original: If i t (cid:54) = 1 , by η t ≤ 2 /γ
Revised: If i t ̸ = 1 , by η t ≤ 2 /γ

Original: we know
Revised: we know

Revised: [Equation]

Revised: t +

Original: [Equation]
Revised: [Equation]

Original: With probability 1 /n
Revised: With probability 1 /n

Original: [Equation]
Revised: [Equation]

Original: the example selected is different
Revised: the example selected is different

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: e. i t = 1 .
Revised: e. i t = 1 .

Original: Then, by the triangle equality and
Revised: Then, by the triangle equality and

Original: Eq.
Revised: Eq.

Original: D.3,
Revised: D.3,

Original: [Equation]
Revised: [Equation]

Original: Combining the above two cases, we can conclude that for every t ,
Revised: Combining the above two cases [Equation] we can conclude that for every t ,

Original: [Equation]
Revised: [Equation]

Original: where I denotes the indicator function.
Revised: where I denotes the indicator function.

Original: Solving recursive inequality gives,
Revised: Solving recursive inequality gives,

Original: [Equation]
Revised: [Equation]

Original: We can apply Lemma D.2 with X k
Revised: We can apply Lemma D.2 with X k

Original: = I
Revised: = I

Original: [ i k =1] , µ = t/n to get the following inequality with probability at least 1 − δt (cid:88)
Revised: [ i k =1] , µ = t/n to get the following inequality with probability at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: Therefore, with probability at least 1 − δ [Equation] there holds
Revised: Therefore, with probability at least 1 − δ , there holds

Original: [Equation]
Revised: [Equation]

Original: By the convexity of the norm (cid:107) · (cid:107) [Equation] we get the following inequality with probability at least 1 − δ
Revised: By the convexity of the norm ∥ · ∥ , we get the following inequality with probability at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: Plugging the inequality back into Eq.
Revised: Plugging the inequality back into Eq.

Original: (D.1), we obtain that, with probability at least 1 − δ | 
Revised: (D.1), we obtain that, with probability at least 1 − δ | f ( w S

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Original: D.7)
Revised: (D.7)

Original: We can combine Eq.
Revised: We can combine Eq.

Original: (D.7) and Theorem 5 to obtain the following inequality with probability at least 1 − δ
Revised: (D.7) and Theorem 5 to obtain the following inequality with probability at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: By the choice of T (cid:16) n , we can get
Revised: By the choice of T ≍ n [Equation] we can get

Original: [Equation]
Revised: [Equation]

Original: The proof is completed.
Revised: The proof is completed.

Original: To prove excess risk bounds, we require the following high-probability bound on optimization error.
Revised: To prove excess risk bounds, we require the following high-probability bound on optimization error.

Original: Notice that optimization error analysis does not depend on the mixing property of the dataset since the randomness is taken with respect to the random indices.
Revised: Notice that optimization error analysis does not depend on the mixing property of the dataset since the randomness is taken with respect to the random indices.

Original: Lemma D.3 (Optimization Error (Lei & Tang, 2018)) .
Revised: Lemma D.3 (Optimization Error (Lei & Tang, 2018)) .

Original: Assume that the loss function f ( · ; z ) is convex and L -Lipschitz for every z .
Revised: Assume that the loss function f ( · ; z ) is convex and L -Lipschitz for every z .

Original: Suppose that we run SGD with step sizes η t = η (cid:16) 1 √ T then with probability at least 1 − δ we have
Revised: Suppose that we run SGD with step sizes η t = η ≍ 1 √ T then with probability at least 1 − δ we have

Original: [Equation]
Revised: [Equation]

Original: Proof of Corollary 8.
Revised: Proof of Corollary 8.

Original: By Corollary 7, we know with probability at least 1 − δ that
Revised: By Corollary 7, we know with probability at least 1 − δ that

Original: [Equation]
Revised: [Equation]

Original: By Lemma A. [Equation] we have the following inequality with probability at least 1 − δ
Revised: By Lemma A.1, we have the following inequality with probability at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: Lemma D.3 shows the following inequality with probability at least 1 − δ
Revised: Lemma D.3 shows the following inequality with probability at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: We plug the above three inequalities back into Eq.
Revised: We plug the above three inequalities back into Eq.

Original: (5.1), and derive the following inequality with probability at least 1 − 3 δ
Revised: (5.1), and derive the following inequality with probability at least 1 − 3 δ

Original: [Equation]
Revised: [Equation]

Original: The proof is completed.
Revised: The proof is completed.

Original: D. P ROOF OF C OROLLARY
Revised: D. P ROOF OF C OROLLARY

Original: In this section, we present the proof on the stability of the iterative localization technique.
Revised: In this section, we present the proof on the stability of the iterative localization technique.

Original: To this aim, we ﬁrst present Algorithm 1.
Revised: To this aim, we first present Algorithm 1.

Original: Algorithm Input: initial point w 0 = 0 , parameter γ > 0 , m = (cid:6) 12 log 2 n [Equation]
Revised: Input: initial point w 0 = 0 , parameter γ > 0 , m = (cid:6) 12 log 2 n (cid:7) 1 for

Original: 1 for i = 1 , 2 , . . . , m do set T i (cid:16) n, γ i = γ 2 i , η t = γ i n t +1 , t ∈ N draw a sample S i of size n from the mixing distribution apply SGD with T i iterations and step size η t to minimize the following problem and get w i 
Revised: , m do set T i ≍ n, γ i = γ 2 i , η t = γ i n t +1 , t ∈ N draw a sample S i of size n from the mixing distribution apply SGD with T i iterations and step size η t to minimize the following problem and get w i (cid:101) F S i ( w ) : = 1 n (cid:88) z ∈ S

Revised: i f (

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Revised: 1 γ

Revised: [Equation]

Revised: n ∥ w − w i

Revised: [Equation]

Revised: 1 ∥ 2 .

Original: 5 end
Revised: 5 end

Original: Then we move to generalization bound for Corollary 9.
Revised: Then we move to generalization bound for Corollary 9.

Original: To this aim, We need to introduce some deﬁnitions for our proof.
Revised: To this aim, We need to introduce some definitions for our proof.

Original: For any i , let
Revised: For any i , let

Original: [Equation]
Revised: [Equation]

Original: where (cid:101) F S i is deﬁned in Algorithm 1.
Revised: where (cid:101) F S i is defined in Algorithm 1.

Original: Lemma D.4 (Optimization Error Bound) .
Revised: Lemma D.4 (Optimization Error Bound) .

Original: Suppose that the function w (cid:55)→ f ( w ; z ) is µ -strongly convex (with respect to (cid:107) · (cid:107) ) and L -Lipschitz.
Revised: Suppose that the function w (cid:55)→ f ( w ; z ) is µ -strongly convex (with respect to ∥ · ∥ ) and L -Lipschitz.

Original: Let { w t } t be produced by SGD on sample S and step size η t = 2 / ( µ ( t + 1)) to minimize F S ( w ) = 1 n (cid:80) ni
Revised: Let { w t } t be produced by SGD on sample S and step size η t = 2 / ( µ ( t + 1)) to minimize F S ( w ) = 1 n (cid:80) ni

Original: =1 f ( w ; z i ) .
Revised: =1 f ( w ; z i ) .

Original: Set ¯ w (cid:48) t 
Revised: Set ¯ w ′ t =

Original: = (cid:16)(cid:80) tj =1 j w j (cid:17) / (cid:80) tj =1 j .
Revised: (cid:16)(cid:80) tj =1 j w j (cid:17) / (cid:80) tj =1 j .

Original: Then, for any δ ∈ (0 , 1) , with probability at least 1 − δ ,
Revised: Then, for any δ ∈ (0 , 1) , with probability at least 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: The proof of Lemma D.4 can be found in Harvey et al.
Revised: The proof of Lemma D.4 can be found in Harvey et al.

Original: According to Algorithm 1, w i is the output by SGD with η t = γ i n/ ( t + 1) to minimize (cid:101) F S i ( w ) , with the iterates weighted as Lemma
Revised: According to Algorithm 1, w i is the output by SGD with η t = γ i n/ ( t + 1) to minimize (cid:101) F S i ( w ) , with the iterates weighted as Lemma

Original: D.4.
Revised: D.4.

Original: The following lemma establishes the bound of Euclidean distance of w i and ˆ w i .
Revised: The following lemma establishes the bound of Euclidean distance of w i and ˆ w i .

Original: Lemma D.5.
Revised: Lemma D.5.

Original: Suppose that the function w (cid:55)−→ f ( w ; z ) is L -Lipschitz and µ -strongly convex.
Revised: Suppose that the function w (cid:55)−→ f ( w ; z ) is L -Lipschitz and µ -strongly convex.

Original: For any δ ∈ (0 , 1) , the following inequality holds with probability at least 1 − δ
Revised: For any δ ∈ (0 , 1) , the following inequality holds with probability at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: From Algorithm 1, we know that (cid:101) F S i is λ
Revised: From Algorithm 1, we know that (cid:101) F S i is λ

Original: i := 2 / ( γ i n ) -strongly convex.
Revised: i := 2 / ( γ i n ) -strongly convex.

Original: According to
Revised: According to

Original: Lemma D.4, the following inequality holds with probability at least 1 − δ
Revised: Lemma D.4, the following inequality holds with probability at least 1 − δ

Original: [Equation]
Revised: [Equation]

Original: It then follows from the deﬁnition of ˆ w i and the strong convexity that
Revised: It then follows from the definition of ˆ w i and the strong convexity that

Original: [Equation]
Revised: [Equation]

Original: and therefore
Revised: and therefore

Original: [Equation]
Revised: [Equation]

Original: The proof is completed.
Revised: The proof is completed.

Original: Lemma D.6 (Bousquet & Elisseeff 2002) .
Revised: Lemma D.6 (Bousquet & Elisseeff 2002) .

Original: Suppose the function f : W × Z (cid:55)→ R takes a structure f = (cid:96) + r , where (cid:96) : W × Z (cid:55)→ R and r : W (cid:55)→ R .
Revised: Suppose the function f : W × Z (cid:55)→ R takes a structure f = ℓ + r , where ℓ : W × Z (cid:55)→ R and r : W (cid:55)→ R .

Original: Assume for all z , we have (cid:107)∇ (cid:96) ( w ; z ) (cid:107) ≤ L .
Revised: Assume for all z , we have ∥∇ ℓ ( w ; z ) ∥ ≤ L .

Original: Suppose F S = 1 n (cid:80) ni
Revised: Suppose F S = 1 n (cid:80) ni

Original: =1 f ( w ; z i ) is µ -strongly convex and deﬁne A as A ( S ) =
Revised: =1 f ( w ; z i ) is µ -strongly convex and define A as A ( S ) =

Original: arg min w ∈W F S ( w ) .
Revised: arg min w ∈W F S ( w ) .

Original: Then A is 4 L 2 nµ -uniformly stable.
Revised: Then A is 4 L 2 nµ -uniformly stable.

Original: Lemma D.7.
Revised: Lemma D.7.

Original: Assume for any z, w (cid:55)→ f ( w ; z ) is L -Lipschitz.
Revised: Assume for any z, w (cid:55)→ f ( w ; z ) is L -Lipschitz.

Original: Let ˆ w i be deﬁned in Eq.
Revised: Let ˆ w i be defined in Eq.

Original: (D.9) .
Revised: (D.9) .

Original: With probability at least 1 − δ/ (2 m ) we have the following inequality uniformly for any w
Revised: With probability at least 1 − δ/ (2 m ) we have the following inequality uniformly for any w

Original: [Equation]
Revised: [Equation]

Original: where k = (cid:100) log 2 n (cid:101) .
Revised: where k = ⌈ log 2 n ⌉ .

Original: Proof.
Revised: Proof.

Original: For any i , deﬁne
Revised: For any i , define

Original: [Equation]
Revised: [Equation]

Original: and w ∗ i =
Revised: and w ∗ i =

Original: arg min w (cid:101) F i ( w ) , where we assume z is independently drawn from the stationary distribution of mixing sequence.
Revised: arg min w (cid:101) F i ( w ) , where we assume z is independently drawn from the stationary distribution of mixing sequence.

Original: Let A i be the algorithm outputting the minimizer of (cid:101) F S
Revised: Let A i be the algorithm outputting the minimizer of (cid:101) F S

Original: i .
Revised: i .

Original: We know (cid:101) F S i is λ i =
Revised: We know (cid:101) F S i is λ i =

Original: 2 / ( γ i n ) -strongly convex.
Revised: 2 / ( γ i n ) -strongly convex.

Original: Then analysis similar to Corollary 6 implies the stated inequality with probability at least 1 − δ/ (2 m ) .
Revised: Then analysis similar to Corollary 6 implies the stated inequality with probability at least 1 − δ/ (2 m ) .

Original: The proof is completed.
Revised: The proof is completed.

Original: Based on the above lemmas, we now turn to proving Corollary 9.
Revised: Based on the above lemmas [Equation] we now turn to proving Corollary 9.

Original: Proof of Corollary 9.
Revised: Proof of Corollary 9.

Original: Let ˆ w 0 = w ∗ and ˆ w i be deﬁned by Eq.
Revised: Let ˆ w 0 = w ∗ and ˆ w i be defined by Eq.

Original: (D.9).
Revised: (D.9).

Original: We can decompose F ( w m ) −
Revised: We can decompose F ( w m ) −

Original: F ( w ∗ ) by
Revised: F ( w ∗ ) by

Original: [Equation]
Revised: [Equation]

Original: Since f is L lipschitz, Lemma D.5 implies the following inequality with probability at least 1 − δ/ (2 m )
Revised: Since f is L lipschitz [Equation] Lemma D.5 implies the following inequality with probability at least 1 − δ/ (2 m )

Original: [Equation]
Revised: [Equation]

Original: Furthermore, we can apply Lemma D.7 with w = ˆ w i − 1 to derive the following inequality with probability at least 1 − δ/ (2 m )
Revised: Furthermore [Equation] we can apply Lemma D.7 with w = ˆ w i − 1 to derive the following inequality with probability at least 1 − δ/ (2 m )

Original: [Equation]
Revised: [Equation]

Original: The following inequality then holds with probability at least 1 − δ/m (cid:88)
Revised: The following inequality then holds with probability at least 1 − δ/m (cid:88)

Original: [Equation]
Revised: [Equation]

Original: By Lemma D.5, we further get the following inequality with probability at least 1 − δ m (cid:88) i =
Revised: By Lemma D.

Original: [Equation]
Revised: [Equation]

Original: =
Revised: we further get the following inequality with probability at least 1 − δ m (cid:88)

Original: [Equation]
Revised: [Equation]

Original: i =

Original: [Equation]
Revised: [Equation]

Original: where in the last two steps we have used γ i
Revised: where in the last two steps we have used γ i

Original: =
Revised: =

Original: γ/ 2 i .
Revised: γ/ 2 i .

Original: We can plug the above inequality and Eq.
Revised: We can plug the above inequality and Eq.

Original: (D.11) back into Eq.
Revised: (D.11) back into Eq.

Original: (D.10), and derive the following inequality with probability 1 − δ
Revised: (D.10), and derive the following inequality with probability 1 − δ

Original: [Equation]
Revised: [Equation]

Original: This gives the stated bound.
Revised: This gives the stated bound.

Original: The proof is completed.
Revised: The proof is completed.

Original: E P ROOF OF L EMMA
Revised: E P ROOF OF L EMMA

Original: Proof of Lemma 3.
Revised: Proof of Lemma 3.

Original: For simplicity, we only consider discrete random variables.
Revised: For simplicity [Equation] we only consider discrete random variables.

Original: Let A ∈ σ n − k
Revised: Let A ∈ σ n − k

Original: −∞ , B ∈ σ ∞ n + k and Z n , z ∈ σ nn .
Revised: −∞ , B ∈ σ ∞ n + k and Z n , z ∈ σ nn .

Original: According to the deﬁnition of mixing sequence, we know
Revised: According to the definition of mixing sequence [Equation] we know

Original: [Equation]
Revised: [Equation]

Original: By Eq.
Revised: By Eq.

Original: (E.1) we know
Revised: (E.1) we know

Original: [Equation]
Revised: [Equation]

Original: It then follows that
Revised: It then follows that

Original: [Equation]
Revised: [Equation]

Original: In a similar way, one can show
Revised: In a similar way, one can show

Original: [Equation]
Revised: [Equation]

Original: The proof is completed by combining the above two inequalities together
Revised: The proof is completed by combining the above two inequalities together.

Original: [Equation]