Original: Fast ConvergenceRates of Random Reshuffling under Interpolation and the Polyak-Łojasiewicz Condition
Revised: Fast Convergence of Random Reshuffling under Interpolation and the Polyak-Łojasiewicz Condition

Original: Abstract
Revised: Abstract

Original: Modern machine learning models are often over-parameterized and as a result they can interpolate the training data.
Revised: Modern machine learning models are often over-parameterized and as a result they can interpolate the training data.

Original: Under such a scenario, we study the convergence properties of a samplingwithout-replacement variant of Stochastic Gradient Descent (SGD), known as Random Reshuffling ( RR ).
Revised: Under such a scenario, we study the convergence properties of a samplingwithout-replacement variant of Stochastic Gradient Descent (SGD), known as Random Reshuffling ( RR ).

Original: Unlike SGD that samples data with replacement at every iteration, RR chooses a random permutation of data at the beginning of each epoch.
Revised: Unlike SGD that samples data with replacement at every iteration, RR chooses a random permutation of data at the beginning of each epoch.

Original: For under-parameterized models, it has been recently shown that RR converges faster than SGD when the number of epochs is larger than the condition number ( κ ) of the problem under standard assumptions like strong convexity.
Revised: For under-parameterized models, it has been recently shown that RR converges faster than SGD when the number of epochs is larger than the condition number ( κ ) of the problem under standard assumptions like strong convexity.

Original: However, previous works do not show that RR outperforms SGD under interpolation for strongly convex objectives.
Revised: However, previous works do not show that RR outperforms SGD under interpolation for strongly convex objectives.

Original: Here, we show that for the class of Polyak-Łojasiewicz (PL) functions that generalizes strong convexity, RR can outperform SGD as long as the number of samples ( n ) is less than the parameter ( ρ ) of a strong growth condition (SGC).
Revised: Here, we show that for the class of Polyak-Łojasiewicz (PL) functions that generalizes strong convexity, RR can outperform SGD as long as the number of samples ( n ) is less than the parameter ( ρ ) of a strong growth condition (SGC).

Original: 1. Introduction
Revised: 1. Introduction

Original: We consider finite-sum minimization problems of the form
Revised: We consider finite-sum minimization problems of the form

Original: [Equation]
Revised: [Equation]

Original: Stochastic gradient descent (SGD) is a popular algorithm for solving machine learning problems of this form.
Revised: Stochastic gradient descent (SGD) is a popular algorithm for solving machine learning problems of this form.

Original: A significant amount of effort has been made to understand its theoretical and empirical properties (Bottou et al., 2018).
Revised: A significant amount of effort has been made to understand its theoretical and empirical properties (Bottou et al., 2018).

Original: SGD has a simple update rule in which a sample i k is chosen randomly with replacement at each iteration to compute x k +

Revised: SGD has a simple update rule in which a sample i k is chosen randomly with replacement at each iteration to compute x k +

Original: = x
Revised: = x

Original: k − η k ∇ f ( x k ; i k ) .
Revised: k − η k ∇ f ( x k ; i k ) .

Original: This is cheaper than using the full gradient at each iteration.
Revised: This is cheaper than using the full gradient at each iteration.

Original: However, it is well known that the convergence rates of
Revised: However, it is well known that the convergence rates of

Original: SGD, O ( 1 k ) and O ( 1 √ k ) for strongly-convex and convex objectives respectively (Nemirovski et al., 2009), are worse than those of full gradient descent.
Revised: SGD, O ( 1 k ) and O ( 1 √ k ) for strongly-convex and convex objectives respectively (Nemirovski et al., 2009), are worse than those of full gradient descent.

Original: Given the increasing complexity of modern learning models, a practically-relevant question to ask is how SGD performs in over-parameterized settings, under which the model fits or interpolates the data completely.
Revised: Given the increasing complexity of modern learning models, a practically-relevant question to ask is how SGD performs in over-parameterized settings, under which the model fits or interpolates the data completely.

Original: Previously, it has been shown that SGD can achieve a linear convergence rate like full gradient descent under various interpolation conditions for strongly-convex functions (Moulines and Bach, 2011, Needell et al., 2014, Schmidt and Roux, 2013).
Revised: Previously, it has been shown that SGD can achieve a linear convergence rate like full gradient descent under various interpolation conditions for strongly-convex functions (Moulines and Bach, 2011, Needell et al., 2014, Schmidt and Roux, 2013).

Original: An assumption that is weaker than strong convexity which allows full gradient descent to achieve a linear rate is the Polyak-Łojasiewicz (PL) condition (Polyak, 1963).
Revised: An assumption that is weaker than strong convexity which allows full gradient descent to achieve a linear rate is the Polyak-Łojasiewicz (PL) condition (Polyak, 1963).

Original: Recently, the PL condition has gained
Revised: Recently, the PL condition has gained

Original: [Table]
Revised: [Table]

Original: This work (SGC)
Revised: This work (SGC)

Original: [Table]
Revised: [Table]

Original: popularity in machine learning (Karimi et al., 2016) and it has been shown that several overparameterized models that interpolate the data satisfy the PL condition (Bassily et al., 2018, Oymak and
Revised: popularity in machine learning (Karimi et al., 2016) and it has been shown that several overparameterized models that interpolate the data satisfy the PL condition (Bassily et al., 2018, Oymak and

Original: Soltanolkotabi, 2019, Soltanolkotabi et al., 2018).
Revised: Soltanolkotabi, 2019, Soltanolkotabi et al., 2018).

Original: Notably, under interpolation and the PL condition
Revised: Notably, under interpolation and the PL condition

Original: SGD can achieve a linear rate similar to full gradient descent (Bassily et al., 2018, Vaswani et al., 2019).
Revised: SGD can achieve a linear rate similar to full gradient descent (Bassily et al., 2018, Vaswani et al., 2019).

Original: This paper shows that further convergence gains can be achieved under the PL condition and an interpolation assumption when a sampling-without-replacement counterpart of SGD, known as
Revised: This paper shows that further convergence gains can be achieved under the PL condition and an interpolation assumption when a sampling-without-replacement counterpart of SGD, known as

Original: Random Reshuffling ( RR ) is used.
Revised: Random Reshuffling ( RR ) is used.

Original: RR has long been known to converge faster than SGD empirically for certain problems (Bottou, 2009, 2012).
Revised: RR has long been known to converge faster than SGD empirically for certain problems (Bottou, 2009, 2012).

Original: However, analyzing RR is more difficult than SGD because (conditioned on the past iterates) each individual gradient is no longer an unbiased estimate of the full gradient.
Revised: However, analyzing RR is more difficult than SGD because (conditioned on the past iterates) each individual gradient is no longer an unbiased estimate of the full gradient.

Original: Thus, the analysis of RR has only emerged in a series of recent efforts (G ¨ urb ¨ uzbalaban et al., 2021, Haochen and Sra, 2019, Nagaraj et al., 2019, Safran and Shamir, 2020).
Revised: Thus, the analysis of RR has only emerged in a series of recent efforts (G ¨ urb ¨ uzbalaban et al., 2021, Haochen and Sra, 2019, Nagaraj et al., 2019, Safran and Shamir, 2020).

Original: Previous works have shown that
Revised: Previous works have shown that

Original: RR outperforms SGD for strongly-convex objectives in various under-parameterized settings, when the the number of epochs ( T ) is larger than the condition number of the problem ( κ ).
Revised: RR outperforms SGD for strongly-convex objectives in various under-parameterized settings, when the the number of epochs ( T ) is larger than the condition number of the problem ( κ ).

Original: However, in the over-parameterized settings current convergence rate analyses do not show that RR is faster than SGD (see Table 1).
Revised: However, in the over-parameterized settings current convergence rate analyses do not show that RR is faster than SGD (see Table 1).

Original: In this work, we address these questions by analyzing RR (and IG) for
Revised: In this work, we address these questions by analyzing RR (and IG) for

Original: PL functions under the strong growth condition (SGC) interpolation condition (Schmidt and Roux, 2013).
Revised: PL functions under the strong growth condition (SGC) interpolation condition (Schmidt and Roux, 2013).

Original: Under the SGC, we give an explicit convergence rate for RR (and IG) that can be faster than the best known rate for SGD.
Revised: Under the SGC, we give an explicit convergence rate for RR (and IG) that can be faster than the best known rate for SGD.

Original: These advantages of RR do not exist for current analyses of convex or strongly-convex functions.
Revised: These advantages of RR do not exist for current analyses of convex or strongly-convex functions.

Original: 1.1. Problem Statement
Revised: 1.1. Problem Statement

Original: The update rule of Random Reshuffling ( RR ) is as follows:
Revised: The update rule of Random Reshuffling ( RR ) is as follows:

Original: [Equation]
Revised: [Equation]

Original: That is, at each epoch t ∈ { 1 , 2 , ..., T } , π t is chosen randomly from the set of all possible permutations, and π tj +1 is sampled without replacement from the set { 1 , 2 , ..., n } for j ∈ { 0 , 1 , ..., n − 1 } .
Revised: That is, at each epoch t ∈ { 1 , 2 , ..., T } , π t is chosen randomly from the set of all possible permutations, and π tj +1 is sampled without replacement from the set { 1 , 2 , ..., n } for j ∈ { 0 , 1 , ..., n − 1 } .

Original: Note that x 0 ≜ x 10 is the initialization and x t + = x tn ∀ t ≥ 1 .
Revised: Note that x 0 ≜ x 10 is the initialization and x t + = x tn ∀ t ≥ 1 .

Original: A deterministic counterpart of RR is the Incremental Gradient method (IG), where π t is deterministic and fixed over all epochs.
Revised: A deterministic counterpart of RR is the Incremental Gradient method (IG), where π t is deterministic and fixed over all epochs.

Original: For IG, we take the ordering to be cyclic, i.e. π tj +1 = j + 1 .
Revised: For IG, we take the ordering to be cyclic, i.e. π tj +1 = j + 1 .

Original: 1.2. Assumptions
Revised: 1.2. Assumptions

Original: In this section, we present the assumptions made in our analyses.
Revised: In this section, we present the assumptions made in our analyses.

Original: First, we give the definition of the
Revised: First, we give the definition of the

Original: Polyak-Łojasiewicz (PL) inequality
Revised: Polyak-Łojasiewicz (PL) inequality

Original: Assumption 1 ( µ -PL)
Revised: Assumption 1 ( µ -PL)

Original: There exists some µ > 0 such that
Revised: There exists some µ > 0 such that

Original: [Equation]
Revised: [Equation]

Original: where f ∗ is the optimal value of f .
Revised: where f ∗ is the optimal value of f .

Original: PL implies that every stationary point of f is a global minimum, but is weaker than strong convexity.
Revised: PL implies that every stationary point of f is a global minimum, but is weaker than strong convexity.

Original: The textbook example of a non-strongly convex function that satisfies PL is that of least-squares (see
Revised: The textbook example of a non-strongly convex function that satisfies PL is that of least-squares (see

Original: Karimi et al., 2016).
Revised: Karimi et al., 2016).

Original: More recent literature shows that PL condition holds for more complex overparameterized models, including several classes of neural networks that interpolate under square loss (Soltanolkotabi et al., 2018).
Revised: More recent literature shows that PL condition holds for more complex overparameterized models, including several classes of neural networks that interpolate under square loss (Soltanolkotabi et al., 2018).

Original: Second, we assume that the individual functions are smooth.
Revised: Second, we assume that the individual functions are smooth.

Original: Assumption 

Revised: Assumption

Original: The objective f is L -smooth and each individual loss f ( · ; i ) is L i -smooth such that ∀ x, x
Revised: The objective f is L -smooth and each individual loss f ( · ; i ) is L i -smooth such that ∀ x, x

Original: [Equation]
Revised: [Equation]

Original: Denote L max = max i
Revised: Denote L max = max i

Original: L i .
Revised: L i .

Original: We assume f is lower bounded by f ∗ , which is achieved at some x ∗ , so f ∗ = f
Revised: We assume f is lower bounded by f ∗ , which is achieved at some x ∗ , so f ∗ = f

Original: ( x ∗ ) .
Revised: ( x ∗ ) .

Original: We also assume that each f ( · ; i ) is lower bounded by some f ∗ i .
Revised: We also assume that each f ( · ; i ) is lower bounded by some f ∗ i .

Original: Next, we formally define interpolation.
Revised: Next, we formally define interpolation.

Original: Assumption 3 We are in the interpolating regime, which we take to mean that
Revised: Assumption 3 We are in the interpolating regime, which we take to mean that

Original: [Equation]
Revised: [Equation]

Original: Thus, by interpolation we mean that stationary points with respect to the function f are also stationary points with respect to the individual functions f ( · ; i ) .
Revised: Thus, by interpolation we mean that stationary points with respect to the function f are also stationary points with respect to the individual functions f ( · ; i ) .

Original: Finally, we introduce the strong growth condition, which is stronger than interpolation as it implies Assumption 3.
Revised: Finally, we introduce the strong growth condition, which is stronger than interpolation as it implies Assumption 3.

Original: Assumption 4 (SGC)
Revised: Assumption 4 (SGC)

Original: There exists a constant ρ ≥ 1 such that the following holds
Revised: There exists a constant ρ ≥ 1 such that the following holds

Original: [Equation]
Revised: [Equation]

Original: For smooth functions in interpolating settings, the SGC is related to PL.
Revised: For smooth functions in interpolating settings, the SGC is related to PL.

Original: Concretely: under interpolation, a smooth and PL objective f also satisfies the SGC with a ρ that is at most L max µ (Vaswani et al., 2019, Prop. 2).
Revised: Concretely: under interpolation, a smooth and PL objective f also satisfies the SGC with a ρ that is at most L max µ (Vaswani et al., 2019, Prop. 2).

Original: There are also other interesting function classes, not necessarily PL, that satisfy the SGC.
Revised: There are also other interesting function classes, not necessarily PL, that satisfy the SGC.

Original: The proposition below extends (Vaswani et al., 2019, Lem. 1) to a function class that includes squared-hinge or logistic losses as special cases.
Revised: The proposition below extends (Vaswani et al., 2019, Lem. 1) to a function class that includes squared-hinge or logistic losses as special cases.

Original: Proposition 1 Assume binary linearly-separable dataset ( a i , y i ) , i ∈ [ n ] of size n with margin τ := arg max ∥ x ∥ 2 =1 min i ∈ [ n ] y i a Ti x > 0 , normalized features ∥ a i ∥ 2 ≤ 1 and y i ∈ {± 1 } .
Revised: Proposition 1 Assume binary linearly-separable dataset ( a i , y i ) , i ∈ [ n ] of size n with margin τ := arg max ∥ x ∥ 2 =1 min i ∈ [ n ] y i a Ti x > 0 , normalized features ∥ a i ∥ 2 ≤ 1 and y i ∈ {± 1 } .

Original: Let f ( x ; i ) =
Revised: Let f ( x ; i ) =

Original: ℓ ( y i a Ti x ) for a smooth monotonic function ℓ .
Revised: ℓ ( y i a Ti x ) for a smooth monotonic function ℓ .

Original: Then, SGC (6) holds with ρ = n/τ 2 .
Revised: Then, SGC (6) holds with ρ = n/τ 2 .

Original: We also consider a relaxation of the SGC that does not require the data to be fit exactly (Cevher and
Revised: We also consider a relaxation of the SGC that does not require the data to be fit exactly (Cevher and

Original: V˜u, 2019, Polyak and Tsypkin, 1973).
Revised: V˜u, 2019, Polyak and Tsypkin, 1973).

Original: Assumption 5 There exists constants ρ ≥ 0 and σ ≥ 0 such that the following holds: ∀ x ∈ dom( f ) ,
Revised: Assumption 5 There exists constants ρ ≥ 0 and σ ≥ 0 such that the following holds: ∀ x ∈ dom( f ) ,

Original: [Equation]
Revised: [Equation]

Original: Assumption 5 reduces to the SGC when σ = 0 , and reduces to the bounded variance assumptionn (cid:80) ni
Revised: Assumption 5 reduces to the SGC when σ = 0 , and reduces to the bounded variance assumptionn (cid:80) ni

Original: =1 ∥∇ f
Revised: =1 ∥∇ f

Original: ( x ; i ) − ∇ f ( x ) ∥ 2 ≤ σ 2 when ρ = 1 , which is commonly used in the analysis of SGD (Bottou et al., 2018).
Revised: ( x ; i ) − ∇ f ( x ) ∥ 2 ≤ σ 2 when ρ = 1 , which is commonly used in the analysis of SGD (Bottou et al., 2018).

Original: 2. Related Work Optimization under the PL condition
Revised: 2. Related Work Optimization under the PL condition

Original: The PL inequality (Assumption 1) was first explored by Polyak (1963) and Lojasiewicz (1963).
Revised: The PL inequality (Assumption 1) was first explored by Polyak (1963) and Lojasiewicz (1963).

Original: It applies to a wide range of important machine learning problems such as least square and logistic regression (over a compact set)
Revised: It applies to a wide range of important machine learning problems such as least square and logistic regression (over a compact set)

Original: (Karimi et al., 2016).
Revised: (Karimi et al., 2016).

Original: More generally, any function of the form f ( x )
Revised: More generally, any function of the form f ( x )

Original: =
Revised: =

Original: g ( Ax ) for a matrix A with a g being µ -strongly convex satisfies the µ -PL condition.
Revised: g ( Ax ) for a matrix A with a g being µ -strongly convex satisfies the µ -PL condition.

Original: Some over-parametrized models such as deep neural networks may contain stationary points that are sub-optimal, which is incompatible with the PL assumption.
Revised: Some over-parametrized models such as deep neural networks may contain stationary points that are sub-optimal, which is incompatible with the PL assumption.

Original: Nevertheless, several works have argued that considering a local PL condition around minimizers can be used as a model for analyzing the effectiveness of SGD in training deep neural networks (Liu et al., 2022, Oymak and Soltanolkotabi, 2019).
Revised: Nevertheless, several works have argued that considering a local PL condition around minimizers can be used as a model for analyzing the effectiveness of SGD in training deep neural networks (Liu et al., 2022, Oymak and Soltanolkotabi, 2019).

Original: Polyak (1963) showed that full gradient descent can achieve linear convergence rate under the PL condition.
Revised: Polyak (1963) showed that full gradient descent can achieve linear convergence rate under the PL condition.

Original: But it has recently been highlighted that the
Revised: But it has recently been highlighted that the

Original: PL condition can be used to show linear convergence rates of a variety of methods (Karimi et al., 2016).
Revised: PL condition can be used to show linear convergence rates of a variety of methods (Karimi et al., 2016).

Original: Typically, the PL condition leads to similar convergence rates as those obtained under the stronger condition of strong convexity.
Revised: Typically, the PL condition leads to similar convergence rates as those obtained under the stronger condition of strong convexity.

Original: In the case of SGD under interpolation, it has been shown that the rate of SGD under the PL condition is linear (Bassily et al., 2018, Vaswani et al., 2019).
Revised: In the case of SGD under interpolation, it has been shown that the rate of SGD under the PL condition is linear (Bassily et al., 2018, Vaswani et al., 2019).

Original: However, the convergence rates for SGD under interpolation for µ -PL functions are slower than those for strongly convex functions (see Table 1).
Revised: However, the convergence rates for SGD under interpolation for µ -PL functions are slower than those for strongly convex functions (see Table 1).

Original: RR for Under-Parameterized Problems
Revised: RR for Under-Parameterized Problems

Original: The difficulty in the analysis of RR arises because of the bias in the conditional expectation of gradients, i.e. E (cid:2) ∇ f ( x t i ; π t i +1 )
Revised: The difficulty in the analysis of RR arises because of the bias in the conditional expectation of gradients, i.e. E (cid:2) ∇ f ( x t i ; π t i +1 )

Original: | x t i (cid:3) ̸ = ∇ f ( x t i ) .
Revised: | x t i (cid:3) ̸ = ∇ f ( x t i ) .

Original: An early attempt to analyze RR by Recht and R´e (2012) was not successful because their noncommutative arithmetic-geometric mean inequality conjecture was proven to be false (Lai and Lim, 2020).
Revised: An early attempt to analyze RR by Recht and R´e (2012) was not successful because their noncommutative arithmetic-geometric mean inequality conjecture was proven to be false (Lai and Lim, 2020).

Original: The non-asymptotic convergence properties of RR have only been addressed recently.
Revised: The non-asymptotic convergence properties of RR have only been addressed recently.

Original: Haochen and Sra
Revised: Haochen and Sra

Original: (2019) give the first convergence result of ˜ O
Revised: (2019) give the first convergence result of ˜ O

Original: ( 1 n 2 T 2 + 1 T 3 ) , where T is the number of epochs.
Revised: ( 1 n 2 T 2 + 1 T 3 ) , where T is the number of epochs.

Original: The rate of RR was improved by Nagaraj et al.
Revised: The rate of RR was improved by Nagaraj et al.

Original: (2019) to ˜ O
Revised: (2019) to ˜ O

Original: ( 1 nT 2 ) when T ≳ κ 2 by assuming componentwise convexity of each f ( · ; i ) , with a matching lower bound of Ω( 1 nT 2 ) given by Rajput et al.
Revised: ( 1 nT 2 ) when T ≳ κ 2 by assuming componentwise convexity of each f ( · ; i ) , with a matching lower bound of Ω( 1 nT 2 ) given by Rajput et al.

Original: Note that this rate is faster than the ˜ O ( 1 nT ) rate of SGD when the large epoch requirement is satisfied.
Revised: Note that this rate is faster than the ˜ O ( 1 nT ) rate of SGD when the large epoch requirement is satisfied.

Original: Mishchenko et al.
Revised: Mishchenko et al.

Original: (2020) obtain the same rate of ˜ O
Revised: (2020) obtain the same rate of ˜ O

Original: ( 1 nT 2 )
Revised: ( 1 nT 2 )

Original: but only require T ≳ κ .
Revised: but only require T ≳ κ .

Original: Their analysis is also dependent on the underlying component-wise convexity structure.
Revised: Their analysis is also dependent on the underlying component-wise convexity structure.

Original: Ahn et al.
Revised: Ahn et al.

Original: (2020) remove this dependence and obtain the same rate with T ≳ κ .
Revised: (2020) remove this dependence and obtain the same rate with T ≳ κ .

Original: However, their analysis relies on each f ( · ; i ) being G -Lipschitz ( ∥∇ f ( · ; i ) ∥ ≤ G for all i ), which may require a constraint on problem (1) and a projection operation is needed to ensure the iterates are bounded (Ahn et al., 2020, Nguyen et al., 2021).
Revised: However, their analysis relies on each f ( · ; i ) being G -Lipschitz ( ∥∇ f ( · ; i ) ∥ ≤ G for all i ), which may require a constraint on problem (1) and a projection operation is needed to ensure the iterates are bounded (Ahn et al., 2020, Nguyen et al., 2021).

Original: Besides this, Nguyen et al.
Revised: Besides this, Nguyen et al.

Original: (2021) have given a unified analysis for shuffling schemes other than RR .
Revised: (2021) have given a unified analysis for shuffling schemes other than RR .

Original: Safran and Shamir (2020) have provided a lower bound of Ω( 1 T 2 + n 2 T 3 ) when f is a sum of n quadratics.
Revised: Safran and Shamir (2020) have provided a lower bound of Ω( 1 T 2 + n 2 T 3 ) when f is a sum of n quadratics.

Original: In a more recent work, they have shown that RR does not significantly improve over SGD unless T is larger than κ in the worst case (Safran and Shamir, 2021).
Revised: In a more recent work, they have shown that RR does not significantly improve over SGD unless T is larger than κ in the worst case (Safran and Shamir, 2021).

Original: Our analysis does not make the component-wise convexity or G-Lipschitzness assumption.
Revised: Our analysis does not make the component-wise convexity or G-Lipschitzness assumption.

Original: RR for Over-Parameterized Problems
Revised: RR for Over-Parameterized Problems

Original: Despite the widespread use of RR for training over-parame terized models, there is relatively little literature analyzing this setting.
Revised: Despite the widespread use of RR for training over-parame terized models, there is relatively little literature analyzing this setting.

Original: Haochen and Sra (2019) show that the convergence rate of RR is at least as fast as SGD under interpolation even without any epoch requirements.
Revised: Haochen and Sra (2019) show that the convergence rate of RR is at least as fast as SGD under interpolation even without any epoch requirements.

Original: However, the result of Haochen and Sra (2019) does not show that a gap in the rates can exist and only applies in the degenerate case where each function is strongly-convex.
Revised: However, the result of Haochen and Sra (2019) does not show that a gap in the rates can exist and only applies in the degenerate case where each function is strongly-convex.

Original: 1 We can also obtain results under interpolation as special cases of the results of Mishchenko et al.
Revised: 1 We can also obtain results under interpolation as special cases of the results of Mishchenko et al.

Original: (2020) and Nguyen et al.
Revised: (2020) and Nguyen et al.

Original: However, in the interpolation setting the rates obtained by these works for convex and strongly-convex functions are slower than the rate obtained by Vaswani et al.
Revised: However, in the interpolation setting the rates obtained by these works for convex and strongly-convex functions are slower than the rate obtained by Vaswani et al.

Original: (2019) for SGD (see Table 1).
Revised: (2019) for SGD (see Table 1).

Original: To our knowledge this is the first work to show RR can outperform SGD in an over-parameterized setting.
Revised: To our knowledge this is the first work to show RR can outperform SGD in an over-parameterized setting.

Original: 3. Contributions
Revised: 3. Contributions

Original: Our main contributions are summarized as follows:
Revised: Our main contributions are summarized as follows:

Original: • For µ -PL functions satisfying the SGC, we derive the sample complexity of RR to be ˜ O
Revised: • For µ -PL functions satisfying the SGC, we derive the sample complexity of RR to be ˜ O

Original: ( L max µ √ n √ ρ
Revised: ( L max µ √ n √ ρ

Original: + 1 + n ) .
Revised: + 1 + n ) .

Original: In comparison, the sample complexity of SGD in this case is ˜ O
Revised: In comparison, the sample complexity of SGD in this case is ˜ O

Original: ( Lµ ρ ) .
Revised: ( Lµ ρ ) .

Original: Hence, RR outperforms SGD when ρ ≫ n and L max ∼ L without requiring a large epoch.
Revised: Hence, RR outperforms SGD when ρ ≫ n and L max ∼ L without requiring a large epoch.

Original: The situation ρ ≫ n happens when there is a large amount of disagreements in the gradients.
Revised: The situation ρ ≫ n happens when there is a large amount of disagreements in the gradients.

Original: This is in clear contrast with the strongly convex case where existing bounds for RR do not show any gains over SGD.
Revised: This is in clear contrast with the strongly convex case where existing bounds for RR do not show any gains over SGD.

Original: • While the choice of RR over SGD under the SGC crucially depends on the relative magnitude of ρ and n , we show that RR consistently outperforms IG for both small and large ρ .
Revised: • While the choice of RR over SGD under the SGC crucially depends on the relative magnitude of ρ and n , we show that RR consistently outperforms IG for both small and large ρ .

Original: Moreover, IG can outperform SGD for the µ -PL case when ρ ≫ n 2 and L max ∼ L .
Revised: Moreover, IG can outperform SGD for the µ -PL case when ρ ≫ n 2 and L max ∼ L .

Original: 4. Theory Results
Revised: 4. Theory Results

Original: We present the convergence results of RR and IG for µ -PL objectives under SGC.
Revised: We present the convergence results of RR and IG for µ -PL objectives under SGC.

Original: Below, we use a constant step size η .
Revised: Below, we use a constant step size η .

Original: When comparing our results to SGD, we assume L max ∼ L , and denote κ =
Revised: When comparing our results to SGD, we assume L max ∼ L , and denote κ =

Original: L max µ ∼ Lµ .
Revised: L max µ ∼ Lµ .

Original: Theorem 1 ( µ -PL + RR ) Suppose Assumptions 1,2,5 hold, and choose a step size η ≤
Revised: Theorem 1 ( µ -PL + RR ) Suppose Assumptions 1,2,5 hold, and choose a step size η ≤

Original: [Equation]
Revised: [Equation]

Original: Then, we have for RR that
Revised: Then, we have for RR that

Original: [Equation]
Revised: [Equation]

Original: Further assuming SGC (Assumption 4), we obtain for η = 2 √ 2 L max √ n ( ρ +1+ n )
Revised: Further assuming SGC (Assumption 4), we obtain for η = 2 √ 2 L max √ n ( ρ +1+ n )

Original: that
Revised: that

Original: [Equation]
Revised: [Equation]

Original: With a proper choice of η , we can in fact translate (8) into a sample complexity of ˜ O
Revised: With a proper choice of η , we can in fact translate (8) into a sample complexity of ˜ O

Original: ( κ √ n √ ρ
Revised: ( κ √ n √ ρ

Original: + 1 + n + κ √ nσ √ µϵ )
Revised: + 1 + n + κ √ nσ √ µϵ )

Original: ; see Corollary 1 in Appendix.
Revised: ; see Corollary 1 in Appendix.

Original: Comparing this with ˜ O
Revised: Comparing this with ˜ O

Original: ( κρ + κ √ ρσ √ µϵ ) for SGD (Vaswani et al., 2019), we see that RR performs better than SGD provided n ≪ ρ .
Revised: ( κρ + κ √ ρσ √ µϵ ) for SGD (Vaswani et al., 2019), we see that RR performs better than SGD provided n ≪ ρ .

Original: Setting σ = 0 reduces to the result shown in Table 1.
Revised: Setting σ = 0 reduces to the result shown in Table 1.

Original: Theorem 2 ( µ -PL + IG) Suppose Assumptions 1,2,5 hold, and choose a step size η ≤
Revised: Theorem 2 ( µ -PL + IG) Suppose Assumptions 1,2,5 hold, and choose a step size η ≤

Original: [Equation]
Revised: [Equation]

Original: 2 nL max √ ρ .
Revised: 2 nL max √ ρ .

Original: Then, we have for IG that
Revised: Then, we have for IG that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Further assuming SGC (Assumption 4), set η = 1 2 nL max √ ρ to obtain f
Revised: Further assuming SGC (Assumption 4), set η = 1 2 nL max √ ρ to obtain f

Original: [Equation]
Revised: [Equation]

Original: The sample complexity of IG for the µ -PL case is ˜ O
Revised: The sample complexity of IG for the µ -PL case is ˜ O

Original: ( κn √ ρ + κnσ √ µϵ ) ; see Corollary 2 in the Appendix.
Revised: ( κn √ ρ + κnσ √ µϵ ) ; see Corollary 2 in the Appendix.

Original: Under SGC, this is worse than RR whether n < ρ or n ≥ ρ .
Revised: Under SGC, this is worse than RR whether n < ρ or n ≥ ρ .

Original: IG is also worse than RR in the case of σ ̸ = 

Revised: IG is also worse than RR in the case of σ ̸ =

Original: due to the extra factor of √ n in the 2 nd term of the sample complexity.
Revised: due to the extra factor of √ n in the 2 nd term of the sample complexity.

Original: On the other hand, IG can outperform SGD under SGC, provided that n 2 ≪ ρ .
Revised: On the other hand, IG can outperform SGD under SGC, provided that n 2 ≪ ρ .

Original: In summary, under SGC and PL,
Revised: In summary, under SGC and PL,

Original: RR always outperforms IG, and also, it outperforms SGD in ill-conditioned or low-sample regimes n ≪ ρ .
Revised: RR always outperforms IG, and also, it outperforms SGD in ill-conditioned or low-sample regimes n ≪ ρ .

Original: Moreover, when n 2 ≪ ρ IG outperforms SGD .
Revised: Moreover, when n 2 ≪ ρ IG outperforms SGD .

Original: 5. Conclusion
Revised: 5. Conclusion

Original: In this paper, we have derived convergence rates of Random Reshuffling under interpolation as implied by the SGC for µ -PL objectives.
Revised: In this paper, we have derived convergence rates of Random Reshuffling under interpolation as implied by the SGC for µ -PL objectives.

Original: In this setting, Random Reshuffling converges faster than
Revised: In this setting, Random Reshuffling converges faster than

Original: SGD provided the key condition ρ ≪ n helps.
Revised: SGD provided the key condition ρ ≪ n helps.

Original: Moreover, we show that IG can outperform SGD when ρ ≪ n 2 , and RR outperforms IG whether ρ is small or large.
Revised: Moreover, we show that IG can outperform SGD when ρ ≪ n 2 , and RR outperforms IG whether ρ is small or large.

Original: We remark that none of these conclusions follows from previous analysis under the strong convexity assumption.
Revised: We remark that none of these conclusions follows from previous analysis under the strong convexity assumption.

Original: Acknowledgement
Revised: Acknowledgement

Original: This research was partially supported by the Canada CIFAR AI Chair Program, the Natural Sciences and Engineering Research Council of Canada (NSERC)
Revised: This research was partially supported by the Canada CIFAR AI Chair Program, the Natural Sciences and Engineering Research Council of Canada (NSERC)

Original: Discovery
Revised: Discovery

Original: Grants RGPIN-2022-03669.
Revised: Grants RGPIN-2022-03669.

Original: Appendix A. Key Lemmas
Revised: Appendix A. Key Lemmas

Original: We draw ideas from Nguyen et al.
Revised: We draw ideas from Nguyen et al.

Original: (2021) and Mishchenko et al.
Revised: (2021) and Mishchenko et al.

Original: (2020) to construct our proofs.
Revised: (2020) to construct our proofs.

Original: The high-level idea is to first bound the decrease in the objective value (see Lemma 2), then bound the progression term (cid:80) n − 1 i =

Revised: The high-level idea is to first bound the decrease in the objective value (see Lemma 2), then bound the progression term (cid:80) n − 1 i =

Original: ∥ x ti − x t 0 ∥ 2 for IG and E [ (cid:80) n − 1 i =

Revised: ∥ x ti − x t 0 ∥ 2 for IG and E [ (cid:80) n − 1 i =

Original: ∥ x ti − x t 0 ∥ 2 ] for RR respectively (see Lemma 3 and Lemma 4).
Revised: ∥ x ti − x t 0 ∥ 2 ] for RR respectively (see Lemma 3 and Lemma 4).

Original: In this section, we present these lemmas that will be used in the theory proofs.
Revised: In this section, we present these lemmas that will be used in the theory proofs.

Original: Lemma 1 is a restatement of (Mishchenko et al., 2020, Lemma 1).
Revised: Lemma 1 is a restatement of (Mishchenko et al., 2020, Lemma 1).

Original: Lemma 1 Let X 1 , ..., X n be a given set of vectors in R d , denote their average to be ¯ X = 1 n (cid:80) ni
Revised: Lemma 1 Let X 1 , ..., X n be a given set of vectors in R d , denote their average to be ¯ X = 1 n (cid:80) ni

Original: =1 X i and population variance to be σ 2 = 1 n (cid:80) ni
Revised: =1 X i and population variance to be σ 2 = 1 n (cid:80) ni

Original: =

Revised: =

Original: ∥ X i − ¯ X ∥ 2 .
Revised: ∥ X i − ¯ X ∥ 2 .

Original: Fix k ∈ { 1 , ..., n } , let X
Revised: Fix k ∈ { 1 , ..., n } , let X

Original: π 1 , ..., X π k be sampled uniformly without replacement from { X 1 , ..., X n } and ¯ X π be their average.
Revised: π 1 , ..., X π k be sampled uniformly without replacement from { X 1 , ..., X n } and ¯ X π be their average.

Original: Then the following hold true
Revised: Then the following hold true

Original: [Equation]
Revised: [Equation]

Original: Lemma 2 is proved in (Nguyen et al., 2021, Lemma 8).
Revised: Lemma 2 is proved in (Nguyen et al., 2021, Lemma 8).

Original: Here we replace L with L max and take the step size η to be constant for all iterations and epochs.
Revised: Here we replace L with L max and take the step size η to be constant for all iterations and epochs.

Original: Lemma 2 Suppose Assumption 2 holds.
Revised: Lemma 2 Suppose Assumption 2 holds.

Original: Given a shuffling scheme { π t } t and a constant step size η such that η ≤ 1 nL , we have the following:
Revised: Given a shuffling scheme { π t } t and a constant step size η such that η ≤ 1 nL , we have the following:

Original: [Equation]
Revised: [Equation]

Original: Taking total expectation over the randomness
Revised: Taking total expectation over the randomness

Original: [Equation]
Revised: [Equation]

Original: The remaining Lemmas 3-4 concern the update rules in Eqn (2).
Revised: The remaining Lemmas 3-4 concern the update rules in Eqn (2).

Original: Recall that x t +
Revised: Recall that x t +

Original: [Equation]
Revised: [Equation]

Original: Also recall that for RR the permutation vector π t is chosen randomly at each epoch t while it is fixed for IG.
Revised: Also recall that for RR the permutation vector π t is chosen randomly at each epoch t while it is fixed for IG.

Original: Lemma 3 will be combined with Lemma 2 in the proofs of IG.
Revised: Lemma 3 will be combined with Lemma 2 in the proofs of IG.

Original: Lemma 3 Suppose Assumptions 2, 5 hold.
Revised: Lemma 3 Suppose Assumptions 2, 5 hold.

Original: For a step size η ≤ 1 √ 2 nL max , it holds for IG that
Revised: For a step size η ≤ 1 √ 2 nL max , it holds for IG that

Original: [Equation]
Revised: [Equation]

Original: Proof By the update
Revised: Proof By the update

Original: [Equation]
Revised: [Equation]

Original: Summing over i =
Revised: Summing over i =

Original: [Equation]
Revised: [Equation]

Original: ..., n − 1 gives n −
Revised: ..., n − 1 gives n −

Original: [Equation]
Revised: [Equation]

Original: Finally, choosing η ≤
Revised: Finally, choosing η ≤

Original: [Equation]
Revised: [Equation]

Original: leads to
Revised: leads to

Original: [Equation]
Revised: [Equation]

Original: Lemma 4 below will be combined with Lemma 2 in the proofs of RR.
Revised: Lemma 4 below will be combined with Lemma 2 in the proofs of RR.

Original: Lemma 4 Suppose Assumption 2, 5 hold.
Revised: Lemma 4 Suppose Assumption 2, 5 hold.

Original: For a step size η ≤ 1 √ 3 nL max , the following holds for RR
Revised: For a step size η ≤ 1 √ 3 nL max , the following holds for RR

Original: [Equation]
Revised: [Equation]

Original: Proof By the update rule
Revised: Proof By the update rule

Original: [Equation]
Revised: [Equation]

Original: Let σ t be a sigma algebra on the iterates { x 10 , ..., x t 0 } .
Revised: Let σ t be a sigma algebra on the iterates { x 10 , ..., x t 0 } .

Original: We take conditional expectation w.r.t σ t and apply Lemma 1 to find that
Revised: We take conditional expectation w.r.t σ t and apply Lemma 1 to find that

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: ∥
Revised: ∥

Original: [Equation]
Revised: [Equation]

Original: Take conditional expectation of (20) and substitute (21) back
Revised: Take conditional expectation of (20) and substitute (21) back

Original: [Equation]
Revised: [Equation]

Original: Next, take total expectation and sum over i = 0 , ..., n − 1 :
Revised: Next, take total expectation and sum over i = 0 , ..., n − 1 :

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: In (24), we have used (cid:80) n − 1 i =

Revised: In (24), we have used (cid:80) n − 1 i =

Original: i ≤ n 2 2 , (cid:80) n − 1 i =

Revised: i ≤ n 2 2 , (cid:80) n − 1 i =

Original: i 2 ≤ n 3 3 , and (cid:80) n − 1 i =

Revised: i 2 ≤ n 3 3 , and (cid:80) n − 1 i =

Original: i ( n − i ) ≤ n 2 ( n − 1).
Revised: i ( n − i ) ≤ n 2 ( n − 1).

Original: Choosing η ≤ 1 √ 3 L max n , we have
Revised: Choosing η ≤ 1 √ 3 L max n , we have

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: Appendix B. Convergence Proofs B.1. Proof of Proposition
Revised: Appendix B. Convergence Proofs B.1. Proof of Proposition

Original: Proof We consider smooth monotonic non-increasing functions of the form
Revised: Proof We consider smooth monotonic non-increasing functions of the form

Original: f i ( x ) =
Revised: f i ( x ) =

Original: l ( y i a Ti x ) =
Revised: l ( y i a Ti x ) =

Original: l ( z Ti x ) , where z i =
Revised: l ( z Ti x ) , where z i =

Original: y i
Revised: y i

Original: a i and a i is the feature vector for the i th sample.
Revised: a i and a i is the feature vector for the i th sample.

Original: We assume the values of z i are properly normalized such that max i ∥
Revised: We assume the values of z i are properly normalized such that max i ∥

Original: z i ∥ ≤ 1 .
Revised: z i ∥ ≤ 1 .

Original: Define x ∗ = argmax ∥ x ∥
Revised: Define x ∗ = argmax ∥ x ∥

Original: [Equation]
Revised: [Equation]

Original: min i z ti x and τ = max ∥ x ∥ 2 =1 min i
Revised: min i z ti x and τ = max ∥ x ∥ 2 =1 min i

Original: z Ti x .
Revised: z Ti x .

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: Note that for a vector u , its 2 -norm is ∥ u ∥ 2 = max ∥ v ∥ 2 =1 v T u .
Revised: Note that for a vector u , its 2 -norm is ∥ u ∥ 2 = max ∥ v ∥ 2 =1 v T u .

Original: Hence, we have the following
Revised: Hence, we have the following

Original: [Equation]
Revised: [Equation]

Original: where the inequality in (29) follows because l is monotonic, that is l ′ ( t 1 )
Revised: where the inequality in (29) follows because l is monotonic, that is l ′ ( t 1 )

Original: l ′
Revised: l ′

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: ≥ 0 ∀ t
Revised: ≥ 0 ∀ t

Original: [Equation]
Revised: [Equation]

Original: t 2 ∈ R
Revised: t 2 ∈ R

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have the following
Revised: Therefore, we have the following

Original: [Equation]
Revised: [Equation]

Original: where ρ = nτ 2 .
Revised: where ρ = nτ 2 .

Original: B.2. Proof of Theorem
Revised: B.2. Proof of Theorem

Original: Apply Lemma 2 and Lemma
Revised: Apply Lemma 2 and Lemma

Original: [Equation]
Revised: [Equation]

Original: Further use the PL inequality in Assumption 1 to arrive at
Revised: Further use the PL inequality in Assumption 1 to arrive at

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: f
Revised: f

Original: [Equation]
Revised: [Equation]

Original: x t +
Revised: x t +

Original: [Equation]
Revised: [Equation]

Original: Choosing η ≤ 2 √ 2 L max √ n ( ρ +1+ n ) gives
Revised: Choosing η ≤ 2 √ 2 L max √ n ( ρ +1+ n ) gives

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Solving (34) recursively for t = 1 , ..., T gives
Revised: Solving (34) recursively for t = 1 , ..., T gives

Original: [Equation]
Revised: [Equation]

Original: This implies (recalling that x 10 = x 0 ):
Revised: This implies (recalling that x 10 = x 0 ):

Original: [Equation]
Revised: [Equation]

Original: Note that for (36) to hold, we require η ≤ min { 2 √ 2 L max √ n (
Revised: Note that for (36) to hold, we require η ≤ min { 2 √ 2 L max √ n (

Original: [Equation]
Revised: [Equation]

Original: √ 3 L max n } because of Lemma 4.
Revised: √ 3 L max n } because of Lemma 4.

Original: This condition becomes η ≤ 2 √ 2 L max √ n ( ρ +1+ n ) as
Revised: This condition becomes η ≤ 2 √ 2 L max √ n ( ρ +1+ n ) as

Original: [Equation]
Revised: [Equation]

Original: Under
Revised: Under

Original: SGC, substitute σ = 0 with η = 2 √ 2 L max √ n ( ρ +1+ n ) to obtain the desired result.
Revised: SGC, substitute σ = 0 with η = 2 √ 2 L max √ n ( ρ +1+ n ) to obtain the desired result.

Original: B.3. Sample Complexity for Strongly-Convex Objective under RR
Revised: B.3. Sample Complexity for Strongly-Convex Objective under RR

Original: Corollary 1 Suppose Assumptions 1,2,5 hold.
Revised: Corollary 1 Suppose Assumptions 1,2,5 hold.

Original: Assume f is µ -strongly convex, choose η = min { 2 √ 2 L max √ n ( ρ +1+ n )
Revised: Assume f is µ -strongly convex, choose η = min { 2 √ 2 L max √ n ( ρ +1+ n )

Original: [Equation]
Revised: [Equation]

Original: 2 nµT log( ( f ( x 0 ) − f ( x ∗ )) µ 3 T 2 n L 2max σ) } , and define κ ≜ L max µ .
Revised: 2 nµT log( ( f ( x 0 ) − f ( x ∗ )) µ 3 T 2 n L 2max σ) } , and define κ ≜ L max µ .

Original: Then we have for RR
Revised: Then we have for RR

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: Proof For the case η =
Revised: Proof For the case η =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: substitute this η into
Revised: substitute this η into

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: For the case η =
Revised: For the case η =

Original: [Equation]
Revised: [Equation]

Original: , we have
Revised: , we have

Original: [Equation]
Revised: [Equation]

Original: Combining (38) and (39)
Revised: Combining (38) and (39)

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: [Equation]
Revised: [Equation]

Original: where ∆ 

Revised: where ∆

Original: = f ( x 0 ) − f ( x ∗ ) .
Revised: = f ( x 0 ) − f ( x ∗ ) .

Original: This translates to a sample complexity of ˜ O
Revised: This translates to a sample complexity of ˜ O

Original: ( κ √ n √ ρ
Revised: ( κ √ n √ ρ

Original: + 1 + n + κ √ nσ √ µϵ ) where logarithmic factors are ignored
Revised: + 1 + n + κ √ nσ √ µϵ ) where logarithmic factors are ignored

Original: [Equation]
Revised: [Equation]

Original: B.4. Proof of Theorem
Revised: B.4. Proof of Theorem

Original: Proof Apply Lemma 2 and Lemma
Revised: Proof Apply Lemma 2 and Lemma

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Choose η ≤ 1 2 nL max √ ρ and apply PL inequality in Assumption
Revised: Choose η ≤ 1 2 nL max √ ρ and apply PL inequality in Assumption

Original: [Equation]
Revised: [Equation]

Original: Solve (42) recursively
Revised: Solve (42) recursively

Original: [Equation]
Revised: [Equation]

Original: This implies
Revised: This implies

Original: [Equation]
Revised: [Equation]

Original: Note for (44) to hold, we require η ≤ min { 1 2 nL max √ ρ , 1 √ 2 nL max } as we used Lemma 3.
Revised: Note for (44) to hold, we require η ≤ min { 1 2 nL max √ ρ , 1 √ 2 nL max } as we used Lemma 3.

Original: This reduces to η ≤ 1 2 nL max √ ρ provided ρ ≥ 1 .
Revised: This reduces to η ≤ 1 2 nL max √ ρ provided ρ ≥ 1 .

Original: Under SGC, set σ = 0 and substitute η = 1 2 nL max √ ρ , we obtain the desired result.
Revised: Under SGC, set σ = 0 and substitute η = 1 2 nL max √ ρ , we obtain the desired result.

Original: B.5. Sample Complexity of Strongly-Convex Objective under IG
Revised: B.5. Sample Complexity of Strongly-Convex Objective under IG

Original: Corollary 2 Suppose Assumptions 1,2,5 hold.
Revised: Corollary 2 Suppose Assumptions 1,2,5 hold.

Original: Assume f is µ -strongly convex, set η = min { 1 2 nL max √ ρ , 2 nµT log( ( f ( x 0 ) − f ( x ∗ )) µ 3 T 2 L 2max σ) } , and define κ ≜ L max µ .
Revised: Assume f is µ -strongly convex, set η = min { 1 2 nL max √ ρ , 2 nµT log( ( f ( x 0 ) − f ( x ∗ )) µ 3 T 2 L 2max σ) } , and define κ ≜ L max µ .

Original: Then we have for IG
Revised: Then we have for IG

Original: [Equation]
Revised: [Equation]

Original: Proof First consider η =
Revised: Proof First consider η =

Original: [Equation]
Revised: [Equation]

Original: For η =
Revised: For η =

Original: [Equation]
Revised: [Equation]

Original: Combine (46) and (47) together
Revised: Combine (46) and (47) together

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where ∆ 

Revised: where ∆

Original: = f ( x 0 ) − f ( x ∗ ) .
Revised: = f ( x 0 ) − f ( x ∗ ) .

Original: This translates to a sample complexity of ˜ O
Revised: This translates to a sample complexity of ˜ O

Original: ( nκ √ ρ + nκσ √ µϵ ) where logarithmic factors are ignored
Revised: ( nκ √ ρ + nκσ √ µϵ ) where logarithmic factors are ignored

Original: [Equation]
Revised: [Equation]