Original: How  Sharpness-Aware Minimization Minimizes Sharpness?
Revised: How Does Sharpness-Aware Minimization Minimizes Sharpness?

Original: author names withheld

Original: Under Review for OPT

Original: Abstract
Revised: Abstract

Original: Sharpness-Aware Minimization (SAM) is a highly effective regularization technique for improving the generalization of deep neural networks for various settings.
Revised: Sharpness-Aware Minimization (SAM) is a highly effective regularization technique for improving the generalization of deep neural networks for various settings.

Original: However, the underlying working of SAM remains elusive because of various intriguing approximations in the theoretical characterizations.
Revised: However, the underlying working of SAM remains elusive because of various intriguing approximations in the theoretical characterizations.

Original: SAM intends to penalize a notion of sharpness of the model but implements a computationally efficient variant; moreover, a third notion of sharpness was used for proving generalization guarantees.
Revised: SAM intends to penalize a notion of sharpness of the model but implements a computationally efficient variant; moreover, a third notion of sharpness was used for proving generalization guarantees.

Original: The subtle differences in these notions of sharpness can indeed lead to significantly different empirical results.
Revised: The subtle differences in these notions of sharpness can indeed lead to significantly different empirical results.

Original: This paper rigorously nails down the exact sharpness notion that SAM regularizes and clarifies the underlying mechanism.
Revised: This paper rigorously nails down the exact sharpness notion that SAM regularizes and clarifies the underlying mechanism.

Original: We also show that the two steps of approximations in the original motivation of SAM individually lead to inaccurate local conclusions, but their combination accidentally reveals the correct effect, when full-batch gradients are applied.
Revised: We also show that the two steps of approximations in the original motivation of SAM individually lead to inaccurate local conclusions, but their combination accidentally reveals the correct effect, when full-batch gradients are applied.

Original: Furthermore, we also prove that the stochastic version of SAM in fact regularizes another notion of sharpness, which is most likely to be the preferred notion for practical performance.
Revised: Furthermore, we also prove that the stochastic version of SAM in fact regularizes another notion of sharpness, which is most likely to be the preferred notion for practical performance.

Original: The key mechanism behind this intriguing phenomenon is the implicit alignment between the gradient and the top eigenvector of Hessian when running SAM.
Revised: The key mechanism behind this intriguing phenomenon is the implicit alignment between the gradient and the top eigenvector of Hessian when running SAM.

Original: 1. Introduction
Revised: 1. Introduction

Original: Modern deep nets are often overparametrized and have the capacity to fit even randomly labeled data [24].
Revised: Modern deep nets are often overparametrized and have the capacity to fit even randomly labeled data [24].

Original: Thus, a small training loss does not necessarily imply good generalization.
Revised: Thus, a small training loss does not necessarily imply good generalization.

Original: Yet, standard gradient-based training algorithms such as SGD are able to find generalizable models.
Revised: Yet, standard gradient-based training algorithms such as SGD are able to find generalizable models.

Original: Recent empirical and theoretical studies suggest that generalization is well-correlated with the sharpness of the loss landscape at the learned parameter [6, 7, 13, 14, 21].
Revised: Recent empirical and theoretical studies suggest that generalization is well-correlated with the sharpness of the loss landscape at the learned parameter [6, 7, 13, 14, 21].

Original: Partly motivated by these studies,
Revised: Partly motivated by these studies,

Original: Foret et al.
Revised: Foret et al.

Original: [9], Wu et al.
Revised: [9], Wu et al.

Original: [23], Zheng et al.
Revised: [23], Zheng et al.

Original: [26] propose to penalize the sharpness of the landscape to improve the generalization.
Revised: [26] propose to penalize the sharpness of the landscape to improve the generalization.

Original: We refer this method to Sharpness-Aware Minimization (SAM) and focus on the version of Foret et al.
Revised: We refer this method to Sharpness-Aware Minimization (SAM) and focus on the version of Foret et al.

Original: [9] in this paper.
Revised: [9] in this paper.

Original: Despite its empirical success, the underlying working of SAM remains elusive because of the various intriguing approximations made in its derivation and analysis.
Revised: Despite its empirical success, the underlying working of SAM remains elusive because of the various intriguing approximations made in its derivation and analysis.

Original: There are three different notions of sharpness involved – SAM intends to optimize the first notion, the sharpness along the worst direction but actually implements a computationally efficient notion, the sharpness along the direction of the gradient.
Revised: There are three different notions of sharpness involved – SAM intends to optimize the first notion, the sharpness along the worst direction but actually implements a computationally efficient notion, the sharpness along the direction of the gradient.

Original: But in the analysis, a third notion of sharpness is actually used to prove generalization guarantees, which admits the first notion as an upper bound.
Revised: But in the analysis, a third notion of sharpness is actually used to prove generalization guarantees, which admits the first notion as an upper bound.

Original: The subtle difference between the three notions can lead to very different explicit biases.
Revised: The subtle difference between the three notions can lead to very different explicit biases.

Original: (see Figure 1 for demonstraion)
Revised: (see Figure 1 for demonstraion)

Original: More concretely, let L be the training loss, x be the parameter and ρ be the perturbation radius , a hyperparameter requiring tuning.
Revised: More concretely, let L be the training loss, x be the parameter and ρ be the perturbation radius , a hyperparameter requiring tuning.

Original: The first notion corresponds to the following optimization problem (1), where we call R max ρ
Revised: The first notion corresponds to the following optimization problem (1), where we call R max ρ

Original: [Equation]
Revised: [Equation]

Original: -direction sharpness at x and thus
Revised: the worst-direction sharpness at x and thus

Original: SAM is intended to minimize the original training loss plus the worst-direction sharpness at x .
Revised: SAM is intended to minimize the original training loss plus the worst-direction sharpness at x .

Original: [Equation]
Revised: [Equation]

Original:  where
Revised: x ) , where

Original: [Equation]
Revised: [Equation]

Original: © .

Original: [Table]
Revised: [Table]

Original: Symbol

Original: Definition

Original: Limiting RegularizersAmong
Revised: Limiting Regularizers

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: However, even evaluation of L max ρ ( x ) is computationally expensive, not to mention optimization.
Revised: However, even evaluation of L max ρ ( x ) is computationally expensive, not to mention optimization.

Original: Thus [9, 26] proposed to approximate the worst perturbation direction by the direction of the gradient and implement the second notion of sharpness, which corresponds to (2).
Revised: Thus [9, 26] proposed to approximate the worst perturbation direction by the direction of the gradient and implement the second notion of sharpness, which corresponds to (2).

Original: We call
Revised: We call

Original: R asc ρ ( x ) =
Revised: R asc ρ ( x ) =

Original: L asc ρ ( x ) − L ( x ) the ascent-direction sharpness at x .
Revised: L asc ρ ( x ) − L ( x ) the ascent-direction sharpness at x .

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: asc
Revised: asc

Original: [Equation]
Revised: [Equation]

Original: Intriguingly, in the generalization analysis of SAM by [9, 23], the first notion of sharpness, i.e. , the worst-direction sharpness, is only used for upper bounding the third notion of sharpness via the PAC Bayesian theory [20].
Revised: Intriguingly, in the generalization analysis of SAM by [9, 23], the first notion of sharpness, i.e. , the worst-direction sharpness, is only used for upper bounding the third notion of sharpness via the PAC Bayesian theory [20].

Original: We call the third notion R avg ρ ( x ) =
Revised: We call the third notion R avg ρ ( x ) =

Original: L avg ρ ( x ) − L ( x ) the averagedirection sharpness at x , where L avg ρ ( x ) =
Revised: L avg ρ ( x ) − L ( x ) the averagedirection sharpness at x , where L avg ρ ( x ) =

Original: E g ∼ N (0 ,I ) L ( x + ρg/ ∥ g ∥ ) .
Revised: E g ∼ N (0 ,I ) L ( x + ρg/ ∥ g ∥ ) .

Original: For further acceleration, Foret et al.
Revised: For further acceleration, Foret et al.

Original: [9], Zheng et al.
Revised: [9], Zheng et al.

Original: [26] omit the gradient through other occurrence of x and approximate the gradient of ascent-direction sharpness by gradient taken after one-step ascent, i.e. , ∇ L asc ρ ( x ) ≈ ∇ L (cid:16) x
Revised: [26] omit the gradient through other occurrence of x and approximate the gradient of ascent-direction sharpness by gradient taken after one-step ascent, i.e. , ∇ L asc ρ ( x ) ≈ ∇ L (cid:16) x

Original: + ρ ∇ L ( x ) ∥∇ L ( x ) ∥ 2 (cid:17) and derive the update rule of SAM, where η is the learning rate.
Revised: + ρ ∇ L ( x ) ∥∇ L ( x ) ∥ 2 (cid:17) and derive the update rule of SAM, where η is the learning rate.

Original: Sharpness-Aware Minimization (SAM):
Revised: Sharpness-Aware Minimization (SAM):

Original: [Equation]
Revised: [Equation]

Original: In this paper, we analyze the explicit bias of various notions of sharpness and the optimization trajectory of SAM.
Revised: In this paper, we analyze the explicit bias of various notions of sharpness and the optimization trajectory of SAM.

Original: Our analysis is performed for small perturbation radius ρ and learning rate η under the setting where the minimizers of loss form a manifold following [8, 16]
Revised: Our analysis is performed for small perturbation radius ρ and learning rate η under the setting where the minimizers of loss form a manifold following [8, 16]

Original: In particular, we make the following theoretical contributions.
Revised: In particular, we make the following theoretical contributions.

Original: We prove that full-batch SAM does minimize worst-direction sharpness.
Revised: We prove that full-batch SAM does minimize worst-direction sharpness.

Original: (Theorem 8) 2.
Revised: (Theorem 8) 2.

Original: Surprisingly, when batch size is 1, SAM minimizes average-direction sharpness.
Revised: Surprisingly, when batch size is 1, SAM minimizes average-direction sharpness.

Original: (Theorem 11) 3.
Revised: (Theorem 11) 3.

Original: We characterize the explicit biases of three notions of sharpness among minimizers when perturbation radius ρ goes to zero.
Revised: We characterize the explicit biases of three notions of sharpness among minimizers when perturbation radius ρ goes to zero.

Original: (Theorems 12 to 14, also see Table 1)
Revised: (Theorems 12 to 14, also see Table 1)

Original: Surprisingly, both heuristic approximations made for the update rule of SAM lead to inaccurate solutions, that is, (1) minimizing worst-direction sharpness and ascent-direction sharpness induce different biases among minimizers, and (2) SAM doesn’t minimize ascent-direction sharpness.
Revised: Surprisingly, both heuristic approximations made for the update rule of SAM lead to inaccurate solutions, that is, (1) minimizing worst-direction sharpness and ascent-direction sharpness induce different biases among minimizers, and (2) SAM doesn’t minimize ascent-direction sharpness.

Original: The key mechanism behind this implicit bias of SAM is an alignment phenomenon between the gradient and the top eigenvector of Hessian when running SAM.
Revised: The key mechanism behind this implicit bias of SAM is an alignment phenomenon between the gradient and the top eigenvector of Hessian when running SAM.

Original: 2. Notations and Assumptions
Revised: 2. Notations and Assumptions

Original: For any integer k , we define C k as the set of k times continuously differentiable functions.
Revised: For any integer k , we define C k as the set of k times continuously differentiable functions.

Original: For any mapping F , we define ∂F ( x )[ u ] and ∂ 2 F ( x )[ u, v ] as the first and second order directional derivative of F ( x ) along the direction u (and v ).
Revised: For any mapping F , we define ∂F ( x )[ u ] and ∂ 2 F ( x )[ u, v ] as the first and second order directional derivative of F ( x ) along the direction u (and v ).

Original: Given a differential submanifold Γ of R D and a point x ∈ Γ , define P x, Γ as the projection operator onto the manifold of the normal space of Γ at x and P ⊤ x, Γ = I D − P x, Γ .
Revised: Given a differential submanifold Γ of R D and a point x ∈ Γ , define P x, Γ as the projection operator onto the manifold of the normal space of Γ at x and P ⊤ x, Γ = I D − P x, Γ .

Original: We fix our initialization as x init and our loss function as L : R D
Revised: We fix our initialization as x init and our loss function as L : R D

Original: → R .
Revised: → R .

Original: Given the loss function, the gradient flow can be defined as mapping ϕ : R D × [0 , ∞ )
Revised: Given the loss function, the gradient flow can be defined as mapping ϕ : R D × [0 , ∞ )

Original: → R D satisfying ϕ
Revised: → R D satisfying ϕ

Original: ( x, τ ) =
Revised: ( x, τ ) =

Original: x − (cid:82) τ 0 ∇ L ( ϕ ( x, t ))
Revised: x − (cid:82) τ 0 ∇ L ( ϕ ( x, t ))

Original: dt .
Revised: dt .

Original: We further define the limiting map Φ : R D
Revised: We further define the limiting map Φ : R D

Original: → R D as Φ( x ) =
Revised: → R D as Φ( x ) =

Original: lim
Revised: lim

Original: τ →∞ ϕ
Revised: τ →∞ ϕ

Original: ( x, τ ) .
Revised: ( x, τ ) .

Original: For any positive definite symmetry matrix A ∈ R D × D , define { λ i ( A ) , v i ( A ) } i ∈
Revised: For any positive definite symmetry matrix A ∈ R D × D , define { λ i ( A ) , v i ( A ) } i ∈

Original: [ D ] as all its eigenvalues and eigenvectors satisfying λ 1 ( A ) ≥ λ 2 ( A ) ... ≥ λ 3 ( A ) and ∥ v i ( A ) ∥ 2 = 1 .
Revised: [ D ] as all its eigenvalues and eigenvectors satisfying λ 1 ( A ) ≥ λ 2 ( A ) ... ≥ λ 3 ( A ) and ∥ v i ( A ) ∥ 2 = 1 .

Original: Our analysis assumes sufficiently small η and ρ and uses O ( · ) to hide constant.
Revised: Our analysis assumes sufficiently small η and ρ and uses O ( · ) to hide constant.

Original: Following Arora et al.
Revised: Following Arora et al.

Original: [2], Fehrman et al.
Revised: [2], Fehrman et al.

Original: [8], Li et al.
Revised: [8], Li et al.

Original: [16], we make the below assumption.
Revised: [16], we make the below assumption.

Original: Assumption 1 Assume loss L : R D
Revised: Assumption 1 Assume loss L : R D

Original: → R belongs to C 4 , and there exists a manifold Γ that is
Revised: → R belongs to C 4 , and there exists a manifold Γ that is

Original: D − M dimensional C 2 − submanifold of R D for some integer 1 ≤ M ≤ D , where for all x ∈ Γ , x is a global minimizer of L , L ( x ) = 0 and rank ( ∇ 2 L ( x ))
Revised: D − M dimensional C 2 − submanifold of R D for some integer 1 ≤ M ≤ D , where for all x ∈ Γ , x is a global minimizer of L , L ( x ) = 0 and rank ( ∇ 2 L ( x ))

Original: =
Revised: =

Original: M .
Revised: M .

Original: The smoothness assumption is met with networks with smooth activation functions and the existence of the manifold is due to the vast overparameterization of the modern neural network.
Revised: The smoothness assumption is met with networks with smooth activation functions and the existence of the manifold is due to the vast overparameterization of the modern neural network.

Original: The full rank assumption is necessary for the analysis to guarantee the differentiability of Φ .
Revised: The full rank assumption is necessary for the analysis to guarantee the differentiability of Φ .

Original: Let
Revised: Let

Original: U = { x ∈ R D | Φ( x ) exists and Φ( x ) ∈ Γ } .
Revised: U = { x ∈ R D | Φ( x ) exists and Φ( x ) ∈ Γ } .

Original: Assumption 1 implies that U is open and Φ is in C 3 on
Revised: Assumption 1 implies that U is open and Φ is in C 3 on

Original: U (from Lemma B.15 [2]).
Revised: U (from Lemma B.15 [2]).

Original: 3. Explicit and Implicit Bias in the Full-batch Setting
Revised: 3. Explicit and Implicit Bias in the Full-batch Setting

Original: Section 3.1 provides a general theorem to properly analyze the explicit bias of various notions of sharpness among different minimizers.
Revised: Section 3.1 provides a general theorem to properly analyze the explicit bias of various notions of sharpness among different minimizers.

Original: We then apply our machinery on ascent-direction sharpness and worst-direction sharpness and show that they have different explicit biases.
Revised: We then apply our machinery on ascent-direction sharpness and worst-direction sharpness and show that they have different explicit biases.

Original: In Section 3.
Revised: In Section 3.

Original: we provide our main theorem in the full-batch setting, that SAM implicitly minimizes the worstdirection sharpness, via characterizing its limiting dynamics as learning rate ρ and η goes to 0 with a Riemmanian gradient flow with respect to the top eigenvalue of the Hessian of the loss on the manifold of local minimizers.
Revised: we provide our main theorem in the full-batch setting, that SAM implicitly minimizes the worstdirection sharpness, via characterizing its limiting dynamics as learning rate ρ and η goes to 0 with a Riemmanian gradient flow with respect to the top eigenvalue of the Hessian of the loss on the manifold of local minimizers.

Original: In Appendix C.1 we sketch the proof of the implicit bias of SAM and identified a key property behind the implicit bias, which is the implicit alignment between the gradient and the top eigenvector of the Hessian throughout the training.
Revised: In Appendix C.1 we sketch the proof of the implicit bias of SAM and identified a key property behind the implicit bias, which is the implicit alignment between the gradient and the top eigenvector of the Hessian throughout the training.

Original: 3.1. Worst- and ascent-direction sharpness have different explicit bias
Revised: 3.1. Worst- and ascent-direction sharpness have different explicit bias

Original: The intuition of approximating R max ρ by R asc ρ comes from the following Taylor expansions [9, 23].
Revised: The intuition of approximating R max ρ by R asc ρ comes from the following Taylor expansions [9, 23].

Original: [Equation]
Revised: [Equation]

Original: For most of points x with non-zero gradient, their leading terms are both the first order term and are the same, since sup ∥ v ∥ 2 ≤ 1 v ⊤ ∇ L ( x )
Revised: For most of points x with non-zero gradient, their leading terms are both the first order term and are the same, since sup ∥ v ∥ 2 ≤ 1 v ⊤ ∇ L ( x )

Original: = ∥∇ L ( x )
Revised: = ∥∇ L ( x )

Original: ∥ 2 .
Revised: ∥ 2 .

Original: Unfortunately, the first order term vanishes when we actually try to minimize the regularized objective, i.e. , the sharpness-aware loss
Revised: Unfortunately, the first order term vanishes when we actually try to minimize the regularized objective, i.e. , the sharpness-aware loss

Original: L asc ρ or L max ρ , because every minimizer of the original loss has zero gradient.
Revised: L asc ρ or L max ρ , because every minimizer of the original loss has zero gradient.

Original: When one attempts to optimize the regularized loss, the original loss must first be optimized, meaning the first order term goes away and the first-order approximation becomes trivial.
Revised: When one attempts to optimize the regularized loss, the original loss must first be optimized, meaning the first order term goes away and the first-order approximation becomes trivial.

Original: A quick way to see this is that any global minimizer of the original loss L will kill the first order term and is a O ( ρ 2 ) -approximate minimizer of the sharpness-aware loss.
Revised: A quick way to see this is that any global minimizer of the original loss L will kill the first order term and is a O ( ρ 2 ) -approximate minimizer of the sharpness-aware loss.

Original: In order to allow the regularizer to actually “regularize” the learning algorithm, the goal must be at least reaching O ( ρ 2 ) error, and what really matters is indeed the second order term.
Revised: In order to allow the regularizer to actually “regularize” the learning algorithm, the goal must be at least reaching O ( ρ 2 ) error, and what really matters is indeed the second order term.

Original: In this section, we aim to understand under Assumption 1 what the explicit biases of various notions of sharpness among different minimizers are.
Revised: In this section, we aim to understand under Assumption 1 what the explicit biases of various notions of sharpness among different minimizers are.

Original: Theorem 4 will be our main theoretic tool to analyze the explicit bias for small perturbation radius ρ .
Revised: Theorem 4 will be our main theoretic tool to analyze the explicit bias for small perturbation radius ρ .

Original: Notation for Regularizers.
Revised: Notation for Regularizers.

Original: Let R ρ : R D → R + ∪ {∞} be a family of regularizers parameterized by ρ .
Revised: Let R ρ : R D → R ∪{∞} be a family of regularizers parameterized by ρ .

Original: If R ρ is not well-defined at some x , then we let R ρ ( x ) = ∞ .
Revised: If R ρ is not well-defined at some x , then we let R ρ ( x ) = ∞ .

Original: This convention will be useful when analyzing ascent-direction sharpness R asc ρ 
Revised: This convention will be useful when analyzing ascent-direction sharpness R asc ρ =

Original: [Equation] − L which is not defined when ∇ L ( x ) = 0 .
Revised: L asc ρ − L which is not defined when ∇ L ( x ) = 0 .

Original: This convention will not change the minimizers of the regularized loss.
Revised: This convention will not change the minimizers of the regularized loss.

Original: Intuitively, a regularizer should always be non-negative, but however, when far away from manifold, regularizer R ρ ( x ) can actually be negative, e.g. , R avg ρ ( x ) 
Revised: Intuitively, a regularizer should always be non-negative, but however, when far away from manifold, regularizer R ρ ( x ) can actually be negative, e.g. , R avg ρ ( x ) ≈

Original: [Equation] Tr[ ∇ 2 L ( x )] .
Revised: ρ 2 2 D Tr[ ∇ 2 L ( x )] .

Original: Therefore we make the following assumption to allow the regularizer to be mildly negative.
Revised: Therefore we make the following assumption to allow the regularizer to be mildly negative.

Original: Assumption 2 Suppose for any bounded closed set B ⊂ U , there exists C > 0 , such that for sufficiently small ρ , ∀ x ∈ B, R ρ ( x ) ≥ − Cρ 2 .
Revised: Assumption 2 Suppose for any bounded closed set B ⊂ U , there exists C > 0 , such that for sufficiently small ρ , ∀ x ∈ B, R ρ ( x ) ≥ − Cρ 2 .

Original: The following concept of limiting regularizer is of crucial role in our analysis.
Revised: The following concept of limiting regularizer is of crucial role in our analysis.

Original: Definition 3 (Limiting Regularizer)
Revised: Definition 3 (Limiting Regularizer)

Original: We define the limiting regularizer of { R ρ } as the function
Revised: We define the limiting regularizer of { R ρ } as the function

Original: [Equation]
Revised: [Equation]

Original: We say the limiting regularizer S of { R ρ } is a good around some x ∗ ∈ Γ , if there is an open set V containing x ∗ , such that S is a non-negative continuous function in V and for any ϵ > 0 ,ϵ > 0 , there is some ρ x ∗ > 0 , it holds that ∀ x ∈ Γ ∩ V,  ρ ≤ ρ x ∗ , 
Revised: We say the limiting regularizer S of { R ρ } is a good around some x ∗ ∈ Γ , if there is an open set V containing x ∗ , such that S is a non-negative continuous function in V and for any ϵ > 0 , there is some ρ x ∗ > 0 , it holds that ∀ x ∈ Γ ∩ V, 0 < ρ ≤ ρ x ∗ , [Equation] S [Equation]

Revised: x ) − inf

Original: [Equation]
Revised: [Equation]

Revised: ≤

Revised: [Equation]

Original: < ϵ.
Revised: < ϵ.

Original: We say the limiting regularizer S is good on Γ , if S is good around every point x ∈ Γ .
Revised: We say the limiting regularizer S is good on Γ , if S is good around every point x ∈ Γ .

Original: The high-level intuition behind the definition of limiting regularizer is to capture the second order term in the Taylor expansion of regularizer R ρ when ρ → 0 .
Revised: The high-level intuition behind the definition of limiting regularizer is to capture the second order term in the Taylor expansion of regularizer R ρ when ρ → 0 .

Original: When the second order term is continuous in x , the definition of S ( x ) can also be simplified as R ρ ( x ) /ρ 2 .
Revised: When the second order term is continuous in x , the definition of S ( x ) can also be simplified as R ρ ( x ) /ρ 2 .

Original: The intuition of the concept of a good limiting regularizer is that, the regularizer should not change very fast, especially in an O ( ρ ) neighborhood of the minimizer.
Revised: The intuition of the concept of a good limiting regularizer is that, the regularizer should not change very fast, especially in an O ( ρ ) neighborhood of the minimizer.

Original: If so, the minimizer of the regularized loss may be Ω( ρ ) away from any minimizer to reduce the regularizer at the cost of increasing the original loss, which makes the limiting regularizer unable to capture the explicit bias of the regularizer.
Revised: If so, the minimizer of the regularized loss may be Ω( ρ ) away from any minimizer to reduce the regularizer at the cost of increasing the original loss, which makes the limiting regularizer unable to capture the explicit bias of the regularizer.

Original: Theorem 4 Let U ′ be any bounded open set such that its closure U ′ is contained in U and that
Revised: Theorem 4 Let U ′ be any bounded open set such that its closure U ′ is contained in U and that

Original: U ′ ∩ Γ = U ′ ∩ Γ .
Revised: U ′ ∩ Γ = U ′ ∩ Γ .

Original: Then for any family of parametrized regularizers { R ρ } admitting a good limiting regularizer on Γ and satisfying Assumption 2 and any ϵ ≥ 0 , there is a ρ 0 > 0 , such that for all u ∈ U ′ and ρ < ρ 0 , it holds that
Revised: Then for any family of parametrized regularizers { R ρ } admitting a good limiting regularizer on Γ and satisfying Assumption 2 and any ϵ ≥ 0 , there is a ρ 0 > 0 , such that for all u ∈ U ′ and ρ < ρ 0 , it holds that

Original: [Equation]
Revised: [Equation]

Original: For the applications we are interested in in this paper, the good limiting regularizer S can be continuously extended to the entire space R D .
Revised: For the applications we are interested in in this paper, the good limiting regularizer S can be continuously extended to the entire space R D .

Original: In such a case, the implication of “ = ⇒ ” of Theorem 4 also admits the following alternative form which doesn’t involve R ρ .
Revised: In such a case, the implication of “ = ⇒ ” of Theorem 4 also admits the following alternative form which doesn’t involve R ρ .

Original: Corollary 5 implies minimizing regularized loss L ( x ) + R ρ ( x ) is equivalent to minimizing the limiting regularizer of { R ρ } ρ , S ( x ) on the global minimizer manifold Γ .
Revised: Corollary 5 implies minimizing regularized loss L ( x ) + R ρ ( x ) is equivalent to minimizing the limiting regularizer of { R ρ } ρ , S ( x ) on the global minimizer manifold Γ .

Original: Corollary 5 Under the setting of Theorem 4, let S be an continuous extension of S to R d , if
Revised: Corollary 5 Under the setting of Theorem 4, let S be an continuous extension of S to R d , if

Original: L ( u ) + R ρ ( u ) ≤ inf x ∈ U ′
Revised: L ( u ) + R ρ ( u ) ≤ inf x ∈ U ′

Original: ( L ( x ) + R ρ ( x ))
Revised: ( L ( x ) + R ρ ( x ))

Original: + ϵρ 

Revised: + ϵρ

Original: + o ( ρ 2 ) , then we have that L ( u ) − inf x ∈ U ′
Revised: + o ( ρ 2 ) , then we have that L ( u ) − inf x ∈ U ′

Original: L ( x ) =
Revised: L ( x ) =

Original: O ( ρ 2 ) and that (cid:12)(cid:12) S ( u ) − inf x ∈ U ′ ∩ Γ S ( x ) (cid:12)(cid:12) =
Revised: O ( ρ 2 ) and that (cid:12)(cid:12) S ( u ) − inf x ∈ U ′ ∩ Γ S ( x ) (cid:12)(cid:12) =

Original: ϵ
Revised: ϵ

Original: + o (1) .
Revised: + o (1) .

Original: Corollary 5 suggests a sharp phase transition of the property of the solution of min x L ( x ) +
Revised: Corollary 5 suggests a sharp phase transition of the property of the solution of min x L ( x ) +

Original: R ρ ( x ) when the optimization error drops from ω ( ρ 2 ) to O ( ρ 2 ) .
Revised: R ρ ( x ) when the optimization error drops from ω ( ρ 2 ) to O ( ρ 2 ) .

Original: when the optimization error is larger than ω ( ρ 2 ) , no regularization effect happens and any minimizer satisfies the requirement.
Revised: when the optimization error is larger than ω ( ρ 2 ) , no regularization effect happens and any minimizer satisfies the requirement.

Original: When the error becomes O ( ρ 2 ) , there is a non-trivial restriction on the (extended) limiting regularizer.
Revised: When the error becomes O ( ρ 2 ) , there is a non-trivial restriction on the (extended) limiting regularizer.

Original: Theorem 6 (Summary of Theorem 13,12 and 14) R asc ρ , R max ρ , R avg ρ satisfy Assumption 2 and admit good limiting regularizers on Γ .
Revised: Theorem 6 (Summary of Theorem 13,12 and 14) R asc ρ , R max ρ , R avg ρ satisfy Assumption 2 and admit good limiting regularizers on Γ .

Original: (see Table 1)
Revised: (see Table 1)

Original: Using Theorem 6, we can apply Corollary 5 to characterize their explicit biases, which are all different.
Revised: Using Theorem 6, we can apply Corollary 5 to characterize their explicit biases, which are all different.

Original: 3.2. SAM provably decreases worst-direction sharpness locally
Revised: 3.2. SAM provably decreases worst-direction sharpness locally

Original: Though ascent-direction sharpness has different explicit bias from worst-direction sharpness, in this subsection we will show that surprisingly, SAM, an heuristic method designed to minimize ascent-direction sharpness, provably decreases worst-direction sharpness.
Revised: Though ascent-direction sharpness has different explicit bias from worst-direction sharpness, in this subsection we will show that surprisingly, SAM, an heuristic method designed to minimize ascent-direction sharpness, provably decreases worst-direction sharpness.

Original: The main result here is an exact characterization of the trajectory of SAM (3) via the following ordinary differential equation (ODE) (6), when learning rate η and perturbation radius ρ are small and the initialization x (0)
Revised: The main result here is an exact characterization of the trajectory of SAM (3) via the following ordinary differential equation (ODE) (6), when learning rate η and perturbation radius ρ are small and the initialization x (0)

Original: =
Revised: =

Original: x init is in U .
Revised: x init is in U .

Original: We call the solution of (6) the limiting flow of SAM, which is exactly the
Revised: We call the solution of (6) the limiting flow of SAM, which is exactly the

Original: Riemannian Gradient Flow on the manifold Γ with respect to λ 1 ( ∇ 2 L ( · )) .
Revised: Riemannian Gradient Flow on the manifold Γ with respect to λ 1 ( ∇ 2 L ( · )) .

Original: In other words, the
Revised: In other words, the

Original: ODE (6) is essentially a projected gradient descent algorithm with loss λ 1 ( ∇ 2 L ( · )) on the constraint set Γ and an infinitesimal learning rate.
Revised: ODE (6) is essentially a projected gradient descent algorithm with loss λ 1 ( ∇ 2 L ( · )) on the constraint set Γ and an infinitesimal learning rate.

Original: [Equation]
Revised: [Equation]

Original: Note λ 1 ( ∇ 2 L ( x )) may not be differentiable at x if λ 1 ( ∇ 2 L ( x ))
Revised: Note λ 1 ( ∇ 2 L ( x )) may not be differentiable at x if λ 1 ( ∇ 2 L ( x ))

Original: =
Revised: =

Original: λ 2 ( ∇ 2 L ( x )) , thus to ensure the (6) is well-defined, we assume there is a positive eigengap for L on Γ .
Revised: λ 2 ( ∇ 2 L ( x )) , thus to ensure the (6) is well-defined, we assume there is a positive eigengap for L on Γ .

Original: Assuming ODE (6) has a solution till time T 3 , we have Theorem 8, which is the main theorem of this section.
Revised: Assuming ODE (6) has a solution till time T 3 , we have Theorem 8, which is the main theorem of this section.

Original: Assumption 7 For x ∈ Γ , there exists a positive eigengap,i.e., λ 1 ( ∇ 2 L ( x )) > λ 2 ( ∇ 2 L ( x )) .
Revised: Assumption 7 For x ∈ Γ , there exists a positive eigengap,i.e., λ 1 ( ∇ 2 L ( x )) > λ 2 ( ∇ 2 L ( x )) .

Original: Theorem 8 (Main, Theorems 46 and 47 stated informally)
Revised: Theorem 8 (Main, Theorems 46 and 47 stated informally)

Original: Let { x ( t ) } be the iterates of SAM (3) with x (0)
Revised: Let { x ( t ) } be the iterates of SAM (3) with x (0)

Original: = x init ∈ U , then under Assumptions 1 and 7, for all η, ρ such that η ln(1 /ρ ) and ρ/η are sufficiently small, the dynamics of SAM can be split into two phases:
Revised: = x init ∈ U , then under Assumptions 1 and 7, for all η, ρ such that η ln(1 /ρ ) and ρ/η are sufficiently small, the dynamics of SAM can be split into two phases:

Original: • Phase I: SAM follows Gradient Flow with respect to L until entering an O ( ηρ ) neighborhood of the manifold Γ in ˜ O
Revised: • Phase I: SAM follows Gradient Flow with respect to L until entering an O ( ηρ ) neighborhood of the manifold Γ in ˜ O

Original: ( 1 η ) steps; • Phase II: SAM tracks the solution X of (6) , the Riemannian Gradient Flow with respect to λ 1 ( ∇ 2 L ( · )) in the O ( ηρ ) neighborhood in the sense
Revised: ( 1 η ) steps; • Phase II: SAM tracks the solution X of (6) , the Riemannian Gradient Flow with respect to λ 1 ( ∇ 2 L ( · )) in the O ( ηρ ) neighborhood in the sense

Original: that max 0 ≤ T ≤ T 3 ∥
Revised: that max 0 ≤ T ≤ T 3 ∥

Original: Φ( x ( ⌈ T/ ( ηρ 2 ) ⌉ ))
Revised: Φ( x ( ⌈ T/ ( ηρ 2 ) ⌉ ))

Original: − X ( T ) ∥
Revised: − X ( T ) ∥

Original: =
Revised: =

Original: O (( η + ρ ) log(1 /ηρ )) .
Revised: O (( η + ρ ) log(1 /ηρ )) .

Original: Moreover, the angle between ∇ L ( x ( t )) and v 1 ( ∇ 2 L ( x ( t ))) is O ( ρ ) .
Revised: Moreover, the angle between ∇ L ( x ( t )) and v 1 ( ∇ 2 L ( x ( t ))) is O ( ρ ) .

Original: Theorem 8 shows that SAM decreases the largest eigenvalue of Hessian of loss locally around the manifold of local minimizers.
Revised: Theorem 8 shows that SAM decreases the largest eigenvalue of Hessian of loss locally around the manifold of local minimizers.

Original: 4. Explicit and Implicit bias in the stochastic setting
Revised: 4. Explicit and Implicit bias in the stochastic setting

Original: In practice, people usually use SAM in the stochastic mini-batch setting, and the test accuracy improves as the batch size decreases [9].
Revised: In practice, people usually use SAM in the stochastic mini-batch setting, and the test accuracy improves as the batch size decreases [9].

Original: Towards explaining this phenomenon, Foret et al.
Revised: Towards explaining this phenomenon, Foret et al.

Original: [9] argues intuitively that stochastic SAM minimizes stochastic worst-direction sharpness.
Revised: [9] argues intuitively that stochastic SAM minimizes stochastic worst-direction sharpness.

Original: In this section, we focus on SGD with batch size 1.
Revised: In this section, we focus on SGD with batch size 1.

Original: We still need Assumption 1 in this section.
Revised: We still need Assumption 1 in this section.

Original: We first by analyzing the explicit bias of the stochastic ascent- and worst-direction sharpness in
Revised: We first by analyzing the explicit bias of the stochastic ascent- and worst-direction sharpness in

Original: Section 4.1 via the tools developed in Section 3.1.
Revised: Section 4.1 via the tools developed in Section 3.1.

Original: It turns out they are all proportional to the trace of hessian as ρ → 0 .
Revised: It turns out they are all proportional to the trace of hessian as ρ → 0 .

Original: In Section 4.2, we show stochastic SAM locally decreases trace of Hessian.
Revised: In Section 4.2, we show stochastic SAM locally decreases trace of Hessian.

Original: Setting.
Revised: Setting.

Original: Let f k ( x ) be the model output on k th data where f k is a C 4 -smooth function and y k be the i th label for l = 1 , . . .
Revised: Let f k ( x ) be the model output on k th data where f k is a C 4 -smooth function and y k be the i th label for l = 1 , . . .

Original: , M .
Revised: , M .

Original: We define the loss on k th data as L k ( x ) =
Revised: We define the loss on k th data as L k ( x ) =

Original: ℓ ( f k ( x ) , y k ) and the total loss L = (cid:80) Mk =1 L k /M , where ℓ ( y ′ , y ) is a C 4 -smooth function satisfying the following properties 1 : 
Revised: ℓ ( f k ( x ) , y k ) and the total loss L = (cid:80) Mk =1 L k /M , where ℓ ( y ′ , y ) is a C 4 -smooth function satisfying the following properties 1 : (1).

Original: arg 
Revised: arg min y ′ ∈ R ℓ ( y ′ , y ) =

Original:  R 
Revised: y , for any y ∈ R ; (2). d 2 ℓ ( y ′ ,y )

Original:  y , y ) =
Revised: d 2 y ′ | y ′ =

Original: y  , for any y ∈ R ; ( y ,y
Revised: y > 0 , for any y ∈ R .

Original: 4.1. Stochastic worst- and ascent-direction sharpness have same explicit bias
Revised: 4.1. Stochastic worst- and ascent-direction sharpness have same explicit bias

Original: Below we specify stochastic worst-direction sharpness and stochastic ascent-direction sharpness as E k
Revised: Below we specify stochastic worst-direction sharpness and stochastic ascent-direction sharpness as E k

Original: [ R max k,ρ ] =
Revised: [ R max k,ρ ] =

Original: E k
Revised: E k

Original: [ L max k,ρ ] − L and E k
Revised: [ L max k,ρ ] − L and E k

Original: [ R asc k,ρ ] = E k
Revised: [ R asc k,ρ ] = E k

Original: [ L asc k,ρ ] − L .
Revised: [ L asc k,ρ ] − L .

Original: Unlike the full-batch setting, these two sharpness have same explicit bias, or more precisely, they have the same limiting regularizers.
Revised: Unlike the full-batch setting, these two sharpness have same explicit bias, or more precisely, they have the same limiting regularizers.

Original: We omit the result on the stochastic average-direction sharpness as it is the same as its counterpart in the full-batch case.
Revised: We omit the result on the stochastic average-direction sharpness as it is the same as its counterpart in the full-batch case.

Original: Theorem
Revised: Theorem

Original: Stochastic worst-direction sharpness E k
Revised: Stochastic worst-direction sharpness E k

Original: [ R max k,ρ ] satisfies Assumption 2 and admits
Revised: [ R max k,ρ ] satisfies Assumption 2 and admits

Original: Tr( ∇ 2 L ( · )) / 2 as a good limiting regularizer on Γ .
Revised: Tr( ∇ 2 L ( · )) / 2 as a good limiting regularizer on Γ .

Original: Theorem 10 Stochastic ascent-direction sharpness E k
Revised: Theorem 10 Stochastic ascent-direction sharpness E k

Original: [ R asc k,ρ ] satisfies Assumption 2 and admits
Revised: [ R asc k,ρ ] satisfies Assumption 2 and admits

Original: Tr( ∇ 2 L ( · )) / 2 as a good limiting regularizer on Γ .
Revised: Tr( ∇ 2 L ( · )) / 2 as a good limiting regularizer on Γ .

Original:  Stochastic SAM minimizesstochastic
Revised: 4.2. Stochastic SAM minimizes

Original:  worst-direction sharpness
Revised: stochastic worst-direction sharpness

Original: Stochastic SAM: Recall L k is the loss on k th data, we use stochastic SAM to denote the following update rule, where k t is sampled i.i.d from uniform distribution on [ M ] .
Revised: Stochastic SAM: Recall L k is the loss on k th data, we use stochastic SAM to denote the following update rule, where k t is sampled i.i.d from uniform distribution on [ M ] .

Original: [Equation]
Revised: [Equation]

Original: The main result of this section is to show stochastic SAM tracks the following Riemannian gradient flow with respect to Tr( ∇ 2 L ( · )) on the manifold for sufficiently small η and ρ ,
Revised: The main result of this section is to show stochastic SAM tracks the following Riemannian gradient flow with respect to Tr( ∇ 2 L ( · )) on the manifold for sufficiently small η and ρ ,

Original: [Equation]
Revised: [Equation]

Original: Theorem 11 Let { x ( t ) } be the iterates defined by SAM (7) and x (0) =
Revised: Theorem 11 Let { x ( t ) } be the iterates defined by SAM (7) and x (0) =

Original: x init ∈ U , then under
Revised: x init ∈ U , then under

Original: Assumption 1, for all η and ρ such that ( η + ρ ) log(1 /ηρ ) is sufficiently small, the dynamics of SAM can be split into two phases:
Revised: Assumption 1, for all η and ρ such that ( η + ρ ) log(1 /ηρ ) is sufficiently small, the dynamics of SAM can be split into two phases:

Original: • Phase I: Stochastic SAM follows Gradient Flow with respect to L until entering an O ( ηρ ) neighborhood of the manifold Γ in ˜ O
Revised: • Phase I: Stochastic SAM follows Gradient Flow with respect to L until entering an O ( ηρ ) neighborhood of the manifold Γ in ˜ O

Original: ( 1 η ) steps, with probability at least 1 − O ( √ ρ ) ; • Phase II: Stochastic SAM tracks the solution X of (8) , the Riemannian Gradient Flow with respect to Tr( ∇ 2 L ( · )) , with time scaling as ∥ Φ( x ( ⌈ T 3 / ( ηρ 2 ) ⌉ ))
Revised: ( 1 η ) steps, with probability at least 1 − O ( √ ρ ) ; • Phase II: Stochastic SAM tracks the solution X of (8) , the Riemannian Gradient Flow with respect to Tr( ∇ 2 L ( · )) , with time scaling as ∥ Φ( x ( ⌈ T 3 / ( ηρ 2 ) ⌉ ))

Original: − X ( T 3 ) ∥ =
Revised: − X ( T 3 ) ∥ =

Original: O (( η + ρ ) log(1 /ηρ ) .
Revised: O (( η + ρ ) log(1 /ηρ ) .

Original: 5. Conclusion
Revised: 5. Conclusion

Original: In this work, we have performed a rigorous mathematical analysis of the explicit bias of various notions of sharpness when used as regularizers and the implicit bias of the SAM algorithm.
Revised: In this work, we have performed a rigorous mathematical analysis of the explicit bias of various notions of sharpness when used as regularizers and the implicit bias of the SAM algorithm.

Original: In particular, we show the explicit biases of worst-, ascent- and average-direction sharpness around the manifold of minimizers are minimizing the largest eigenvalue , the smallest nonzero eigenvalue, and the trace of Hessian of the loss function.
Revised: In particular, we show the explicit biases of worst-, ascent- and average-direction sharpness around the manifold of minimizers are minimizing the largest eigenvalue, the smallest nonzero eigenvalue, and the trace of Hessian of the loss function.

Original: We show that in the full-batch setting, SAM provably decreases the largest eigenvalue of Hessian, while in the stochastic setting when batch size is 1,
Revised: We show that in the full-batch setting, SAM provably decreases the largest eigenvalue of Hessian, while in the stochastic setting when batch size is 1,

Original: SAM provably decreases the trace of Hessian
Revised: SAM provably decreases the trace of Hessian

Original: C Proof overview
Revised: C Proof overview

Original: C.1 Proof Sketch for Phase II in Theorem. . . . . . . . . . . . . . . . . . . . . . .
Revised: C.1 Proof Sketch for Phase II in Theorem. . . . . . . . . . . . . . . . . . . . . . .

Original: C.2 Proof Sketch for Stochastic Case . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: C.2 Proof Sketch for Stochastic Case . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: D Proof Details
Revised: D Proof Details

Original: D.1 Setup . .
Revised: D.1 Setup . .

Original: D.2 Lemma about Φ . .
Revised: D.2 Lemma about Φ . .

Original: D.3 Explicit Bias . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: D.3 Explicit Bias . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: D.4 Full-batch SAM on Quadratic Loss: Proof of Theorem 21 . . . . . . . . . . . . . .
Revised: D.4 Full-batch SAM on Quadratic Loss: Proof of Theorem 21 . . . . . . . . . . . . . .

Original: D.4.
Revised: D.4.

Original: Preparation Phase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Preparation Phase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: D.4.
Revised: D.4.

Original: Alignment Phase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Alignment Phase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: D.4.
Revised: D.4.

Original: Length Convergence . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Length Convergence . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: D.5 Full-batch SAM on General Loss: Proof of Theorem 8 . . . . . . . . . . . . . . .
Revised: D.5 Full-batch SAM on General Loss: Proof of Theorem 8 . . . . . . . . . . . . . . .

Original: D.5.
Revised: D.5.

Original: Phase I: Proof of Theorem 46 . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Phase I: Proof of Theorem 46 . . . . . . . . . . . . . . . . . . . . . . . .

Original: D.5.
Revised: D.5.

Original: Phase II: Proof of Theorem 47 . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Phase II: Proof of Theorem 47 . . . . . . . . . . . . . . . . . . . . . . . .

Original: D.5.
Revised: D.5.

Original: Proof of corollary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Proof of corollary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: D.6 Stochastic SAM: Proof of Theorem 11 . . . . . . . . . . . . . . . . . . . . . . . .
Revised: D.6 Stochastic SAM: Proof of Theorem 11 . . . . . . . . . . . . . . . . . . . . . . . .

Original: D.6.
Revised: D.6.

Original: Phase I: Proof of Theorem 51 . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Phase I: Proof of Theorem 51 . . . . . . . . . . . . . . . . . . . . . . . .

Original: D.6.
Revised: D.6.

Original: Phase II: Proof of Theorem 52 . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Phase II: Proof of Theorem 52 . . . . . . . . . . . . . . . . . . . . . . . .

Original: D.6.
Revised: D.6.

Original: Proof of corollary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Proof of corollary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: D.7 Technical Lemmas . .
Revised: [Table] [Table]

Original: Appendix A. Related Works
Revised: Appendix A. Related Works

Original: Sharpness and Generalization.
Revised: Sharpness and Generalization.

Original: The study on the connection between sharpness and generalization can be traced back to [10].
Revised: The study on the connection between sharpness and generalization can be traced back to [10].

Original: Keskar et al.
Revised: Keskar et al.

Original: [14] observed a positive correlation between the batch size, the generalization error, and the sharpness of the loss landscape when changing the batch size.
Revised: [14] observed a positive correlation between the batch size, the generalization error, and the sharpness of the loss landscape when changing the batch size.

Original: Jastrzebski et al.
Revised: Jastrzebski et al.

Original: [12] extends this by finding a correlation of the sharpness and the ratio between learning rate to batch size.
Revised: [12] extends this by finding a correlation of the sharpness and the ratio between learning rate to batch size.

Original: Dinh et al.
Revised: Dinh et al.

Original: [6] shows that one can easily construct networks with good generalization but with arbitrary large sharpness by reparametrization.
Revised: [6] shows that one can easily construct networks with good generalization but with arbitrary large sharpness by reparametrization.

Original: Dziugaite and
Revised: Dziugaite and

Original: Roy [7], Neyshabur et al.
Revised: Roy [7], Neyshabur et al.

Original: [21], Wei and Ma [22] give theoretical guarantees on the generalization error using sharpness-related measures.
Revised: [21], Wei and Ma [22] give theoretical guarantees on the generalization error using sharpness-related measures.

Original: [13] performs a large-scale empirical study on various generalization measures and showed that sharpness-based measures have the highest correlation with generalization.
Revised: [13] performs a large-scale empirical study on various generalization measures and showed that sharpness-based measures have the highest correlation with generalization.

Original: Background on Sharpness Aware-Minimization.
Revised: Background on Sharpness Aware-Minimization.

Original: Foret et al.
Revised: Foret et al.

Original: [9], Zheng et al.
Revised: [9], Zheng et al.

Original: [26] concurrently proposed to minimize the loss at the perturbed from current parameter towards the worst direction to improve generalization.
Revised: [26] concurrently proposed to minimize the loss at the perturbed from current parameter towards the worst direction to improve generalization.

Original: Wu et al.
Revised: Wu et al.

Original: [23] proposed the almost identical method for a different purpose, robust generalization of adversarial training.
Revised: [23] proposed the almost identical method for a different purpose, robust generalization of adversarial training.

Original: [15] proposed a different metric for SAM to fix the rescaling problem pointed out by [6].
Revised: [15] proposed a different metric for SAM to fix the rescaling problem pointed out by [6].

Original: Liu et al.
Revised: Liu et al.

Original: [17] proposed an more computationally efficient version of SAM.
Revised: [17] proposed an more computationally efficient version of SAM.

Original: Zhao et al.
Revised: Zhao et al.

Original: [25] proposed to improve generalization by penalizing gradient norm.
Revised: [25] proposed to improve generalization by penalizing gradient norm.

Original: Their proposed algorithm can be viewed as a generalization of SAM.
Revised: Their proposed algorithm can be viewed as a generalization of SAM.

Original: Andriushchenko and
Revised: Andriushchenko and

Original: Flammarion [1] studied a variant of SAM where the step size of ascent step is ρ instead of ρ ∥∇ L ( x ) ∥ 2 .
Revised: Flammarion [1] studied a variant of SAM where the step size of ascent step is ρ instead of ρ ∥∇ L ( x ) ∥ 2 .

Original: They showed that for a simple model this variant of SAM has a stronger regularization effect when batch size is 1 compared to the full-batch case and argued that this might be the explanation that
Revised: They showed that for a simple model this variant of SAM has a stronger regularization effect when batch size is 1 compared to the full-batch case and argued that this might be the explanation that

Original: SAM generalizes better with small batch sizes.
Revised: SAM generalizes better with small batch sizes.

Original: Sharpness Minimization as Implicit Bias.
Revised: Sharpness Minimization as Implicit Bias.

Original: Recent theoretical works Blanc et al.
Revised: Recent theoretical works Blanc et al.

Original: [3], Damian et al.
Revised: [3], Damian et al.

Original: [4], Li et al.
Revised: [4], Li et al.

Original: [16] showed that SGD with label noise is implicitly biased to local minimizers with a smaller trace of Hessian.
Revised: [16] showed that SGD with label noise is implicitly biased to local minimizers with a smaller trace of Hessian.

Original: Arora et al.
Revised: Arora et al.

Original: [2] showed that normalized GD implicitly decreases the largest eigenvalue of the Hessian.
Revised: [2] showed that normalized GD implicitly decreases the largest eigenvalue of the Hessian.

Original: Lyu et al.
Revised: Lyu et al.

Original: [18] showed that GD with weight decay on a scale invariant loss function implicitly decreases the spherical sharpness, i.e. , the largest eigenvalue of the
Revised: [18] showed that GD with weight decay on a scale invariant loss function implicitly decreases the spherical sharpness, i.e. , the largest eigenvalue of the

Original: Hessian evaluated at the normalized parameter.
Revised: Hessian evaluated at the normalized parameter.

Original: Appendix B. Discussion and Implication of main result B.1. Limiting Regularizer
Revised: Appendix B. Discussion and Implication of main result B.1. Limiting Regularizer

Original: Theorem 12 Worst-direction sharpness R max ρ satisfies Assumption 2 and admits λ 1 ( ∇ 2 L ( · )) / 2 as a good limiting regularizer on Γ .
Revised: Theorem 12 Worst-direction sharpness R max ρ satisfies Assumption 2 and admits λ 1 ( ∇ 2 L ( · )) / 2 as a good limiting regularizer on Γ .

Original: Theorem 13 Ascent-direction sharpness R asc ρ satisfies Assumption 2 and admits λ M ( ∇ 2 L ( · )) /as
Revised: Theorem 13 Ascent-direction sharpness R asc ρ satisfies Assumption 2 and admits λ M ( ∇ 2 L ( · )) /as

Original: a good limiting regularizer on Γ .
Revised: a good limiting regularizer on Γ .

Original: Theorem 14 Average-direction sharpness R avg ρ satisfies Assumption 2 and admits Tr( ∇ 2 L ( · )) / (2 D ) as a good limiting regularizer on Γ .
Revised: Theorem 14 Average-direction sharpness R avg ρ satisfies Assumption 2 and admits Tr( ∇ 2 L ( · )) / (2 D ) as a good limiting regularizer on Γ .

Original: When R ρ is continuous at some x ∈ Γ , the definition of S ( x ) can be simplified as lim ρ → 0 R ρ ( x ) /ρ 2 .
Revised: When R ρ is continuous at some x ∈ Γ , the definition of S ( x ) can be simplified as lim ρ → 0 R ρ ( x ) /ρ 2 .

Original: Worst- and average- direction sharpness fall into this type and the limiting regularizer can be solved straightforwardly by Taylor expansion.
Revised: Worst- and average- direction sharpness fall into this type and the limiting regularizer can be solved straightforwardly by Taylor expansion.

Original: The analysis for ascent-direction sharpness is more tricky as R asc ρ ( x ) = ∞ and thus is not continuous for any x ∈ Γ .
Revised: The analysis for ascent-direction sharpness is more tricky as R asc ρ ( x ) = ∞ and thus is not continuous for any x ∈ Γ .

Original: To minimize R asc ρ around x , we can pick x ′ → x to make ∥∇ L ( x ) ∥ 

Revised: To minimize R asc ρ around x , we can pick x ′ → x to make ∥∇ L ( x ) ∥

Original: →but not equal to 0.
Revised: →but not equal to 0.

Original: By (5), we have R asc ρ ( x ′ )
Revised: By (5), we have R asc ρ ( x ′ )

Original: ≈ ρ 2 / 2 · ∇ L ( x ′ )
Revised: ≈ ρ 2 / 2 · ∇ L ( x ′ )

Original: ⊤ ∇ 2 L ( x ) ∇ L ( x ′ )
Revised: ⊤ ∇ 2 L ( x ) ∇ L ( x ′ )

Original: / ∥∇ L ( x ′ )
Revised: / ∥∇ L ( x ′ )

Original: ∥ 2 .
Revised: ∥ 2 .

Original: Here the crucial step of the proof is that because of Assumption 1, ∇ L ( x ) / ∥∇ L ( x ) ∥ 2 must almost lie in the column span of ∇ 2 L ( x ) , and thus implies inf x ′ ∇ L ( x ′ )
Revised: Here the crucial step of the proof is that because of Assumption 1, ∇ L ( x ) / ∥∇ L ( x ) ∥ 2 must almost lie in the column span of ∇ 2 L ( x ) , and thus implies inf x ′ ∇ L ( x ′ )

Original: ⊤ ∇ 2 L ( x ) ∇ L ( x ′ )
Revised: ⊤ ∇ 2 L ( x ) ∇ L ( x ′ )

Original: / ∥∇ L ( x ′ )
Revised: / ∥∇ L ( x ′ )

Original: ∥ 2 ρ → 0 → λ M ( ∇ 2 L ( x )) .
Revised: ∥ 2 ρ → 0 → λ M ( ∇ 2 L ( x )) .

Original: The above alignment property between the gradient and the column space of Hessian can be checked directly for any non-negative quadratic function and the maximal Hessian rank assumption in Assumption 1 ensures this property extends to general losses.
Revised: The above alignment property between the gradient and the column space of Hessian can be checked directly for any non-negative quadratic function and the maximal Hessian rank assumption in Assumption 1 ensures this property extends to general losses.

Original: Unlike in the full-batch setting where the implicit regularizer of ascent-direction sharpness and worst-direction sharpness have different explicit bias, in the stochastic case they are the same because there is no difference between the maximum and minimum of its non-zero eigenvalue for rank-1 Hessian of each individual loss, and that the average of limiting regularizers is equal to the limiting regularizer of the average regularizers by definition.
Revised: Unlike in the full-batch setting where the implicit regularizer of ascent-direction sharpness and worst-direction sharpness have different explicit bias, in the stochastic case they are the same because there is no difference between the maximum and minimum of its non-zero eigenvalue for rank-1 Hessian of each individual loss, and that the average of limiting regularizers is equal to the limiting regularizer of the average regularizers by definition.

Original: B.2. Full-batch setting
Revised: B.2. Full-batch setting

Original: As a corollary of Theorem 8, we can also show that the largest eigenvalue of the limiting flow closely tracks the regularized training loss.
Revised: As a corollary of Theorem 8, we can also show that the largest eigenvalue of the limiting flow closely tracks the regularized training loss.

Original: Corollary 15 For all T ′ 3 > 0 , for all ρ, η such that η ln(1 /ρ ) and ρ/η are sufficiently small we have ∀ T ′
Revised: Corollary 15 For all T ′ 3 > 0 , for all ρ, η such that η ln(1 /ρ ) and ρ/η are sufficiently small we have ∀ T ′

Original: < ηρ
Revised: < ηρ

Original: 2 t ≤ T 3 , ∥ R max ρ
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Recall Theorem 13 shows that the largest eigenvalue of Hessian is the limiting regularizer of the worst-direction sharpness , leveraging the equivalence relationship in Theorem 4, below we show that full-batch SAM provably minimizes worst-direction sharpness if we additionally assume the limiting flow converges to a local minimizer of the top eigenvalue of Hessian.
Revised: Recall Theorem 13 shows that the largest eigenvalue of Hessian is the limiting regularizer of the worst-direction sharpness , leveraging the equivalence relationship in Theorem 4, below we show that full-batch SAM provably minimizes worst-direction sharpness if we additionally assume the limiting flow converges to a local minimizer of the top eigenvalue of Hessian.

Original: Corollary 16 Define U ′ as in Theorem 4, suppose X ( ∞ ) =
Revised: Corollary 16 Define U ′ as in Theorem 4, suppose X ( ∞ ) =

Original: lim t →∞ X ( t ) exists and is a minimizer of λ 1 ( ∇ 2 L ( x )) in U ′ ∩ Γ , for all ϵ > 0 , there exists a constant T > 0 , then for all ρ, η such that η ln(1 /ρ ) and ρ/η are sufficiently small we have L max ρ
Revised: lim t →∞ X ( t ) exists and is a minimizer of λ 1 ( ∇ 2 L ( x )) in U ′ ∩ Γ , for all ϵ > 0 , there exists a constant T > 0 , then for all ρ, η such that η ln(1 /ρ ) and ρ/η are sufficiently small we have L max ρ

Original: [Equation]
Revised: [Equation]

Original: B.3. Stochastic Setting
Revised: B.3. Stochastic Setting

Original: Corollary 17 and 18 below are stochastic counterparts of Corollary 15 and 16, which says that the trace of Hessian of the limiting flow of well-tracks the stochastic worst-direction sharpness, and therefore when the limiting flow converges to a local minimizer of trace of Hessian, stochastic SAM minimizes the stochastic worst-direction sharpness.
Revised: Corollary 17 and 18 below are stochastic counterparts of Corollary 15 and 16, which says that the trace of Hessian of the limiting flow of well-tracks the stochastic worst-direction sharpness, and therefore when the limiting flow converges to a local minimizer of trace of Hessian, stochastic SAM minimizes the stochastic worst-direction sharpness.

Original: Corollary 17 For all T ′ 3 > 0 , for all ρ, η such that ( η + ρ ) log(1 /ηρ ) are sufficiently small we have ∀ T ′ 3 ≤ ηρ 2 t ≤ T 3 ,
Revised: Corollary 17 For all T ′ 3 > 0 , for all ρ, η such that ( η + ρ ) log(1 /ηρ ) are sufficiently small we have ∀ T ′ 3 ≤ ηρ 2 t ≤ T 3 ,

Original: ∥ E k
Revised: ∥ E k

Original: [ R max k,ρ ]( x ( t )) − ρ 2 Tr( ∇ 2 L ( X ( ηρ 2 t ))) / 2 ∥ =
Revised: [ R max k,ρ ]( x ( t )) − ρ 2 Tr( ∇ 2 L ( X ( ηρ 2 t ))) / 2 ∥ =

Original: ˜ O
Revised: ˜ O

Original: (( η + ρ )
Revised: (( η + ρ )

Original: ρ 2 ) .
Revised: ρ 2 ) .

Original: Corollary 18 Define U ′ as in Theorem 4, suppose X ( ∞ ) =
Revised: Corollary 18 Define U ′ as in Theorem 4, suppose X ( ∞ ) =

Original: lim t →∞ X ( t ) exists and is a minimizer of
Revised: lim t →∞ X ( t ) exists and is a minimizer of

Original: Tr( ∇ 2 L ( x )) in U ′ ∩ Γ , for all ϵ > 0 , there exists a constant T > 0 , then for all ρ, η such that ( η + ρ ) log(1 /ηρ ) are sufficiently small we have E
Revised: Tr( ∇ 2 L ( x )) in U ′ ∩ Γ , for all ϵ > 0 , there exists a constant T > 0 , then for all ρ, η such that ( η + ρ ) log(1 /ηρ ) are sufficiently small we have E

Original: k
Revised: k

Original: [ L max k,ρ ]( ⌈ T/ ( ηρ 2 ) ⌉ )) ≤ ϵρ 2 + inf x ∈ U ′
Revised: [ L max k,ρ ]( ⌈ T/ ( ηρ 2 ) ⌉ )) ≤ ϵρ 2 + inf x ∈ U ′

Original: E k
Revised: E k

Original: [ L max k,ρ ]( x )
Revised: [ L max k,ρ ]( x )

Original: Appendix C. Proof overview C.1. Proof Sketch for Phase II in Theorem
Revised: Appendix C. Proof overview C.1. Proof Sketch for Phase II in Theorem

Original: Now we sketch the proof for the ODE-based characterization of the trajectory of SAM in Phase II.
Revised: Now we sketch the proof for the ODE-based characterization of the trajectory of SAM in Phase II.

Original: The framework of the analysis is similar to Arora et al.
Revised: The framework of the analysis is similar to Arora et al.

Original: [2], Lyu et al.
Revised: [2], Lyu et al.

Original: [18], where the high-level idea is to use Φ( x ( t )) as a proxy for x ( t ) and study the dynamics of Φ( x ( t )) via Taylor expansion, which turns out to be dependent on the alignment between the gradient and the eigenvectors of the
Revised: [18], where the high-level idea is to use Φ( x ( t )) as a proxy for x ( t ) and study the dynamics of Φ( x ( t )) via Taylor expansion, which turns out to be dependent on the alignment between the gradient and the eigenvectors of the

Original: Hessian.
Revised: Hessian.

Original: In particular, like Arora et al.
Revised: In particular, like Arora et al.

Original: [2], Lyu et al.
Revised: [2], Lyu et al.

Original: [18], we show that the gradient aligns to the top eigenvector of Hessian, and thus encourages the SAM dynamics to reduce the top eigenvalue of the Hessian.
Revised: [18], we show that the gradient aligns to the top eigenvector of Hessian, and thus encourages the SAM dynamics to reduce the top eigenvalue of the Hessian.

Original: Taylor Expansion on Φ .
Revised: Taylor Expansion on Φ .

Original: In Phase II, it can be shown that ∥ x ( t ) − Φ( x ( t )) ∥
Revised: In Phase II, it can be shown that ∥ x ( t ) − Φ( x ( t )) ∥

Original: = O ( ηρ ) holds for every step, this implies ∥ x ( t + 1) − x ( t ) ∥ 2 =
Revised: = O ( ηρ ) holds for every step, this implies ∥ x ( t + 1) − x ( t ) ∥ 2 =

Original: O ( ρη ) .
Revised: O ( ρη ) .

Original: (See Lemma 35)
Revised: (See Lemma 35)

Original: Therefore we have that
Revised: Therefore we have that

Original: [Equation]
Revised: [Equation]

Original: Now we apply Taylor expansion on ∇ L (cid:16) x
Revised: Now we apply Taylor expansion on ∇ L (cid:16) x

Original: + ρ ∇ L ( x ) ∥∇ L ( x ) ∥ 2 (cid:17) around x and get ∇ L
Revised: + ρ ∇ L ( x ) ∥∇ L ( x ) ∥ 2 (cid:17) around x and get ∇ L

Original: [Equation]
Revised: [Equation]

Original: Lemma 19 (Lemma B.16 [2])
Revised: Lemma 19 (Lemma B.16 [2])

Original: For x ∈ U , ∂ Φ( x ) ∇ L ( x ) = 0 , ∂ Φ( x ) ∇ 2 L ( x ) ∇ L ( x ) = − ∂ 2 Φ( x )[ ∇ L ( x ) , ∇ L ( x )] .
Revised: For x ∈ U , ∂ Φ( x ) ∇ L ( x ) = 0 , ∂ Φ( x ) ∇ 2 L ( x ) ∇ L ( x ) = − ∂ 2 Φ( x )[ ∇ L ( x ) , ∇ L ( x )] .

Original: Now we plug (10) into (9) and simplify the expression using Lemma 19.
Revised: Now we plug (10) into (9) and simplify the expression using Lemma 19.

Original: We have that 
Revised: We have that Φ

Original: [Equation]
Revised: [Equation]

Original: where in the last step we use the property that ∥ x ( t ) − Φ( x ( t ))
Revised: where in the last step we use the property that ∥ x ( t ) − Φ( x ( t ))

Original: ∥
Revised: ∥

Original: =
Revised: =

Original: O ( ηρ ) , which further implies ∥∇ L ( x ( t ))
Revised: O ( ηρ ) , which further implies ∥∇ L ( x ( t ))

Original: ∥ 2 =
Revised: ∥ 2 =

Original: O ( ηρ ) in Phase II (See Lemma 30).
Revised: O ( ηρ ) in Phase II (See Lemma 30).

Original: Lemma 20 For x ∈ Γ , ∂ Φ( x ) =
Revised: Lemma 20 For x ∈ Γ , ∂ Φ( x ) =

Original: P ⊤ x, Γ , ∂ Φ( x ) ∇ 2 L ( x ) = 0 , where P ⊤ x, Γ is the orthogonal projection matrix of the tangent space of Γ at x .
Revised: P ⊤ x, Γ , ∂ Φ( x ) ∇ 2 L ( x ) = 0 , where P ⊤ x, Γ is the orthogonal projection matrix of the tangent space of Γ at x .

Original: Next we use Lemma 20 to further simplify (11).
Revised: Next we use Lemma 20 to further simplify (11).

Original: Since the entire phase II happens in an O ( ηρ ) - neighborhood of manifold Γ , we have that ∥ x ( t ) − Φ( x ( t ))
Revised: Since the entire phase II happens in an O ( ηρ ) - neighborhood of manifold Γ , we have that ∥ x ( t ) − Φ( x ( t ))

Original: ∥ 2 = O ( ηρ ) , thus both (cid:13)(cid:13) ∂
Revised: ∥ 2 = O ( ηρ ) , thus both (cid:13)(cid:13) ∂

Original: Φ( x ( t )) ∇ 2 L ( x ( t ))
Revised: Φ( x ( t )) ∇ 2 L ( x ( t ))

Original: (cid:13)(cid:13) and ∥∇ L ( x ( t ))
Revised: (cid:13)(cid:13) and ∥∇ L ( x ( t ))

Original: ∥ 2 are O ( ηρ ) .
Revised: ∥ 2 are O ( ηρ ) .

Original: So far, the proof is almost completed, with the implicit alignment between gradient and top eigenvector of Hessian being the last missing piece.
Revised: So far, the proof is almost completed, with the implicit alignment between gradient and top eigenvector of Hessian being the last missing piece.

Original: Suppose we have (cid:13)(cid:13)(cid:13) ∇ L ( x ) ∥∇ L ( x ) ∥ 2 − v
Revised: Suppose we have (cid:13)(cid:13)(cid:13) ∇ L ( x ) ∥∇ L ( x ) ∥ 2 − v

Original: [Equation]
Revised: [Equation]

Original: ∇ 2 L ( x ))
Revised: ∇ 2 L ( x ))

Original: (cid:13)(cid:13)(cid:13) =
Revised: (cid:13)(cid:13)(cid:13) =

Original: O ( ρ ) , then it holds that
Revised: O ( ρ ) , then it holds that

Original: [Equation]
Revised: [Equation]

Original: where the second to last step we use the following property about the derivative of eigenvalue (Lemma 57) and the last step is due to Taylor expansion.
Revised: where the second to last step we use the following property about the derivative of eigenvalue (Lemma 57) and the last step is due to Taylor expansion.

Original: Implicit Hessian-gradient Alignment.
Revised: Implicit Hessian-gradient Alignment.

Original: It remains to explain why the gradient implicitly aligns to the top eigenvector of Hessian, which is the key component of the analysis in Phase II.
Revised: It remains to explain why the gradient implicitly aligns to the top eigenvector of Hessian, which is the key component of the analysis in Phase II.

Original: The proof strategy here is to first show alignment for a quadratic loss function, and then generalize its proof to general loss functions satisfying Assumption 1.
Revised: The proof strategy here is to first show alignment for a quadratic loss function, and then generalize its proof to general loss functions satisfying Assumption 1.

Original: Below we first give the formal statement of the implicit alignment on quadratic loss Theorem 21.
Revised: Below we first give the formal statement of the implicit alignment on quadratic loss Theorem 21.

Original: Note this alignment property is an implicit property of the SAM algorithm – it is not due to the fact that SAM is an approximation of GD on
Revised: Note this alignment property is an implicit property of the SAM algorithm – it is not due to the fact that SAM is an approximation of GD on

Original: L asc ρ , because optimizing L asc ρ would rather make the gradient align to the smallest eigenvector!
Revised: L asc ρ , because optimizing L asc ρ would rather make the gradient align to the smallest eigenvector!

Original: Theorem 21 Suppose A is a positive definite symmetric matrix with unique top eigenvalue.
Revised: Theorem 21 Suppose A is a positive definite symmetric matrix with unique top eigenvalue.

Original: Consider running SAM on loss L ( x ) :
Revised: Consider running SAM on loss L ( x ) :

Original: = 12 x T Ax as (13) , then for almost every x (0) , we have x ( t ) converges in direction to v 1 ( A ) and lim t →∞ ∥ x ( t ) ∥
Revised: = 12 x T Ax as (13) , then for almost every x (0) , we have x ( t ) converges in direction to v 1 ( A ) and lim t →∞ ∥ x ( t ) ∥

Original: = ηρλ 1 ( A ) 2 − ηλ 1 ( A ) with ηλ 1 ( A ) < 1 .
Revised: = ηρλ 1 ( A ) 2 − ηλ 1 ( A ) with ηλ 1 ( A ) < 1 .

Original: [Equation]
Revised: [Equation]

Original: Equivalently, we can reformulate the update rule through the lens of the gradient ∇ L ( x ) =
Revised: Equivalently, we can reformulate the update rule through the lens of the gradient ∇ L ( x ) =

Original: Ax :
Revised: Ax :

Original: [Equation]
Revised: [Equation]

Original: and the result of Theorem 21 becomes the alignment between gradient and the top eigenvector of
Revised: and the result of Theorem 21 becomes the alignment between gradient and the top eigenvector of

Original: Hessian.
Revised: Hessian.

Original: This property can be generalized to the analysis of general loss as the dynamic of SAM near a fixed point on Γ is similar to the quadratic case with small perturbation.
Revised: This property can be generalized to the analysis of general loss as the dynamic of SAM near a fixed point on Γ is similar to the quadratic case with small perturbation.

Original: To see this, we apply
Revised: To see this, we apply

Original: Taylor expansion on the update rule of SAM (3):
Revised: Taylor expansion on the update rule of SAM (3):

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Since phase II happens in an O ( ηρ ) -neighborhood of manifold Γ , we have ∥ x ( t + 1) − x ( t ) ∥ 2 =
Revised: Since phase II happens in an O ( ηρ ) -neighborhood of manifold Γ , we have ∥ x ( t + 1) − x ( t ) ∥ 2 =

Original: O ( ηρ ) .
Revised: O ( ηρ ) .

Original: Then by (15) and Taylor expansion on ∇ L ( x ( t + 1)) at x ( t ) , we have that
Revised: Then by (15) and Taylor expansion on ∇ L ( x ( t + 1)) at x ( t ) , we have that

Original: [Equation]
Revised: [Equation]

Original: Equation (16) is a O ( ηρ 2 ) -perturbed version of the update rule in the quadratic case.
Revised: Equation (16) is a O ( ηρ 2 ) -perturbed version of the update rule in the quadratic case.

Original: Note this is a higher order term comparing to the other two terms, which have orders Θ( η 2 ρ ) and Θ( ηρ ) respectively, the implicit alignment between Hessian and gradient happens for the same reason as in the quadratic case.
Revised: Note this is a higher order term comparing to the other two terms, which have orders Θ( η 2 ρ ) and Θ( ηρ ) respectively, the implicit alignment between Hessian and gradient happens for the same reason as in the quadratic case.

Original: We further show once this alignment happens, it will be kept until the end of our analysis, which is Θ( η − 1 ρ − 2 ) steps.
Revised: We further show once this alignment happens, it will be kept until the end of our analysis, which is Θ( η − 1 ρ − 2 ) steps.

Original: C.2. Proof Sketch for Stochastic Case
Revised: C.2. Proof Sketch for Stochastic Case

Original: Given our results in Section 3, it’s natural to ask if we can justify the implicit Hessian-gradient alignment in the stochastic setting.
Revised: Given our results in Section 3, it’s natural to ask if we can justify the implicit Hessian-gradient alignment in the stochastic setting.

Original: Unfortunately, such alignment is not possible in the most general setting.
Revised: Unfortunately, such alignment is not possible in the most general setting.

Original: For example, take a simple quadratic loss L ( x ) =
Revised: For example, take a simple quadratic loss L ( x ) =

Original: L 1 ( x )+ L 2 ( x ), where L k ( x ) = 0 .
Revised: L 1 ( x )+ L 2 ( x ), where L k ( x ) = 0 .

Original: 5 x ⊤ A k x
Revised: 5 x ⊤ A k x

Original: and A k is a positive definite matrix for k = 1 , 2 .
Revised: and A k is a positive definite matrix for k = 1 , 2 .

Original: If A 1 and A 2 have different top eigendirection, then no x can simultaneously satisfy that ∇ L k ( x ) =
Revised: If A 1 and A 2 have different top eigendirection, then no x can simultaneously satisfy that ∇ L k ( x ) =

Original: A k x aligns to the top eigenvector of ∇ 2 L k ( x ) =
Revised: A k x aligns to the top eigenvector of ∇ 2 L k ( x ) =

Original: A k .
Revised: A k .

Original: Yet when the batch size is 1, we can prove rigorously that stochastic SAM minimizes stochastic worst-direction sharpness (Section 4.2).
Revised: Yet when the batch size is 1, we can prove rigorously that stochastic SAM minimizes stochastic worst-direction sharpness (Section 4.2).

Original: The fundamental difference between stochastic SAM (7) with batch size 1 and that in the general setting is that the rank of the Hessian of each stochastic loss at minimizers is only rank-1, which enforces the gradient ∇ L k ( x ) ≈ ∇ 2 L k ( x )( x − Φ( x )) to (almost) lie in the direction of the top eigenvalue of the Hessian.
Revised: The fundamental difference between stochastic SAM (7) with batch size 1 and that in the general setting is that the rank of the Hessian of each stochastic loss at minimizers is only rank-1, which enforces the gradient ∇ L k ( x ) ≈ ∇ 2 L k ( x )( x − Φ( x )) to (almost) lie in the direction of the top eigenvalue of the Hessian.

Original: Lemma 22 formally states this property.
Revised: Lemma 22 formally states this property.

Original: For convenience, we denote d 2 ℓ ( y ′ ,y k ) d 2 y ′ | y ′ =
Revised: For convenience, we denote d 2 ℓ ( y ′ ,y k ) d 2 y ′ | [Equation]

Original: f k ( x ) ∇ f k ( x ) / ∥∇ f k ( x ) ∥ as Λ k ( x ) , w k ( x ) .
Revised: f k [Equation] / ∥∇ f k ( x ) ∥ as Λ k ( x ) , w k ( x ) .

Original: Lemma 

Revised: Lemma

Original: Under Assumption 1, for any x ∈ U and p ∈ Γ , ∇ 2 L k ( p ) =
Revised: Under Assumption 1, for any x ∈ U and p ∈ Γ , ∇ 2 L k ( p ) =

Original: Λ k ( p ) w k ( p ) w k ( p ) T and ∃ s ∈
Revised: Λ k ( p ) w k ( p ) w k ( p ) T and ∃ s ∈

Original: [Equation]
Revised: [Equation]

Original: With Lemma 22, we can show Φ( x ( t + 1))
Revised: With Lemma 22, we can show Φ( x ( t + 1))

Original: − Φ( x ( t ))
Revised: − Φ( x ( t ))

Original: = − ηρ 2 P ⊤ t, Γ ∇ (Λ k ( x )) / 2 + ˜ O
Revised: = − ηρ 2 P ⊤ t, Γ ∇ (Λ k ( x )) / 2 + ˜ O

Original: η 2 ρ 2 ) when ∥ x ( t + 1) − x ( t ) ∥
Revised: η 2 ρ 2 ) when ∥ x ( t + 1) − x ( t ) ∥

Original: =
Revised: =

Original: ˜ O ( ηρ ) , via a similar but slightly more complicated analysis on Φ( x ( t + 1)) − Φ( x ( t )) as in Section 3.2.
Revised: ˜ O ( ηρ ) , via a similar but slightly more complicated analysis on Φ( x ( t + 1)) − Φ( x ( t )) as in Section 3.2.

Original: As we have E [Λ k ( x )]
Revised: As we have E [Λ k ( x )]

Original: = ( (cid:80) k
Revised: = ( (cid:80) k

Original: Λ k ( x )) /M
Revised: Λ k ( x )) /M

Original: =
Revised: =

Original: Tr( ∇ 2 L ( x )) , we have that locally SAM is essentially performing a stochastic projected gradient descent with respect to
Revised: Tr( ∇ 2 L ( x )) , we have that locally SAM is essentially performing a stochastic projected gradient descent with respect to

Original: Tr( ∇ 2 L ( · )) .
Revised: Tr( ∇ 2 L ( · )) .

Original: It still remains to prove stochastic SAM gets ˜ O ( ηρ ) close of the manifold, which is addressed by Theorem 51.
Revised: It still remains to prove stochastic SAM gets ˜ O ( ηρ ) close of the manifold, which is addressed by Theorem 51.

Original: Appendix D. Proof Details D.1. Setup
Revised: Appendix D. Proof Details D.1. Setup

Original: We will first restate our main assumptions in Section 2.
Revised: We will first restate our main assumptions in Section 2.

Original: Assumption 23 Assume loss L : R D
Revised: Assumption 23 Assume loss L : R D

Original: → R belongs to C 4 , and there exists a manifold Γ that is
Revised: → R belongs to C 4 , and there exists a manifold Γ that is

Original: D − M dimensional C 2 − submanifold of R D for some integer 1 ≤ M ≤ D , where for all x ∈ Γ , x is a local minimizer of L , L ( x ) = 0 and rank ( ∇ 2 L ( x ))
Revised: D − M dimensional C 2 − submanifold of R D for some integer 1 ≤ M ≤ D , where for all x ∈ Γ , x is a local minimizer of L , L ( x ) = 0 and rank ( ∇ 2 L ( x ))

Original: =
Revised: =

Original: M .
Revised: M .

Original: [Figure]
Revised: [Figure]

Original: {+L  -  -
 L  -  -
 L+} Assumption 24 For x ∈ Γ , there exists a positive eigengap,i.e., λ 1 ( x ) > λ 2 ( x ) .
Revised: {+L  -  -
 L  -  -
 L+} Assumption 24 For x ∈ Γ , there exists a positive eigengap,i.e., λ 1 ( x ) > λ 2 ( x ) .

Original: For stochastic loss, we use notation in Assumption 49, a general assumption containing our setup in Theorem 11.
Revised: For stochastic loss, we use notation in Assumption 49, a general assumption containing our setup in Theorem 11.

Original: We additionally define Φ k as the gradient flow with respect to L k .
Revised: We additionally define Φ k as the gradient flow with respect to L k .

Original: We abuse the notation and define λ i ( x ) , v i ( x ) as λ i ( ∇ 2 L (Φ( x ))) , v i ( ∇ 2 L (Φ( x ))) whenever the latter is well defined.
Revised: We abuse the notation and define λ i ( x ) , v i ( x ) as λ i ( ∇ 2 L (Φ( x ))) , v i ( ∇ 2 L (Φ( x ))) whenever the latter is well defined.

Original: When x ( t ) and Γ is clear from context, we also use λ
Revised: When x ( t ) and Γ is clear from context, we also use λ

Original: i ( t ) :
Revised: i ( t ) :

Original: = λ i ( x ( t )) , v i ( t ) :
Revised: = λ i ( x ( t )) , v i ( t ) :

Original: = v i ( x ( t )) , P ⊤ t, Γ := P ⊤ Φ( x ( t )) , Γ , P t, Γ := P Φ( x ( t )) , Γ .
Revised: = v i ( x ( t )) , P ⊤ t, Γ := P ⊤ Φ( x ( t )) , Γ , P t, Γ := P Φ( x ( t )) , Γ .

Original: In our proof, we repeatedly discuss compact set in Γ and their neighborhoods.
Revised: In our proof, we repeatedly discuss compact set in Γ and their neighborhoods.

Original: We will use
Revised: We will use

Original: K ⊂ Γ to denote a compact set.
Revised: K ⊂ Γ to denote a compact set.

Original: This notation may have different meanings in different proof and will be clearly stated.
Revised: This notation may have different meanings in different proof and will be clearly stated.

Original: We further define K d
Revised: We further define K d

Original: = { x | Dist( x, K ) ≤ d } .
Revised: = { x | Dist( x, K ) ≤ d } .

Original: We first present some lemmas regularizing the behavior of L and Γ near C .
Revised: We first present some lemmas regularizing the behavior of L and Γ near C .

Original: Definition 25 A function L is µ − PL in a set U iff ∀ x ∈ U , ∥∇ L ( x ) ∥ 2 ≥ 2 µ ( L ( x ) − inf x ∈ U L ( x )) .
Revised: Definition 25 A function L is µ − PL in a set U iff ∀ x ∈ U , ∥∇ L ( x ) ∥ 2 ≥ 2 µ ( L ( x ) − inf x ∈ U L ( x )) .

Original: Definition 

Revised: Definition

Original: The spectral 2-norm of a k -order tensor Γ i 1 ,...,i k ∈ R d 1 × ...
Revised: The spectral 2-norm of a k -order tensor Γ i 1 ,...,i k ∈ R d 1 × ...

Original: × d k is defined as
Revised: × d k is defined as

Original: [Equation]
Revised: [Equation]

Original: Lemma 

Revised: Lemma

Original: Given C , there is a sufficiently small r ( C ) > 0 such that
Revised: Given C , there is a sufficiently small r ( C ) > 0 such that

Original: K r ∩ Γ is compact 2.
Revised: K r ∩ Γ is compact 2.

Original: K r ⊂ U ∩ ( ∩ k U k ) 3.
Revised: K r ⊂ U ∩ ( ∩ k U k ) 3.

Original: L is µ − PL on K r 4.
Revised: L is µ − PL on K r 4.

Original: inf x ∈ K r ( λ 1 ( ∇ 2 L ( x )) − λ 2 ( ∇ 2 L ( x ))) ≥ ∆ >5.
Revised: inf x ∈ K r ( λ 1 ( ∇ 2 L ( x )) − λ 2 ( ∇ 2 L ( x ))) ≥ ∆ >5.

Original: inf x ∈ K r λ M ( ∇ 2 L ( x )) ≥ µ >6.
Revised: inf x ∈ K r λ M ( ∇ 2 L ( x )) ≥ µ >6.

Original: inf x ∈ K r λ 1 ( ∇ 2 L k ( x )) ≥ µ >
Revised: inf x ∈ K r λ 1 ( ∇ 2 L k ( x )) ≥ µ >

Original: We further assume
Revised: We further assume

Original: [Equation]
Revised: [Equation]

Original: and all these constants are greater than 1 .
Revised: and all these constants are greater than 1 .

Original: We also abuse the notation and use notations like ζ k to denote the same norm for stochastic loss.
Revised: We also abuse the notation and use notations like ζ k to denote the same norm for stochastic loss.

Original: Lemma 

Revised: Lemma

Original: Given C , there is a sufficiently small h ( C ) > 0 such that
Revised: Given C , there is a sufficiently small h ( C ) > 0 such that

Original: supx
Revised: sup

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original:  ∈ K h , Φ( x ) ∈ K r
Revised: x ∈ K h , Φ( x ) ∈ K r

Original: [Equation]
Revised: [Equation]

Original: Then we have
Revised: Then we have

Original: Lemma 

Revised: Lemma

Original: For any x ∈ K h , we have
Revised: For any x ∈ K h , we have

Original: The entire gradient flow trajectory with respect to every stochastic loss L k and L lies in K r . 2.
Revised: The entire gradient flow trajectory with respect to every stochastic loss L k and L lies in K r . 2.

Original: The whole segment x Φ( x ) and x Φ k ( x ) lies in K r .
Revised: The whole segment x Φ( x ) and x Φ k ( x ) lies in K r .

Original: The proof of these lemmas can be found in [2].
Revised: The proof of these lemmas can be found in [2].

Original: D.2. Lemma about Φ
Revised: D.2. Lemma about Φ

Original: In this section, we will introduce some geometric lemma about SAM , which will be heavily used in the analysis below.
Revised: In this section, we will introduce some geometric lemma about SAM , which will be heavily used in the analysis below.

Original: Lemma 30 For x ∈ K h , we have
Revised: Lemma 30 For x ∈ K h , we have

Original: [Equation]
Revised: [Equation]

Original: Lemma 31 For x ∈ K h , we have
Revised: Lemma 31 For x ∈ K h , we have

Original: [Equation]
Revised: [Equation]

Original: , Γ ∇ ( λ 1 ( ∇ 2 ( L ( x ) [Equation] ) , x ∈ Γ .
Revised: P ⊥ X, Γ ∇ ( λ 1 ( ∇ 2 ( L ( x )))) , x ∈ Γ .

Original: Lemma 

Revised: Lemma

Original: At any point x ∈ K h , we have
Revised: At any point x ∈ K h , we have

Original: [Equation]
Revised: [Equation]

Original: The proof of above lemmas can be found in [2]
Revised: The proof of above lemmas can be found in [2]

Original: [Equation]
Revised: [Equation]

Original: Lemma 33 For x ∈ K h ,
Revised: Lemma 33 For x ∈ K h ,

Original: [Equation]
Revised: [Equation]

Original: Proof Consider doing Taylor expansion,
Revised: Proof Consider doing Taylor expansion,

Original: [Equation]
Revised: [Equation]

Revised: and

Revised: [Equation]

Original: Here we use Lemma 28 to ensure the approximation is correct
Revised: Here we use Lemma 28 to ensure the approximation is correct

Original: [Equation]
Revised: [Equation]

Original: Lemma 34 There exists h 1 > 0 , for x ∈ K h and p ∈ C, ∇ 2 L k ( p ) =
Revised: Lemma 34 There exists h 1 > 0 , for x ∈ K h and p ∈ C, ∇ 2 L k ( p ) =

Original: Λ k ( p ) w k ( p ) w k ( p ) T , suppose ∥ x − p
Revised: Λ k ( p ) w k ( p ) w k ( p ) T , suppose ∥ x − p

Original: ∥ < h 1 , there exists s ∈ { 1 , − 1 } ,
Revised: ∥ < h 1 , there exists s ∈ { 1 , − 1 } ,

Original: [Equation]
Revised: [Equation]

Original: Further if we have | w Tk ( x − p ) | ≥ ∥ x − p ∥ 3 / 2 , then we have s = sign ( w Tk ( x − p )) .
Revised: Further if we have | w Tk ( x − p ) | ≥ ∥ x − p ∥ 3 / 2 , then we have s = sign ( w Tk ( x − p )) .

Original: This implies
Revised: This implies

Original: [Equation]
Revised: [Equation]

Original: Proof There are two ways we may use to estimate the direction ∇ L k ( x ) ∥∇ L k ( x ) ∥ .
Revised: Proof There are two ways we may use to estimate the direction ∇ L k ( x ) ∥∇ L k ( x ) ∥ .

Original: First Way According to Lemma 32,
Revised: First Way According to Lemma 32,

Original: [Equation]
Revised: [Equation]

Original: Suppose ∇ 2 L k (Φ k ( x ))
Revised: Suppose ∇ 2 L k (Φ k ( x ))

Original: =
Revised: =

Original: Λ k
Revised: Λ k

Original: (Φ k ( x ))
Revised: (Φ k ( x ))

Original: w k (Φ k ( x ))
Revised: w k (Φ k ( x ))

Original: w k (Φ k ( x )) T , then
Revised: w k (Φ k ( x )) T , then

Original: [Equation]
Revised: [Equation]

Original: Define ∇ 2 L k ( O )
Revised: Define ∇ 2 L k ( O )

Original: = v i v Ti , using Davis-Kahan Theorem 55, we would have ∃ s ∈ {− 1 , 1 } , such that ∥ 
Revised: = v i v Ti , using Davis-Kahan Theorem 55, we would have ∃ s ∈ {− 1 , 1 } , such that ∥ w k

Original: [Equation]
Revised: [Equation]

Original: According to Lemma 30, we have ∥ x − Φ k ( x ) ∥ ≤ ∥∇ L k ( x ) ∥ µ ≤
Revised: According to Lemma 30, we have ∥ x − Φ k ( x ) ∥ ≤ ∥∇ L k ( x ) ∥ µ ≤

Original: [Equation]
Revised: [Equation]

Original: This implies,
Revised: This implies,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Second Way There is another direct way to consider the direction of ∇ L k ( x ) ∥∇ L k ( x [Equation] ∥ .
Revised: Second Way There is another direct way to consider the direction of ∇ L k ( x ) ∥∇ L k ( x ) ∥ .

Original: Doing a Taylor expansion at O ,
Revised: Doing a Taylor expansion at O ,

Original: [Equation]
Revised: [Equation]

Original: That being said, when | w Tk ( x − p ) | ≥ ∥ x − p ∥ 3 / 2 , we have
Revised: That being said, when | w Tk ( x − p ) | ≥ ∥ x − p ∥ 3 / 2 , we have

Original: [Equation]
Revised: [Equation]

Original: Hence we have
Revised: Hence we have

Original: [Equation]
Revised: [Equation]

Original:  Comparing (18) and (19), we have there exists h 1 , such that if ∥ x − p ∥ ≤ h 1 , s = sign ( w Tk ( x − p )) when | w Tk ( x − p ) | ≥ ∥ x − p ∥ 3 / 2 .
Revised: [Equation] Comparing (18) and (19), we have there exists h 1 , such that if ∥ x − p ∥ ≤ h 1 , s = sign ( w Tk ( x − p )) when | w Tk ( x − p ) | ≥ ∥ x − p ∥ 3 / 2 .

Original: The final inequality is self-explanatory.
Revised: The final inequality is self-explanatory.

Original: We will abuse notation slightly and suppose h 1 in Lemma 34 satisfies h 1 ≥ h .
Revised: We will abuse notation slightly and suppose h 1 in Lemma 34 satisfies h 1 ≥ h .

Original: Lemma 35 Suppose x ∈ K h and y = x − η ∇ L (cid:16) x
Revised: Lemma 35 Suppose x ∈ K h and y = x − η ∇ L (cid:16) x

Original: + ρ ∇ L ( x ) ∥∇ L ( x ) ∥ (cid:17) ,
Revised: + ρ ∇ L ( x ) ∥∇ L ( x ) ∥ (cid:17) ,

Original: [Equation]
Revised: [Equation]

Original: Proof Using Taylor Expansion and Lemma 28,
Revised: Proof Using Taylor Expansion and Lemma 28,

Original: [Equation]
Revised: [Equation]

Original: Further y 
Revised: Further y − x = − η ∇ L (cid:18) x

Original: [Equation]
Revised: + ρ ∇ L ( x ) ∥∇ L ( x ) ∥ (cid:19) = − η ∇ L ( x ) − ηρ ∇ 2 L ( x ) ∇ L ( x ) ∥∇ L ( x ) ∥ − ηρ 2 ∂ ∇ 2 L ( x )[ ∇ L ( x ) ∥∇ L ( x ) ∥ , ∇ L ( x ) ∥∇ L ( x ) ∥ ] / 2 + O ( ηρ 3 Υ)

Original: ⇒∥
Revised: ⇒∥

Original: [Equation]
Revised: y − x + η ∇ L ( x ) + ηρ ∇ 2 L ( x ) ∇ L ( x ) ∥∇ L ( x ) ∥ + ηρ 2 ∂ ∇ 2 L ( x )[ ∇ L ( x ) ∥∇ L ( x ) ∥ , ∇ L ( x ) ∥∇ L ( x ) ∥ ] / 2 ∥ ≤ ηρ 3 Υ

Original: This implies
Revised: This implies

Original: [Equation]
Revised: [Equation]

Original: Using Lemma 31,
Revised: Using Lemma 31,

Original: [Equation]
Revised: [Equation]

Original: Putting together we have
Revised: Putting together we have

Original: [Equation]
Revised: [Equation]

Original: Lemmas in this section will be repeatedly used in our proofs.
Revised: Lemmas in this section will be repeatedly used in our proofs.

Original: D.3. Explicit Bias
Revised: D.3. Explicit Bias

Original: We will first prove the generic Theorem 4 and then apply this theorems to characterize a variety of regularizers.
Revised: We will first prove the generic Theorem 4 and then apply this theorems to characterize a variety of regularizers.

Original: Lemma 36 Let U ′ be any bounded open set such that its closure U ′ ⊆
Revised: Lemma 36 Let U ′ be any bounded open set such that its closure U ′ ⊆

Original: U .
Revised: U .

Original: Further assume U ′ ∩ Γ =
Revised: Further assume U ′ ∩ Γ =

Original: U ′ ∩ Γ , then for all h 2 > 0 , ∃ ρ 0 > 0 if x ∈ U ′ , dist( x, Γ) ≤ ρ 0 ⇒ dist( x, U ′ ∩ Γ) ≤ h
Revised: U ′ ∩ Γ , then for all h 2 > 0 , ∃ ρ 0 > 0 if x ∈ U ′ , dist( x, Γ) ≤ ρ 0 ⇒ dist( x, U ′ ∩ Γ) ≤ h

Original: Proof Prove by contradiction.
Revised: Proof Prove by contradiction.

Original: If there exists a list of ρ 1 , ..., ρ k , ... , such that ρ k → 0 and there exists x k ∈ U ′ , such that dist( x k , Γ) ≤ ρ k and dist( x k , U ′ ∩ Γ) ≥ h 2 .
Revised: If there exists a list of ρ 1 , ..., ρ k , ... , such that ρ k → 0 and there exists x k ∈ U ′ , such that dist( x k , Γ) ≤ ρ k and dist( x k , U ′ ∩ Γ) ≥ h 2 .

Original: Then consider accumulation point of x k x ∗ in U ′ .
Revised: Then consider accumulation point of x k x ∗ in U ′ .

Original: Then we would have x ∗ ∈ Γ ∩ U ′ =
Revised: Then we would have x ∗ ∈ Γ ∩ U ′ =

Original: U ′ ∩ Γ .
Revised: U ′ ∩ Γ .

Original: Lemma 37 Let U ′ be any bounded open set such that its closure U ′ ⊆
Revised: Lemma 37 Let U ′ be any bounded open set such that its closure U ′ ⊆

Original: U .
Revised: U .

Original: then for all h 2 > 0 , ∃ ρ 1 > 0 if x ∈ U ′ , L ( x ) ≤ ρ 1 ⇒ dist( x, U ′ ∩ Γ) ≤ h
Revised: then for all h 2 > 0 , ∃ ρ 1 > 0 if x ∈ U ′ , L ( x ) ≤ ρ 1 ⇒ dist( x, U ′ ∩ Γ) ≤ h

Original: Proof Prove by contradiction.
Revised: Proof Prove by contradiction.

Original: If there exists a list of ρ 1 , ..., ρ k , ... , such that ρ k → 0 and there exists x k ∈ U ′ , such that L ( x k ) ≤ ρ k and dist( x k , U ′ ∩ Γ) ≥ h 2 .
Revised: If there exists a list of ρ 1 , ..., ρ k , ... , such that ρ k → 0 and there exists x k ∈ U ′ , such that L ( x k ) ≤ ρ k and dist( x k , U ′ ∩ Γ) ≥ h 2 .

Original: Then consider accumulation point of x k x ∗ in U ′ .
Revised: Then consider accumulation point of x k x ∗ in U ′ .

Original: Then we would have x ∗ ∈ Γ ∩ U ′ =
Revised: Then we would have x ∗ ∈ Γ ∩ U ′ =

Original: U ′ ∩ Γ .
Revised: U ′ ∩ Γ .

Original: Proof [Proof of Theorem 4 ] We will first prove ⇒ side.
Revised: Proof [Proof of Theorem 4 ] We will first prove ⇒ side.

Original: The proof consists of four steps.
Revised: The proof consists of four steps.

Original: Define h as the constant in Lemma 29 with K = U ′
Revised: Define h as the constant in Lemma 29 with K = U ′

Original: ∩ H .
Revised: ∩ H .

Original: Step  Consider x  ∈ U ′ ∩ Γ , satisfying
Revised: Step 1 Consider x 0 ∈ U ′ ∩ Γ , satisfying

Original: [Equation]
Revised: [Equation]

Original: Then using definition 3, we have there exists sufficiently small ϵ ρ < ρ 2 , such that exists ∥ x 1 − x 0 ∥ ≤ ϵ ρ < ρ 2 and ∥ S ( x 0 ) − R ρ ( x 1 ) /ρ 

Revised: Then using definition 3, we have there exists sufficiently small ϵ ρ < ρ 2 , such that exists ∥ x 1 − x 0 ∥ ≤ ϵ ρ < ρ 2 and ∥ S ( x 0 ) − R ρ ( x 1 ) /ρ

Original: ∥ =
Revised: ∥ =

Original: O ( ρ 2 ) .
Revised: O ( ρ 2 ) .

Original: Now that as x 0 ∈ U ′ , we have for sufficiently small ρ , x 1 ∈ U ′ .
Revised: Now that as x 0 ∈ U ′ , we have for sufficiently small ρ , x 1 ∈ U ′ .

Original: By Lemma 36, we have for sufficiently small ρ , x 0 x 1 ∈ U ′
Revised: By Lemma 36, we have for sufficiently small ρ , x 0 x 1 ∈ U ′

Original: ∩ H h , then consider doing Taylor Expansion, we would have L ( x 1 ) + R ρ ( x 1 ) ≤ ρ 2 S ( x 0 )
Revised: ∩ H h , then consider doing Taylor Expansion, we would have L ( x 1 ) + R ρ ( x 1 ) ≤ ρ 2 S ( x 0 )

Original: + o (
Revised: + o (

Original: ρ 2 ) .
Revised: ρ 2 ) .

Original: Step  Now consider L ( u ) + R ρ ( u ) ≤ ρ 2 inf x ∈ U ′ ∩ Γ S ( x ) + ϵρ [-
-] + o ( ρ 2 ) .
Revised: Step 2 Now consider L ( u ) + R ρ ( u ) ≤ ρ 2 inf x ∈ U ′ ∩ Γ S ( x ) + ϵρ [-
-]2 + o ( ρ 2 ) .

Original: We easily have L ( u ) ≤ O ( ρ 2 ) by Assumption 2.
Revised: We easily have L ( u ) ≤ O ( ρ 2 ) by Assumption 2.

Original: By Lemma 37, we have u Φ( u ) ∈ K h and that ∥ u − Φ( u ) ∥ =
Revised: By Lemma 37, we have u Φ( u ) ∈ K h and that ∥ u − Φ( u ) ∥ =

Original: O ( ρ ) by
Revised: O ( ρ ) by

Original: Lemma 30.
Revised: Lemma 30.

Original: This implies dist( U ′ ∩ H, Φ( u ))
Revised: This implies dist( U ′ ∩ H, Φ( u ))

Original: =
Revised: =

Original: o (1) .
Revised: o (1) .

Original: Notice now, we have shown R ρ ( u ) ≤ ρ 2 inf S ( x ) + ρ 2 ϵ
Revised: Notice now, we have shown R ρ ( u ) ≤ ρ 2 inf S ( x ) + ρ 2 ϵ

Original: + o ( ρ 2 )
Revised: + o ( ρ 2 )

Original: Step  Now applying definition of a good limiting regularizer and Finite covering theorem, we have this further implies S (Φ( u ))
Revised: Step 3 Now applying definition of a good limiting regularizer and Finite covering theorem, we have this further implies S (Φ( u ))

Original: ρ 2 ≤ R ρ ( u ) + o ( ρ 2 ) .
Revised: ρ 2 ≤ R ρ ( u ) + o ( ρ 2 ) .

Original: We also have S (Φ( u )) ≥ inf x ∈ U ′
Revised: We also have S (Φ( u )) ≥ inf x ∈ U ′

Original: ∩ Γ S ( x ) − o (1) as
Revised: ∩ Γ S ( x ) − o (1) as

Original: S ∈ C 0 .
Revised: S ∈ C 0 .

Original: Finally R ρ ( u ) ≥ S (Φ( u ))
Revised: Finally R ρ ( u ) ≥ S (Φ( u ))

Original: ρ 2 − o ( ρ 2 ) ≥ ρ 2 inf S ( x ) − o ( ρ 2 ) .
Revised: ρ 2 − o ( ρ 2 ) ≥ ρ 2 inf S ( x ) − o ( ρ 2 ) .

Original: We also have L ( u ) ≤ o ( ρ 2 ) , proving our claim.
Revised: We also have L ( u ) ≤ o ( ρ 2 ) , proving our claim.

Original: For the ⇐ side, we need to provide a lower bound for L ( x ) + R ρ ( x ) .
Revised: For the ⇐ side, we need to provide a lower bound for L ( x ) + R ρ ( x ) .

Original: Define constant C 1 such that for all x , ∥ x − Φ( x ) ∥ ≥ C 1 ρ , we have L ( x ) ≥ ( C U ′ + inf x ∈ U ′
Revised: Define constant C 1 such that for all x , ∥ x − Φ( x ) ∥ ≥ C 1 ρ , we have L ( x ) ≥ ( C U ′ + inf x ∈ U ′

Original: ∩ Γ S ( x ) + 1) ρ 2 where C U ′ is the constant in Assumption 2.
Revised: ∩ Γ S ( x ) + 1) ρ 2 where C U ′ is the constant in Assumption 2.

Original: Then we have for x such that ∥ x − Φ( x ) ∥ ≥ C 1 ρ , we have L ( x )
Revised: Then we have for x such that ∥ x − Φ( x ) ∥ ≥ C 1 ρ , we have L ( x )

Original: +
Revised: +

Original: R ρ ( x ) ≥ (inf x ∈ U ′
Revised: R ρ ( x ) ≥ (inf x ∈ U ′

Original: ∩ Γ S ( x ) + 1) ρ 2 .
Revised: ∩ Γ S ( x ) + 1) ρ 2 .

Original: For ∥ x − Φ( x ) ∥ ≤ C 1 ρ , we have by definition of good limiting regularizer and Finite covering theorem, R ρ ( x ) ≥ ρ 2 S (Φ( x ))
Revised: For ∥ x − Φ( x ) ∥ ≤ C 1 ρ , we have by definition of good limiting regularizer and Finite covering theorem, R ρ ( x ) ≥ ρ 2 S (Φ( x ))

Original: − o ( ρ 2 ) .
Revised: − o ( ρ 2 ) .

Original: As distΦ( x ) , U ′ ∩ Γ = o (1) , we have R ρ ( x ) ≥ inf x ∈ U ′ ∩ Γ S ( x ) − o ( ρ 2 ) , hence L ( x ) + R ρ ( x ) ≥ inf x ∈ U ′
Revised: As distΦ( x ) , U ′ ∩ Γ = o (1) , we have R ρ ( x ) ≥ inf x ∈ U ′ ∩ Γ S ( x ) − o ( ρ 2 ) , hence L ( x ) + R ρ ( x ) ≥ inf x ∈ U ′

Original: ∩ Γ S ( x ) ρ 2 .
Revised: ∩ Γ S ( x ) ρ 2 .

Original: Theorem 4 implies Corollary 38 saying that the minimum of the regularized loss is approximately the sum of the minimum of loss and the minimum of limiting regularizer.
Revised: Theorem 4 implies Corollary 38 saying that the minimum of the regularized loss is approximately the sum of the minimum of loss and the minimum of limiting regularizer.

Original: Corollary 38 Under the setting of Theorem 4,
Revised: Corollary 38 Under the setting of Theorem 4,

Original: [Equation]
Revised: [Equation]

Original: Proof [Proof of Corollary 38]
Revised: Proof [Proof of Corollary 38]

Original: In Theorem 4, choose ϵ = 0 , and choose u such that L ( u )+ R ρ ( u ) ≤ inf x ∈ U ′
Revised: In Theorem 4, choose ϵ = 0 , and choose u such that L ( u )+ R ρ ( u ) ≤ inf x ∈ U ′

Original: ( L ( x ) + R ρ ( x ))
Revised: ( L ( x ) + R ρ ( x ))

Original: + o (
Revised: + o (

Original: ρ 2 ) , the result is then clear.
Revised: ρ 2 ) , the result is then clear.

Original: We will then prove the Corollary 5.
Revised: We will then prove the Corollary 5.

Original: Proof [Proof of Corollary 5] By Theorem 4, we have ∥ L ( u ) − inf x ∈ U ′
Revised: Proof [Proof of Corollary 5] By Theorem 4, we have ∥ L ( u ) − inf x ∈ U ′

Original: L ( x ) ∥
Revised: L ( x ) ∥

Original: =
Revised: =

Original: O ( ρ 2 ) .
Revised: O ( ρ 2 ) .

Original: This implies ∥ u − Φ( u ) ∥
Revised: This implies ∥ u − Φ( u ) ∥

Original: =
Revised: =

Original: O ( ρ ) and also dist( U ′ ∩ H, Φ( u ))
Revised: O ( ρ ) and also dist( U ′ ∩ H, Φ( u ))

Original: =
Revised: =

Original: o (1) .
Revised: o (1) .

Original: We also have ρ 2 S (Φ( x ))
Revised: We also have ρ 2 S (Φ( x ))

Original: − o ( ρ 2 ) ≤ R ρ ( u ) ≤ ρ 2 inf x ∈ U ′
Revised: − o ( ρ 2 ) ≤ R ρ ( u ) ≤ ρ 2 inf x ∈ U ′

Original: ∩ H S ( x ) .
Revised: ∩ H S ( x ) .

Original: We have ∥ S (Φ( x ))
Revised: We have ∥ S (Φ( x ))

Original: − inf x ∈ U ′
Revised: − inf x ∈ U ′

Original: ∩ H S ( x ) ∥
Revised: ∩ H S ( x ) ∥

Original: =
Revised: =

Original: o (1) .
Revised: o (1) .

Original: This further implies ∥ S (Φ( x )) − inf x ∈ U ′
Revised: This further implies ∥ S (Φ( x )) − inf x ∈ U ′

Original: ∩ H S ( x ) ∥
Revised: ∩ H S ( x ) ∥

Original: =
Revised: =

Original: o (1)
Revised: o (1)

Original: Proof [Proof of Theorem 12]
Revised: Proof [Proof of Theorem 12]

Original: Step 1 For assumption
Revised: Step 1 For assumption

Original: [Equation]
Revised: [Equation]

Original: Step 2 For definition of good limiting regularizer, by Davis-Kahan Theorem and assumption 1, S ( x ) =
Revised: Step 2 For definition of good limiting regularizer, by Davis-Kahan Theorem and assumption 1, S ( x ) =

Original: λ 1 ( x ) / 2 is non-negative and continuous on Γ .
Revised: λ 1 ( x ) / 2 is non-negative and continuous on Γ .

Original: We also have
Revised: We also have

Original: [Equation]
Revised: [Equation]

Original: Further for x ∗ ∈ Γ , consider a sufficiently small open set V containing x ∗ in which ∥∇ 3 L ∥ is bounded, for x ∈ V ∩ Γ , for ∥ x ′ − x ∥ ≤ Cρ , we have 
Revised: Further for x ∗ ∈ Γ , consider a sufficiently small open set V containing x ∗ in which ∥∇ 3 L ∥ is bounded, for x ∈ V ∩ Γ , for ∥ x ′ − x ∥ ≤ Cρ , we have [Equation] max ρ

Original: [Equation]
Revised: [Equation]

Original: On the other hand
Revised: On the other hand

Original: [Equation]
Revised: [Equation]

Original: Proof [Proof of Theorem 13]
Revised: Proof [Proof of Theorem 13]

Original: Step 1 For assumption
Revised: Step 1 For assumption

Original: [Equation]
Revised: [Equation]

Original: This constant is by taking minimizer of λ 1 ( ∇ 2 L ) over U ′.
Revised: This constant is by taking minimizer of λ 1 ( ∇ 2 L ) over U ′

Original: Step 2 For definition of good limiting regularizer, by Davis-Kahan Theoremm 55 and assumption 1, S ( x ) =
Revised: Step 2 For definition of good limiting regularizer, by Davis-Kahan Theoremm 55 and assumption 1, S ( x ) =

Original: λ M ( x ) / 2 is non-negative and continuous on Γ .
Revised: λ M ( x ) / 2 is non-negative and continuous on Γ .

Original: Further for x ∗ ∈ Γ , consider a sufficiently small open set V containing x ∗ in which ∥∇ 3 L ∥ is bounded, for x ∈ V ∩ Γ , for ∥ x ′ − x ∥ ≤ Cρ , then we easily have ∥ x ′ − Φ( x ) ∥
Revised: Further for x ∗ ∈ Γ , consider a sufficiently small open set V containing x ∗ in which ∥∇ 3 L ∥ is bounded, for x ∈ V ∩ Γ , for ∥ x ′ − x ∥ ≤ Cρ , then we easily have ∥ x ′ − Φ( x ) ∥

Original: ≤ O ( ρ ) , we have
Revised: ≤ O ( ρ ) , we have

Original: [Equation]
Revised: [Equation]

Original: max
Revised: max

Original: [Equation]
Revised: [Equation]

Original: Using Lemma 32 and Theorem 55, we have
Revised: Using Lemma 32 and Theorem 55, we have

Original: [Equation]
Revised: [Equation]

Original: We also have
Revised: We also have

Original: [Equation]
Revised: [Equation]

Original: Combining we have S ( x ) is a good limiting regularizer of R ( x ) .
Revised: Combining we have S ( x ) is a good limiting regularizer of R ( x ) .

Original: Proof [Proof of Theorem 14]
Revised: Proof [Proof of Theorem 14]

Original: Step 1 For assumption
Revised: Step 1 For assumption

Original: [Equation]
Revised: [Equation]

Original: This constant is by taking minimizer of Tr( ∇ 2 L ) over U ′ .
Revised: This constant is by taking minimizer of Tr( ∇ 2 L ) over U ′ .

Original: Step 2 For definition of good limiting regularizer, by Davis-Kahan Theoremm 55 and assumption 1,
Revised: Step 2 For definition of good limiting regularizer, by Davis-Kahan Theoremm 55 and assumption 1,

Original: S ( x ) =
Revised: S ( x ) =

Original: Tr( x ) / (2 D ) is non-negative and continuous on Γ .
Revised: Tr( x ) / (2 D ) is non-negative and continuous on Γ .

Original: Further for x ∗ ∈ Γ , consider a sufficiently small open set V containing x ∗ in which ∥∇ 3 L ∥ is bounded, for x ∈ V ∩ Γ , for ∥ x ′ − x ∥ ≤ Cρ , we have
Revised: Further for x ∗ ∈ Γ [Equation] consider a sufficiently small open set V containing x ∗ in which ∥∇ 3 L ∥ is bounded, for x ∈ V ∩ Γ , for ∥ x ′ − x ∥ ≤ Cρ , we have

Original: [Equation]
Revised: [Equation]

Original: We also have
Revised: We also have

Original: [Equation]
Revised: [Equation]

Original:  ∇ 2 L ( x )) / 2 D
Revised: = Tr( ∇ 2 L ( x )) / 2 D

Original: Combining we have S ( x ) is a good limiting regularizer of R ( x ) .
Revised: Combining we have S ( x ) is a good limiting regularizer of R ( x ) .

Original: Proof [Proof of Theorem 9]
Revised: Proof [Proof of Theorem 9]

Original: By Theorem 50, Assumption 49 holds.
Revised: By Theorem 50, Assumption 49 holds.

Original: Easily deducted from Theorem 12 Λ k ( x ) is a good limiting regularizer for R max k,ρ on Γ k .
Revised: Easily deducted from Theorem 12 Λ k ( x ) is a good limiting regularizer for R max k,ρ on Γ k .

Original: Then as Γ ⊂ Γ k , Λ k
Revised: Then as Γ ⊂ Γ k , Λ k

Original: ( x ) is a good limiting regularizer for R max k,ρ on Γ .
Revised: ( x ) is a good limiting regularizer for R max k,ρ on Γ .

Original: Hence S ( x ) = (cid:80) k Λ k
Revised: Hence S ( x ) = (cid:80) k Λ k

Original: ( x ) / 2 M =
Revised: ( x ) / 2 M =

Original: Tr( ∇ 2 L(x)) / 2 is a good limiting regularizer of E k [ R max k,ρ ]( x ) on Γ .
Revised: Tr( ∇ 2 L(x)) / 2 is a good limiting regularizer of E k [ R max k,ρ ]( x ) on Γ .

Original: Proof [Proof of Theorem 10]
Revised: Proof [Proof of Theorem 10]

Original: By Theorem 50, Assumption 49 holds.
Revised: By Theorem 50, Assumption 49 holds.

Original: Easily deducted from Theorem 13 Λ k ( x ) is a good limiting regularizer for R asc k,ρ on Γ k as the codimension of Γ k is 1.
Revised: Easily deducted from Theorem 13 Λ k ( x ) is a good limiting regularizer for R asc k,ρ on Γ k as the codimension of Γ k is 1.

Original: Then as Γ ⊂ Γ k , Λ k
Revised: Then as Γ ⊂ Γ k , Λ k

Original: ( x ) is a good limiting regularizer for R max k,ρ on Γ .Hence
Revised: ( x ) is a good limiting regularizer for R max k,ρ on Γ .Hence

Original: S ( x ) =
Revised: S ( x ) =

Original: (cid:80)
Revised: (cid:80)

Original: k Λ k
Revised: k Λ k

Original: ( x ) / 2 M =
Revised: ( x ) / 2 M =

Original: Tr( ∇ 2 L(x)) / 2 is a good limiting regularizer of E k
Revised: Tr( ∇ 2 L(x)) / 2 is a good limiting regularizer of E k

Original: [ R asc k,ρ ]( x ) on Γ .
Revised: [ R asc k,ρ ]( x ) on Γ .

Original: D.4. Full-batch SAM on Quadratic Loss: Proof of Theorem
Revised: D.4. Full-batch SAM on Quadratic Loss: Proof of Theorem

Original: We first simplify the iterate as
Revised: We first simplify the iterate as

Original: [Equation]
Revised: [Equation]

Original: Define ˜ x ( t ) =
Revised: Define ˜ x ( t ) =

Original: Ax ( t ) ρ .
Revised: Ax ( t ) ρ .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Our proof consists of three steps
Revised: Our proof consists of three steps

Original: (1) Preparation Phase ∃ T 1 , ∀ t > T 1 , ∥ P ( j : D ) ˜ x
Revised: (1) Preparation Phase ∃ T 1 , ∀ t > T 1 , ∥ P ( j : D ) ˜ x

Original: ( t ) ∥ ≤ ηλ 2 j (2) Alignment Phase Define S  = { t |∥ ˜ x ( t ) ∥ ≤
Revised: ( t ) ∥ ≤ ηλ 2 j (2) Alignment Phase Define S 1 = { t |∥ ˜ x ( t ) ∥ ≤

Original: ηλ 21 2 − ηλ 1 , t > T 1 } , suppose t, t ′ ∈ S 1 , t ≤ t ′ , then | ˜ x 1 ( t ) | < | ˜ x1 ( t′ )
Revised: ηλ 21 2 − ηλ 1 , t > T 1 } , suppose t, t ′ ∈ S 1 , t ≤ t ′ , then | ˜ x 1 ( t ) | < | ˜ x t

Original:  | (Lemma 45) and we have t ∈ S 1 or t + 1 ∈ S 1 for t ≥ T 1 (Lemma 43).
Revised: ) | (Lemma 45) and we have t ∈ S 1 or t + 1 ∈ S 1 for t ≥ T 1 (Lemma 43).

Original:  Length Convergence ∥ ˜ x ( t ) ∥ will converge to ηλ 21 2 − ηλ
Revised: (3) Length Convergence ∥ ˜ x ( t ) ∥ will converge to ηλ 21 2 − ηλ

Original: D.4.1. P REPARATION P HASE
Revised: D.4.1. P REPARATION P HASE

Original: We define I j = { ˜ x |∥ P ( j : D )
Revised: We define I j = { ˜ x |∥ P ( j : D )

Original: ˜ x
Revised: ˜ x

Original: ∥ ≤ ηλ 2 j } and we will prove the following two lemmas.
Revised: ∥ ≤ ηλ 2 j } and we will prove the following two lemmas.

Original: Lemmawill show this is an invariant set for update rule 20 and Lemma 40 will show that all vectors not in this set will shrink exponentially in norm.
Revised: Lemmawill show this is an invariant set for update rule 20 and Lemma 40 will show that all vectors not in this set will shrink exponentially in norm.

Original: Lemma 

Revised: Lemma

Original: If ˜ x ( t ) ∈ I j , using update rule 20, ˜ x ( t + 1) ∈
Revised: If ˜ x ( t ) ∈ I j , using update rule 20, ˜ x ( t + 1) ∈

Original: I j
Revised: I j

Original: Proof We have by update rule 20,
Revised: Proof We have by update rule 20,

Original: [Equation]
Revised: [Equation]

Original: Hence 
Revised: Hence ∥ P

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: We have ∥ ˜ x
Revised: We have ∥ ˜ x

Original: ( t ) ∥ ≤ ηλ 2 j 1 − ηλ j by assumption,
Revised: ( t ) ∥ ≤ ηλ 2 j 1 − ηλ j by assumption,

Original: I (1 − ηλ j − η 
Revised: I (1 − ηλ j − η λ

Revised: [Equation]

Revised: ∥ P

Revised: [Equation]

Revised: ˜

Original: [Equation]
Revised: [Equation]

Revised: t

Original:  Then we have ∥ I − P ( j : D ) ηA − η P ( j : D )
Revised: [Equation] ≺ [Equation] Then we have ∥ I − P ( j : D ) ηA − η P ( j : D )

Original: A 2 ∥
Revised: A 2 ∥

Original: ˜ x
Revised: ˜ x

Original: ( t ) ∥
Revised: ( t ) ∥

Original: ∥ ≤ max(1 , ηλ j + η λ 
Revised: ∥ ≤ max(1 , ηλ j + η λ 2 j ∥ P ( j : D )

Original: [Equation]
Revised: ˜

Original: j ∥ P ( j : D ) ˜ x ( t ) ∥ − 1)Hence ∥ P ( j : D [Equation]
Revised: x ( t ) ∥ − 1)

Original:  ∥ P
Revised: Hence ∥ P

Original: [Equation]
Revised: [Equation]

Original: j : D)
Revised: j : D

Original: [Equation]
Revised: [Equation]

Original: t )

Original: ∥

Original: [Equation]
Revised: [Equation]

Original: ηλ
Revised: max

Original: [Equation]
Revised: [Equation]

Original: 1 −
Revised: P

Original: [Equation]
Revised: [Equation]

Original: P
Revised: j : D ) ˜ x ( t ) ∥ , ηλ 2 j −

Original: [Equation]
Revised: [Equation]

Revised: 1 − ηλ j ) ∥

Original:  j : D )
Revised: P ( j : D )

Original: [Equation]
Revised: [Equation]

Revised: ( t ) ∥ ) ≤ ηλ 2 j

Original: Here the last equation use 1 − ηλ j ≥ 0 .
Revised: Here the last equation use 1 − ηλ j ≥ 0 .

Original: We have by definition ˜
Revised: We have by definition ˜

Original: x ( t + 1) ∈ I j
Revised: x ( t + 1) ∈ I j

Original: Lemma 

Revised: Lemma

Original: If ˜ x ( t ) ̸∈ I j , then ∥ P ( j : D )
Revised: If ˜ x ( t ) ̸∈ I j , then ∥ P ( j : D )

Original: ˜ x
Revised: ˜ x

Original: ( t + 1) ∥ ≤ max (1 − ηλ D , ηλ j )
Revised: ( t + 1) ∥ ≤ max (1 − ηλ D , ηλ j )

Original: ∥ P ( j : D ) ˜ x
Revised: ∥ P ( j : D ) ˜ x

Original: ( t ) ∥
Revised: ( t ) ∥

Original: Proof
Revised: Proof

Original: [Equation]
Revised: [Equation]

Original: We have ∥ ˜ x
Revised: We have ∥ ˜ x

Original: ( t ) ∥ ≥ ∥ P ( j : D ) ˜ x ( t ) ∥ > ηλ 2 j , hence η P ( j : D )
Revised: ( t ) ∥ ≥ ∥ P ( j : D ) ˜ x ( t ) ∥ > ηλ 2 j , hence η P ( j : D )

Original: A 2 ∥
Revised: A 2 ∥

Original: ˜ x
Revised: ˜ x

Original: ( t ) ∥ 
Revised: ( t ) ∥ ≺ η P ( j

Original: [Equation]
Revised: [Equation]

Revised: j ≺

Revised: I

Original: This implies
Revised: This implies

Original: [Equation]
Revised: [Equation]

Original: Hence we have
Revised: Hence we have

Original: [Equation]
Revised: [Equation]

Original: Lemma 41 Choosing T 1 = max
Revised: Lemma 41 Choosing T 1 = max

Original: j
Revised: j

Original: [Equation]
Revised: [Equation]

Original:  , then ∀ t ≥ T 1 , D > j ≥ 1 , ˜ x ( t ) ∈
Revised: (cid:19) , then ∀ t ≥ T 1 , D > j ≥ 1 , ˜ x ( t ) ∈

Original: I j
Revised: I j

Original: Proof Proof by contradiction, suppose ∃ j, T > T 1 , ˜ x
Revised: Proof Proof by contradiction, suppose ∃ j, T > T 1 , ˜ x

Original: ( T )
Revised: ( T )

Original: ̸∈ I j .
Revised: ̸∈ I j .

Original: By Lemma 39, ∀ t < T, ˜ x ( t )
Revised: By Lemma 39, ∀ t < T, ˜ x ( t )

Original: ̸∈ I j .
Revised: ̸∈ I j .

Original: Then by Lemma 40, ∥ P ( j : D ) ˜ x ( T ) ∥ ≤ (1 − ηλ j ) T ∥
Revised: Then by Lemma 40, ∥ P ( j : D ) ˜ x ( T ) ∥ ≤ (1 − ηλ j ) T ∥

Original: P ( j : D ) ˜ x (0) ∥ ≤ max (1 − ηλ D , ηλ j ) T ∥
Revised: P ( j : D ) ˜ x (0) ∥ ≤ max (1 − ηλ D , ηλ j ) T ∥

Original: P ( j : D ) ˜ x (0) ∥ ≤ ηλ 2 j , which is a contradiction.
Revised: P ( j : D ) ˜ x (0) ∥ ≤ ηλ 2 j , which is a contradiction.

Original: D.4.2. A LIGNMENT P HASE
Revised: D.4.2. A LIGNMENT P HASE

Original: Define 
Revised: Define θ ( t ) = arccos( |⟨ ˜ x ( t ) ,e

Original: [Equation]
Revised: [Equation]

Original: = arccos
Revised: | ∥ ˜ x ( t ) ∥ ) , [Equation] x i

Original: [Equation]
Revised: [Equation]

Revised: t ) = ⟨

Revised: [Equation]

Revised: x, e i ⟩

Original: We will first show the following lemma.
Revised: We will first show the following lemma.

Original: Lemma 42 When ∥
Revised: Lemma 42 When ∥

Original: ˜ x
Revised: ˜ x

Original: ( t ) ∥ ≤ ηλ 21 2 − ηλ 1 , using update rule
Revised: ( t ) ∥ ≤ ηλ 21 2 − ηλ 1 , using update rule

Original: [Equation]
Revised: [Equation]

Original: Proof We have | ˜ x 1 ( t + 1) | = | 1 − ηλ 1 − η λ 21 ∥
Revised: Proof We have | ˜ x 1 ( t + 1) | = | 1 − ηλ 1 − η λ 21 ∥

Original: ˜ x ( t ) ∥
Revised: ˜ x ( t ) ∥

Original: || ˜ x 1 ( t ) |
Revised: || ˜ x 1 ( t ) |

Original: We also have η λ 21 ∥
Revised: We also have η λ 21 ∥

Original: ˜ x ( t ) ∥ > 2 − ηλ 21 , hence 1 − ηλ 1 − η λ 21 ∥
Revised: ˜ x ( t ) ∥ > 2 − ηλ 21 , hence 1 − ηλ 1 − η λ 21 ∥

Original: ˜ x ( t ) ∥ < − 1 .
Revised: ˜ x ( t ) ∥ < − 1 .

Original: Hence we have | ˜ x 1 ( t + 1) | > | ˜ x 1 ( t ) |
Revised: Hence we have | ˜ x 1 ( t + 1) | > | ˜ x 1 ( t ) |

Original: On the other hand, ∥ P (2: D ) ˜ x
Revised: On the other hand, ∥ P (2: D ) ˜ x

Original: ( t +1) ∥ ≤ max (cid:16) | 1 − ηλ 2 − η λ 22 ∥
Revised: ( t +1) ∥ ≤ max (cid:16) | 1 − ηλ 2 − η λ 22 ∥

Original: ˜ x ( t ) ∥ | , | 1 − ηλ D − η λ 2 D ∥
Revised: ˜ x ( t ) ∥ | , | 1 − ηλ D − η λ 2 D ∥

Original: ˜ x ( t ) ∥ | (cid:17) ∥ P (2: D )
Revised: ˜ x ( t ) ∥ | (cid:17) ∥ P (2: D )

Original: ˜ x ( t ) ∥
Revised: ˜ x ( t ) ∥

Original: Notice that
Revised: Notice that

Original: [Equation]
Revised: [Equation]

Original: ∥
Revised: ∥

Original: [Equation]
Revised: [Equation]

Original: −
Revised: ( t

Original: [Equation]
Revised: [Equation]

Original: ηλ
Revised: ∥ ≤

Original: [Equation]
Revised: [Equation]

Original:  λ∥ ˜ x
Revised: + η λ∥ ˜ x

Original: ( t ) ∥ −
Revised: ( t ) ∥ −

Original: Hence max (cid:16) | 1 − ηλ 2 − η λ 22 ∥
Revised: Hence max (cid:16) | 1 − ηλ 2 − η λ 22 ∥

Original: ˜ x ( t ) ∥ | , | 1 − ηλ D − η λ 2 D ∥
Revised: ˜ x ( t ) ∥ | , | 1 − ηλ D − η λ 2 D ∥

Original: ˜ x ( t ) ∥ | (cid:17) < | 1 − ηλ 1 − η λ 21 ∥
Revised: ˜ x ( t ) ∥ | (cid:17) < | 1 − ηλ 1 − η λ 21 ∥

Original: ˜ x ( t
Revised: ˜ x ( t

Original: [Equation]
Revised: [Equation]

Original: ∥ | .
Revised: ∥ | .

Original: Lemma 43 ∥
Revised: Lemma 43 ∥

Original: ˜ x ( t ) ∥ > ηλ 21 2 − ηλ 1 , ˜ x
Revised: ˜ x ( t ) ∥ > ηλ 21 2 − ηλ 1 , ˜ x

Original: ( t ) ∈
Revised: ( t ) ∈

Original: ∩ I j , then using update rule 20,∥ ˜
Revised: ∩ I j , then using update rule 20,

Original: [Equation]
Revised: [Equation]

Original: max(

Original: [Equation]

Original: Proof
Revised: Proof

Original: [Equation]
Revised: [Equation]

Original: j = (cid:0) (1 − ηλ j 
Revised: j = (cid:0) (1 − ηλ j ) ∥

Original: ) ∥ ˜ x 
Revised: ˜ x (

Original: t
Revised: [Equation]

Original: ∥ −ηλ
Revised: ∥ −

Original: 2 j (cid:1)
Revised: [Equation]

Original: ˜
Revised: ˜

Original: x j ( t ) e j
Revised: [Equation] j

Original: Consider the following three cases.
Revised: Consider the following three cases.

Original: Case 1 ∀ i, (cid:12)(cid:12) (1 − ηλ 1 ) ∥
Revised: Case 1 ∀ i, (cid:12)(cid:12) (1 − ηλ 1 ) ∥

Original: ˜ x ( t ) ∥ − ηλ 21 (cid:12)(cid:12) ≥ (cid:12)(cid:12) (1 − ηλ i ) ∥
Revised: ˜ x ( t ) ∥ − ηλ 21 (cid:12)(cid:12) ≥ (cid:12)(cid:12) (1 − ηλ i ) ∥

Original: ˜ x ( t ) ∥ − ηλ 2 i (cid:12)(cid:12)
Revised: ˜ x ( t ) ∥ − ηλ 2 i (cid:12)(cid:12)

Original: In this case we have ∥ ˜ x
Revised: In this case we have ∥ ˜ x

Original: ( t +
Revised: ( t +

Original: ∥ ≤ (cid:12)(cid:12)
Revised: ∥ ≤ (cid:12)(cid:12)

Original: 1 − ηλ 1 ) ∥
Revised: (1 − ηλ 1 ) ∥

Original: ˜ x ( t ) ∥ − ηλ 21 (cid:12)(cid:12) =
Revised: ˜ x ( t ) ∥ − ηλ 21 (cid:12)(cid:12) =

Original: ηλ 21 − (1 − ηλ 1 ) ∥
Revised: ηλ 21 − (1 − ηλ 1 ) ∥

Original: ˜ x ( t ) ∥
Revised: ˜ x ( t ) ∥

Original: A more detailed analysis would show ∥ ˜ x
Revised: A more detailed analysis would show ∥ ˜ x

Original: ( t + 1) ∥ is upper bounded by
Revised: ( t + 1) ∥ is upper bounded by

Original: (cid:113)(cid:0) ηλ 21 − (1 − ηλ 1 ) ∥
Revised: (cid:113)(cid:0) ηλ 21 − (1 − ηλ 1 ) ∥

Original: ˜ x ( t ) ∥ (cid:1) 2 cos 2 ( θ ( t ))
Revised: ˜ x ( t ) ∥ (cid:1) 2 cos 2 ( θ ( t ))

Original: [Equation]
Revised: [Equation]

Original: 1 − ηλ
Revised: ηλ

Original: [Equation]
Revised: [Equation]

Original: ˜ x

Original: [Equation]
Revised: [Equation]

Original:  t ) ∥|} sin
Revised: ηλ 2 ) ∥ ˜ x ( t ) ∥|

Original: [Equation]
Revised: [Equation]

Original:  θ ( t ))
Revised: ηλ 2 D − (1 − ηλ D ) ∥ ˜ x ( t ) ∥|} sin 2 ( θ ( t ))

Original: Case
Revised: Case

Original: 2 ∃ i, (cid:12)(cid:12) (1 − ηλ 1 ) ∥
Revised: 2 ∃ i, (cid:12)(cid:12) (1 − ηλ 1 ) ∥

Original: ˜ x ( t ) ∥ − ηλ 21 (cid:12)(cid:12) < (cid:12)(cid:12) (1 − ηλ i ) ∥
Revised: ˜ x ( t ) ∥ − ηλ 21 (cid:12)(cid:12) < (cid:12)(cid:12) (1 − ηλ i ) ∥

Original: ˜ x ( t ) ∥ − ηλ 2 i (cid:12)(cid:12) ,suppose WLOG,
Revised: ˜ x ( t ) ∥ − ηλ 2 i (cid:12)(cid:12) ,

Original:  i is the smallest among such index.
Revised: suppose WLOG, i is the smallest among such index.

Original: As
Revised: As

Original: ηλ [Equation] i ) ∥
Revised: ηλ 2 i − (1 − ηλ i ) ∥

Original: ˜ x ( t ) ∥ [Equation] ηλ [Equation]
Revised: ˜ x ( t ) ∥ < ηλ 21 − (1 − ηλ 1 ) ∥

Original: − (

Original: [Equation] ηλ [Equation] t ) 
Revised: ˜ x ( t ) ∥

Original: [Equation] [Equation] [Equation]
Revised: = (cid:12)(cid:12) (1 − ηλ 1 ) ∥

Original: ˜ x ( t ) ∥ [Equation] 21 (cid:12)(cid:12)
Revised: ˜ x ( t ) ∥ − ηλ 21 (cid:12)(cid:12)

Original: We have
Revised: We have

Original: − ηλ 2 i + (1 − ηλ i ) ∥ ˜ x ( t ) ∥ > ηλ 21 − (1 − ηλ 1 ) ∥
Revised: [Equation]

Original: ˜ x ( t ) ∥ .
Revised: x ( t ) ∥ .

Original: Equivalently,∥
Revised: Equivalently,

Original: [Equation]

Original: x

Original: [Equation]
Revised: [Equation]

Original: i [Equation] Combining with ˜ x ( t ) ∈ I 1 ⇒ ∥
Revised: Combining with ˜ x ( t ) ∈ I 1 ⇒ ∥

Original: ˜ x ( t ) ∥ ≤ ηλ 21 , we have η < λ 1 − λ i λ 21 .
Revised: ˜ x ( t ) ∥ ≤ ηλ 21 , we have η < λ 1 − λ i λ 21 .

Original: Now consider the following vertors,
Revised: Now consider the following vertors,

Original: [Equation]
Revised: [Equation]

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: By assumption
Revised: By assumption

Original: [Equation]
Revised: [Equation]

Original: we have ˜ x ( t ) ∈
Revised: we have ˜ x ( t ) ∈

Original: ∩ I j , hence we have
Revised: ∩ I j , hence we have

Original: [Equation]

Original: t )

Original: ∥

Original: [Equation]
Revised: [Equation]

Original: Using AM-GM inequality
Revised: Using AM-GM inequality

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Hence
Revised: Hence

Original: [Equation]
Revised: [Equation]

Original: So,
Revised: So,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: λ
Revised: λ

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: 2 λ

Original: We further discuss three cases
Revised: We further discuss three cases

Original: Case 2.1 ηλ i
Revised: Case 2.1 ηλ i

Original: [Equation]
Revised: [Equation]

Original: In this case we have ∥ ˜ x
Revised: In this case we have ∥ ˜ x

Original: ( t ) ∥ 
Revised: ( t ) ∥ > ηλ 21 + ηλ [Equation] − ηλ

Original: [Equation]
Revised: [Equation]

Original:  i (cid:114)12 λ
Revised: > ηλ i (cid:114)

Revised: 12 λ 2 i + λ1 − ηλ 1 ,then ∥

Revised: ˜ x ( t + 1) ∥ ≤ ηλ 21 + ηλ 2 i (2 − ηλ 1 − 2 3 ηλ i ) − (1 − ηλ 1 ) ∥

Revised: ˜ x ( t ) ∥ − η

Original: [Equation]
Revised: [Equation]

Original: 1 − ηλ 1 ,then ∥ ˜
Revised: λ 2 i (12 λ

Original: [Equation]
Revised: [Equation]

Revised: + λ 21 )

Revised: [Equation]

Revised: η λ

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: λ

Original: [Equation]
Revised: [Equation]

Original: The second line is because (1 − ηλ 1 ) ∥
Revised: The second line is because (1 − ηλ 1 ) ∥

Original: ˜ x ( t ) ∥
Revised: ˜ x ( t ) ∥

Original: + η 2 λ 2 i ( 12 λ 2 i + λ 21 ) 1 ∥
Revised: + η 2 λ 2 i ( 12 λ 2 i + λ 21 ) 1 ∥

Original: ˜ x ( t ) ∥ monotonously increase w.r.t ∥ ˜ x ( t ) ∥ when ∥ ˜ x
Revised: ˜ x ( t ) ∥ monotonously increase w.r.t ∥ ˜ x ( t ) ∥ when ∥ ˜ x

Original: ( t ) ∥ > ηλ i (cid:114)12 λ
Revised: ( t ) ∥ > ηλ i (cid:114)

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ηλ 1 .

Original: The last line is due to technical lemma Lemma 59.
Revised: The last line is due to technical lemma Lemma 59.

Original: [Equation]
Revised: [Equation]

Original: The second line is because of AM-GM inequality.
Revised: The second line is because of AM-GM inequality.

Original: The last line is due to technical lemma
Revised: The last line is due to technical lemma

Original: Lemma 61.
Revised: Lemma 61.

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: In this case we have ∥ ˜ x
Revised: In this case we have ∥ ˜ x

Original: ( t ) ∥ < ηλ 21 <
Revised: ( t ) ∥ < ηλ 21 <

Original: ηλ i (cid:114)
Revised: ηλ i (cid:114)

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: 2 i +

Original: [Equation]
Revised: [Equation]

Original:  ηλ 1 , then
Revised: 1 − ηλ 1 , then

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: λ 4 D 2 λ
Revised: λ [Equation] 2 λ

Original: [Equation]
Revised: [Equation]

Original: 4 D2 λ
Revised: 4 D

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: The second line is because (1 − ηλ 1 ) ∥
Revised: The second line is because (1 − ηλ 1 ) ∥

Original: ˜ x ( t ) ∥
Revised: ˜ x ( t ) ∥

Original: + η 2 λ 2 i ( 12 λ 2 i + λ 21 ) 1 ∥˜ x ( [Equation]
Revised: + η 2 λ 2 i ( 12 λ 2 i + λ 21 ) 1 ∥

Original:  ) ∥ monotonously decrease w.r.t ∥ ˜ x ( t ) ∥ when ∥ ˜ x
Revised: ˜ x ( t ) ∥ monotonously decrease w.r.t ∥ ˜ x ( t ) ∥ when ∥ ˜ x

Original: ( t ) ∥ < ηλ i {+(cid:114) 
+}
Revised: ( t ) ∥ < ηλ i {+(cid:114) 
+}

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original:  ηλ 1 .
Revised: 1 − ηλ 1 .

Original: The last line is due to technical lemma Lemma 60.
Revised: The last line is due to technical lemma Lemma 60.

Original: Lemma 

Revised: Lemma

Original: If ∥ ˜ x
Revised: If ∥ ˜ x

Original: ( t ) ∥ ≤ ηλ 21 2 − ηλ 1 , ˜ x ( t ) ∈
Revised: ( t ) ∥ ≤ ηλ 21 2 − ηλ 1 , ˜ x ( t ) ∈

Original: ∩ I j , then ∥
Revised: ∩ I j , then ∥

Original: ˜ x
Revised: ˜ x

Original: ( t + 1) ∥ ≤ ηλ 21 − (1 − ηλ 1 ) ∥
Revised: ( t + 1) ∥ ≤ ηλ 21 − (1 − ηλ 1 ) ∥

Original: ˜ x ( t ) ∥
Revised: ˜ x ( t ) ∥

Original: Proof This can be directly inferred from the proof of lemma
Revised: Proof This can be directly inferred from the proof of lemma

Original: Lemma 45 Define S 

Revised: Lemma 45 Define S

Original: =
Revised: =

Original: { t |∥ ˜ x ( t ) ∥ ≤ ηλ 21 2 − ηλ 1 , t > T 1 } , suppose t, t ′ ∈ S 1 , t ≤ t ′ , then | ˜ x 1 ( t ) | < | ˜ x 1 ( t ′ )
Revised: { t |∥ ˜ x ( t ) ∥ ≤ ηλ 21 2 − ηλ 1 , t > T 1 } , suppose t, t ′ ∈ S 1 , t ≤ t ′ , then | ˜ x 1 ( t ) | < | ˜ x 1 ( t ′ )

Original: |
Revised: |

Original: Proof For t ∈ S 1 , by Lemma 43, t + 1 ∈ S 1 or t + 1 ̸∈ S 1 , t + 2 ∈ S 1 .
Revised: Proof For t ∈ S 1 , by Lemma 43, t + 1 ∈ S 1 or t + 1 ̸∈ S 1 , t + 2 ∈ S 1 .

Original: Case 1 t + 1 ∈ S 1 , we can use Lemma 42 to show | ˜ x 1 ( t ) | < | ˜ x 1 ( t + 1) | .
Revised: Case 1 t + 1 ∈ S 1 , we can use Lemma 42 to show | ˜ x 1 ( t ) | < | ˜ x 1 ( t + 1) | .

Original: Case 2 t + 1 ̸∈ S 1 , t + 2 ∈ S 1 .
Revised: Case 2 t + 1 ̸∈ S 1 , t + 2 ∈ S 1 .

Original: |
Revised: |

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: | ˜ x
Revised: | ˜ x

Original: ( t ) |
Revised: ( t ) |

Original: We only need to prove ( ηλ 21 − (1 − ηλ 1 ) ∥
Revised: We only need to prove ( ηλ 21 − (1 − ηλ 1 ) ∥

Original: ˜ x ( t ) ∥ )( ηλ 21 − (1 − ηλ 1 ) ∥
Revised: ˜ x ( t ) ∥ )( ηλ 21 − (1 − ηλ 1 ) ∥

Original: ˜ x ( t + 1) ∥ ) > ∥ ˜ x ( t ) ∥∥
Revised: ˜ x ( t + 1) ∥ ) > ∥ ˜ x ( t ) ∥∥

Original: ˜ x ( t + 1) ∥
Revised: ˜ x ( t + 1) ∥

Original: ⇐⇒ η 2 λ 41 − ηλ 21 (1 − ηλ 1 )( ∥ ˜ x ( t ) ∥
Revised: ⇐⇒ η 2 λ 41 − ηλ 21 (1 − ηλ 1 )( ∥ ˜ x ( t ) ∥

Original: + ∥ ˜ x ( t + 1) ∥ )
Revised: + ∥ ˜ x ( t + 1) ∥ )

Original: + ( − 2 ηλ 1 + η 2 λ 21 ) ∥
Revised: + ( − 2 ηλ 1 + η 2 λ 21 ) ∥

Original: ˜ x ( t ) ∥∥
Revised: ˜ x ( t ) ∥∥

Original: ˜ x ( t + 1) ∥ ≥⇐⇒
Revised: ˜ x ( t + 1) ∥ ≥⇐⇒

Original: η 2 λ 41 − ηλ 21 (1 − ηλ 1 ) ∥
Revised: η 2 λ 41 − ηλ 21 (1 − ηλ 1 ) ∥

Original: ˜ x ( t ) ∥ ≥ (cid:0) (2 ηλ 1 − η 2 λ 21 ) ∥
Revised: ˜ x ( t ) ∥ ≥ (cid:0) (2 ηλ 1 − η 2 λ 21 ) ∥

Original: ˜ x ( t ) ∥
Revised: ˜ x ( t ) ∥

Original: + ηλ 21 (1 − ηλ 1 ) (cid:1) ∥
Revised: + ηλ 21 (1 − ηλ 1 ) (cid:1) ∥

Original: ˜ x ( t + 1) ∥
Revised: ˜ x ( t + 1) ∥

Original: Now using Lemma 44, we only need to prove, η 2 λ 41 − ηλ 21 (1 − ηλ 1 ) ∥
Revised: Now using Lemma 44, we only need to prove, η 2 λ 41 − ηλ 21 (1 − ηλ 1 ) ∥

Original: ˜ x ( t ) ∥ ≥ (cid:0) (2 ηλ 1 − η 2 λ 21 ) ∥
Revised: ˜ x ( t ) ∥ ≥ (cid:0) (2 ηλ 1 − η 2 λ 21 ) ∥

Original: ˜ x ( t ) ∥
Revised: ˜ x ( t ) ∥

Original: + ηλ 21 (1 − ηλ 1 ) (cid:1) (cid:0) ηλ 21 − (1 − ηλ 1 ) ∥
Revised: + ηλ 21 (1 − ηλ 1 ) (cid:1) (cid:0) ηλ 21 − (1 − ηλ 1 ) ∥

Original: ˜ x ( t ) ∥ (cid:1)
Revised: ˜ x ( t ) ∥ (cid:1)

Original: Through some calculation, this is equivalent to ((2 − ηλ 1 ) ∥
Revised: Through some calculation, this is equivalent to ((2 − ηλ 1 ) ∥

Original: ˜ x ( t ) ∥ − ηλ21 )(
Revised: ˜ x ( t ) ∥ − ηλ

Original: [Equation]
Revised: [Equation]

Original: which holds for ∥ ˜ x ( t ) ∥ ≤ ηλ 21 2 − ηλ 1 .
Revised: which holds for ∥ ˜ x ( t ) ∥ ≤ ηλ 21 2 − ηλ 1 .

Original: Concluding the two cases and use induction, we can get the desired result.
Revised: Concluding the two cases and use induction, we can get the desired result.

Original: D.4.3. L ENGTH C ONVERGENCE
Revised: D.4.3. L ENGTH C ONVERGENCE

Original: As Lemma 45 show, ∥ ˜ x 1 ( t ) ∥ increase monotonously for t ∈ S 1 .
Revised: As Lemma 45 show, ∥ ˜ x 1 ( t ) ∥ increase monotonously for t ∈ S 1 .

Original: We can inferred from Lemma 43,
Revised: We can inferred from Lemma 43,

Original: S 1 is infinite.
Revised: S 1 is infinite.

Original: ∀ ϵ > 0 , ∃ T ϵ satisfies ∀ t,
Revised: ∀ ϵ > 0 , ∃ T ϵ satisfies ∀ t,

Original: [Equation]
Revised: [Equation]

Original: Then ∀ t > T ϵ , we have
Revised: Then ∀ t > T ϵ , we have

Original: [Equation]
Revised: [Equation]

Original: or 1 + 
Revised: or 1 + ϵ ≥

Original: [Equation]
Revised: [Equation]

Revised: +

Revised: [Equation]

Revised: ) ∥ ∥

Revised: ˜ x 1 ( t ) ∥

Revised: = ( ηλ 21 − (1 − ηλ 1 ) ∥

Original:  ≥ 
Revised: ˜ x ( t ) ∥ )( ηλ 21 − (1 − ηλ 1 ) ∥ ˜ x ( t + 1) ∥ ) ∥ ˜ x ( t ) ∥∥ ˜ x ( t + 1) ∥ ≥ ( ηλ

Revised: [Equation]

Revised: (1 − ηλ

Revised: [Equation]

Revised: [Equation]

Revised: ( t )

Original: [Equation]
Revised: [Equation]

Revised: ) (cid:0) ηλ

Revised: [Equation]

Original: ∥ ∥
Revised: [Equation] ) (cid:0) ηλ 21 − ( [Equation] ηλ 1 ) ∥

Original: [Equation]
Revised: [Equation]

Original: =
Revised: x [Equation] t ) ∥ (cid:1)(cid:1) ∥

Original: [Equation]
Revised: [Equation]

Revised: ( t )

Revised: [Equation]

Revised: [Equation]

Revised: (1 − ηλ

Revised: [Equation]

Revised: [Equation]

Original: ∥ ˜ x ( t ) ∥
Revised: x [Equation] t ) [Equation]

Revised: (cid:1) =

Revised: ηλ 21 − (1 − ηλ 1 ) (cid:0) ηλ 21 − (1 − ηλ

Revised: [Equation]

Revised: ) ∥ ˜ x ( t ) ∥ (cid:1) ∥

Revised: ˜ x ( t ) ∥

Original: Hence ∥ ˜ x
Revised: Hence ∥ ˜ x

Original: ( t ) ∥ ≥ min (cid:16) ηλ2 − ηλ 21 + ϵ , η 2 λ(2 − λ 1 η ) λ 1 η + ϵ (cid:17) , ∀ t > T ϵ , t ∈ S 1 .
Revised: ( t ) ∥ ≥ min (cid:16) ηλ2 − ηλ 21 + ϵ , η 2 λ(2 − λ 1 η ) λ 1 η + ϵ (cid:17) , ∀ t > T ϵ , t ∈ S 1 .

Original: As ∀ t ̸∈ S 1 , t > T ϵ we have ∥ ˜ x
Revised: As ∀ t ̸∈ S 1 , t > T ϵ we have ∥ ˜ x

Original: ( t ) ∥ ≥ ηλ i (cid:114) 12 λ 2 i + λ1 − ηλ 1 .
Revised: ( t ) ∥ ≥ ηλ i (cid:114) 12 λ 2 i + λ1 − ηλ 1 .

Original: Hence we have ∀ t > T ϵ , ∥
Revised: Hence we have ∀ t > T ϵ , ∥

Original: ˜ x ( t ) ∥ ≥ min (cid:16) ηλ2 − ηλ 21 + ϵ , η 2 λ(2 − λ 1 η ) λ 1 η + ϵ (cid:17)
Revised: ˜ x ( t ) ∥ ≥ min (cid:16) ηλ2 − ηλ 21 + ϵ , η 2 λ(2 − λ 1 η ) λ 1 η + ϵ (cid:17)

Original: Further by Lemma 44, we can prove ∀ t > T ϵ +1 , ∥
Revised: Further by Lemma 44, we can prove ∀ t > T ϵ +1 , ∥

Original: ˜ x ( t ) ∥ ≤ ηλ 21 − (1 − ηλ 1 ) min (cid:16) ηλ2 − ηλ 21 + ϵ , η 2 λ(2 − λ 1 η ) λ 1 η + ϵ (cid:17) .
Revised: ˜ x ( t ) ∥ ≤ ηλ 21 − (1 − ηλ 1 ) min (cid:16) ηλ2 − ηλ 21 + ϵ , η 2 λ(2 − λ 1 η ) λ 1 η + ϵ (cid:17) .

Original: Combining both bound, we have lim t →∞ ∥
Revised: Combining both bound, we have lim t →∞ ∥

Original: ˜ x ( t ) ∥ =
Revised: ˜ x ( t ) ∥ =

Original: ηλ 21 2 − ηλ 1 .
Revised: ηλ 21 2 − ηλ 1 .

Original: Notice that ∥ P (2: D ) ˜
Revised: Notice that ∥ P (2: D ) ˜

Original: x
Revised: x

Original: ( t +1) ∥ ≤ max (cid:16) | 1 − ηλ 2 − η λ 22 ∥
Revised: ( t +1) ∥ ≤ max (cid:16) | 1 − ηλ 2 − η λ 22 ∥

Original: ˜ x ( t ) ∥ | , | 1 − ηλ D − η λ 2 D ∥
Revised: ˜ x ( t ) ∥ | , | 1 − ηλ D − η λ 2 D ∥

Original: ˜ x ( t ) ∥ | (cid:17) ∥ P (2: D )
Revised: ˜ x ( t ) ∥ | (cid:17) ∥ P (2: D )

Original: ˜ x ( t ) ∥ .
Revised: ˜ x ( t ) ∥ .

Original: When ∥ ˜ x
Revised: When ∥ ˜ x

Original: ( t ) ∥ > ηλ 22 2 − ηλ 2 − δ , − 1 + δ ≤ 1 − ηλ 2 − η λ∥ ˜ x
Revised: ( t ) ∥ > ηλ 22 2 − ηλ 2 − δ , − 1 + δ ≤ 1 − ηλ 2 − η λ∥ ˜ x

Original: ( t ) ∥ ≤ 1 − ηλ D − η λ 2 D ∥
Revised: ( t ) ∥ ≤ 1 − ηλ D − η λ 2 D ∥

Original: ˜ x ( t ) ∥ ≤ 1 − ηλ D ∥ P (2: D )
Revised: ˜ x ( t ) ∥ ≤ 1 − ηλ D ∥ P (2: D )

Original: ˜ x
Revised: ˜ x

Original: ( t + 1) ∥ ≤ max(1 − ηλ D , 1 − δ )
Revised: ( t + 1) ∥ ≤ max(1 − ηλ D , 1 − δ )

Original: ∥ P (2: D ) ˜ x
Revised: ∥ P (2: D ) ˜ x

Original: ( t ) ∥
Revised: ( t ) ∥

Original: Hence for sufficiently large t , ∥ P (2: D ) ˜ x
Revised: Hence for sufficiently large t , ∥ P (2: D ) ˜ x

Original: ( t ) ∥ shrinks exponentially, showing
Revised: ( t ) ∥ shrinks exponentially, showing

Original: that lim t →∞ ∥ ˜ x 1 ( t ) ∥ = ηλ 21 2 − ηλ
Revised: that lim t →∞ ∥ ˜ x 1 ( t ) ∥ = ηλ 21 2 − ηλ

Original: D.5. Full-batch SAM on General Loss: Proof of Theorem
Revised: D.5. Full-batch SAM on General Loss: Proof of Theorem

Original: To prove the theorem, we will separate the dynamic of SAM on general loss L to two phases.
Revised: To prove the theorem, we will separate the dynamic of SAM on general loss L to two phases.

Original: Define
Revised: Define

Original: [Equation]
Revised: [Equation]

Original: which is the length projection of x − Φ( x ) on button − k eigenspace of ∇ 2 L (Φ( x ))
Revised: which is the length projection of x − Φ( x ) on button − k eigenspace of ∇ 2 L (Φ( x ))

Original: We will provide a fine-grained convergence bound on R j ( x ) .
Revised: We will provide a fine-grained convergence bound on R j ( x ) .

Original: Theorem 46 (Phase I) Let { x ( t ) } be the iterates defined by SAM (3) and x (0) =
Revised: Theorem 46 (Phase I) Let { x ( t ) } be the iterates defined by SAM (3) and x (0) =

Original: x init ∈ U , then under Assumption 1 there exists a constant T 1 , such that for any T ′ 1 > T 1 , it holds for all η, ρ such that ( η + ρ ) log(1 /ηρ ) is sufficiently small, we have
Revised: x init ∈ U , then under Assumption 1 there exists a constant T 1 , such that for any T ′ 1 > T 1 , it holds for all η, ρ such that ( η + ρ ) log(1 /ηρ ) is sufficiently small, we have

Original: [Equation]
Revised: [Equation]

Original: Theorem 46 implies SAM will converge to an O ( ηρ ) neighbor of Γ .
Revised: Theorem 46 implies SAM will converge to an O ( ηρ ) neighbor of Γ .

Original: Notice in the time frame defined by Theorem 46, x ( t ) effectively operates at a local regime around Φ( ⌈− T 1 log ηρ/η ⌉ ) , this allows us to approximate L with the quadratic Taylor expansion of L at Φ( ⌈− T 1 log ηρ/η ⌉ ) and give us the following theorem.
Revised: Notice in the time frame defined by Theorem 46, x ( t ) effectively operates at a local regime around Φ( ⌈− T 1 log ηρ/η ⌉ ) , this allows us to approximate L with the quadratic Taylor expansion of L at Φ( ⌈− T 1 log ηρ/η ⌉ ) and give us the following theorem.

Original: Theorem 47 (Phase II) Let { x ( t ) } be the iterates defined by SAM (3) under Assumptions 1 and 7, further assuming that (1) max j R j ( x (0))
Revised: Theorem 47 (Phase II) Let { x ( t ) } be the iterates defined by SAM (3) under Assumptions 1 and 7, further assuming that (1) max j R j ( x (0))

Original: =
Revised: =

Original: O ( ηρ 2 ) , (2) ∥
Revised: O ( ηρ 2 ) , (2) ∥

Original: Φ( x (0)) − Φ( x init ) ∥
Revised: Φ( x (0)) − Φ( x init ) ∥

Original: =
Revised: =

Original: O (( η + ρ ) log(1 /ηρ )) and (3) |⟨
Revised: O (( η + ρ ) log(1 /ηρ )) and (3) |⟨

Original: x (0) − Φ( x (0)) , v 1 ( x (0))
Revised: x (0) − Φ( x (0)) , v 1 ( x (0))

Original: ⟩| ≥ Ω( ρ 2 ) , then there exists constant T 2 > 0 , for any T 3 >till which solution of (6) exists, for all η, ρ such that η ln(1 /ρ )
Revised: ⟩| ≥ Ω( ρ 2 ) , then there exists constant T 2 > 0 , for any T 3 >till which solution of (6) exists, for all η, ρ such that η ln(1 /ρ )

Original: and ρ/η is sufficiently small,
Revised: and ρ/η is sufficiently small,

Original: [Equation]
Revised: [Equation]

Original: In this section we will define K as { X ( t ) } where X is the solution of (6).
Revised: In this section we will define K as { X ( t ) } where X is the solution of (6).

Original: D.5.1. P HASE I: P ROOF OF T HEOREM
Revised: D.5.1. P HASE I: P ROOF OF T HEOREM

Original: The proof of Theorem 46 is further split into three subphases.
Revised: The proof of Theorem 46 is further split into three subphases.

Original: In Subphase A , we will show that the trajectory of SAM will track gradient flow to the working zone K h .
Revised: In Subphase A , we will show that the trajectory of SAM will track gradient flow to the working zone K h .

Original: This subphase will take time O ( 1 η ) for sufficiently small η and ρ .
Revised: This subphase will take time O ( 1 η ) for sufficiently small η and ρ .

Original: At the end of this subphase x ( t ) − Φ( x ( t ))
Revised: At the end of this subphase x ( t ) − Φ( x ( t ))

Original: =
Revised: =

Original: O (1) , Φ( x ( t ))
Revised: O (1) , Φ( x ( t ))

Original: − Φ( x (0))
Revised: − Φ( x (0))

Original: =
Revised: =

Original: O ( η + ρ ) .
Revised: O ( η + ρ ) .

Original: In Subphase B , we will show that in the working zone, the loss will continue to decrease until ∥∇ L ∥ =
Revised: In Subphase B , we will show that in the working zone, the loss will continue to decrease until ∥∇ L ∥ =

Original: O ( ρ ) .
Revised: O ( ρ ) .

Original: This will take time O
Revised: This will take time O

Original: ( − log ρ η ) for sufficiently small η and ρ .
Revised: ( − log ρ η ) for sufficiently small η and ρ .

Original: At the end of this subphase x ( t ) − Φ( x ( t ))
Revised: At the end of this subphase x ( t ) − Φ( x ( t ))

Original: =
Revised: =

Original: O ( ρ ) , Φ( x ( t )) − Φ( x (0))
Revised: O ( ρ ) , Φ( x ( t )) − Φ( x (0))

Original: =
Revised: =

Original: O ( − ( η + ρ ) log ρ ) .
Revised: O ( − ( η + ρ ) log ρ ) .

Original: In Subphase C , we will show that R j ( x ) will shrink exponentially to O ( ηρ 3 / 2 ) and x ( t ) will stay in the invariant sets I j
Revised: In Subphase C , we will show that R j ( x ) will shrink exponentially to O ( ηρ 3 / 2 ) and x ( t ) will stay in the invariant sets I j

Original: = { R j ( x ) ≤ O ( ηρ 3 / 2 ) } .
Revised: = { R j ( x ) ≤ O ( ηρ 3 / 2 ) } .

Original: Subphase A
Revised: Subphase A

Original: We know ∃ T 0 , ∥ Φ( x init , T 0 ) − Φ( x init )
Revised: We know ∃ T 0 , ∥ Φ( x init , T 0 ) − Φ( x init )

Original: ∥ ≤ h 4 , L (Φ( x init , T 0 )) <
Revised: ∥ ≤ h 4 , L (Φ( x init , T 0 )) <

Original: h 2 µ 16 .
Revised: h 2 µ 16 .

Original: Using standard approximation theory, ∃ η 0 , ρ 0 > 0 , such that
Revised: Using standard approximation theory, ∃ η 0 , ρ 0 > 0 , such that

Original: [Equation]
Revised: [Equation]

Original: This further implies
Revised: This further implies

Original: [Equation]
Revised: [Equation]

Original: Subphase B
Revised: Subphase B

Original: Define
Revised: Define

Original: [Equation]
Revised: [Equation]

Original: In this subphase
Revised: In this subphase

Original: [Equation]
Revised: [Equation]

Original: we will track the descent of loss to show that ∃ t , such that
Revised: we will track the descent of loss to show that ∃ t [Equation] such that

Original: [Equation]
Revised: [Equation]

Original: for sufficiently small η .
Revised: for sufficiently small η .

Original: We require η ≤ 12 ζ .
Revised: We require η ≤ 12 ζ .

Original: We assume inf x ∈ U L ( x ) = 0 .
Revised: We assume inf x ∈ U L ( x ) = 0 .

Original: We also requires − η log ρ being sufficiently small, i.e ρ is not too small compared to η .
Revised: We also requires − η log ρ being sufficiently small, i.e ρ is not too small compared to η .

Original: So that
Revised: So that

Original: [Equation]
Revised: [Equation]

Original: We will prove the following proposition,
Revised: We will prove the following proposition,

Original: [Equation]
Revised: [Equation]

Original: Proof
Revised: Proof

Original: By lemma [Equation] we have
Revised: By lemma 31, we have

Original: [Equation]
Revised: [Equation]

Original: Further, given the choice of η, ρ , ∥ Φ( x ( t )) − Φ( x init )
Revised: Further, given the choice of η, ρ , ∥ Φ( x ( t )) − Φ( x init )

Original: ∥ ≤ h 4 .
Revised: ∥ ≤ h 4 .

Original: Hence we have x ( t ) ∈ K 3 h 4 .
Revised: Hence we have x ( t ) ∈ K 3 h 4 .

Original: By Lemma 35.
Revised: By Lemma 35.

Original: we have x ( t ) x
Revised: we have x ( t ) x

Original: ( t + 1) ⊂
Revised: ( t + 1) ⊂

Original: K h
Revised: K h

Original: Under update rule (3), using the smoothness of L , we have
Revised: Under update rule (3), using the smoothness of L , we have

Original: L(
Revised: L

Original: [Equation]
Revised: [Equation]

Original: ( t
Revised: We have that

Original: [Equation]
Revised: [Equation]

Original: [Equation] We have that
Revised: ( x ( t

Original: [Equation]
Revised: [Equation]

Original: Hence
Revised: Hence

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: By induction hypothesis, we have
Revised: By induction hypothesis, we have

Original: [Equation]
Revised: [Equation]

Original: Hence as x ( t + 1) x ( t ) ∈
Revised: Hence as x ( t + 1) x ( t ) ∈

Original: K h
Revised: K h

Original: [Equation]
Revised: [Equation]

Original: ≤

Original: [Equation]

Original: This implies
Revised: This implies

Original: [Equation]
Revised: [Equation]

Original: Using Lemma
Revised: Using Lemma

Original: [Equation]
Revised: [Equation]

Original: The induction is complete.
Revised: The induction is complete.

Original: Now define t 1 the minimal t ≥ T 0 η , such that ∥∇ L ( x ( t ))
Revised: Now define t 1 the minimal t ≥ T 0 η , such that ∥∇ L ( x ( t ))

Original: ∥ ≤ 4 ζρ .
Revised: ∥ ≤ 4 ζρ .

Original: If t 1 > T 0 η + log 1 − ηµ 8 ( 64 ζ 2 ρ 2 h 2 ) , then by the induction, 
Revised: If t 1 > T 0 η + log 1 − ηµ 8 ( 64 ζ 2 ρ 2 h 2 ) , then by the induction, L ( Tη + log

Original: [Equation]
Revised: [Equation]

Original: This is a contradiction
Revised: This is a contradiction

Original: [Equation]
Revised: [Equation]

Original: Subphase C
Revised: Subphase C

Original: Recall the definition of R j ( x ) ,
Revised: Recall the definition of R j ( x ) ,

Original: [Equation]
Revised: [Equation]

Original: In this subphase, we will show that R j ( x ) will shrink exponentially to O ( ηρ 2 ) and x ( t ) will stay in the invariant sets I j
Revised: In this subphase, we will show that R j ( x ) will shrink exponentially to O ( ηρ 2 ) and x ( t ) will stay in the invariant sets I j

Original: = { R j ( x ) ≤ O ( ηρ 2 + ( ηρ ) 3 / 2 ) } .
Revised: = { R j ( x ) ≤ O ( ηρ 2 + ( ηρ ) 3 / 2 ) } .

Original: Define ˆ 
Revised: Define ˆ x ( t ) =

Original: [Equation]
Revised: x ( t ) − Φ( x ( t )) , A ( t ) = ∇ 2 L (Φ( x ( t ))) , ˜ x ( t ) = A ( t )ˆ x ( t ) We will prove the induction hypothesis for t 1 ≤ t ≤ t 1 + 10 log 1 − ηµ ηµ 3 4 ζ 2 , ∥

Original: [Equation]
Revised: [Equation]

Original: We will prove the induction hypothesis for t 1 ≤
Revised: ∥

Original: [Equation]
Revised: [Equation]

Original: As we have ∥ ˜ x
Revised: As we have ∥ ˜ x

Original: ( t 1 ) ∥ = 
Revised: ( t 1 ) ∥ = ∥ A

Original: [Equation]
Revised: [Equation]

Original:  Combining with the induction hypothesis, we have ∥ ˜ x
Revised: µ . Combining with the induction hypothesis, we have ∥ ˜ x

Original: ( t ) ∥ ≤ 4 ζ 2 ρ µ .
Revised: ( t ) ∥ ≤ 4 ζ 2 ρ µ .

Original: Then we have, ∥ Φ
Revised: Then we have, ∥ Φ

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: − Φ

Original: [Equation]
Revised: [Equation]

Original: ≤

Original: [Equation]
Revised: [Equation]

Original: L
Revised: ζ

Original: [Equation]
Revised: [Equation]

Original: η
Revised: η

Original: [Equation]
Revised: [Equation]

Original: ρ
Revised: ρ

Original: [Equation]
Revised: [Equation]

Original: ≤ [Equation] ) − [Equation]
Revised: c

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: ∥ [Equation] As t 1 ≤ t ≤ t 1 + 10 log 1 − ηµ [Equation]
Revised: As t 1 ≤ t ≤ t 1 + 10 log 1 − ηµ ηµ 3 4 ζ 2 ≤ t 1 + 10 − log

Original: [Equation]
Revised: [Equation]

Original: ζ 2 ≤ t 1 + 10 − log

Original: [Equation]
Revised: [Equation]

Original: 3 4 ζηµ , this implies
Revised: ηµ , this implies

Original: [Equation]
Revised: [Equation]

Original: We have x ( t ) ∈ K h 2 Using Lemma 35, we conclude that x ( t ) x ( t + 1)
Revised: We have x ( t ) ∈ K h 2 Using Lemma 35, we conclude that x ( t ) x ( t + 1)

Original: ⊂ K h .
Revised: ⊂ K h .

Original: [Equation]
Revised: [Equation]

Original: Now Using Lemma 32, we have 
Revised: Now Using Lemma 32, we have ∥

Original: [Equation]
Revised: [Equation]

Original: Further we have ∥ ∇ 
Revised: Further we have ∥ ∇ L ( x

Original: [Equation]
Revised: [Equation]

Original: ) ∥∇
Revised: ) ∥∇

Original: [Equation]
Revised: [Equation]

Revised: t )))

Original: [Equation]
Revised: [Equation]

Original: Hence
Revised: Hence

Original: [Equation]
Revised: [Equation]

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: This implies,
Revised: This implies,

Original: [Equation]
Revised: [Equation]

Original: Also by Lemma
Revised: Also by Lemma

Original: [Equation]
Revised: [Equation]

Original: )) − ˜ x ( t + 1) + ˜ x ( t ) ∥ =
Revised: x ( t )) − ˜ x [Equation] t [Equation]

Revised: [Equation]

Revised: ˜

Revised: [Equation]

Revised: t

Revised: [Equation]

Revised: ∥ =

Original: ∥
Revised: ∥

Original: A ( t )( x ( t + 1) − x ( t ))
Revised: A ( t [Equation] 1) − x [Equation]

Original:  − A ( t+ 1) x
Revised: t )) − A [Equation] t

Original: ( t + 1)+ A ( t ) x ( t) +
Revised: [Equation] 1) x [Equation] t

Original:  A ( t  + 1)Φ
Revised: [Equation] + A ( t ) x ( t ) + A ( t [Equation]

Original: ( x ( t+ 1))
Revised: 1)Φ( x ( t

Revised: [Equation]

Original: − A ( t )Φ( x ( t ))
Revised: − A ( t )Φ( x ( t ))

Original: ∥ =
Revised: ∥ =

Original: ∥ ( A ( t )− A
Revised: ∥ ( A ( t )

Original: ( t + 1)) ( x ( t + 1) − Φ( x ( t + 1)))
Revised: [Equation]

Original: + A ( t)(Φ( x ( t + 1))
Revised: A ( t

Original: − Φ( x ( t ))) ∥ ≤ ν ∥ Φ( x ( t + 1)) − Φ( x ( t ))
Revised: [Equation]

Original: ∥∥ x ( t
Revised: ( x ( t

Original: [Equation]
Revised: [Equation]

Original: 1) − Φ( x ( t
Revised: 1) − Φ( x ( t

Original: [Equation]
Revised: [Equation]

Original: ∥

Original: [Equation]
Revised: [Equation]

Original: ζ ∥ Φ( x ( t 
Revised: ( t )

Original: [Equation]
Revised: [Equation]

Original: 1)) − Φ( x ( t ))

Original: ∥ ≤ ( νh + ζ ) ∥

Original: Φ( x ( t
Revised: x ( t

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Φ( x ( 
Revised: ( x ( t )))

Original: [Equation]
Revised: [Equation]

Original: ∥ ≤ (
Revised: Combining with induction hypothesis

Original: [Equation]
Revised: [Equation]

Original: Combining with induction hypothesis, we know exists constant c 1 , such that
Revised: we know exists constant c 1 , such that

Original: [Equation]
Revised: [Equation]

Original: We first bound ∥ ˜ x ( t ) ∥ , as in quadratic case, if ∥ ˜ x
Revised: We first bound ∥ ˜ x ( t ) ∥ , as in quadratic case, if ∥ ˜ x

Original: ( t ) ∥ > ηρλ 21 ( t ) , we would have
Revised: ( t ) ∥ > ηρλ 21 ( t ) , we would have

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Choosing ρ small enough, we have
Revised: Choosing ρ small enough

Original: [Equation]
Revised: [Equation]

Original:  If∥ ˜
Revised: we have [Equation] If

Original: [Equation]
Revised: [Equation]

Original:  ( t )
Revised: ˜ x ( t )

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: ( t )
Revised: ( t )

Original: [Equation]
Revised: [Equation]

Original: Here we usemax i
Revised: Here we use

Original: [Equation]
Revised: [Equation]

Original: Now define t 2 the minimal t ≥ t 1 , such that ∥ ˜ x
Revised: Now define t 2 the minimal t ≥ t 1 , such that ∥ ˜ x

Original: ( t ) ∥ ≤ ηρλ 21 ( t ) .
Revised: ( t ) ∥ ≤ ηρλ 21 ( t ) .

Original: If [Equation] t
Revised: If t 2 > t

Original: [Equation]
Revised: [Equation]

Original: log 1 − ηµ ( ηµ [-
 [Equation]-] 2 ) , then by the induction,
Revised: log 1 − ηµ ( ηµ [-
 [Equation]-]3 4 ζ 2 ) , then by the induction,

Original: [Equation]
Revised: [Equation]

Original: This is a contradiction.
Revised: This is a contradiction.

Original: Following the induction, we further have for t 2 ≤ t ≤ t 1 + T ′ 1 log 1 − ηµ ( ηµ
Revised: Following the induction, we further have for t 2 ≤ t ≤ t 1 + T ′ [Equation] 1 − ηµ ( ηµ

Original: [Equation]
Revised: [Equation]

Original: ζ 2 ) ,
Revised: ζ 2 ) ,

Original: [Equation]
Revised: [Equation]

Original: We will now use a quantization technique separating [ M ] into disjoint continuous subset S 1 , ..., S p such that ∀ i ̸
Revised: We will now use a quantization technique separating [ M ] into disjoint continuous subset S 1 , ..., S p such that ∀ i ̸

Original: = j ,
Revised: = j ,

Original: [Equation]
Revised: [Equation]

Original: We would have
Revised: We would have

Original: [Equation]
Revised: [Equation]

Revised: 99 ρ

Original: We would then have for t ≥ t 2 (analogous to proof in Section D.4.1)
Revised: We would then have for t ≥ t 2 (analogous to proof in Section D.4.1)

Original: If 
Revised: If (cid:113)(cid:80)

Original: [Equation]
Revised: [Equation]

Original: i
Revised: (cid:118)(cid:117)(cid:117)(cid:116)

Original: [Equation]
Revised: [Equation]

Original: j
Revised: i

Original: [Equation]
Revised: [Equation]

Original: If (cid:113)(cid:80) [Equation]{+pi = j ∥ P ( t ) S ( i ) ˜ x ( t ) ∥ 
+}
Revised: If (cid:113)(cid:80) {+pi = j ∥ P ( t ) S ( i ) ˜ x ( t ) ∥ 
+}

Original: ≤ max 
Revised: ≤ max k ∈ S

Original: [Equation]
Revised: j λ 2 k ( t ) ηρ (cid:118)(cid:117)(cid:117)(cid:116) p (cid:88) i = j ∥ P ( t ) S ( i ) ˜ x ( t + 1) ∥ 2 ≤ max k ∈ S j λ 2 k ( t ) ηρ + c 1 ηρ≤

Original: λ
Revised: max k ∈ S

Original: [Equation] + [Equation]
Revised: j λ 2 k ( t + 1) ηρ + 2 c 1 ηρ

Original: Further we have ∥ P ( t ) s k − P
Revised: Further we have ∥ P ( t ) s k − P

Original: ( t +1)
Revised: ( t +1)

Original: s k
Revised: s k

Original: ∥ ≤ O ( νηρ ) by the Lemma
Revised: ∥ ≤ O ( νηρ ) by the Lemma

Original: So we have
Revised: So we have

Original: If (cid:113)(cid:80) pi = j ∥ P ( t ) S ( i ) ˜ x ( t ) ∥ [Equation] >
Revised: If (cid:113)(cid:80) pi = j ∥ P ( t ) S ( i ) ˜ x ( t ) ∥ 2 >

Original: max k ∈ S
Revised: max k ∈ S

Original: j λ 2 k ( t ) ηρ
Revised: j λ 2 k ( t ) ηρ

Original: (cid:118)(cid:117) (cid:117) (cid:116) p (cid:88) i =
Revised: (cid:118)(cid:117) (cid:117) (cid:116) p (cid:88) i =

Original: j ∥ P ( t +1) [Equation] i 
Revised: j ∥ P ( t +1) S ( i ) ˜

Original: [Equation] + 1) [Equation] max 
Revised: x ( t + 1) ∥ 2 ≤ max { (1 − ηλ D ) ∥

Original: [Equation] ∥ [Equation] (cid:88) 
Revised: p (cid:88) i =

Original: [Equation] = [Equation]
Revised: j

Original: P [Equation] η max [Equation]
Revised: P ( t ) S ( i ) ˜ x ( t ) ∥ − ηρλ 2 D , η max k ∈ S j λ k ∥ p (cid:88) i =

Original: λ
Revised: j

Original: [Equation] ηρ[Equation]
Revised: P ( t ) S ( i ) ˜ x ( t ) ∥} + c 1 ηρ

Original: If (cid:113)(cid:80) pi = j ∥ P ( t ) S ( i ) ˜ x ( t ) ∥ 

Revised: If (cid:113)(cid:80) pi = j ∥ P ( t ) S ( i ) ˜ x ( t ) ∥

Original: ≤ max k ∈ S
Revised: ≤ max k ∈ S

Original: j λ [Equation] k ( t ) ηρ (cid:118)(cid:117)(cid:117)(cid:116) p 
Revised: j λ 2 k ( t ) ηρ (cid:118)(cid:117)(cid:117)(cid:116) p (cid:88) i =

Original: [Equation] i = [Equation]
Revised: j ∥ P ( t +1) S ( i ) ˜

Original: max [Equation] ≤ max 
Revised: x ( t + 1) ∥ 2 ≤ max k ∈ S

Revised: j λ 2 k ( t ) ηρ + c 1 ηρ 2 + O ( ηρ 2 )

Original: [Equation]
Revised: [Equation]

Original: Finally taking into quantization error, as all the eigenvalue in the same group at most differ Dρ , we would have
Revised: Finally taking into quantization error, as all the eigenvalue in the same group at most differ Dρ , we would have

Original: If R 
Revised: If R k ( x ( t ) [Equation]

Original: [Equation]
Revised: [Equation]

Original: If R k ( x ( t )) <
Revised: If R k ( x ( t )) <

Original: [Equation]
Revised: [Equation]

Original: Similar to the proof of existence of t 2 , we can show the existence of t 3 ≤ t 2 + log 1 − ηµ ( ηµ 3 4 ζ 2 ) , such that for t 3 ≤ t ≤ t 3 + T ′ 1 log 1 −ηµ ( ηµ 3 4 ζ
Revised: Similar to the proof of existence of t 2 , we can show the existence of t 3 ≤ t 2 + log 1 − ηµ ( ηµ 3 4 ζ 2 ) , such that for t 3 ≤ t ≤ t 3 + T ′ 1 log [Equation] −

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: D.5.2. P HASE II: P ROOF OF T HEOREM
Revised: D.5.2. P HASE II: P ROOF OF T HEOREM

Original: The proof consists of two subphase.
Revised: The proof consists of two subphase.

Original: In subphase A , we will show x ( t ) − Φ( x ( t )) aligns with v 1 ( t ) in O (log(1 /ρ ) /η ) steps.
Revised: In subphase A , we will show x ( t ) − Φ( x ( t )) aligns with v 1 ( t ) in O (log(1 /ρ ) /η ) steps.

Original: In subphase B , we will show that the alignment continues to hold and x ( t ) moves as a timerescaled version of solution of Equation (6).
Revised: In subphase B , we will show that the alignment continues to hold and x ( t ) moves as a timerescaled version of solution of Equation (6).

Original: Subphase A
Revised: Subphase A

Original: This proof is analogous to Section D.4.2.
Revised: This proof is analogous to Section D.4.2.

Original: To maintain consistency with previous section, we abuse notation and change the starting step to t 3 .
Revised: To maintain consistency with previous section, we abuse notation and change the starting step to t 3 .

Original: The time frame we are discussing is t 3 ≤ t ≤ [Equation] 3 + T ′ 1 log [Equation]
Revised: The time frame we are discussing is t 3 ≤ t ≤ t 3 + T ′ 1 log 1 − ηµ ( ηµ 3 4 ζ 2 ) .

Original: [-ηµ ( ηµ 
-]
Revised: [-ηµ ( ηµ 
-]

Original: [Equation]

Original: First Induction
Revised: First Induction

Original: We will inductively prove ∥ x 1 ( t ) ∥ is Ω( ρ 2 ) and that there exists step t 4 for t 4 ≤ t ≤ t 3 + T ′ 1 log 1 − ηµ ( ηµ 3 4 ζ 2 ) that x 1 ( t ) ≥ 14 (cid:16) ηλ 21 2 − ηλ 1 + 3 ηλ 22 2 − ηλ 2 (cid:17) ρ − O ( ηρ 2 )
Revised: We will inductively prove ∥ x 1 ( t ) ∥ is Ω( ρ 2 ) and that there exists step t 4 for t 4 ≤ t ≤ t 3 + T ′ 1 log 1 − ηµ ( ηµ 3 4 ζ 2 ) that x 1 ( t ) ≥ 14 (cid:16) ηλ 21 2 − ηλ 1 + 3 ηλ 22 2 − ηλ 2 (cid:17) ρ − O ( ηρ 2 )

Original: For step t , we fixed the quadratic function as ⟨ x − Φ( x ( t )) , ∇ 2 L (Φ( x ( t )))
Revised: For step t , we fixed the quadratic function as ⟨ x − Φ( x ( t )) , ∇ 2 L (Φ( x ( t )))

Original: ( x − Φ( x ( t )) ⟩ ,
Revised: ( x − Φ( x ( t )) ⟩ ,

Original: Define ¯ x = ∇ 2 L (Φ( x ( t )))
Revised: Define ¯ x = ∇ 2 L (Φ( x ( t )))

Original: ( x − Φ( x ( t ))
Revised: ( x − Φ( x ( t ))

Original: ρ , A = ∇ 2 L ( t )
Revised: ρ , A = ∇ 2 L ( t )

Original: We have for t ≤ t 4 − 1 , ∥ ¯ x 1 ( t ) ∥ ≤ 12 (cid:16) ηλ 21 2 − ηλ 1 + ηλ 22 2 − ηλ 2 (cid:17) .
Revised: We have for t ≤ t 4 − 1 , ∥ ¯ x 1 ( t ) ∥ ≤ 12 (cid:16) ηλ 21 2 − ηλ 1 + ηλ 22 2 − ηλ 2 (cid:17) .

Original: By assumption, we have ∥ ¯ x 1 ( t )
Revised: By assumption, we have ∥ ¯ x 1 ( t )

Original: ∥ ≥ Ω( ρ ) .
Revised: ∥ ≥ Ω( ρ ) .

Original: We have ∥( x ( t + 1) − x ( t )) + η ∇ 2 L (Φ( x ( t ))) ( x ( t ) − Φ( x ( t ))) + ηρ ∇
Revised: We have ∥

Original: [Equation]
Revised: [Equation]

Original: (Φ( x ( t ))) ∇ 2 L (Φ( x ( t )))
Revised: ∥∇

Original: ( x ( t ) − Φ( x ( t ))) ∥∇ 2 L (Φ( x ( t ))) ( x ( t ) − Φ( x ( t ))) ∥∥ ≤ νρ 2 η + νη ∥ x ( t ) − Φ( x ( t )) ∥ 2 + ηρ 2 ζν µ ∥ x ( t ) − Φ( x ( t ))
Revised: [Equation]

Original: ∥ + νηρ
Revised: ∥∥ ≤ νρ

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: Further we have, there exists constant c 4 such that
Revised: Further we have, there exists constant c 4 such that

Original: [Equation]
Revised: [Equation]

Original: Now we have
Revised: Now we have

Original: [Equation]
Revised: [Equation]

Original: So here our goal is to discuss the dynamics of the following perturbed version of quadratic
Revised: So here our goal is to discuss the dynamics of the following perturbed version of quadratic

Original: SAM.
Revised: SAM.

Original: [Equation]
Revised: [Equation]

Original: Define ˆ x ( t + 1) ,
Revised: Define ˆ x ( t + 1) ,

Original: [Equation]
Revised: [Equation]

Original: In the quadratic case, we have Lemma 43 to show ∥ ˜ x ( t ) ∥ can’t stay greater ηλ 21 2 − ηλ 1 for two steps.
Revised: In the quadratic case, we have Lemma 43 to show ∥ ˜ x ( t ) ∥ can’t stay greater ηλ 21 2 − ηλ 1 for two steps.

Original: Hence here we have if ∥
Revised: Hence here we have if ∥

Original: ¯ x ( t ) ∥ ≥ ηλ 21 2 − ηλ 1 , then ∥ ˆ x ( t + 1) ∥ ≤ ηλ [-
 [Equation]-] ηλ 1 − O ( η ) , which leads ∥ ¯ x ( t + 1) ∥ ≤ ∥ ˆ( x ( t + 1))
Revised: ¯ x ( t ) ∥ ≥ ηλ 21 2 − ηλ 1 , then ∥ ˆ x ( t + 1) ∥ ≤ ηλ [-
 [Equation]-]21 2 − ηλ 1 − O ( η ) , which leads ∥ ¯ x ( t + 1) ∥ ≤ ∥ ˆ( x ( t + 1))

Original: ∥ + ˜ O ( ηρ ) ≤ ηλ 21 2 − ηλ 1 .
Revised: ∥ + ˜ O ( ηρ ) ≤ ηλ 21 2 − ηλ 1 .

Original: Here we can in fact prove a more subtle version of this lemma showing,
Revised: Here we can in fact prove a more subtle version of this lemma showing,

Original: Lemma
Revised: Lemma

Original: [Equation] ηλ2 − ηλ
Revised: [Equation] [Equation]

Original: Proof
Revised: Proof

Original: If ∥ ¯ x ( t ) ∥ 
Revised: If ∥ ¯ x ( t ) ∥ ≥ ηλλ

Original: [Equation]
Revised: [Equation]

Original: ηλ
Revised: λ

Original: [Equation]
Revised: [Equation]

Revised: λ

Revised: [Equation]

Revised: 1 − ηλ 1 )

Original: Using Lemma 43, we have ∥
Revised: Using Lemma 43, we have ∥

Original: [Equation]
Revised: [Equation]

Revised: 2 λ 21 ,

Original: [Equation]
Revised: [Equation]

Revised: λ

Original: [Equation]
Revised: [Equation]

Original:  ηλ2 − ηλc 6 ( λ 1 , λD ) <
Revised: ) ηλ2 − ηλc [Equation] λ

Original: If ∥ ¯ x ( t ) ∥ ≤ ηλλ 21 (1 −
Revised: [Equation] λ

Original: [Equation]
Revised: [Equation]

Original: D ) 
Revised: ) <

Original:  + ( λ 21 − λ 2 D )(1 − ηλ 1 )
Revised: If ∥ ¯ x ( t ) ∥ ≤ ηλλ 21 (1 − ηλ D ) + ( λ 21 − λ 2 D )(1 − ηλ 1 )

Original: Then we have − [Equation]
Revised: Then we have − ηλ 2 D

Original: [Equation] [Equation] D ) 
Revised: + (1 − ηλ D ) ∥ ¯ x ( t ) ∥ ηλ 21 − (1 − ηλ 1 )

Original: ∥
Revised: ∥

Original: ¯ x ( t ) ∥ 
Revised: ¯ x ( t ) ∥ ≤

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Revised: D ληλ 22 − (1 − ηλ 2 ) ∥

Original: ∥ ¯ x ( t ) ∥ 
Revised: ¯ x ( t ) ∥ ηλ

Original: [Equation]
Revised: [Equation]

Original: ληλ
Revised: (1 −

Original: [Equation]
Revised: [Equation]

Original: [Equation] ∥ ¯ x ( t ) ∥ 
Revised: ∥ ¯ x ( t ) ∥ ≤ λλ

Revised: This implies, ∥ ˆ

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Revised: −

Original: [Equation]
Revised: [Equation]

Original: ≤ λλ
Revised: λ

Original: This implies, ∥

Original: [Equation]
Revised: [Equation]

Original: λ
Revised: λ

Original: [Equation]
Revised: [Equation]

Revised: λλ 21 }

Original: As we suppose
Revised: As we suppose

Original: [Equation]
Revised: [Equation]

Original: This implies
Revised: This implies

Original: [Equation]
Revised: [Equation]

Original: Combining we have
Revised: Combining we have

Original: [Equation]
Revised: [Equation]

Original: This implies
Revised: This implies

Original: [Equation]
Revised: [Equation]

Original: As ∃ c 8 , ∥ ˆ x ( t + 1)−
Revised: As ∃ c 8 , ∥ ˆ x ( t + 1)

Original: [Equation]
Revised: [Equation]

Original: +

Original: [Equation]
Revised: [Equation]

Original: c 8 ηρ .

Original: We can conclude that
Revised: We can conclude that

Original: [Equation]
Revised: [Equation]

Original: Define T = { t |∥ ¯ x ( t ) ∥ ≤ 12 (cid:16) ηλ[-
-]
Revised: ηλ[-
-]

Original: [Equation]
Revised: [Equation]

Original: 1 +

Original: [Equation] [Equation] [Equation] } , S = { t |∥ ¯ x ( t ) ∥ ≤ ηλ [-
-]
Revised: Define T = { t |∥ ¯ x ( t ) ∥ ≤ 12 (cid:16) ηλ 21 2 − ηλ 1 + ηλ 22 2 − ηλ 2 (cid:17) } , S = { t |∥ ¯ x ( t ) ∥ ≤ ηλ [-
-]21 2 − ηλ 1 } ,

Original: [Equation]

Original: For s ∈ S, n ( s ) :
Revised: For s ∈ S, n ( s ) :

Original: =
Revised: =

Original: min j>s { j ∈ S } .
Revised: min j>s { j ∈ S } .

Original: We will show when ∥ ¯ x 1 ( t ) ∥ ≤ 12 (cid:16) ηλ 21 2 − ηλ 1 + ηλ 22 2 − ηλ 2 (cid:17) , ∃ c 4 ( λ 1 , λ 2 , λ D ) > 1 , ∥ ¯ x 1 ( n ( s ))
Revised: We will show when ∥ ¯ x 1 ( t ) ∥ ≤ 12 (cid:16) ηλ 21 2 − ηλ 1 + ηλ 22 2 − ηλ 2 (cid:17) , ∃ c 4 ( λ 1 , λ 2 , λ D ) > 1 , ∥ ¯ x 1 ( n ( s ))

Original: ∥ ≥ c 4 ∥ ¯ x 1 ( s ) ∥ or ∥ ¯ x 1 ( n ( n ( s )))
Revised: ∥ ≥ c 4 ∥ ¯ x 1 ( s ) ∥ or ∥ ¯ x 1 ( n ( n ( s )))

Original: ∥ ≥ c 4 ∥ ¯ x 1 ( s ) ∥
Revised: ∥ ≥ c 4 ∥ ¯ x 1 ( s ) ∥

Original: Define x ( t + 1)
Revised: Define x ( t + 1)

Original: =
Revised: =

Original: ˆ x ( t ) − ηA ˆ x ( t ) − ηA 2 ˆ x ( t )
Revised: ˆ x ( t ) − ηA ˆ x ( t ) − ηA 2 ˆ x ( t )

Original: ∥
Revised: ∥

Original: ˆ x ( t ) ∥
Revised: ˆ x ( t ) ∥

Original: Consider two cases
Revised: Consider two cases

Original: Case 1 ∥
Revised: Case 1 ∥

Original: ¯ x ( s ) ∥ ≤ C 1 ηλ 21 2 − ηλ∥ ˆ x 1 ( s + 1) ∥ 
Revised: ¯ x ( s ) ∥ ≤ C 1 ηλ 21 2 − ηλ∥ ˆ x 1 ( s + 1) ∥ ∥ ¯ x 1 ( t ) ∥ = ηλ 21 − (1 − ηλ 1 ) ∥ ¯ x ( t ) ∥

Revised: ∥ ¯ x

Original: [Equation]
Revised: [Equation]

Original: ∥ 
Revised: ∥ x

Original: [Equation]
Revised: [Equation]

Original: ∥ ∥
Revised: ∥ ∥

Original: [Equation]
Revised: [Equation]

Original: C
Revised: ≥

Original: [Equation]
Revised: [Equation]

Original: [Equation] C
Revised: −

Original: [Equation]
Revised: [Equation]

Original:  η
Revised: C ≥ 1 + C 3 η

Original: Here we require C 3 ≥ 

Revised: Here we require C 3 ≥

Original: As n ( s ) = s + 1 or n ( s ) =
Revised: As n ( s ) = s + 1 or n ( s ) =

Original: s + 

Revised: s +

Original: and we have
Revised: and we have

Revised: [Equation]

Revised: 9 ηρ

Original: [Equation]
Revised: [Equation]

Original:  Also | ¯ x 1 ( s ) | ≥ Ω( ρ )
Revised: ηρ Also | ¯ x 1 ( s ) | ≥ Ω( ρ )

Original: We have ∥ ¯ x 1 ( n ( s ))
Revised: We have ∥ ¯ x 1 ( n ( s ))

Original: ∥ ≥ (1 + C 3 η/ 2) ∥
Revised: ∥ ≥ (1 + C 3 η/ 2) ∥

Original: ¯ x 1 ( s ) ∥
Revised: ¯ x 1 ( s ) ∥

Original: Case 2 ∥
Revised: Case 2 ∥

Original: ¯ x ( s ) ∥ > C 1 ηλ 21 2 − ηλ 1 , then ∥
Revised: ¯ x ( s ) ∥ > C 1 ηλ 21 2 − ηλ 1 , then ∥

Original: ¯ x ( s + 1) ∥ ≤ C 2 ηλ 21 2 − ηλ 1 , n ( s )
Revised: ¯ x ( s + 1) ∥ ≤ C 2 ηλ 21 2 − ηλ 1 , n ( s )

Original: = s +
Revised: = s +

Original: Similar to case 1 We have ∥ ˆ x 1 ( s + 1)
Revised: Similar to case 1 We have ∥ ˆ x 1 ( s + 1)

Original: ∥ ≥ ∥
Revised: ∥ ≥ ∥

Original: ˜ x 1 ( s ) ∥
Revised: ˜ x 1 ( s ) ∥

Original: So in fact we have ¯ x 1 ( n ( s )) ≥ ¯ x 1 ( s ) − O ( ηρ ) , suppose ¯ x 1 ( t 3 )
Revised: So in fact we have ¯ x 1 ( n ( s )) ≥ ¯ x 1 ( s ) − O ( ηρ ) , suppose ¯ x 1 ( t 3 )

Original: ρ is a sufficiently large constant, then we can assume | ¯ x 1 ( n ( s ))
Revised: ρ is a sufficiently large constant, then we can assume | ¯ x 1 ( n ( s ))

Original: | ≥ (1 − C 3 η/ 8) | ¯ x 1 ( s ) | .
Revised: | ≥ (1 − C 3 η/ 8) | ¯ x 1 ( s ) | .

Original: As ∥ ¯ x ( n ( s ))
Revised: As ∥ ¯ x ( n ( s ))

Original: ∥ ≤ C 2 ηλ 21 2 − ηλ 1 , similar to case 1, ∥ ¯ x 1 ( n ( n ( s )))
Revised: ∥ ≤ C 2 ηλ 21 2 − ηλ 1 , similar to case 1, ∥ ¯ x 1 ( n ( n ( s )))

Original: ∥ ≥ (1 + C 3 η/ 2) ∥
Revised: ∥ ≥ (1 + C 3 η/ 2) ∥

Original: ¯ x 1 ( n ( s ))
Revised: ¯ x 1 ( n ( s ))

Original: ∥ ≥ (1 + C 3 η/ 2) ∥
Revised: ∥ ≥ (1 + C 3 η/ 2) ∥

Original: ¯ x 1 ( s ) ∥
Revised: ¯ x 1 ( s ) ∥

Original: In conclusion, if ∥ ¯ x 1 ( s ) ∥ ≤ 12 (cid:16) ηλ 21 2 − ηλ 1 + ηλ 22 2 − ηλ 2 (cid:17) , we would have ∃ c 4 ( λ 1 , λ 2 , λ D ) > 0 , ∥ ¯ x 1 ( n ( s ))
Revised: In conclusion, if ∥ ¯ x 1 ( s ) ∥ ≤ 12 (cid:16) ηλ 21 2 − ηλ 1 + ηλ 22 2 − ηλ 2 (cid:17) , we would have ∃ c 4 ( λ 1 , λ 2 , λ D ) > 0 , ∥ ¯ x 1 ( n ( s ))

Original: ∥ ≥ (1 + c 4 η ) ∥
Revised: ∥ ≥ (1 + c 4 η ) ∥

Original: ¯ x 1 ( s ) ∥ or ∥ ¯ x 1 ( n ( n ( s )))
Revised: ¯ x 1 ( s ) ∥ or ∥ ¯ x 1 ( n ( n ( s )))

Original: ∥ ≥ (1 + c 4 η ) ∥ ¯ x 1 ( s ) ∥
Revised: ∥ ≥ (1 + c 4 η ) ∥ ¯ x 1 ( s ) ∥

Original: This implies ∃ t 4 ≤ t 3 − 4(log ρ ) / ( c 4 η ) , such that
Revised: This implies ∃ t 4 ≤ t 3 − 4(log ρ ) / ( c 4 η ) , such that

Original: [Equation]
Revised: [Equation]

Original: As we have ∥ ¯ x 1 ( n ( t ))
Revised: As we have ∥ ¯ x 1 ( n ( t ))

Original: ∥ ≥ ∥ ¯ x 1 ( t ) ∥− max { c 8 , c 9 } ( ηρ + η 3 / 2 ρ 1 / 2 ) for ∥
Revised: ∥ ≥ ∥ ¯ x 1 ( t ) ∥− max { c 8 , c 9 } ( ηρ + η 3 / 2 ρ 1 / 2 ) for ∥

Original: ¯ x 1 ( t ) ∥ ≥ 12 (cid:16) ηλ 21 2 − ηλ 1 + ηλ 22 2 − ηλ 2 (cid:17) , we would have[Equation]
Revised: ¯ x 1 ( t ) ∥ ≥ 12 (cid:16) ηλ 21 2 − ηλ 1 + ηλ 22 2 − ηλ 2 (cid:17) , we would have

Original: [Equation]
Revised: [Equation]

Original: for t 3 + T ′ 1 log 1 − ηµ ( ηµ 3 4 ζ 2 ) ≥ t ≥ t 4 and the first induction is complete.
Revised: for t 3 + T ′ 1 log 1 − ηµ ( ηµ 3 4 ζ 2 ) ≥ t ≥ t 4 and the first induction is complete.

Original: Second Induction
Revised: Second Induction

Original: Define ˆ x and ¯ x as before.
Revised: Define ˆ x and ¯ x as before.

Original: This implies for t ≥ t 4 ,
Revised: This implies for t ≥ t 4 ,

Original: [Equation]
Revised: [Equation]

Original: As ∥ P (2: D ) ¯ x ( t ) ∥ ≤ 2 λ 22 η We can inductively show that for t ≥ t 4 +log max(1 − λ 2 D 2 λ [Equation] 1 − c 10 ( λ
Revised: As ∥ P (2: D ) ¯ x ( t ) ∥ ≤ 2 λ 22 η We can inductively show that for t ≥ t 4 +log max(1 − λ 2 D 2 λ 21 , 1 − c 10 ( λ

Original: [Equation]
Revised: [Equation]

Original: λ 2 ))
Revised: λ 2 ))

Original: ρ
Revised: ρ

Original: [Equation]
Revised: [Equation]

Original: ζ
Revised: ζ

Original: [Equation]
Revised: [Equation]

Original: iteration, ∥ P (2: D ) ¯ x ( t + 1) ∥ ≤ 2 c 5 ( ηρ )
Revised: iteration, ∥ P (2: D ) ¯ x ( t + 1) ∥ ≤ 2 c 5 ( ηρ )

Original: Now as we have
Revised: Now as we have

Original: [Equation]
Revised: [Equation]

Original: This implies we have for t ≥ t 4 + O (log max(1 − λ 2 D 2 λ 21 , 
Revised: This implies we have for t ≥ t 4 + O (log max(1 − λ 2 D 2 λ 21 , 1 − c 10 ( λ 1 ,

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Subphase B
Revised: Subphase B

Original: We are now ready to show that Φ( x ( t )) will track the solution of (6).
Revised: We are now ready to show that Φ( x ( t )) will track the solution of (6).

Original: The main principal of this proof have been introduced in Section C.1.
Revised: The main principal of this proof have been introduced in Section C.1.

Original: To simplify our writing define θ ( t ) =
Revised: To simplify our writing define θ ( t ) =

Original: arccos( ⟨ v 1 , ∇ L ( x ) ∥∇ L ( x ) ∥ ⟩ )
Revised: arccos( ⟨ v 1 , ∇ L ( x ) ∥∇ L ( x ) ∥ ⟩ )

Original: We can inductively prove the following statement
Revised: We can inductively prove the following statement

Original: [Equation]
Revised: [Equation]

Original: The initial condition is satisfied by assumption.
Revised: The initial condition is satisfied by assumption.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Using Lemma 31, we have ηρ∂
Revised: Using Lemma 31, we have ηρ∂

Original: [Equation]
Revised: [Equation]

Original: This implies, 
Revised: This implies, ∥

Original: [Equation]
Revised: [Equation]

Original: ( x )[ ∇ L ( x)
Revised: ∇ [Equation] x

Original: [Equation]
Revised: [Equation]

Original: L ( x ) ∥ , ∇ L ( x ) ∥∇ L ( x ) ∥ ] / 2 ∥ ≤ ηρ 3 Γ + ηρκ ∥∇ L ( x ) ∥ .
Revised: ≤ [Equation] ∥∇ L ( x ) ∥ .

Original: Further
Revised: Further

Original: [Equation]
Revised: [Equation]

Original: By induction we have 0 ≤ t ≤ T 3 ηρ 2 , we have ∥ x − Φ( x ) 
Revised: By induction we have 0 ≤ t ≤ T 3 ηρ 2 , we have ∥ x − Φ( x ) ∥

Original: [Equation]
Revised: =

Original: Θ( ηρ ) , θ = O ( ρ )
Revised: Θ( ηρ ) , θ = O ( ρ )

Original: So we have ∥
Revised: So we have ∥

Original: [Equation]
Revised: [Equation]

Original: Further we have
Revised: Further we have

Original: [Equation]
Revised: [Equation]

Revised: ≤ ηρ

Revised: [Equation]

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: This implies
Revised: This implies

Original: [Equation]
Revised: [Equation]

Original: Hence we can perform the induction and the accumulated approximation error will be of order
Revised: Hence we can perform the induction and the accumulated approximation error will be of order

Original: O ( ρ + η ) .
Revised: O ( ρ + η ) .

Original: D.5.3. P ROOF OF COROLLARY
Revised: D.5.3. P ROOF OF COROLLARY

Original: Proof [Proof of Corollary 15] We will do a Taylor expansion on L max ρ ( x ) .
Revised: Proof [Proof of Corollary 15] We will do a Taylor expansion on L max ρ ( x ) .

Original: By Theorem 46 and 47, for t > T ′ 3 /ηρ 2 , we have ∥ X ( ηρ 2 t ) − x ( t ) ∥ =
Revised: By Theorem 46 and 47, for t > T ′ 3 /ηρ 2 , we have ∥ X ( ηρ 2 t ) − x ( t ) ∥ =

Original: ˜ O
Revised: ˜ O

Original: ( η + ρ ) and ∥ x ( t ) − Φ( x 
Revised: ( η + ρ ) and ∥ x ( t ) − Φ( x ( t

Original: [Equation]
Revised: [Equation]

Original:  ( ηρ )[Equation] ρ
Revised: = O ( ηρ )

Original: [Equation]
Revised: [Equation]

Original: Then as ∥ v T ∇ L ( x ) ∥ =
Revised: Then as ∥ v T ∇ L ( x ) ∥ =

Original: O ( ηρ ) [Equation] this implies
Revised: O ( ηρ ) , this implies

Original: R max

Original: [Equation]
Revised: [Equation]

Original: Proof [Proof of Corollary 16]
Revised: Proof [Proof of Corollary 16]

Original: Choose T such that X ( T ) is sufficiently close to X ( ∞ ) , such that λ 1 ( X ( T )) ≤ λ 1 ( X ( ∞ ))
Revised: Choose T such that X ( T ) is sufficiently close to X ( ∞ ) , such that λ 1 ( X ( T )) ≤ λ 1 ( X ( ∞ ))

Original: +2 ϵ
Revised: +2 ϵ

Original: By corollary 15, we have ∥ R max ρ 
Revised: By corollary 15, we have ∥ R max ρ (

Revised: [Equation]

Revised: ⌈ T/

Original: [Equation]
Revised: [Equation]

Revised: ηρ 2 ) ⌉ ))

Revised: − ρ 2 λ 1 ( X ( T )) / 2 ∥ ≤ ˜ O

Original: ( ηρ 2 ) .
Revised: ( ηρ 2 ) .

Original: This further implies ∥ R max ρ
Revised: This further implies ∥ R max ρ

Original: [Equation]
Revised: [Equation]

Original:  ηρ 2 ) .
Revised: ˜ O ( ηρ 2 ) .

Original: We also have ∥ L ( x ( ⌈ T/ ( ηρ 2 ) ⌉ ))
Revised: We also have ∥ L ( x ( ⌈ T/ ( ηρ 2 ) ⌉ ))

Original: ∥ =
Revised: ∥ =

Original: O ( η 2 ρ 2 ) .
Revised: O ( η 2 ρ 2 ) .

Original: Then we can leverage Theorem 4 and Theorem 12 to get the desired bound.
Revised: Then we can leverage Theorem 4 and Theorem 12 to get the desired bound.

Original: D.6. Stochastic SAM: Proof of Theorem
Revised: D.6. Stochastic SAM: Proof of Theorem

Original: We will first prove Lemma 22,
Revised: We will first prove Lemma 22,

Original: Proof [Proof of Lemma 22] Note that dℓ ( y ′ ,y k ) dy ′
Revised: Proof [Proof of Lemma 22] Note that dℓ ( y ′ ,y k ) dy ′

Original: | y ′
Revised: | y ′

Original: =
Revised: =

Original: f k ( p ) = 0 , we have ∇ 2 L k ( p ) =
Revised: f k ( p ) = 0 , we have ∇ 2 L k ( p ) =

Original: Λ k ( p ) w k ( p ) w k ( p ) T .
Revised: Λ k ( p ) w k ( p ) w k ( p ) T .

Original: Also note ∇ L k ( x ) / ∥∇ L k ( x ) ∥
Revised: Also note ∇ L k ( x ) / ∥∇ L k ( x ) ∥

Original: = sign( dℓ ( y ′ ,y k ) dy ′
Revised: = sign( dℓ ( y ′ ,y k ) dy ′

Original: | y ′
Revised: | y ′

Original: =
Revised: =

Original: f k ( x ) ) ∇ f k ( x ) / ∥∇ f k ( x ) ∥ .
Revised: f k ( x ) ) ∇ f k ( x ) / ∥∇ f k ( x ) ∥ .

Original: By Assumption 1, ∇ 2 L ( p ) =
Revised: By Assumption 1, ∇ 2 L ( p ) =

Original: (cid:80) k ∇ 2 L k ( p ) /M
Revised: (cid:80) k ∇ 2 L k ( p ) /M

Original: = (cid:80) k
Revised: = (cid:80) k

Original: Λ k ( p ) w k ( p ) w k ( p ) T /M has rank M , this implies ∀ k, ∇ f k ( p ) ̸
Revised: Λ k ( p ) w k ( p ) w k ( p ) T /M has rank M , this implies ∀ k, ∇ f k ( p ) ̸

Original: = 0 , hence ∇ f k ( x ) / ∥∇ f k ( x ) ∥ is in C 1 near p and we have proved our claim.
Revised: = 0 , hence ∇ f k ( x ) / ∥∇ f k ( x ) ∥ is in C 1 near p and we have proved our claim.

Original: We will prove under a more general assumption.
Revised: We will prove under a more general assumption.

Original: Assumption 49 Assume loss L = (cid:80) k
Revised: Assumption 49 Assume loss L = (cid:80) k

Original: L k /M
Revised: L k /M

Original: and L k belongs to C 4 , and there exists a manifold Γ k that is D − 1 dimensional C 2 − submanifold of R D , where for all x ∈ Γ , x is a global minimizer of L k , L k ( x ) = 0 and rank ( ∇ 2 L k ( x ))
Revised: and L k belongs to C 4 , and there exists a manifold Γ k that is D − 1 dimensional C 2 − submanifold of R D , where for all x ∈ Γ , x is a global minimizer of L k , L k ( x ) = 0 and rank ( ∇ 2 L k ( x ))

Original: = 1 .
Revised: = 1 .

Original: Let U k
Revised: Let U k

Original: = { x ∈ R D | Φ( x ) exists and Φ k ( x ) ∈ Γ k } .
Revised: = { x ∈ R D | Φ( x ) exists and Φ k ( x ) ∈ Γ k } .

Original: We have U k is open and Φ k is in C 3 on U k .(from
Revised: We have U k is open and Φ k is in C 3 on U k .(from

Original: Lemma B.15 [2])
Revised: Lemma B.15 [2])

Original: Theorem 50 shows that Setting in Theorem 11 satisfies assumption 49.
Revised: Theorem 50 shows that Setting in Theorem 11 satisfies assumption 49.

Original: Theorem 50 Suppose L ( x ) =
Revised: Theorem 50 Suppose L ( x ) =

Original: (cid:80) Mk =1 L k ( x ) /M
Revised: (cid:80) Mk =1 L k ( x ) /M

Original: and manifold Γ satisfy Assumption 1, and that f k ( x ) =
Revised: and manifold Γ satisfy Assumption 1, and that f k ( x ) =

Original: y k for every x ∈ Γ , k ∈ [ M ] .
Revised: y k for every x ∈ Γ , k ∈ [ M ] .

Original: Then there exists ( D − 1) -dimensional
Revised: Then there exists ( D − 1) -dimensional

Original: C 2 submanifolds of R D , such that ∩ Mk =1 Γ k = Γ and for every x ∈ Γ k and k ∈
Revised: C 2 submanifolds of R D , such that ∩ Mk =1 Γ k = Γ and for every x ∈ Γ k and k ∈

Original: [ M ] , f k ( x ) =
Revised: [ M ] , f k ( x ) =

Original: y k , that is, Γ k is a manifold of global minimizers of L k .
Revised: y k , that is, Γ k is a manifold of global minimizers of L k .

Original: Proof [Proof of Theorem 50]
Revised: Proof [Proof of Theorem 50]

Original: By standard calculus, for x ∈ Γ , we have ∇ 2 L ( x ) =
Revised: By standard calculus, for x ∈ Γ , we have ∇ 2 L ( x ) =

Original: (cid:80) Mk =1 w k w Tk /M .
Revised: (cid:80) Mk =1 w k w Tk /M .

Original: By Assumption 1, ∇ 2 L ( x ) is full rank, this implies w k ̸ = 0 .
Revised: By Assumption 1, ∇ 2 L ( x ) is full rank, this implies w k ̸ = 0 .

Original: Then we have in an open set V ( x ) containing x , ∇ f k ( x ) ̸ = 0 .
Revised: Then we have in an open set V ( x ) containing x , ∇ f k ( x ) ̸ = 0 .

Original: Then consider V = ∪ x ∈ Γ V ( x ) , which is an open set and in which ∇ f k ̸ = 0 .
Revised: Then consider V = ∪ x ∈ Γ V ( x ) , which is an open set and in which ∇ f k ̸ = 0 .

Original: Now apply preimage theorem, we would have Γ k
Revised: Now apply preimage theorem, we would have Γ k

Original: = { x ∈ V | f k ( x ) ̸ = y k } forms C 2 dimensional sub-manifolds.
Revised: = { x ∈ V | f k ( x ) ̸ = y k } forms C 2 dimensional sub-manifolds.

Original: We also easily have ∩ Γ k
Revised: We also easily have ∩ Γ k

Original: = Γ from definitions.
Revised: = Γ from definitions.

Original: The following theorems shows that stochastic SAM (7) essentially minimize trace of Hessian of loss.
Revised: The following theorems shows that stochastic SAM (7) essentially minimize trace of Hessian of loss.

Original: Analogous to the full-batch setting, we will split the trajectory into two phase.
Revised: Analogous to the full-batch setting, we will split the trajectory into two phase.

Original: Theorem 51 (Phase I) Let { x ( t ) } be the iterates defined by SAM (3) and x (0) =
Revised: Theorem 51 (Phase I) Let { x ( t ) } be the iterates defined by SAM (3) and x (0) =

Original: x init ∈ U , then under Assumption 1 and 49 there exists a constant T 1 , it holds for sufficiently small − ( η + ρ ) log( ηρ ) , we have with probability 1 − O ( √ ρ ) , min − T 1 log ρ/η ≥ t ∥ x ( t ) − Φ( x ( t ))
Revised: x init ∈ U , then under Assumption 1 and 49 there exists a constant T 1 , it holds for sufficiently small − ( η + ρ ) log( ηρ ) , we have with probability 1 − O ( √ ρ ) , min − T 1 log ρ/η ≥ t ∥ x ( t ) − Φ( x ( t ))

Original: ∥
Revised: ∥

Original: =
Revised: =

Original: O ( − ( ηρ + ρ 2 ) log ηρ ) and max − T 1 log ρ/η ≥ t ∥
Revised: O ( − ( ηρ + ρ 2 ) log ηρ ) and max − T 1 log ρ/η ≥ t ∥

Original: Φ( x init ) − Φ( x ( t )) ∥
Revised: Φ( x init ) − Φ( x ( t )) ∥

Original: =
Revised: =

Original: O ( − ( η + ρ ) log( ηρ )) .
Revised: O ( − ( η + ρ ) log( ηρ )) .

Original: Theorem 51 shows that SAM will converges to an ˜ O ( ηρ ) neighborhood of the manifold without getting far away from Φ( x (0)) , where we can perform a local analysis on the trajectory of Φ( x ( t )) .
Revised: Theorem 51 shows that SAM will converges to an ˜ O ( ηρ ) neighborhood of the manifold without getting far away from Φ( x (0)) , where we can perform a local analysis on the trajectory of Φ( x ( t )) .

Original: Under Assumptions 1 and 49, we have Tr( ∇ 2 L k ( x ))
Revised: Under Assumptions 1 and 49, we have Tr( ∇ 2 L k ( x ))

Original: =
Revised: =

Original: λ 1 ( ∇ 2 L k ( x )) is differentiable for x ∈ Γ i .
Revised: λ 1 ( ∇ 2 L k ( x )) is differentiable for x ∈ Γ i .

Original: Hence Tr( ∇ 2 L ( x )) =
Revised: Hence Tr( ∇ 2 L ( x )) =

Original: (cid:80) i
Revised: (cid:80) i

Original: Tr( ∇ 2 L k ( x )) is also differentiable and we have (8) is well defined for some finite time T 2 .
Revised: Tr( ∇ 2 L k ( x )) is also differentiable and we have (8) is well defined for some finite time T 2 .

Original: Theorem 52 (Phase II) Let { x ( t ) } be the iterates defined by SAM (7) under Assumptions 1 and 49, assuming (1) ∀ t, k,
Revised: Theorem 52 (Phase II) Let { x ( t ) } be the iterates defined by SAM (7) under Assumptions 1 and 49, assuming (1) ∀ t, k,

Original: L k ( x ( t )) ̸ = 0 , (2) ∥ x (0) − Φ( x (0))
Revised: L k ( x ( t )) ̸ = 0 , (2) ∥ x (0) − Φ( x (0))

Original: ∥
Revised: ∥

Original: =
Revised: =

Original: O ( − ( ηρ + ρ 2 ) log ηρ ) and (3) ∥
Revised: O ( − ( ηρ + ρ 2 ) log ηρ ) and (3) ∥

Original: Φ( x init ) − Φ( x (0))
Revised: Φ( x init ) − Φ( x (0))

Original: ∥
Revised: ∥

Original: =
Revised: =

Original: O ( − ( η + ρ ) log( ηρ )) , then for any T 2 > 0 till which solution of (8) exists, for sufficiently small − ( η + ρ ) log( ηρ ) , we have with probability 1 − O ( ηρ ) , for all ηρ 2 t < T 2 , ∥ Φ( x ( t ))
Revised: O ( − ( η + ρ ) log( ηρ )) , then for any T 2 > 0 till which solution of (8) exists, for sufficiently small − ( η + ρ ) log( ηρ ) , we have with probability 1 − O ( ηρ ) , for all ηρ 2 t < T 2 , ∥ Φ( x ( t ))

Original: − X ( ηρ 2 t ) ∥
Revised: − X ( ηρ 2 t ) ∥

Original: =
Revised: =

Original: O ( − ( η + ρ ) log ηρ ) and ∥ x ( t ) − Φ( x ( t ))
Revised: O ( − ( η + ρ ) log ηρ ) and ∥ x ( t ) − Φ( x ( t ))

Original: ∥
Revised: ∥

Original: =
Revised: =

Original: O ( − ( ηρ + ρ 2 ) log ηρ ) .
Revised: O ( − ( ηρ + ρ 2 ) log ηρ ) .

Original: In this section we will define K as { X ( t ) } where X is the solution of (8).
Revised: In this section we will define K as { X ( t ) } where X is the solution of (8).

Original: We will prove these theorems respectively in the following sections.
Revised: We will prove these theorems respectively in the following sections.

Original: D.6.1. P HASE I: P ROOF OF T HEOREM
Revised: D.6.1. P HASE I: P ROOF OF T HEOREM

Original: We will now discuss the convergence of 1-SAM to the manifold of minimizer.
Revised: We will now discuss the convergence of 1-SAM to the manifold of minimizer.

Original: We will separate the dynamics into the following phase.
Revised: We will separate the dynamics into the following phase.

Original: Define ϕ
Revised: Define ϕ

Original: as the gradient flow projection as in deterministic case.
Revised: as the gradient flow projection as in deterministic case.

Original: Subphase A Gradient flow approximation, using standard approximation, we can show that x ( t ) with high probability falls into a region K h where the loss satisfies PL condition.
Revised: Subphase A Gradient flow approximation, using standard approximation, we can show that x ( t ) with high probability falls into a region K h where the loss satisfies PL condition.

Original: Subphase B After reaching the region, a detailed analysis will show that loss continue to decrease until ∥ x ( t ) − Φ( x ( t )) ∥
Revised: Subphase B After reaching the region, a detailed analysis will show that loss continue to decrease until ∥ x ( t ) − Φ( x ( t )) ∥

Original: =
Revised: =

Original: ˜ O ( ρ ) .
Revised: ˜ O ( ρ ) .

Original: Subphase C Consider a quadratic approximation and we will get with high probability x
Revised: Subphase C Consider a quadratic approximation and we will get with high probability x

Original: ( t ) will falls into
Revised: ( t ) will falls into

Original: O ( ηρ + ρ 2 ) neighbor of Φ( x ( t )) .
Revised: O ( ηρ + ρ 2 ) neighbor of Φ( x ( t )) .

Original: Subphase A
Revised: Subphase A

Original: This is analogous to Subphase A in Section D.5.1 and we can suppose ∃ t 1 , x ( t 1 ) ∈
Revised: This is analogous to Subphase A in Section D.5.1 and we can suppose ∃ t 1 , x ( t 1 ) ∈

Original: K h
Revised: K h

Original: Subphase B
Revised: Subphase B

Original: Define event A ( t ) as {∇ L ( x ( τ )) ≥ 4 ζρ, ∀ τ ≤ t } .
Revised: Define event A ( t ) as {∇ L ( x ( τ )) ≥ 4 ζρ, ∀ τ ≤ t } .

Original: We have if ∥∇ L ( x ( t ))
Revised: We have if ∥∇ L ( x ( t ))

Original: ∥ ≥ 4 ζρ
Revised: ∥ ≥ 4 ζρ

Original: E [ L ( x ( t + 1))
Revised: E [ L ( x ( t + 1))

Original: | x ( t )] =
Revised: | x ( t )] =

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: L

Original: [Equation]

Original: x (

Original: [Equation]

Original: We have
Revised: We have

Original: E [

Original: [Equation]
Revised: [Equation]

Original: A ( t )] .

Original: We can then conclude that with t 2 = 2log
Revised: We can then conclude that with t 2 = 2log

Original: [Equation]

Original: ρ

Original: [Equation]
Revised: [Table]

Original: Mηµ
Revised: Mηµ

Original: [Equation]
Revised: [Equation]

Original: t16 ζ[-
-]
Revised: 16 ζ[-
-]

Original: ρ 2 µP ( A ( t 2 + 1))
Revised: [Equation] µ

Original: [Equation]
Revised: [Equation]

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: With an abuse of notation, suppose ∇ L ( x ( t 2 )) ≤ 4 ζρ , which implies ∥ x ( t 2 ) − Φ( x ( t 2 ))
Revised: With an abuse of notation, suppose ∇ L ( x ( t 2 )) ≤ 4 ζρ , which implies ∥ x ( t 2 ) − Φ( x ( t 2 ))

Original: ∥ =
Revised: ∥ =

Original: O ( ρ ) .
Revised: O ( ρ ) .

Original: Subphase C
Revised: Subphase C

Original: After ∥∇ L ( x ( t 2 ))
Revised: After ∥∇ L ( x ( t 2 ))

Original: ∥ ≤ 4 ζρ , it becomes difficult to prove the loss continue to decrease.
Revised: ∥ ≤ 4 ζρ , it becomes difficult to prove the loss continue to decrease.

Original: We proceed by consider a quadratic approximation.
Revised: We proceed by consider a quadratic approximation.

Original: Now consider
Revised: Now consider

Original: [Equation]
Revised: [Equation]

Original: Iteratively define t 2 ,j , for 1 ≤ j ≤ 3 , t 2 , 1 = t 2 .
Revised: Iteratively define t 2 ,j , for 1 ≤ j ≤ 3 , t 2 , 1 = t 2 .

Original: For j ≤ 2 , inductively suppose ∥ x ( t 2 ,j ) − Φ( x ( t 2 ,j ))
Revised: For j ≤ 2 , inductively suppose ∥ x ( t 2 ,j ) − Φ( x ( t 2 ,j ))

Original: ∥ ≤ O ( ρ ( j +1) / 2 + Mηρ ) , let p j
Revised: ∥ ≤ O ( ρ ( j +1) / 2 + Mηρ ) , let p j

Original: =
Revised: =

Original: Φ( x ( t 2 ,j )) , further assume ∇ L k (Φ( x ( t 2 ,j )))
Revised: Φ( x ( t 2 ,j )) , further assume ∇ L k (Φ( x ( t 2 ,j )))

Original: = v i v Ti .
Revised: = v i v Ti .

Original: Further suppose N as the normal space of Γ at p j and T as the tangent space.
Revised: Further suppose N as the normal space of Γ at p j and T as the tangent space.

Original: Define P N and P T as projection to the space.
Revised: Define P N and P T as projection to the space.

Original: We have ∥ P T ( x ( t 2 ,j ) − p j ) ∥
Revised: We have ∥ P T ( x ( t 2 ,j ) − p j ) ∥

Original: =
Revised: =

Original: O (( x ( t 2 ,j ) − p j ) 2 ) .
Revised: O (( x ( t 2 ,j ) − p j ) 2 ) .

Original: Consider t 2 ,j ≤ t ≤ t 2 ,j + ρ j −
Revised: Consider t 2 ,j ≤ t ≤ t 2 ,j + ρ [Equation]

Original: Mη
Revised: Mη

Original: Define event A j ( t ) =
Revised: Define event A j ( t ) =

Original: {∥ x ( τ ) − p j ∥ ≥ ρ ( j +2) /
Revised: {∥ x ( τ ) − p j ∥ ≥ ρ ( j +2) /

Original: 2 |∀ τ ≤ t }
Revised: 2 |∀ τ ≤ t }

Original: We have ∥ P N ( x ( t ) − p j ) ∥
Revised: We have ∥ P N ( x ( t ) − p j ) ∥

Original: =
Revised: =

Original: O ( ρ k +

Revised: O ( ρ k +

Original: + Mηρ 2 ( t − t 2 ,j ))
Revised: + Mηρ 2 ( t − t 2 ,j ))

Original: ≤
Revised: ≤

Original: O ( ρ ( j +2) / 2 )
Revised: O ( ρ ( j +2) / 2 )

Original: Under A j ( t ) , we would have ∥ P T ( x ( t ) − p j ) ∥
Revised: Under A j ( t ) , we would have ∥ P T ( x ( t ) − p j ) ∥

Original: =
Revised: =

Original: O ( ∥ x ( t ) − p j ∥ )
Revised: O ( ∥ x ( t ) − p j ∥ )

Original: By Lemma 34, we have
Revised: By Lemma 34, we have

Original: [Equation]
Revised: [Equation]

Original: We also have
Revised: We also have

Original: [Equation]
Revised: [Equation]

Original: Now by Taylor Expansion, 
Revised: Now by Taylor Expansion, x

Original: [Equation]
Revised: [Equation]

Original: We then have
Revised: We then have

Original: E [ ∥ 
Revised: E [ ∥ x ( [Equation] [Equation] Λ

Original: [Equation]
Revised: [Equation]

Original: We lower bound (cid:80) k
Revised: We lower bound (cid:80) k

Original: Λ k s k ( t ) v Ti ( x ( t )
Revised: Λ k s k ( t ) v Ti ( x ( t )

Original: − p j ) again by Lemma 34, there exists C such that (cid:88)
Revised: − p j ) again by Lemma [Equation] there exists C such that (cid:88)

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: E [∥ [Equation] [Equation] −
Revised: E [

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Hence we have if ∥ x ( t ) − p j ∥ ≥ O ( ηρ )
Revised: Hence we have if ∥ x ( t ) − p j ∥ ≥ O ( ηρ )

Original: E [ ∥ x ( t + 1) − p j ∥| x ( t )] ≤
Revised: E [ ∥ x ( t + 1) − p j ∥| x ( t )] ≤

Original: ∥ x
Revised: ∥ x

Original: ( t ) − p j ∥ − αηρ.
Revised: ( t ) − p j ∥ − αηρ.

Original: which implies,
Revised: which implies,

Original: [Equation]
Revised: [Equation]

Original: Hence αρ
Revised: Hence αρ

Original: [Equation]

Original: [Equation]

Original: P

Original: [Equation]
Revised: [Equation]

Original: t

Original: [Equation]

Original: [Equation]

Original: η

Original: [Equation]

Original: t = t
Revised: t = t

Original: [Equation]
Revised: [Equation]

Original: − 2)

Original: [Equation]
Revised: [Equation]

Original: (cid:88) t
Revised: (cid:88) t

Original: = t
Revised: = t

Original: [Equation]
Revised: [Equation]

Original: P ( A j ( t )) ≤ ∥ x ( t 2 ,j ) − Φ( x ( t 2 ,j )) ∥
Revised: P ( A j ( t )) ≤ ∥ x ( t 2 ,j ) − Φ( x ( t 2 ,j )) ∥

Original: =
Revised: =

Original: O ( ρ ( j +1) / 2 )
Revised: O ( ρ ( j +1) / 2 )

Original: Hence with probability O ( √ ρ ) , there exists t 2 ,j +1 , such that ∥ x ( t 2 ,j +1 ) − p j ∥ ≤ O ( ρ ( j +2) / 2 ) , which further implies ∥ x
Revised: Hence with probability O ( √ ρ ) , there exists t 2 ,j +1 , such that ∥ x ( t 2 ,j +1 ) − p j ∥ ≤ O ( ρ ( j +2) / 2 ) , which further implies ∥ x

Original: ( t 2 ,j +1 ) − Φ( x ( t 2 ,j +1 ))
Revised: ( t 2 ,j +1 ) − Φ( x ( t 2 ,j +1 ))

Original: ∥ ≤ O ( ρ ( j +2) / 2 ) using Lemma 30.
Revised: ∥ ≤ O ( ρ ( j +2) / 2 ) using Lemma 30.

Original: Define t 3 = t 2 ,
Revised: Define t 3 = t 2 ,

Original: D.6.2. P HASE II: P ROOF OF T HEOREM
Revised: D.6.2. P HASE II: P ROOF OF T HEOREM

Original: We will inductively prove the following claim holds with probability 1 − O ( ηρ ) ,
Revised: We will inductively prove the following claim holds with probability 1 − O ( ηρ ) ,

Original: [Equation]
Revised: [Equation]

Original: To be more precise, the induction mainly consists of two parts.
Revised: To be more precise, the induction mainly consists of two parts.

Original: The first part shows that x ( t ) will stay close to the manifold with large probability and the second part shows the direction Φ( x ( t )) moves.
Revised: The first part shows that x ( t ) will stay close to the manifold with large probability and the second part shows the direction Φ( x ( t )) moves.

Original: To be more succinct with previous section, we abuse notation and suppose the iteration starts at t 3 .
Revised: To be more succinct with previous section, we abuse notation and suppose the iteration starts at t 3 .

Original: Part I: Convergence Near Manifold
Revised: Part I: Convergence Near Manifold

Original: We have ∥ x ( t 3 ) − Φ( x ( t 3 ))
Revised: We have ∥ x ( t 3 ) − Φ( x ( t 3 ))

Original: ∥
Revised: ∥

Original: =
Revised: =

Original: O ( ηρ + ρ 2 ) .
Revised: O ( ηρ + ρ 2 ) .

Original: By induction hypothesis, we have x ( t ) ∈
Revised: By induction hypothesis, we have x ( t ) ∈

Original: K h .
Revised: K h .

Original: According to Lemma
Revised: According to Lemma

Original: [Equation]
Revised: [Equation]

Original: Using the same argument in previous section, we have for sufficienly large constant A ,
Revised: Using the same argument in previous section, we have for sufficienly large constant A ,

Original: if A ( ηρ + ρ 2 ) log(1 /ηρ ) ≥ ∥ x ( t ) − Φ( x ( t ))
Revised: if A ( ηρ + ρ 2 ) log(1 /ηρ ) ≥ ∥ x ( t ) − Φ( x ( t ))

Original: ∥ ≥ A ( ηρ + ρ 2 ) , then there exists constant α, B independent of
Revised: ∥ ≥ A ( ηρ + ρ 2 ) , then there exists constant α, B independent of

Original: A ,
Revised: A ,

Original: [Equation]
Revised: [Equation]

Original: Bηρ .

Original: We then have
Revised: We then have

Original: P r ( y ( t + 1) ≥ A ( ηρ + ρ 2 ) log(1 /ηρ )) 
Revised: P r ( y ( t + 1) ≥ A ( ηρ + ρ 2 ) log(1 /ηρ )) =

Original: = t (cid:88) τ
Revised: [Equation]

Original:  =t
Revised: τ =

Revised: [Equation]

Original: P r ( y ( t + 1) ≥ A ( ηρ + ρ 2 ) log(1 /ηρ ) and y ( τ ) <
Revised: P r ( y ( t + 1) ≥ A ( ηρ + ρ 2 ) log(1 /ηρ ) and y ( τ ) <

Original: A ( ηρ + ρ 2 ) and ∀ t + 1 ≥ τ ′ ≥ τ + 1 , y ( τ ′ ) >
Revised: A ( ηρ + ρ 2 ) and ∀ t + 1 ≥ τ ′ ≥ τ + 1 , y ( τ ′ ) >

Original: A ( ηρ + ρ 2 )
Revised: A ( ηρ + ρ 2 )

Original: We then consider each term,
Revised: We then consider each term,

Original: P r ( y ( t + 1) ≥ A ( ηρ + ρ 2 ) log(1 /ηρ ) and y ( τ ) <
Revised: P r ( y ( t + 1) ≥ A ( ηρ + ρ 2 ) log(1 /ηρ ) and y ( τ ) <

Original: A ( ηρ + ρ 2 ) and ∀ t + 1 ≥ τ ′ ≥ τ + 1 , y ( τ ′ ) >
Revised: A ( ηρ + ρ 2 ) and ∀ t + 1 ≥ τ ′ ≥ τ + 1 , y ( τ ′ ) >

Original: A ( ηρ + ρ 2 )) ≤ P r ( y ( t + 1) ≥
Revised: A ( ηρ + ρ 2 )) ≤ P r ( y ( t + 1) ≥

Original: A ( ηρ + ρ 2 ) log(1 /ηρ ) and ∀ t + 1 ≥ τ ′ ≥ τ + 1 , y ( τ ′ ) >
Revised: A ( ηρ + ρ 2 ) log(1 /ηρ ) and ∀ t + 1 ≥ τ ′ ≥ τ + 1 , y ( τ ′ ) >

Original: A ( ηρ + ρ 2 ) |
Revised: A ( ηρ + ρ 2 ) |

Original: A ( ηρ + ρ 2 ) <
Revised: A ( ηρ + ρ 2 ) <

Original: y ( τ + 1) <
Revised: y ( τ + 1) <

Original: ( A + B )( ηρ + ρ 2 ))
Revised: ( A + B )( ηρ + ρ 2 ))

Original: Define a coupled process ˜
Revised: Define a coupled process ˜

Original: y ( τ + 1) =
Revised: y ( τ + 1) =

Original: y ( τ + 1) and ˜
Revised: y ( τ + 1) and ˜

Original: [Equation]
Revised: [Equation]

Original: if otherwise
Revised: if otherwise

Original: Then clearly
Revised: Then clearly

Original: P 
Revised: P r

Original: [Equation]
Revised: [Equation]

Revised: y

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: and

Revised: [Equation]

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Now applying Azuma-Hoeffding bound(Lemma 56), we have
Revised: Now applying Azuma-Hoeffding bound(Lemma 56), we have

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Choosing h = αηρ ( t − τ ) − y ( τ + 1) + A ( ηρ + ρ 2 ) log(1 /ηρ )
Revised: Choosing h = αηρ ( t − τ ) − y ( τ + 1) + A ( ηρ + ρ 2 ) log(1 /ηρ )

Original: P (˜ y ( t + 1) ≥ A ( ηρ + ρ 2 ) log(1 /ηρ )) ≤ exp ( − 2( αηρ ( t − τ ) − y ( τ + 1) + A ( ηρ + ρ 2 ) log(1 /ηρ ))
Revised: P (˜ y ( t + 1) ≥ A ( ηρ + ρ 2 ) log(1 /ηρ )) ≤ exp ( − 2( αηρ ( t − τ ) − y ( τ + 1) + A ( ηρ + ρ 2 ) log(1 /ηρ ))

Original: ( t − τ ) B 2 η [Equation] ≤ 
Revised: ( t − τ ) B 2 η 2 ρ) ≤ exp

Original: [Equation]

Original: ( −

Original: [Equation]
Revised: [Equation]

Original: We then have
Revised: We then have

Original: [Equation]
Revised: [Equation]

Original: Part II: Direction of Φ( x ( t + 1)) − Φ( x ( t ))
Revised: Part II: Direction of Φ( x ( t + 1)) − Φ( x ( t ))

Original: We shall do a Taylor expansion and show that
Revised: We shall do a Taylor expansion and show that

Original: [Equation]
Revised: [Equation]

Original: Now by induction we have,
Revised: Now by induction we have,

Original: [Equation]

Original: x (

Original: [Equation]
Revised: [Equation]

Original: , then by Lemma 35, it implies
Revised: , then by Lemma 35, it implies

Original: [Equation]
Revised: [Equation]

Original: Then we have∥
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: Using Lemma [Equation] we have
Revised: Using Lemma 33, we have

Original: [Equation]
Revised: [Equation]

Original: Hence 
Revised: Hence ∥

Original: [Equation]
Revised: [Equation]

Original: Notice finally that by Lemma
Revised: Notice finally that by Lemma

Original: [Equation]
Revised: [Equation]

Original:  ( t )
Revised: Φ( x ( t )) ∇ ( λ

Original: [Equation]
Revised: [Equation]

Revised: ∇

Original: [Equation]
Revised: [Equation]

Revised: L k (Φ( x ( t )))))

Revised: + O ( ∥ x ( t ) − Φ( x ( t ))

Revised: ∥ )

Original: Hence we have
Revised: Hence we have

Original: [Equation]
Revised: [Equation]

Original: Notice finally,
Revised: Notice finally,

Original: [Equation]
Revised: [Equation]

Original: Together with standard concentration bound, we would have x ( t ) follows the Riemannian gradient flow on Γ for loss
Revised: Together with standard concentration bound, we would have x ( t ) follows the Riemannian gradient flow on Γ for loss

Original: [Equation]
Revised: [Equation]

Original: D.6.3. P ROOF OF COROLLARY
Revised: D.6.3. P ROOF OF COROLLARY

Original: Proof [Proof of Corollary 17] We will do a Taylor expansion on E k
Revised: Proof [Proof of Corollary 17] We will do a Taylor expansion on E k

Original: [ L max k,ρ ]( x ) .
Revised: [ L max k,ρ ]( x ) .

Original: By Theorem 51 and 52, for t > ηρ 2 T ′ 3 , we have ∥ X ( ηρ 2 t ) − x ( t ) ∥ =
Revised: By Theorem 51 and 52, for t > ηρ 2 T ′ 3 , we have ∥ X ( ηρ 2 t ) − x ( t ) ∥ =

Original: ˜ O
Revised: ˜ O

Original: ( η + ρ ) and ∥ x ( t ) − Φ( x ( t ))
Revised: ( η + ρ ) and ∥ x ( t ) − Φ( x ( t ))

Original: ∥
Revised: ∥

Original: =
Revised: =

Original: ˜ O ( ηρ + ρ 2 )
Revised: ˜ O ( ηρ + ρ 2 )

Revised: [Equation]

Revised: [Equation]

Revised: =

Original: [Equation]
Revised: [Equation]

Original: Then as L k ( x ) =
Revised: Then as L k ( x ) =

Original: ˜ O
Revised: ˜ O

Original: ( η 2 ρ 2 + ρ 4 ) and ∥ v T ∇ L k ( x ) ∥ =
Revised: ( η 2 ρ 2 + ρ 4 ) and ∥ v T ∇ L k ( x ) ∥ =

Original: ˜ O ( ηρ + ρ 2 ) , this implies
Revised: ˜ O ( ηρ + ρ 2 ) , this implies

Revised: E k

Revised: [ R max k,

Original: [Equation]
Revised: [Equation]

Original: [ R
Revised: [Equation] =

Original: [Equation]
Revised: [Equation]

Original: Proof [Proof of Corollary 18]
Revised: Proof [Proof of Corollary 18]

Original: Choose T such that X ( T ) is sufficiently close to X ( ∞ ) , such that Tr( X ( T ))
Revised: Choose T such that X ( T ) is sufficiently close to X ( ∞ ) , such that Tr( X ( T ))

Original: ≤
Revised: ≤

Original: Tr( X ( ∞ ))
Revised: Tr( X ( ∞ ))

Original: +2 ϵ
Revised: +2 ϵ

Original: By corollary 17, we have ∥ E k
Revised: By corollary 17, we have ∥ E k

Original: [ R max k,ρ ]( x ( ⌈ T/ ( ηρ 2 ) ⌉ ))
Revised: [ R max k,ρ ]( x ( ⌈ T/ ( ηρ 2 ) ⌉ ))

Original: − ρ 2 Tr( X ( T )) / 2 ∥ ≤ ˜ O
Revised: − ρ 2 Tr( X ( T )) / 2 ∥ ≤ ˜ O

Original: ( ηρ 2 ) .
Revised: ( ηρ 2 ) .

Original: This further implies ∥ E k
Revised: This further implies ∥ E k

Original: [ R max k,ρ ]( x ( ⌈ T/ ( ηρ 2 ) ⌉ ))
Revised: [ R max k,ρ ]( x ( ⌈ T/ ( ηρ 2 ) ⌉ ))

Original: − ρ 2 Tr( X ( ∞ )) / 2 ∥ ≤ ϵρ 2 + ˜
Revised: − ρ 2 Tr( X ( ∞ )) / 2 ∥ ≤ ϵρ 2 + ˜

Original: O
Revised: O

Original: ( ηρ 2 ) .
Revised: ( ηρ 2 ) .

Original: We also have ∥ L ( x ( ⌈ T/ ( ηρ 2 ) ⌉ ))
Revised: We also have ∥ L ( x ( ⌈ T/ ( ηρ 2 ) ⌉ ))

Original: ∥
Revised: ∥

Original: =
Revised: =

Original: O ( η 2 ρ 2 ) .
Revised: O ( η 2 ρ 2 ) .

Original: Then we can leverage Theorem 4 and Theorem 9 to get the desired bound.
Revised: Then we can leverage Theorem 4 and Theorem 9 to get the desired bound.

Original: D.7. Technical Lemmas
Revised: D.7. Technical Lemmas

Original: Lemma 53 (Cor.
Revised: Lemma 53 (Cor.

Original: 4.3.15 in [11]) Let Σ , ˆΣ ∈ R D
Revised: 4.3.15 in [11]) Let Σ , ˆΣ ∈ R D

Original: × D be symmetric and non-negative with eigenvalues λ 1 ≥ ... ≥ λ D and ˆ λ 1 ≥ ... ≥ ˆ λ D , then for any i ,
Revised: × D be symmetric and non-negative with eigenvalues λ 1 ≥ ... ≥ λ D and ˆ λ 1 ≥ ... ≥ ˆ λ D , then for any i ,

Original: [Equation]
Revised: [Equation]

Original: Definition 54 (Unitary invariant norms)
Revised: Definition 54 (Unitary invariant norms)

Original: A matrix norm ∥ · ∥ ∗ on the space of matrices in R p × d is unitary invariant if for any matrix K ∈ R p × d , ∥
Revised: A matrix norm ∥ · ∥ ∗ on the space of matrices in R p × d is unitary invariant if for any matrix K ∈ R p × d , ∥

Original: UKW ∥ ∗ =
Revised: UKW ∥ ∗ =

Original: ∥ K ∥ ∗ for any unitary matrices
Revised: ∥ K ∥ ∗ for any unitary matrices

Original: U ∈ R p × p , W ∈ R d × d .
Revised: U ∈ R p × p , W ∈ R d × d .

Original: Theorem 55 [Davis-Kahan sin( θ ) theorem [5]] Let Σ , ˆΣ ∈ R p × p be symmetric, with eigenvalues λ 1 ≥ . . .
Revised: Theorem 55 [Davis-Kahan sin( θ ) theorem [5]] Let Σ , ˆΣ ∈ R p × p be symmetric, with eigenvalues λ 1 ≥ . . .

Original: ≥ λ p and ˆ λ 1 ≥ . .
Revised: ≥ λ p and ˆ λ 1 ≥ . .

Original: ≥ ˆ λ p respectively.
Revised: ≥ ˆ λ p respectively.

Original: Fix 1 ≤ r ≤ s ≤ p , let d := s − r +and let V = ( v r , v r +1 , . .
Revised: Fix 1 ≤ r ≤ s ≤ p , let d := s − r +and let V = ( v r , v r +1 , . .

Original: . , v s ) ∈ R p ×
Revised: . , v s ) ∈ R p ×

Original: d and ˆ V =
Revised: d and ˆ V =

Original: (ˆ v r , ˆ v r +1 , . . .
Revised: (ˆ v r , ˆ v r +1 , . . .

Original: , ˆ v s ) ∈
Revised: , ˆ v s ) ∈

Original: R p × d have orthonormal columns satisfying Σ v j = λ
Revised: R p × d have orthonormal columns satisfying Σ v j = λ

Original: j v j and ˆΣˆ v j = ˆ λ j ˆ v j for j = r, r + 1 , . .
Revised: j v j and ˆΣˆ v j = ˆ λ j ˆ v j for j = r, r + 1 , . .

Original: s. Define ∆ := min (cid:110) max { 0 , λ s − ˆ λ s +1 } , max { 0 , ˆ λ r − 1 −
Revised: s. Define ∆ := min (cid:110) max { 0 , λ s − ˆ λ s +1 } , max { 0 , ˆ λ r − 1 −

Original: λ r } (cid:111) , where ˆ λ 0 := ∞ and ˆ λ p
Revised: λ r } (cid:111) , where ˆ λ 0 := ∞ and ˆ λ p

Original: +1 :=
Revised: +1 :=

Original: −∞ , we have for any unitary invariant norm ∥ · ∥ ∗ ,
Revised: −∞ , we have for any unitary invariant norm ∥ · ∥ ∗ ,

Original: [Equation]
Revised: [Equation]

Original: Here Θ( ˆ V , V ) ∈ R d × d , with Θ( ˆ V , V ) j,j = arccos σ j for any j ∈ [ d ] and Θ( ˆ V , V ) i,j = 0 for all i ̸ = j ∈ [ d ] .
Revised: Here Θ( ˆ V , V ) ∈ R d × d , with Θ( ˆ V , V ) j,j = arccos σ j for any j ∈ [ d ] and Θ( ˆ V , V ) i,j = 0 for all i ̸ = j ∈ [ d ] .

Original: σ 1 ≥ σ 2 ≥ · · · ≥ σ d denotes the singular values of ˆ V ⊤ V. [sin Θ] ij is defined as sin(Θ ij ) .
Revised: σ 1 ≥ σ 2 ≥ · · · ≥ σ d denotes the singular values of ˆ V ⊤ V. [sin Θ] ij is defined as sin(Θ ij ) .

Original: Lemma 

Revised: Lemma

Original: (Azuma-Hoeffding Bound)
Revised: (Azuma-Hoeffding Bound)

Original: Suppose Z n is a super-martingale, suppose − α ≤ Z i +1 −
Revised: Suppose Z n is a super-martingale, suppose − α ≤ Z i +1 −

Original: Z i ≤ β , then for all n > 0 , a > 0 , we have
Revised: Z i ≤ β , then for all n > 0 , a > 0 , we have

Original: [Equation]
Revised: [Equation]

Original: Lemma 57 ([19]) Let A : R D → R D
Revised: Lemma 57 ([19]) Let A : R D → R D

Original: × D be any C 1 symmetric matrix function and x ∗ ∈ R D satisfying λ 

Revised: × D be any C 1 symmetric matrix function and x ∗ ∈ R D satisfying λ

Original: ( A ( x ∗ )) > λ 2 ( A ( x ∗ )) and v 1 be the top eigenvector of A ( x ∗ ) .
Revised: ( A ( x ∗ )) > λ 2 ( A ( x ∗ )) and v 1 be the top eigenvector of A ( x ∗ ) .

Original: It holds that ∇ λ 1 ( A ( x )) | x = x ∗ = ∇ ( v ⊤ 1 A ( x ) v 1 ) | x = x ∗ .
Revised: It holds that ∇ λ 1 ( A ( x )) | x = x ∗ = ∇ ( v ⊤ 1 A ( x ) v 1 ) | x = x ∗ .

Original: We then present some of the technical lemmas we required to prove Lemma 43.
Revised: We then present some of the technical lemmas we required to prove Lemma 43.

Original: Lemma 

Revised: Lemma

Original: If 0 < c < b − a b 2 , a (cid:113) a 2 +2 b 2 2(1 − cb ) ≥ a 2 + b 2 2 − ca − cb , then a > 12 b, cb ≤
Revised: If 0 < c < b − a b 2 , a (cid:113) a 2 +2 b 2 2(1 − cb ) ≥ a 2 + b 2 2 − ca − cb , then a > 12 b, cb ≤

Original: Proof Notice that
Revised: Proof Notice that

Original: [Equation]
Revised: [Equation]

Original: So
Revised: So

Original: [Equation]
Revised: [Equation]

Original: As c < b − a b 2 , we have 1 > 1 − cb > ab .
Revised: As c < b − a b 2 , we have 1 > 1 − cb > ab .

Original: So
Revised: So

Original: [Equation]
Revised: [Equation]

Original: The above inequality implies a ≥ 12 b .
Revised: The above inequality implies a ≥ 12 b .

Original: As c < b − a b 2 , cb ≤
Revised: As c < b − a b 2 , cb ≤

Original: Lemma
Revised: Lemma

Original: When 0 < a < b, 0 < c < b − a b 2 , we have
Revised: When 0 < a < b, 0 < c < b − a b 2 , we have

Original: [Equation]
Revised: [Equation]

Original: Proof
Revised: Proof

Original: Equivalently, we are going to prove (1 − cb ) b
Revised: Equivalently, we are going to prove (1 − cb ) b(cid:18)

Original: [Equation]
Revised: [Equation]

Original: Further simplifying
Revised: Further simplifying

Original: [Equation]
Revised: [Equation]

Original: we only need to prove
Revised: we only need to prove

Original: [Equation]
Revised: [Equation]

Original: We have the following auxiliary inequalities,
Revised: We have the following auxiliary inequalities,

Original: [Equation]
Revised: [Equation]

Original: Using the above anxiliary inequalities we have
Revised: Using the above anxiliary inequalities we have

Original: [Equation]
Revised: [Equation]

Original: Lemma
Revised: Lemma

Original: When 0 < a < b

Original: [Equation]

Original:  0 < c < b − a b [Equation] a (cid:113) a 2 +2 b 2 2(1 − cb ) ≥ a 2 + b 2 2 − ca − cb , we have
Revised: When 0 < a < b, 0 < c < b − a b 2 , a (cid:113) a 2 +2 b 2 2(1 − cb ) ≥ a 2 + b 2 2 − ca − cb , we have

Original: [Equation]
Revised: [Equation]

Original: Proof Equivalently
Revised: Proof Equivalently

Original: [Equation]
Revised: [Equation]

Original: we are going to prove,
Revised: we are going to prove,

Original: [Equation]
Revised: [Equation]

Original: We have the auxiliary inequality
Revised: We have the auxiliary inequality

Original: [Equation]
Revised: [Equation]