Original: Chaotic Regularization and Heavy-Tailed Limits for Deterministic Gradient Descent
Revised: Chaotic Regularization and Heavy-Tailed Limits for Deterministic Gradient Descent

Original: Abstract
Revised: Abstract

Original: Recent studies have shown that gradient descent (GD) can achieve improved generalization when its dynamics exhibits a chaotic behavior.
Revised: Recent studies have shown that gradient descent (GD) can achieve improved generalization when its dynamics exhibits a chaotic behavior.

Original: However, to obtain the desired effect, the step-size should be chosen sufﬁciently large, a task which is problem dependent and can be difﬁcult in practice.
Revised: However, to obtain the desired effect, the step-size should be chosen sufﬁciently large, a task which is problem dependent and can be difﬁcult in practice.

Original: In this study, we incorporate a chaotic component to GD in a controlled manner, and introduce multiscale perturbed GD (MPGD), a novel optimization framework where the GD recursion is augmented with chaotic perturbations that evolve via an independent dynamical system.
Revised: In this study, we incorporate a chaotic component to GD in a controlled manner, and introduce multiscale perturbed GD (MPGD), a novel optimization framework where the GD recursion is augmented with chaotic perturbations that evolve via an independent dynamical system.

Original: We analyze MPGD from three different angles: (i) By building up on recent advances in rough paths theory, we show that, under appropriate assumptions, as the step-size decreases, the MPGD recursion converges weakly to a stochastic differential equation (SDE) driven by a heavy-tailed Lévy-stable process.
Revised: We analyze MPGD from three different angles: (i) By building up on recent advances in rough paths theory, we show that, under appropriate assumptions, as the step-size decreases, the MPGD recursion converges weakly to a stochastic differential equation (SDE) driven by a heavy-tailed Lévy-stable process.

Original: (ii) By making connections to recently developed generalization bounds for heavy-tailed processes, we derive a generalization bound for the limiting SDE and relate the worst-case generalization error over the trajectories of the process to the parameters of MPGD.
Revised: (ii) By making connections to recently developed generalization bounds for heavy-tailed processes, we derive a generalization bound for the limiting SDE and relate the worst-case generalization error over the trajectories of the process to the parameters of MPGD.

Original: (iii)
Revised: (iii)

Original: We analyze the implicit regularization effect brought by the dynamical regularization and show that, in the weak perturbation regime, MPGD introduces terms that penalize the Hessian of the loss function.
Revised: We analyze the implicit regularization effect brought by the dynamical regularization and show that, in the weak perturbation regime, MPGD introduces terms that penalize the Hessian of the loss function.

Original: Empirical results are provided to demonstrate the advantages of MPGD.
Revised: Empirical results are provided to demonstrate the advantages of MPGD.

Original:  Introduction
Revised:  Introduction

Original: Many important problems in supervised learning can be expressed by the following population risk minimization problem:
Revised: Many important problems in supervised learning can be expressed by the following population risk minimization problem:

Original: [Equation]
Revised: [Equation]

Original: where x ∈ R d denotes the learnable parameter, z ∈ Z denotes a data sample coming from an unknown data distribution D , Z denotes the space of data points, and (cid:96) :
Revised: where x ∈ R d denotes the learnable parameter, z ∈ Z denotes a data sample coming from an unknown data distribution D , Z denotes the space of data points, and (cid:96) :

Original: R d × Z → R + is the loss 36th Conference on Neural Information Processing Systems (NeurIPS 2022).
Revised: R d × Z → R + is the loss 36th Conference on Neural Information Processing Systems (NeurIPS 2022).

Original: function, measuring the quality of the parameter x .
Revised: function, measuring the quality of the parameter x .

Original: Since the data distribution D is not known, as a proxy for solving (1), we consider the empirical risk minimization (ERM) problem, given as follows:
Revised: Since the data distribution D is not known, as a proxy for solving (1), we consider the empirical risk minimization (ERM) problem, given as follows:

Original: [Equation]
Revised: [Equation]

Original: where S n := { z 1 , . . .
Revised: where S n := { z 1 , . . .

Original: , z n } denotes a training set of n points that are independently and identically distributed (i.i.d.) and sampled from D .
Revised: , z n } denotes a training set of n points that are independently and identically distributed (i.i.d.) and sampled from D .

Original: To solve the ERM problem, typically gradient-based optimization algorithms are being used in practice.
Revised: To solve the ERM problem, typically gradient-based optimization algorithms are being used in practice.

Original: In this study, we will consider the gradient descent (GD) algorithm
Revised: In this study, we will consider the gradient descent (GD) algorithm

Original: [C + 47], which is based on the following deterministic recursion:
Revised: [C + 47], which is based on the following deterministic recursion:

Original: [Equation]
Revised: [Equation]

Original: Here, k represents the iteration number, η k > 0 is the step-size (learning rate), and x 0 ∈ R
Revised: Here, k represents the iteration number, η k > 0 is the step-size (learning rate), and x 0 ∈ R

Original: d is the initialization.
Revised: d is the initialization.

Original: Even though stochastic counterparts of GD (e.g., stochastic GD [BCN18]) have been more popular due to their reduced computational requirements, the GD algorithm has attracted increasing attention in the past few years, as it has been illustrated that GD can achieve similar generalization performance, as measured by the generalization error | ˆ R ( x k , S n ) − R ( x k ) | , as long as certain design choices are implemented [GGP + 21].
Revised: Even though stochastic counterparts of GD (e.g., stochastic GD [BCN18]) have been more popular due to their reduced computational requirements, the GD algorithm has attracted increasing attention in the past few years, as it has been illustrated that GD can achieve similar generalization performance, as measured by the generalization error | ˆ R ( x k , S n ) − R ( x k ) | , as long as certain design choices are implemented [GGP + 21].

Original: These design choices mainly include using large and/or oscillating learning rates and using explicit regularization of the empirical risk.
Revised: These design choices mainly include using large and/or oscillating learning rates and using explicit regularization of the empirical risk.

Original: Contrary to the wisdom of optimization theory, which typically suggests using smaller learning rates, the fact that large learning rates often provide better generalization performance in modern nonconvex optimization problems has opened up several research directions.
Revised: Contrary to the wisdom of optimization theory, which typically suggests using smaller learning rates, the fact that large learning rates often provide better generalization performance in modern nonconvex optimization problems has opened up several research directions.

Original: In this context, [CKL + 21] showed that large learning rates drive the GD dynamics towards the ‘edge of stability’, meaning that the learning rate should be chosen large enough to make the algorithm diverge.
Revised: In this context, [CKL + 21] showed that large learning rates drive the GD dynamics towards the ‘edge of stability’, meaning that the learning rate should be chosen large enough to make the algorithm diverge.

Original: Recently, the large learning rates have been extended to exotic learning rate schedules (i.e., non-trivial evolution of η k ), such as fractal scheduling [AGZ21] and cyclic scheduling [Oym21, Smi17].
Revised: Recently, the large learning rates have been extended to exotic learning rate schedules (i.e., non-trivial evolution of η k ), such as fractal scheduling [AGZ21] and cyclic scheduling [Oym21, Smi17].

Original: It has been shown that the use of large learning rates introduces a chaotic behavior in the GD dynamics [KT20].
Revised: It has been shown that the use of large learning rates introduces a chaotic behavior in the GD dynamics [KT20].

Original: Hence, a possible explanation for the improvements brought by large learning rates can be partially attributed to such chaotic behavior, since chaotic systems exhibit a stochastic-like evolutions (even though they are fully deterministic), which might be an underlying cause of the improvements in generalization obtained in non-convex settings.
Revised: Hence, a possible explanation for the improvements brought by large learning rates can be partially attributed to such chaotic behavior, since chaotic systems exhibit a stochastic-like evolutions (even though they are fully deterministic), which might be an underlying cause of the improvements in generalization obtained in non-convex settings.

Original: However, achieving the desired chaotic dynamics is a problem-dependent task and can be often difﬁcult in practice.
Revised: However, achieving the desired chaotic dynamics is a problem-dependent task and can be often difﬁcult in practice.

Original: In this paper, we develop a novel optimization framework, coined multiscale perturbed gradient descent (MPGD), where a chaotic component is introduced to GD in a controlled manner.
Revised: In this paper, we develop a novel optimization framework, coined multiscale perturbed gradient descent (MPGD), where a chaotic component is introduced to GD in a controlled manner.

Original: Our approach is based on extending the GD recursion via generic regularization terms, where the regularization coefﬁcients are modulated by using external deterministic dynamical systems that exhibit chaotic characteristics.
Revised: Our approach is based on extending the GD recursion via generic regularization terms, where the regularization coefﬁcients are modulated by using external deterministic dynamical systems that exhibit chaotic characteristics.

Original: Our main contributions are as follows.
Revised: Our main contributions are as follows.

Original: • We build up on recent advances in rough paths theory and homogenization techniques [GM13, CFK + 16, CFKM20], and show that, under appropriate assumptions, as the step-size decreases, the MPGD recursion converges weakly to a stochastic differential equation (SDE) driven by a heavy-tailed Lévy-stable process (Theorem 4.1; see also Theorem C.1 in Appendix).
Revised: • We build up on recent advances in rough paths theory and homogenization techniques [GM13, CFK + 16, CFKM20], and show that, under appropriate assumptions, as the step-size decreases, the MPGD recursion converges weakly to a stochastic differential equation (SDE) driven by a heavy-tailed Lévy-stable process (Theorem 4.1; see also Theorem C.1 in Appendix).

Original: • We draw connections to the recent theoretical links between heavy-tailed processes and generalization [SSG19, ¸SSDE20, H¸SKM21]
Revised: • We draw connections to the recent theoretical links between heavy-tailed processes and generalization [SSG19, ¸SSDE20, H¸SKM21]

Original: and under certain topological regularity assumptions, we derive a generalization bound (Theorem 4.2) for the limiting SDE and relate the worst-case generalization error over its trajectories to the parameters of MPGD.
Revised: and under certain topological regularity assumptions, we derive a generalization bound (Theorem 4.2) for the limiting SDE and relate the worst-case generalization error over its trajectories to the parameters of MPGD.

Original: • We show that MPGD exhibits a natural form of implicit regularization by deriving an appropriate explicit regularizer in the weak perturbation regime (Theorem 4.3).
Revised: • We show that MPGD exhibits a natural form of implicit regularization by deriving an appropriate explicit regularizer in the weak perturbation regime (Theorem 4.3).

Original: Our analysis shows that, in this regime, the derived regularizer introduces terms that penalize the Hessian of the loss function.
Revised: Our analysis shows that, in this regime, the derived regularizer introduces terms that penalize the Hessian of the loss function.

Original: • We empirically demonstrate the advantages of MPGD on different settings and provide further support for the developed framework and the theory (see Section 5).
Revised: • We empirically demonstrate the advantages of MPGD on different settings and provide further support for the developed framework and the theory (see Section 5).

Original: We emphasize that our primary goal is to understand the behavior of deterministic GD in a chaotic setting.
Revised: We emphasize that our primary goal is to understand the behavior of deterministic GD in a chaotic setting.

Original: In this respect, we shall underline that, at this stage, our goal is not to develop a competitive algorithm that would outperform SGD, but rather to provide a solid theoretical analysis for how stochastic behavior can emerge from GD with suitable deterministic components, as well as their implication in terms of generalization error.
Revised: In this respect, we shall underline that, at this stage, our goal is not to develop a competitive algorithm that would outperform SGD, but rather to provide a solid theoretical analysis for how stochastic behavior can emerge from GD with suitable deterministic components, as well as their implication in terms of generalization error.

Original:  Preliminaries and Technical Background
Revised:  Preliminaries and Technical Background

Original: Notation.
Revised: Notation.

Original: For a vector v ∈ R d , we denote its i th component by either v i or [ v ] i and the Euclidean norm by | v | (or (cid:107) v (cid:107) ).
Revised: For a vector v ∈ R d , we denote its i th component by either v i or [ v ] i and the Euclidean norm by | v | (or (cid:107) v (cid:107) ).

Original: diag(v) denotes the diagonal matrix with i th diagonal entry of v i .
Revised: diag(v) denotes the diagonal matrix with i th diagonal entry of v i .

Original: (cid:104)· , ·(cid:105) denotes the dot product of two vectors.
Revised: (cid:104)· , ·(cid:105) denotes the dot product of two vectors.

Original: U ( d ) denotes uniform distribution on d dimension, tr denotes trace, and the superscript T denotes transposition.
Revised: U ( d ) denotes uniform distribution on d dimension, tr denotes trace, and the superscript T denotes transposition.

Original: ◦ and (cid:12) denote composition and Hadamard product respectively.
Revised: ◦ and (cid:12) denote composition and Hadamard product respectively.

Original: ∇ and ∇ 2 denote gradient and Hessian respectively.
Revised: ∇ and ∇ 2 denote gradient and Hessian respectively.

Original: E denotes expectation.
Revised: E denotes expectation.

Original: (d) = and (d) → denote equivalence and convergence in the sense of distribution respectively.
Revised: (d) = and (d) → denote equivalence and convergence in the sense of distribution respectively.

Original: I denotes the identity matrix, and 1 A denotes indicator function of the set A .
Revised: I denotes the identity matrix, and 1 A denotes indicator function of the set A .

Original: Stable distributions.
Revised: Stable distributions.

Original: Let α ∈ (0 , 2] be a stability parameter and d ≥ 1 .
Revised: Let α ∈ (0 , 2] be a stability parameter and d ≥ 1 .

Original: A random variable X ∈ R d is α -stable if X 1 , X 2 , . .
Revised: A random variable X ∈ R d is α -stable if X 1 , X 2 , . .

Original: are independent copies of X , then n − 1 /α
Revised: are independent copies of X , then n − 1 /α

Original: (cid:80) nj =1 X j (d) =
Revised: (cid:80) nj =1 X j (d) =

Original: X for all n ≥[Sam17].
Revised: X for all n ≥[Sam17].

Original: Stable distributions appear as the limiting distribution in the generalized central limit theorem (CLT)
Revised: Stable distributions appear as the limiting distribution in the generalized central limit theorem (CLT)

Original: [GK54].
Revised: [GK54].

Original: The case α = 2 corresponds to the Gaussian distribution, while α =corresponds to the Cauchy distribution.
Revised: The case α = 2 corresponds to the Gaussian distribution, while α =corresponds to the Cauchy distribution.

Original: The p -th moment of a stable random variable is ﬁnite if and only if p < α .
Revised: The p -th moment of a stable random variable is ﬁnite if and only if p < α .

Original: Here we are interested in the case α ∈ (1 , 2) , such that E [ | X | ] < ∞ and E [ | X | 2 ] = ∞ .
Revised: Here we are interested in the case α ∈ (1 , 2) , such that E [ | X | ] < ∞ and E [ | X | 2 ] = ∞ .

Original: If X is symmetric, then there exists a scale parameter c > 0 and the characteristic function of X is given by E
Revised: If X is symmetric, then there exists a scale parameter c > 0 and the characteristic function of X is given by E

Original: [exp( i (cid:104) ξ, X (cid:105) )]
Revised: [exp( i (cid:104) ξ, X (cid:105) )]

Original: = exp( −| cξ | α ) for all ξ ∈ R d .
Revised: = exp( −| cξ | α ) for all ξ ∈ R d .

Original: If X is a random vector with independent components, then there exists a scale vector c
Revised: If X is a random vector with independent components, then there exists a scale vector c

Original: = ( c 1 , . . .
Revised: = ( c 1 , . . .

Original: , c d ) ∈ R d + and the characteristic function of
Revised: , c d ) ∈ R d + and the characteristic function of

Original: X is given by E
Revised: X is given by E

Original: [exp( i (cid:104) ξ, X (cid:105)
Revised: [exp( i (cid:104) ξ, X (cid:105)

Original: [Equation]
Revised: [Equation]

Original: Lévy processes.
Revised: Lévy processes.

Original: Lévy processes are stochastic processes (L t ) t ≥ 0 with independent and stationary increments, and are deﬁned as follows [App09]:
Revised: Lévy processes are stochastic processes (L t ) t ≥ 0 with independent and stationary increments, and are deﬁned as follows [App09]:

Original: For N ∈ N and t 0 < t 1 < . . .
Revised: For N ∈ N and t 0 < t 1 < . . .

Original: <
Revised: <

Original: t N , the increments (L t i − L t i − 1 ) are independent for all i .
Revised: t N , the increments (L t i − L t i − 1 ) are independent for all i .

Original: For any t > s > 0 , (L t − L s ) and L t − s have the same distribution.
Revised: For any t > s > 0 , (L t − L s ) and L t − s have the same distribution.

Original: L t is continuous in probability, i
Revised: L t is continuous in probability, i

Original: e., for all δ > 0 and s ≥ 0 , P ( | L t − L s | > δ ) → 0 as t → s .
Revised: e., for all δ > 0 and s ≥ 0 , P ( | L t − L s | > δ ) → 0 as t → s .

Original: Typical examples of Lévy processes include Brownian motion and the α -stable processes.
Revised: Typical examples of Lévy processes include Brownian motion and the α -stable processes.

Original: By the
Revised: By the

Original: Lévy-Khintchine formula, a Lévy process (L t ) t ≥ 0 with L 0 = 0 is determined by a triplet ( b, Σ , ν ) for some b ∈ R d ,
Revised: Lévy-Khintchine formula, a Lévy process (L t ) t ≥ 0 with L 0 = 0 is determined by a triplet ( b, Σ , ν ) for some b ∈ R d ,

Original: Σ ∈ R d × d positive semi-deﬁnite and a measure ν on R d \ { 0 } such that (cid:82) x
Revised: Σ ∈ R d × d positive semi-deﬁnite and a measure ν on R d \ { 0 } such that (cid:82) x

Original: (cid:54) =

Revised: (cid:54) =

Original: min { 1 , | x | 2 } ν ( dx ) < ∞ .
Revised: min { 1 , | x | 2 } ν ( dx ) < ∞ .

Original: The characteristic function of L t is therefore exp( − t Ψ( ξ )) , with the characteristic exponent Ψ : R d → C deﬁned by
Revised: The characteristic function of L t is therefore exp( − t Ψ( ξ )) , with the characteristic exponent Ψ : R d → C deﬁned by

Original: [Equation]
Revised: [Equation]

Original: In the Lévy–Itô decomposition of a Lévy process, b denotes a constant drift, Σ is the covariance matrix of a Brownian motion and ν characterises a Lévy jump process, which is modiﬁed to be càdlàg (right-continuous and has left limits everywhere) with countably many jumps.
Revised: In the Lévy–Itô decomposition of a Lévy process, b denotes a constant drift, Σ is the covariance matrix of a Brownian motion and ν characterises a Lévy jump process, which is modiﬁed to be càdlàg (right-continuous and has left limits everywhere) with countably many jumps.

Original: In particular, if b = 0 R d , Σ = 0 R d × d and ν ( dx ) =
Revised: In particular, if b = 0 R d , Σ = 0 R d × d and ν ( dx ) =

Original: | x | − α − 1 dx for some α ∈ (0 , 2) , then the triplet (0 R d , 0 R d × d , | x | − α − 1 dx ) gives a symmetric α -stable process, denoted by (L αt ) t ≥ 0 .
Revised: | x | − α − 1 dx for some α ∈ (0 , 2) , then the triplet (0 R d , 0 R d × d , | x | − α − 1 dx ) gives a symmetric α -stable process, denoted by (L αt ) t ≥ 0 .

Original: Marcus differential equations.
Revised: Marcus differential equations.

Original: Passing to the limit of the driving processes with jumps in an SDE is not trivial, as one needs to be careful with the meaning of integration [CP14].
Revised: Passing to the limit of the driving processes with jumps in an SDE is not trivial, as one needs to be careful with the meaning of integration [CP14].

Original: Unfortunately, the common Itô and Stratonovich integration (which are based on Riemann-Stieltjes sums with the integrand evaluated at the left end point and the midpoint of the partition intervals respectively) fail to provide the desired convergence.
Revised: Unfortunately, the common Itô and Stratonovich integration (which are based on Riemann-Stieltjes sums with the integrand evaluated at the left end point and the midpoint of the partition intervals respectively) fail to provide the desired convergence.

Original: To this end, we resort to Marcus differential equations [Mar81], which in their basic form are given by:
Revised: To this end, we resort to Marcus differential equations [Mar81], which in their basic form are given by:

Original: [Equation]
Revised: [Equation]

Original: where (cid:5) denotes integration in the Marcus sense.
Revised: where (cid:5) denotes integration in the Marcus sense.

Original: Marcus integrals involve sums over inﬁnitely many jumps and transform under the usual laws of calculus [App09], and thus play a similar role for Lévy processes as the Stratonovich integral for Brownian motion.
Revised: Marcus integrals involve sums over inﬁnitely many jumps and transform under the usual laws of calculus [App09], and thus play a similar role for Lévy processes as the Stratonovich integral for Brownian motion.

Original: The solution map of Marcus differential equations is continuous with respect to the driving process under certain variants of the Skorokhod topology.
Revised: The solution map of Marcus differential equations is continuous with respect to the driving process under certain variants of the Skorokhod topology.

Original: Note that if b ( X t ) is constant in X t , then there is no difference in the solutions of the SDE in the sense of Marcus, Itô and Stratonovich.
Revised: Note that if b ( X t ) is constant in X t , then there is no difference in the solutions of the SDE in the sense of Marcus, Itô and Stratonovich.

Original: We refer to Appendix B for more details on these.
Revised: We refer to Appendix B for more details on these.

Original: Generating stable laws using the Thaler map.
Revised: Generating stable laws using the Thaler map.

Original: Let γ := 1 /α .
Revised: Let γ := 1 /α .

Original: It was shown in [GM21] that α -stable laws with α ∈ (1 , 2) can be generated using a deterministic dynamical system whose states y k are obtained by iterating the Thaler map T : [0 , 1] → [0 , 1] [Tha80]:
Revised: It was shown in [GM21] that α -stable laws with α ∈ (1 , 2) can be generated using a deterministic dynamical system whose states y k are obtained by iterating the Thaler map T : [0 , 1] → [0 , 1] [Tha80]:

Original: [Equation]
Revised: [Equation]

Original: with any source of randomness coming solely from the initial condition y 0 ∈ [0 , 1] .
Revised: with any source of randomness coming solely from the initial condition y 0 ∈ [0 , 1] .

Original: The map has the following properties.
Revised: The map has the following properties.

Original: Let y ∗ ∈
Revised: Let y ∗ ∈

Original: (0 , 1) be the unique solution to the equation
Revised: (0 , 1) be the unique solution to the equation

Original: [Equation]
Revised: [Equation]

Original: There are two increasing branches on the intervals [0 , y ∗ ] , [ y ∗ , 1] .
Revised: There are two increasing branches on the intervals [0 , y ∗ ] , [ y ∗ , 1] .

Original: For γ ∈ [0 , 1) , there exists a unique invariant probability density h ( y )
Revised: For γ ∈ [0 , 1) , there exists a unique invariant probability density h ( y )

Original: = 1 − γ 2 1 − γ
Revised: = 1 − γ 2 1 − γ

Original: ( y − γ + (1 + y ) − γ ) .
Revised: ( y − γ + (1 + y ) − γ ) .

Original: Moreover, the density deﬁnes a ﬁnite measure.
Revised: Moreover, the density deﬁnes a ﬁnite measure.

Original: For γ ∈ (0 , 1) , the map is nonuniformly expanding with a neutral ﬁxed point at y =and correlations decay algebraically with rate 1 /k
Revised: For γ ∈ (0 , 1) , the map is nonuniformly expanding with a neutral ﬁxed point at y =and correlations decay algebraically with rate 1 /k

Original: α − 1 (which is sharp)
Revised: α − 1 (which is sharp)

Original: [GM21].
Revised: [GM21].

Original: Heuristically, the trajectory spends prolonged period of times near y = 0 (laminar dynamics near the ﬁxed point), slowing down the decay of correlation as α decreases.
Revised: Heuristically, the trajectory spends prolonged period of times near y = 0 (laminar dynamics near the ﬁxed point), slowing down the decay of correlation as α decreases.

Original: Let the observable v : [0 , 1] → R be a Hölder function with zero mean with respect to the invariant measure of T , i.e., d ν = h d y , and consider the sequence of random variables ( v ◦ T k
Revised: Let the observable v : [0 , 1] → R be a Hölder function with zero mean with respect to the invariant measure of T , i.e., d ν = h d y , and consider the sequence of random variables ( v ◦ T k

Original: ( y 0 ))
Revised: ( y 0 ))

Original: k ≥ 0 , with the randomness coming from the initial state y 0 .
Revised: k ≥ 0 , with the randomness coming from the initial state y 0 .

Original: When γ ∈ (1 / 2 , 1) , the correlations are not summable and CLT breaks down if v (0) (cid:54) = 

Revised: When γ ∈ (1 / 2 , 1) , the correlations are not summable and CLT breaks down if v (0) (cid:54) =

Original: (it ”sees” the ﬁxed point at y = 0 ).
Revised: (it ”sees” the ﬁxed point at y = 0 ).

Original: Heuristically, the
Revised: Heuristically, the

Original: Birkhoff sum (cid:80) k − 1 j =0 v ◦ T j is ballistic with almost linear behavior near x = 0 and the small jumps of size v (0) accumulate into a single large jump, contradicting CLT [GM21].
Revised: Birkhoff sum (cid:80) k − 1 j =0 v ◦ T j is ballistic with almost linear behavior near x = 0 and the small jumps of size v (0) accumulate into a single large jump, contradicting CLT [GM21].

Original: Instead, if v is properly normalized, then it was shown in [Gou04] that the one-sided stable limit law 1 k γ (cid:80) k − 1 j =0 v ◦ T j (d) →
Revised: Instead, if v is properly normalized, then it was shown in [Gou04] that the one-sided stable limit law 1 k γ (cid:80) k − 1 j =0 v ◦ T j (d) →

Original: X α,β as k → ∞ , with β = sign( v (0)) , holds instead, where X α,β denotes the stable law whose characteristic function is given by:
Revised: X α,β as k → ∞ , with β = sign( v (0)) , holds instead, where X α,β denotes the stable law whose characteristic function is given by:

Original: [Equation]
Revised: [Equation]

Original: In this paper, we focus on MPGD with chaotic components whose limiting behavior at each time is described by the class of stable laws X α,β whose characteristic function is given by (6) with the stability parameter α ∈ (1 , 2) (i.e., γ ∈ (1 / 2 , 1) ) and the skewness parameter β ∈ [ − 1 , 1] .
Revised: In this paper, we focus on MPGD with chaotic components whose limiting behavior at each time is described by the class of stable laws X α,β whose characteristic function is given by (6) with the stability parameter α ∈ (1 , 2) (i.e., γ ∈ (1 / 2 , 1) ) and the skewness parameter β ∈ [ − 1 , 1] .

Original: The X α,β is centered (i.e., E X α,β = 0 ), totally skewed (or one-sided) if β = ± 1 , and symmetric if β = 0 .
Revised: The X α,β is centered (i.e., E X α,β = 0 ), totally skewed (or one-sided) if β = ± 1 , and symmetric if β = 0 .

Original:  Multiscale Perturbed Gradient Descent (MPGD)
Revised:  Multiscale Perturbed Gradient Descent (MPGD)

Original: In full generality, we start by considering the following family (parametrized by m > 0 ) of deterministic fast-slow dynamical systems on R d × Y , with Y a bounded metric space:
Revised: In full generality, we start by considering the following family (parametrized by m > 0 ) of deterministic fast-slow dynamical systems on R d × Y , with Y a bounded metric space:

Original: [Equation]
Revised: [Equation]

Original: (7) for k = 0 , 1 , 2 , . . .
Revised: (7) for k = 0 , 1 , 2 , . . .

Original: , where the α i ∈ (1 , 2) , a m : R d → R d , b ( i ) m : R d → R d × r i , v i :
Revised: , where the α i ∈ (1 , 2) , a m : R d → R d , b ( i ) m : R d → R d × r i , v i :

Original: Y → R r i , and
Revised: Y → R r i , and

Original: T i : Y → Y .
Revised: T i : Y → Y .

Original: The initial conditions x ( m ) 0 , y ( i ) 0 are independent random variables.
Revised: The initial conditions x ( m ) 0 , y ( i ) 0 are independent random variables.

Original: One important example of recursions of the form (7) is the GD deﬁned in (3).
Revised: One important example of recursions of the form (7) is the GD deﬁned in (3).

Original: Other than GD, the above family of recursions (7) includes various variants of GD as special cases upon specifying suitable choices of the a m , b ( i ) m , v i , T i , and the dimensions.
Revised: Other than GD, the above family of recursions (7) includes various variants of GD as special cases upon specifying suitable choices of the a m , b ( i ) m , v i , T i , and the dimensions.

Original: Of particular interest are new algorithms that can be studied within the above framework.
Revised: Of particular interest are new algorithms that can be studied within the above framework.

Original: To this end, we introduce multiscale perturbed gradient descent (MPGD), whose dynamics are described by the following recursion:
Revised: To this end, we introduce multiscale perturbed gradient descent (MPGD), whose dynamics are described by the following recursion:

Original: [Equation]
Revised: [Equation]

Original: where T is the Thaler map (5) with γ = α − 1 ∈ (1 / 2 , 1) , the v 1 , v 2 are observable maps, µ ≥ 0 and σ ∈ R are tunable parameters.
Revised: where T is the Thaler map (5) with γ = α − 1 ∈ (1 / 2 , 1) , the v 1 , v 2 are observable maps, µ ≥ 0 and σ ∈ R are tunable parameters.

Original: The above recursion is of the form (7), with q = 2 , r 1 = r 2 =
Revised: The above recursion is of the form (7), with q = 2 , r 1 = r 2 =

Original: d , a m ( x ) =
Revised: d , a m ( x ) =

Original: −∇ ˆ R ( x, S n ) , b (1) m ( x ) = − µ diag(x) , and b (2) m ( x ) =
Revised: −∇ ˆ R ( x, S n ) , b (1) m ( x ) = − µ diag(x) , and b (2) m ( x ) =

Original: σI independent of m .
Revised: σI independent of m .

Original: Another class of algorithms that falls within this framework is when m is set to n , the number of training data points, in which case the empirical risk converges to the population risk as m = n → ∞ .
Revised: Another class of algorithms that falls within this framework is when m is set to n , the number of training data points, in which case the empirical risk converges to the population risk as m = n → ∞ .

Original: To complete the description of MPGD, it remains to specify the choice for the observables v i ( y ( i ) k ) in (8) whose rescaled version can be shown to generate stable laws X α,β , with α ∈ (1 , 2) and β ∈ [ − 1 , 1] , in the limit m → ∞ .
Revised: To complete the description of MPGD, it remains to specify the choice for the observables v i ( y ( i ) k ) in (8) whose rescaled version can be shown to generate stable laws X α,β , with α ∈ (1 , 2) and β ∈ [ − 1 , 1] , in the limit m → ∞ .

Original: Following [GM21], we take them to be independent realizations of the observable v ( k ) described as follows.
Revised: Following [GM21], we take them to be independent realizations of the observable v ( k ) described as follows.

Original: Let d α = α α (1 − γ )Γ(1 − α )cos( απ/ 2) 2 1 −
Revised: Let d α = α α (1 − γ )Γ(1 − α )cos( απ/ 2) 2 1 −

Original: −, and δ 0 , δ 1 , . .
Revised: −, and δ 0 , δ 1 , . .

Original: . be independent copies of the random variable δ
Revised: . be independent copies of the random variable δ

Original: where P ( δ = ± 1) =
Revised: where P ( δ = ± 1) =

Original: 12 (1 ± β ) .
Revised: 12 (1 ± β ) .

Original: The observables v ( k ) are then deﬁned to be: v ( k ) =
Revised: The observables v ( k ) are then deﬁned to be: v ( k ) =

Original: χ ( k ) v ◦ T k , where v : [0 , 1] → R is the mean zero observable given by v ( y ) =
Revised: χ ( k ) v ◦ T k , where v : [0 , 1] → R is the mean zero observable given by v ( y ) =

Original: d − γα (1 − 2 γ − 1 ) − γ ˜ v ( y ) , with ˜ v ( y ) =
Revised: d − γα (1 − 2 γ − 1 ) − γ ˜ v ( y ) , with ˜ v ( y ) =

Original: (cid:26)if y ≤ y ∗ , (1 − 2 1 − γ )
Revised: (cid:26)if y ≤ y ∗ , (1 − 2 1 − γ )

Original: −if y > y ∗ , χ ( k ) =
Revised: −if y > y ∗ , χ ( k ) =

Original: χ k − 1 · · · χ 0 ∈ {± 1 } with χ j = (cid:26)if T j y ≤ y ∗ , δ j if T j y > y ∗ .
Revised: χ k − 1 · · · χ 0 ∈ {± 1 } with χ j = (cid:26)if T j y ≤ y ∗ , δ j if T j y > y ∗ .

Original: In particular, the random variables χ ( k ) get updated only when the trajectory visits ( y ∗ , 1] and are not changed during the phase in [0 , y ∗ ] .
Revised: In particular, the random variables χ ( k ) get updated only when the trajectory visits ( y ∗ , 1] and are not changed during the phase in [0 , y ∗ ] .

Original: The initial condition can be, in theory, equally well chosen using the invariant probability measure ν or the uniform Lebesgue measure.
Revised: The initial condition can be, in theory, equally well chosen using the invariant probability measure ν or the uniform Lebesgue measure.

Original: Empirically, convergence of the probability density is faster if it is drawn using ν .
Revised: Empirically, convergence of the probability density is faster if it is drawn using ν .

Original: Hence, we consider initial conditions drawn using ν .
Revised: Hence, we consider initial conditions drawn using ν .

Original: Therefore, for MPGD we propagate uniformly distributed initial conditions y (cid:48) 

Revised: Therefore, for MPGD we propagate uniformly distributed initial conditions y (cid:48)

Original: ∈ [0 , 1] under 10,000 iterations of the Thaler map and take the initial condition as y 

Revised: ∈ [0 , 1] under 10,000 iterations of the Thaler map and take the initial condition as y

Original: = T 10000 y (cid:48) 0 .
Revised: = T 10000 y (cid:48) 0 .

Original: Figure 1 illustrates the dynamics of a realization of the Thaler iterates and the observable (perturbations used in MPGD), as well as the asymmetry and heavy tail nature of the distribution of the corresponding Birkhoff sum.
Revised: Figure 1 illustrates the dynamics of a realization of the Thaler iterates and the observable (perturbations used in MPGD), as well as the asymmetry and heavy tail nature of the distribution of the corresponding Birkhoff sum.

Original:  Theoretical Analysis
Revised:  Theoretical Analysis

Original: In this section, we analyze MPGD via three different angles.
Revised: In this section, we analyze MPGD via three different angles.

Original: We start by presenting results on superdiffusive limit (homogenization [PS08]) for a class of fast-slow deterministic systems that include MPGD as a special case in Subsection 4.1.
Revised: We start by presenting results on superdiffusive limit (homogenization [PS08]) for a class of fast-slow deterministic systems that include MPGD as a special case in Subsection 4.1.

Original: We then study generalization properties of MPGD by deriving a generalization bound for the limiting dynamics in Subsection 4.2.
Revised: We then study generalization properties of MPGD by deriving a generalization bound for the limiting dynamics in Subsection 4.2.

Original: Lastly, we study the regularizing effects of MPGD via the lens of implicit regularization in Subsection 4.3.
Revised: Lastly, we study the regularizing effects of MPGD via the lens of implicit regularization in Subsection 4.3.

Original: 4.1 Superdiffusive Limits for the Fast-Slow Systems and MPGD
Revised: 4.1 Superdiffusive Limits for the Fast-Slow Systems and MPGD

Original: Without loss of generality, we set q = 1 in (7) and consider the following family of recursions:
Revised: Without loss of generality, we set q = 1 in (7) and consider the following family of recursions:

Original: [Equation]
Revised: [Equation]

Original: where T is the Thaler map and v is the observable constructed in Section 3.
Revised: where T is the Thaler map and v is the observable constructed in Section 3.

Original: The following result shows that stochastic dynamics emerges in the above family of deterministic dynamical systems in the limit m → ∞ .
Revised: The following result shows that stochastic dynamics emerges in the above family of deterministic dynamical systems in the limit m → ∞ .

Original: Moreover, the limiting dynamics are described by a stochastic differential equation (SDE) driven by a heavy-tailed Lévy-stable process.
Revised: Moreover, the limiting dynamics are described by a stochastic differential equation (SDE) driven by a heavy-tailed Lévy-stable process.

Original: Theorem 4.1 (Superdiffusive Limit – Informal) .
Revised: Theorem 4.1 (Superdiffusive Limit – Informal) .

Original: Let α ∈ (1 , 2) and assume that the coefﬁcients a m , b m and the initial condition x
Revised: Let α ∈ (1 , 2) and assume that the coefﬁcients a m , b m and the initial condition x

Original: ( m )of (9) are well-behaved as m → ∞ .
Revised: ( m )of (9) are well-behaved as m → ∞ .

Original: Then, under certain regularity conditions on the a m , b m , the process ( x ( m ) (cid:98) mt (cid:99) )
Revised: Then, under certain regularity conditions on the a m , b m , the process ( x ( m ) (cid:98) mt (cid:99) )

Original: t ≥ 0 converges in distribution to the solution ( X t ) t ≥of the SDE:
Revised: t ≥ 0 converges in distribution to the solution ( X t ) t ≥of the SDE:

Original: [Equation]
Revised: [Equation]

Original: as m → ∞ , where a = lim m →∞ a m , b = lim m →∞ b m , x 0 =
Revised: as m → ∞ , where a = lim m →∞ a m , b = lim m →∞ b m , x 0 =

Original: lim m →∞ x ( m ) 0 , (cid:5) denotes the Marcus integration, deﬁned in Section B.2.1, and (L α,βt ) t ≥ 0 is a Lévy process with the characteristic exponent
Revised: lim m →∞ x ( m ) 0 , (cid:5) denotes the Marcus integration, deﬁned in Section B.2.1, and (L α,βt ) t ≥ 0 is a Lévy process with the characteristic exponent

Original: [Equation]
Revised: [Equation]

Original: In particular, Theorem 4.1 implies convergence in distribution of x ( m ) (cid:98) mt (cid:99) to X t at ﬁxed t .
Revised: In particular, Theorem 4.1 implies convergence in distribution of x ( m ) (cid:98) mt (cid:99) to X t at ﬁxed t .

Original: It is a special case of Theorem C.1, which covers a large class of the maps T and observables v , and extends the results in [GM21, CFKM20] to the case where the coefﬁcients are dependent on m .
Revised: It is a special case of Theorem C.1, which covers a large class of the maps T and observables v , and extends the results in [GM21, CFKM20] to the case where the coefﬁcients are dependent on m .

Original: We provide technical details on the topology used for the convergence and the proof in Appendix B-C.
Revised: We provide technical details on the topology used for the convergence and the proof in Appendix B-C.

Original: Specializing Theorem 4.1 (extended to the case of q = 2 ) to the MPGD recursion (8), we see that the rescaled MPGD iterate process, ( x ( m ) (cid:98) mt (cid:99) )
Revised: Specializing Theorem 4.1 (extended to the case of q = 2 ) to the MPGD recursion (8), we see that the rescaled MPGD iterate process, ( x ( m ) (cid:98) mt (cid:99) )

Original: t ≥ 0 , converges in distribution to the solution ( X t ) t ≥ 0 of the following SDE:
Revised: t ≥ 0 , converges in distribution to the solution ( X t ) t ≥ 0 of the following SDE:

Original: [Equation]
Revised: [Equation]

Original: in the limit m → ∞ , which is equivalent to sending the step-size, η := 1 /m , in the MPGD to zero.
Revised: in the limit m → ∞ , which is equivalent to sending the step-size, η := 1 /m , in the MPGD to zero.

Original: Here (L α 1 ,β 1 t ) t ≥ 0 and (˜L α 2 ,β 2 t ) t ≥ 0 are independent Lévy processes with characteristic exponent (10), and β 1 , β 2 ∈ [ − 1 , 1] are the parameters appearing in the construction of the observables v 1 and v 2 respectively.
Revised: Here (L α 1 ,β 1 t ) t ≥ 0 and (˜L α 2 ,β 2 t ) t ≥ 0 are independent Lévy processes with characteristic exponent (10), and β 1 , β 2 ∈ [ − 1 , 1] are the parameters appearing in the construction of the observables v 1 and v 2 respectively.

Original: Therefore, we see that MPGD converges to a gradient ﬂow driven by heavy-tailed
Revised: Therefore, we see that MPGD converges to a gradient ﬂow driven by heavy-tailed

Original: Lévy-stable processes in the considered scaling limit.
Revised: Lévy-stable processes in the considered scaling limit.

Original: The driving processes contain both additive and multiplicative components.
Revised: The driving processes contain both additive and multiplicative components.

Original: The multiplicative component can be interpreted as a dynamical version of the weight decay, whereas the additive component can be viewed as gradient noise [NVL + 15].
Revised: The multiplicative component can be interpreted as a dynamical version of the weight decay, whereas the additive component can be viewed as gradient noise [NVL + 15].

Original: 4.2 Generalization Properties of MPGD
Revised: 4.2 Generalization Properties of MPGD

Original: In this section, we shift our focus to the generalization properties of MPGD.
Revised: In this section, we shift our focus to the generalization properties of MPGD.

Original: Our main roadmap will be to consider the limiting SDE arising from Theorem 4.1 (with β = 0 ) and relate it to the recently developed generalization bounds for heavy-tailed random processes [¸SSDE20, H¸SKM21].
Revised: Our main roadmap will be to consider the limiting SDE arising from Theorem 4.1 (with β = 0 ) and relate it to the recently developed generalization bounds for heavy-tailed random processes [¸SSDE20, H¸SKM21].

Original: For mathematical convenience, we consider the following special case of (7) 1 :
Revised: For mathematical convenience, we consider the following special case of (7) 1 :

Original: [Equation]
Revised: [Equation]

Original: In the light of Theorem 4.1, the recursion (11) converges weakly to the following SDE:
Revised: In the light of Theorem 4.1, the recursion (11) converges weakly to the following SDE:

Original: [Equation]
Revised: [Equation]

Original: (12) where L αt denotes the symmetric α -stable process in R d (i.e., β = 0 ).
Revised: (12) where L αt denotes the symmetric α -stable process in R d (i.e., β = 0 ).

Original: Following the previous work [¸SSDE20, H¸SKM21], we are interested in bounding the worst-case generalization error over the trajectories of (12).
Revised: Following the previous work [¸SSDE20, H¸SKM21], we are interested in bounding the worst-case generalization error over the trajectories of (12).

Original: More precisely, let ( X t ) 

Revised: More precisely, let ( X t )

Original: ≤ t ≤ 1 denote the solution of (12) 2 , and deﬁne the set X so that it contains all the points visited by the trajectory:
Revised: ≤ t ≤ 1 denote the solution of (12) 2 , and deﬁne the set X so that it contains all the points visited by the trajectory:

Original: [Equation]
Revised: [Equation]

Original: Then, our goal is to analyze sup x ∈X | ˆ R ( x, S n ) − R ( x ) | .
Revised: Then, our goal is to analyze sup x ∈X | ˆ R ( x, S n ) − R ( x ) | .

Original: Since we are mainly interested in the case when a ( x ) =
Revised: Since we are mainly interested in the case when a ( x ) =

Original: −∇ ˆ R ( x, S n ) , the SDE (12) and hence the trajectory X will depend on the dataset S n .
Revised: −∇ ˆ R ( x, S n ) , the SDE (12) and hence the trajectory X will depend on the dataset S n .

Original: Therefore, in the generalization bound, we will need to control the statistical dependence of X on S n , through the notion of ρ -mutual information, deﬁned as follows.
Revised: Therefore, in the generalization bound, we will need to control the statistical dependence of X on S n , through the notion of ρ -mutual information, deﬁned as follows.

Original: Let X, Y be two random elements, let P X,Y denote their joint distribution, and let P X , P Y denote their respective marginal distributions.
Revised: Let X, Y be two random elements, let P X,Y denote their joint distribution, and let P X , P Y denote their respective marginal distributions.

Original: Then the ρ -mutual information between X and Y is deﬁned as: I ρ ( X, Y ) =
Revised: Then the ρ -mutual information between X and Y is deﬁned as: I ρ ( X, Y ) =

Original: D ρ ( P X,Y (cid:107) P X ⊗ P Y ) , where D ρ is the ρ -Renyi divergence: 3 D ρ ( µ,
Revised: D ρ ( P X,Y (cid:107) P X ⊗ P Y ) , where D ρ is the ρ -Renyi divergence: 3 D ρ ( µ,

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: , and d µ d λ denotes the Radon-Nikodym derivative.
Revised: , and d µ d λ denotes the Radon-Nikodym derivative.

Original: Finally, we require two technical regularity assumptions (Assumptions 2 and 3, given in Appendix D) for our generalization bound.
Revised: Finally, we require two technical regularity assumptions (Assumptions 2 and 3, given in Appendix D) for our generalization bound.

Original: Informally, Assumption 2 is a topological regularity condition over the trajectory X , whereas Assumption 3 is a statistical regularity assumption and imposes that in the neighborhood of a local minimum x (cid:63) of ˆ R ( x, S n ) , the statistical behavior of the process X t can be approximated by the process solving the SDE d ˜ X t
Revised: Informally, Assumption 2 is a topological regularity condition over the trajectory X , whereas Assumption 3 is a statistical regularity assumption and imposes that in the neighborhood of a local minimum x (cid:63) of ˆ R ( x, S n ) , the statistical behavior of the process X t can be approximated by the process solving the SDE d ˜ X t

Original: = b ( x (cid:63) ) (cid:5) dL αt .
Revised: = b ( x (cid:63) ) (cid:5) dL αt .

Original: This requires b to be regular in the neighborhood of x (cid:63) and further imposes that a ( x (cid:63) )
Revised: This requires b to be regular in the neighborhood of x (cid:63) and further imposes that a ( x (cid:63) )

Original: = 0 , which is natural when a ( x ) =
Revised: = 0 , which is natural when a ( x ) =

Original: −∇ ˆ R ( x, S n ) .
Revised: −∇ ˆ R ( x, S n ) .

Original: We now ready to present our generalization bound.
Revised: We now ready to present our generalization bound.

Original: Theorem 4.2 (Generalization bound) .
Revised: Theorem 4.2 (Generalization bound) .

Original: Assume that
Revised: Assume that

Original: Assumptions 2 and 3 given in Section D hold, and that (cid:96) is bounded by B > 0 and L -Lipschitz continuous.
Revised: Assumptions 2 and 3 given in Section D hold, and that (cid:96) is bounded by B > 0 and L -Lipschitz continuous.

Original: There exists a constant K 1 > 0 such that with probability at least 1 − δ ,
Revised: There exists a constant K 1 > 0 such that with probability at least 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: where I ∞ ( X, Y ) :
Revised: where I ∞ ( X, Y ) :

Original: =
Revised: =

Original: lim
Revised: lim

Original: ρ →∞ I ρ ( X, Y ) .
Revised: ρ →∞ I ρ ( X, Y ) .

Original: Furthermore, there exists K 2 > 0 such that
Revised: Furthermore, there exists K 2 > 0 such that

Original: [Equation]
Revised: [Equation]

Original: This result shows that the generalization error is mainly bounded by two terms: the tail exponent of the limiting Lévy process L αt , and the statistical dependence of X on S n .
Revised: This result shows that the generalization error is mainly bounded by two terms: the tail exponent of the limiting Lévy process L αt , and the statistical dependence of X on S n .

Original: The bound suggests that, provided the mutual information between X and S n is ﬁxed, the generalization error can be reduced by using a smaller α in MPGD.
Revised: The bound suggests that, provided the mutual information between X and S n is ﬁxed, the generalization error can be reduced by using a smaller α in MPGD.

Original: However, it has been illustrated that a smaller α can induce signiﬁcant error on ˆ R ( x, S n ) [SZTG20, CWZ + 21], hence a reasonable value of α should be chosen to balance both the empirical risk and the generalization error.
Revised: However, it has been illustrated that a smaller α can induce signiﬁcant error on ˆ R ( x, S n ) [SZTG20, CWZ + 21], hence a reasonable value of α should be chosen to balance both the empirical risk and the generalization error.

Original: The algorithm parameters µ, σ interact with the bound via the constant terms K 1 , K 2 and the mutual information terms I 1 , I ∞ .
Revised: The algorithm parameters µ, σ interact with the bound via the constant terms K 1 , K 2 and the mutual information terms I 1 , I ∞ .

Original: Unfortunately, their dependence is quite implicit and we are not able to provide quantitative estimations.
Revised: Unfortunately, their dependence is quite implicit and we are not able to provide quantitative estimations.

Original: Finally, the main difference between Theorem 4.2 and [¸SSDE20, H¸SKM21] is that, in our bounds we have explicit access to the characteristic exponent of L αt since we construct it manually through our dynamics; whereas in the prior work generic processes were used as approximations for SGD, hence their results are not as explicit.
Revised: Finally, the main difference between Theorem 4.2 and [¸SSDE20, H¸SKM21] is that, in our bounds we have explicit access to the characteristic exponent of L αt since we construct it manually through our dynamics; whereas in the prior work generic processes were used as approximations for SGD, hence their results are not as explicit.

Original: 4.3 Implicit Regularization of MPGD
Revised: 4.3 Implicit Regularization of MPGD

Original: To understand MPGD and its connection to the vanilla GD (3) better, we provide additional analysis for the behavior of loss functions optimized under the recursion (8) via the lens of implicit regularization [Mah12, LEHM21].
Revised: To understand MPGD and its connection to the vanilla GD (3) better, we provide additional analysis for the behavior of loss functions optimized under the recursion (8) via the lens of implicit regularization [Mah12, LEHM21].

Original: By this, we mean regularization imposed implicitly by the learning strategy, without explicitly modifying the loss.
Revised: By this, we mean regularization imposed implicitly by the learning strategy, without explicitly modifying the loss.

Original: We shall achieve this by deriving an appropriate explicit regularizer through a perturbation analysis in the weak perturbation regime.
Revised: We shall achieve this by deriving an appropriate explicit regularizer through a perturbation analysis in the weak perturbation regime.

Original: To this end, we work in the regime where the parameters µ, σ are small, keeping m ﬁxed.
Revised: To this end, we work in the regime where the parameters µ, σ are small, keeping m ﬁxed.

Original: We set µ = µ 0 (cid:15) and σ = σ 0 (cid:15) , where (cid:15) > 0 is a small parameter, and analyze the loss optimized using the recursion (8) in the small (cid:15) regime.
Revised: We set µ = µ 0 (cid:15) and σ = σ 0 (cid:15) , where (cid:15) > 0 is a small parameter, and analyze the loss optimized using the recursion (8) in the small (cid:15) regime.

Original: In the sequel, we let x ( m ) k denote the states of the unperturbed
Revised: In the sequel, we let x ( m ) k denote the states of the unperturbed

Original: GD, satisfying the recursion x ( m ) k
Revised: GD, satisfying the recursion x ( m ) k

Original: +1 =
Revised: +1 =

Original: x ( m ) k − 1 m ˆ R
Revised: x ( m ) k − 1 m ˆ R

Original: ( x ( m ) k , S n ) , for k = 0 , 1 , . . . .
Revised: ( x ( m ) k , S n ) , for k = 0 , 1 , . . . .

Original: To simplify notation, we shall denote ˆ R ( x ) :
Revised: To simplify notation, we shall denote ˆ R ( x ) :

Original: =
Revised: =

Original: ˆ R ( x, S n ) .
Revised: ˆ R ( x, S n ) .

Original: The following result relates the loss function, averaged over realizations of the injected perturbations, evolved under training with MPGD to that of unperturbed GD in the weak perturbation regime.
Revised: The following result relates the loss function, averaged over realizations of the injected perturbations, evolved under training with MPGD to that of unperturbed GD in the weak perturbation regime.

Original: Theorem 4.3 (Implicit regularization) .
Revised: Theorem 4.3 (Implicit regularization) .

Original: Let m > 0 , k ∈ N , and S n be given.
Revised: Let m > 0 , k ∈ N , and S n be given.

Original: Assume that r 1 =and r 2 =
Revised: Assume that r 1 =and r 2 =

Original: d in Eq.
Revised: d in Eq.

Original: (8) , and E [ v 1 ( y (1) k )]
Revised: (8) , and E [ v 1 ( y (1) k )]

Original: =
Revised: =

Original: E [ v 2 ( y (2) k )]
Revised: E [ v 2 ( y (2) k )]

Original: = 0 for all k .
Revised: = 0 for all k .

Original: Then, for a scalar-valued loss function ˆ R with ∇ ˆ R having Lipschitz continuous partial derivatives in each coordinate up to order three (inclusive),
Revised: Then, for a scalar-valued loss function ˆ R with ∇ ˆ R having Lipschitz continuous partial derivatives in each coordinate up to order three (inclusive),

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: as (cid:15) → 0 , where the expectation is with respect to the realization of the y ( i ) , the λ ( m ) k are the vector with the l th component:
Revised: as (cid:15) → 0 , where the expectation is with respect to the realization of the y ( i ) , the λ ( m ) k are the vector with the l th component:

Original: [Equation]
Revised: [Equation]

Original: the Φ ( m ) i := (cid:81) k − 1 j = i
Revised: the Φ ( m ) i := (cid:81) k − 1 j = i

Original: ( I − 1 m ∇ 2 ˆ R ( x (
Revised: ( I − 1 m ∇ 2 ˆ R ( x (

Original: [Equation]
Revised: [Equation]

Original: j )) , with the empty product taken to be the identity
Revised: j )) , with the empty product taken to be the identity

Original: [Equation]
Revised: [Equation]

Original: and the
Revised: and the

Original: C ( m ) k are covariance matrices with the ( p, q ) -entry of (cid:80) ki
Revised: C ( m ) k are covariance matrices with the ( p, q ) -entry of (cid:80) ki

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Theorem 4.3 implies that the loss function optimized under MPGD at a given iteration and m is, on average, approximately equivalent to a regularized objective functional.
Revised: Theorem 4.3 implies that the loss function optimized under MPGD at a given iteration and m is, on average, approximately equivalent to a regularized objective functional.

Original: Moreover, ∇ ˆ R ( x ( m ) k ) T v ( m ) k tr (cid:16) C ( m ) k ∇ 2 ˆ R ( x ( m ) k ) (cid:17) ≤ C m for some constant C independent of m , suggesting that the trace term is the dominant explicit regularizer for large m .
Revised: Moreover, ∇ ˆ R ( x ( m ) k ) T v ( m ) k tr (cid:16) C ( m ) k ∇ 2 ˆ R ( x ( m ) k ) (cid:17) ≤ C m for some constant C independent of m , suggesting that the trace term is the dominant explicit regularizer for large m .

Original: This explicit regularizer is solely determined by the discrete-time ﬂow generated by the Jacobians I − 1 m ∇ 2 ˆ R ( x ( m ) j ) , the covariances C ( m ) k , and the Hessian of the loss function, all evaluated along the dynamics of the unperturbed GD.
Revised: This explicit regularizer is solely determined by the discrete-time ﬂow generated by the Jacobians I − 1 m ∇ 2 ˆ R ( x ( m ) j ) , the covariances C ( m ) k , and the Hessian of the loss function, all evaluated along the dynamics of the unperturbed GD.

Original: We can therefore expect the use of the perturbations in MPGD as a regularization mechanism should reduce the Hessian of the loss function according to the perturbation levels ( µ 0 , σ 0 ) and the correlations
Revised: We can therefore expect the use of the perturbations in MPGD as a regularization mechanism should reduce the Hessian of the loss function according to the perturbation levels ( µ 0 , σ 0 ) and the correlations

Original: [Equation]
Revised: [Equation]

Original: Reducing the Hessian of the loss function can lead to ﬂatter minima in the loss landscape, which is widely believed to associate with better generalization in deep learning [KMN + 16, JNM + 19].
Revised: Reducing the Hessian of the loss function can lead to ﬂatter minima in the loss landscape, which is widely believed to associate with better generalization in deep learning [KMN + 16, JNM + 19].

Original: Lastly, we observe that in the weak perturbation regime and for non-convex losses, MPGD is more beneﬁcial than GD in the sense that it reduces the Hessian of the loss, thereby promoting ﬂatter minima.
Revised: Lastly, we observe that in the weak perturbation regime and for non-convex losses, MPGD is more beneﬁcial than GD in the sense that it reduces the Hessian of the loss, thereby promoting ﬂatter minima.

Original: Hence, in order to observe the beneﬁts of our scheme, we suspect that the optimization problem should be non-convex and local minima should have different curvatures.
Revised: Hence, in order to observe the beneﬁts of our scheme, we suspect that the optimization problem should be non-convex and local minima should have different curvatures.

Original:  Empirical Results
Revised:  Empirical Results

Original: In this section, we illustrate the advantages of MPGD compared to other schemes on the tasks of (1) minimizing the widening valley loss, (2) regression on the Airfoil Self-Noise Dataset, and (3) classiﬁcation on CIFAR-10.
Revised: In this section, we illustrate the advantages of MPGD compared to other schemes on the tasks of (1) minimizing the widening valley loss, (2) regression on the Airfoil Self-Noise Dataset, and (3) classiﬁcation on CIFAR-10.

Original: We also provide additional results and details in Appendix F.
Revised: We also provide additional results and details in Appendix F.

Original: 5.1 Minimizing the Widening Valley Loss
Revised: 5.1 Minimizing the Widening Valley Loss

Original: We consider the problem of minimizing the widening valley loss, given by (cid:96) ( u, v ) = v 2 (cid:107) u (cid:107) 2 / 2 .
Revised: We consider the problem of minimizing the widening valley loss, given by (cid:96) ( u, v ) = v 2 (cid:107) u (cid:107) 2 / 2 .

Original: The gradient of l is given by ∇ (cid:96) ( u, v ) =
Revised: The gradient of l is given by ∇ (cid:96) ( u, v ) =

Original: ( v 2 u, (cid:107) u (cid:107) 2 v ) , and the trace of the Hessian is dv 2 + (cid:107)
Revised: ( v 2 u, (cid:107) u (cid:107) 2 v ) , and the trace of the Hessian is dv 2 + (cid:107)

Original: u
Revised: u

Original: (cid:107) 2 , where d is the dimension of u .
Revised: (cid:107) 2 , where d is the dimension of u .

Original: The trace of the Hessian measures the ﬂatness of the minima, which is monotonously changing in this case.
Revised: The trace of the Hessian measures the ﬂatness of the minima, which is monotonously changing in this case.

Original: The loss has a valley of minima with zero loss for all ( u, v ) with v = 0 .
Revised: The loss has a valley of minima with zero loss for all ( u, v ) with v = 0 .

Original: The smaller the norm of u , the ﬂatter the minimum.
Revised: The smaller the norm of u , the ﬂatter the minimum.

Original: Vanilla GD gets stuck when it ﬁrst enters this valley, and the intuition is that injecting suitable perturbations should help convergence to a ﬂat (with small (cid:107) u (cid:107) ) part of the valley.
Revised: Vanilla GD gets stuck when it ﬁrst enters this valley, and the intuition is that injecting suitable perturbations should help convergence to a ﬂat (with small (cid:107) u (cid:107) ) part of the valley.

Original: All ( u, v ) with v = 0 are minima, but we also need (cid:107) u (cid:107) to be minimized in order to minimize the trace of the Hessian.
Revised: All ( u, v ) with v = 0 are minima, but we also need (cid:107) u (cid:107) to be minimized in order to minimize the trace of the Hessian.

Original: In high dimension (large d ), the GD path is biased towards making v small and not optimizing u since the direction along v is the most curved.
Revised: In high dimension (large d ), the GD path is biased towards making v small and not optimizing u since the direction along v is the most curved.

Original: For the experiments, we start optimizing from the point ( u 0 , 0) , where u 0 ∼ 5 · U ( d ) with d = 10 , and use the learning rate η = 0 .
Revised: For the experiments, we start optimizing from the point ( u 0 , 0) , where u 0 ∼ 5 · U ( d ) with d = 10 , and use the learning rate η = 0 .

Original: We study and compare the behavior of the following schemes: (i) baseline (vanilla GD), (ii) GD with uncorrelated Gaussian noise injection instead, and (iii) MPGD.
Revised: We study and compare the behavior of the following schemes: (i) baseline (vanilla GD), (ii) GD with uncorrelated Gaussian noise injection instead, and (iii) MPGD.

Original: Figure 2 demonstrates that MPGD can lead to successful optimization of the widening loss whereas the baseline GD and GD with Gaussian perturbations lead to poor solutions.
Revised: Figure 2 demonstrates that MPGD can lead to successful optimization of the widening loss whereas the baseline GD and GD with Gaussian perturbations lead to poor solutions.

Original: This is in agreement with our analysis of implicit regularization for MPGD, showing that the injected perturbations effectively favor small trace of the loss Hessian, thereby biasing the solution to ﬂatter region of the loss landscape.
Revised: This is in agreement with our analysis of implicit regularization for MPGD, showing that the injected perturbations effectively favor small trace of the loss Hessian, thereby biasing the solution to ﬂatter region of the loss landscape.

Original: 5.2 Airfoil Self-Noise Prediction
Revised: 5.2 Airfoil Self-Noise Prediction

Original: [Table]
Revised: [Table]

Original: We consider the Airfoil Self-Noise Dataset [DG17] from the UCI repository.
Revised: We consider the Airfoil Self-Noise Dataset [DG17] from the UCI repository.

Original: It comprises of differently sized airfoils at various wind tunnel speeds and angles of attack.
Revised: It comprises of differently sized airfoils at various wind tunnel speeds and angles of attack.

Original: This is a regression problem aiming to predict scaled sound pressure level, in decibels, of the airfoil based on the the features: frequency, angle of attack, chord length, free-stream velocity, and suction side displacement thickness.
Revised: This is a regression problem aiming to predict scaled sound pressure level, in decibels, of the airfoil based on the the features: frequency, angle of attack, chord length, free-stream velocity, and suction side displacement thickness.

Original: We use 1202 samples for training and 301 samples for testing.
Revised: We use 1202 samples for training and 301 samples for testing.

Original: For the training, we use a fully connected shallow neural network of width 16 with ReLU activation and train for 3000 epochs with the learning rate η = 0 . 1 , using mean square error (MSE) as the loss and choosing β = 0 .
Revised: For the training, we use a fully connected shallow neural network of width 16 with ReLU activation and train for 3000 epochs with the learning rate η = 0 . 1 , using mean square error (MSE) as the loss and choosing β = 0 .

Original: Table 1 reports the average root MSE (RMSE) and the RMSE gap (deﬁned as test RMSE - train RMSE) evaluated for models that are trained with 5 different seed values for this task.
Revised: Table 1 reports the average root MSE (RMSE) and the RMSE gap (deﬁned as test RMSE - train RMSE) evaluated for models that are trained with 5 different seed values for this task.

Original: We can see that MPGD leads to both lower test RMSE and RMSE gap when compared with vanilla GD (baseline) and GD with uncorrelated Gaussian perturbations (see the results not in parenthesis in Table 1; here µ = 0 .
Revised: We can see that MPGD leads to both lower test RMSE and RMSE gap when compared with vanilla GD (baseline) and GD with uncorrelated Gaussian perturbations (see the results not in parenthesis in Table 1; here µ = 0 .

Original: 01 , σ = 0 .
Revised: 01 , σ = 0 .

Original: Using the form of the perturbations in (11) instead can also give lower RMSE gap (see the results in parenthesis in Table 1; here σ = µ = 0 . 01 ).
Revised: Using the form of the perturbations in (11) instead can also give lower RMSE gap (see the results in parenthesis in Table 1; here σ = µ = 0 . 01 ).

Original: Overall these results support our generalization theory for MPGD.
Revised: Overall these results support our generalization theory for MPGD.

Original: Lastly, we note that using larger values of γ naively does not guarantee better test performance: one has to ﬁne tune the parameters β , µ , σ , η appropriately to achieve favorable trade-off between training stability and test performance.
Revised: Lastly, we note that using larger values of γ naively does not guarantee better test performance: one has to ﬁne tune the parameters β , µ , σ , η appropriately to achieve favorable trade-off between training stability and test performance.

Original: 5.3 CIFAR-10 Classiﬁcation
Revised: 5.3 CIFAR-10 Classiﬁcation

Original: We consider training a ResNet-18 classiﬁer [HZRS16] on the CIFAR-10 data set [Kri09].
Revised: We consider training a ResNet-18 classiﬁer [HZRS16] on the CIFAR-10 data set [Kri09].

Original: We follow the setup used in [GGP + 21], where a ResNet-18 is trained using batch size of 50K (the entire training dataset).
Revised: We follow the setup used in [GGP + 21], where a ResNet-18 is trained using batch size of 50K (the entire training dataset).

Original: As in [GGP + 21], we consider a standard ResNet-18 with the parameters in the linear layer randomly initialized and the parameters in the batch normalization initialized with mean zero and unit variance, except for the last in each residual branch which is initialized to zero.
Revised: As in [GGP + 21], we consider a standard ResNet-18 with the parameters in the linear layer randomly initialized and the parameters in the batch normalization initialized with mean zero and unit variance, except for the last in each residual branch which is initialized to zero.

Original: The default random CIFAR-10 data ordering is kept as is, and every image is randomly augmented by horizontal ﬂips and random crops after padding by 4 pixels.
Revised: The default random CIFAR-10 data ordering is kept as is, and every image is randomly augmented by horizontal ﬂips and random crops after padding by 4 pixels.

Original: Using the setup of [GGP + 21], the reference mini-batch SGD is trained using a batch size of(sampling without replacement), Nesterov momentum of 0.9 and weight decay of 0.0005.
Revised: Using the setup of [GGP + 21], the reference mini-batch SGD is trained using a batch size of(sampling without replacement), Nesterov momentum of 0.9 and weight decay of 0.0005.

Original: The learning rate is warmed up from 0.0 to 0.1 over the ﬁrst 5 epochs and then reduced via cosine annealing to 0 over the course of training for 300 epochs (resulting in 390x300=117,000 update steps).
Revised: The learning rate is warmed up from 0.0 to 0.1 over the ﬁrst 5 epochs and then reduced via cosine annealing to 0 over the course of training for 300 epochs (resulting in 390x300=117,000 update steps).

Original: The validation accuracy obtained in [GGP + 21] is 95.7%.
Revised: The validation accuracy obtained in [GGP + 21] is 95.7%.

Original: For the full-batch GDs, we replace the mini-batch updates by full batches and accumulate the gradients over all mini-batches.
Revised: For the full-batch GDs, we replace the mini-batch updates by full batches and accumulate the gradients over all mini-batches.

Original: For the
Revised: For the

Original: Gaussian scheme and MPGD, we inject the perturbations into the GD recursion for parameters of the input layer of ResNet-18, choosing µ = 0 .
Revised: Gaussian scheme and MPGD, we inject the perturbations into the GD recursion for parameters of the input layer of ResNet-18, choosing µ = 0 .

Original: 03 , σ = 0 .
Revised: 03 , σ = 0 .

Original: 01 and β = 0 .
Revised: 01 and β = 0 .

Original: 5 (for MPGD).
Revised: 5 (for MPGD).

Original: In our experiments,
Revised: In our experiments,

Original: [Table]
Revised: [Table]

Original: we ﬁx the learning rate to 0.1 throughout training, and do not add momentum and weight decay.
Revised: we ﬁx the learning rate to 0.1 throughout training, and do not add momentum and weight decay.

Original: This is in constrast to [GGP + 21], where the additional tricks applied in the mini-batch SGD are also used for the full-batch GD, leading to higher test accuracies (see Table 2 in [GGP + 21]).
Revised: This is in constrast to [GGP + 21], where the additional tricks applied in the mini-batch SGD are also used for the full-batch GD, leading to higher test accuracies (see Table 2 in [GGP + 21]).

Original: Table 2 shows that MPGD can lead to better test performance and lower accuracy gap when compared to the baseline (full batch GD) and Gaussian noise-perturbed GD.
Revised: Table 2 shows that MPGD can lead to better test performance and lower accuracy gap when compared to the baseline (full batch GD) and Gaussian noise-perturbed GD.

Original: Using the perturbations in (11) instead can also achieve lower accuracy gap.
Revised: Using the perturbations in (11) instead can also achieve lower accuracy gap.

Original: Therefore, adding perturbations of MPGD can be potentially a useful trick to improve training of deep architectures on benchmark data sets.
Revised: Therefore, adding perturbations of MPGD can be potentially a useful trick to improve training of deep architectures on benchmark data sets.

Original: Lastly, we note that additional tricks, such as gradient clipping and tuning of optimization hyperparameters, can be applied to improve our test accuracies and close the gap to results obtained with the reference mini-batch SGD.
Revised: Lastly, we note that additional tricks, such as gradient clipping and tuning of optimization hyperparameters, can be applied to improve our test accuracies and close the gap to results obtained with the reference mini-batch SGD.

Original: However, our main focus here is not on competitive performance, but rather on demonstrating the effects of the perturbations in MPGD when compared to full batch GDs and
Revised: However, our main focus here is not on competitive performance, but rather on demonstrating the effects of the perturbations in MPGD when compared to full batch GDs and

Original: Gaussian noise-perturbed full-batch GD to support our theory.
Revised: Gaussian noise-perturbed full-batch GD to support our theory.

Original: Lastly, we remark that we did not tune the step-size for any of the algorithms.
Revised: Lastly, we remark that we did not tune the step-size for any of the algorithms.

Original: Since our generalization bound applies to the asymptotic SDE, which is obtained when the step-size goes to zero, in order to stay close to the theory, we chose a small enough step-size to be both not far from the continuous dynamics, and large-enough so that the algorithm converges in a reasonable amount of time.
Revised: Since our generalization bound applies to the asymptotic SDE, which is obtained when the step-size goes to zero, in order to stay close to the theory, we chose a small enough step-size to be both not far from the continuous dynamics, and large-enough so that the algorithm converges in a reasonable amount of time.

Original: That being said, we have tried a range of step-sizes for both algorithms, and we observed that the proposed scheme consistently outperforms vanilla GD for smaller step-sizes.
Revised: That being said, we have tried a range of step-sizes for both algorithms, and we observed that the proposed scheme consistently outperforms vanilla GD for smaller step-sizes.

Original: Whereas if we use a large stepsize, both algorithms perform similarly.
Revised: Whereas if we use a large stepsize, both algorithms perform similarly.

Original: In this regime, vanilla GD as well potentially emits a chaotic behavior, which might also indicate the importance of the "implicit randomness".
Revised: In this regime, vanilla GD as well potentially emits a chaotic behavior, which might also indicate the importance of the "implicit randomness".

Original: However, as we indicated in the introduction, this regime is not easily controllable, and our purpose is to introduce a controlled chaotic component, with a clear theoretical understanding.
Revised: However, as we indicated in the introduction, this regime is not easily controllable, and our purpose is to introduce a controlled chaotic component, with a clear theoretical understanding.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we introduce and study a class of slow-fast deterministic dynamical systems which homogenize to a limiting SDE driven by a heavy-tailed Lévy-stable process in an appropriate scaling limit as a rigorous framework for perturbed GDs.
Revised: In this paper, we introduce and study a class of slow-fast deterministic dynamical systems which homogenize to a limiting SDE driven by a heavy-tailed Lévy-stable process in an appropriate scaling limit as a rigorous framework for perturbed GDs.

Original: Within this framework, we introduce MPGD, a novel version of perturbed GD, which we show to have good generalization and regularization properties.
Revised: Within this framework, we introduce MPGD, a novel version of perturbed GD, which we show to have good generalization and regularization properties.

Original: We further demonstrate the advantages of MPGD empirically in various optimization tasks.
Revised: We further demonstrate the advantages of MPGD empirically in various optimization tasks.

Original: Our framework can provide useful tools for identifying implicit randomness in deterministic optimization algorithms and inspire other promising algorithms.
Revised: Our framework can provide useful tools for identifying implicit randomness in deterministic optimization algorithms and inspire other promising algorithms.

Original: It would be interesting to investigate the interactions of the chaotic perturbations in MPGD with the noise arising from mini-batch sampling in SGD [WHX + 20] and other optimization tricks.
Revised: It would be interesting to investigate the interactions of the chaotic perturbations in MPGD with the noise arising from mini-batch sampling in SGD [WHX + 20] and other optimization tricks.

Original: Since this is a theoretical paper studying a framework for perturbed GDs, there are no potential negative societal impacts of our work.
Revised: Since this is a theoretical paper studying a framework for perturbed GDs, there are no potential negative societal impacts of our work.

Original: Acknowledgements.
Revised: Acknowledgements.

Original: We are grateful to the computational resources provided by the Swedish
Revised: We are grateful to the computational resources provided by the Swedish

Original: National Infrastructure for Computing (SNIC) at Chalmers Centre for Computational Science and
Revised: National Infrastructure for Computing (SNIC) at Chalmers Centre for Computational Science and

Original: Engineering (C3SE) partially funded by the Swedish Research Council through grant agreement no.
Revised: Engineering (C3SE) partially funded by the Swedish Research Council through grant agreement no.

Original: S. H. Lim would like to acknowledge the WINQ Fellowship, the SNIC AI/ML grant, and the Swedish Research Council (VR/2021-03648) for providing support of this work.
Revised: S. H. Lim would like to acknowledge the WINQ Fellowship, the SNIC AI/ML grant, and the Swedish Research Council (VR/2021-03648) for providing support of this work.

Original: U. ¸S.’s research was funded in part by the French government under management of Agence Nationale de la Recherche as part of the “Investissements d’avenir” program, reference ANR-19-P3IA-0001 (PRAIRIE 3IAInstitute), and the European Research Council Starting Grant DYNASTY – 101039676. Appendix A Overview Organizational Details.
Revised: U. ¸S.’s research was funded in part by the French government under management of Agence Nationale de la Recherche as part of the “Investissements d’avenir” program, reference ANR-19-P3IA-0001 (PRAIRIE 3IA

Original: This Appendix is organized as follows.

Original: • In Section B, we provide technical background materials that will be needed for proving the homogenization results.

Original: • In Section C, we present a general theorem that includes Theorem 4.1 as a special case and provide the proof.

Original: • In Section D, we provide proof for the generalization bound (Theorem 4.2).

Original: • In Section E, we provide proof for the implicit regularization result (Theorem 4.3).

Original: • In Section F, we provide additional empirical results and details.

Original: In addition to the notations introduced in the main paper, we shall need the following notations.

Original: Notation.

Original: Denote the space of càdlàg (right continuous with left limits) functions from [0 , 1] to R d by D ([0 , 1] , R d ) , on which the Skorokhod-type metrics are denoted by σ · .

Original: The left limit of f ∈ D ([0 , 1] , R d ) at t is written by f ( t − ) :

Original: = lim s ↑ t f ( s ) .

Original: The identity function is denoted by id : t (cid:55)→ t, t ∈ [0 , 1] .

Original: For a continuous function f deﬁned on R d , denote the uniform norm by (cid:107) f (cid:107) ∞ =

Original: sup x ∈ R d | f ( x ) | .

Original: Denote by D ([0 , 1] , R d ) the space of admissible path-trajectory pairs in

Original: Deﬁnition B.5, whose metrics are denoted by α · .

Original: Denote by C p -var ([0 , 1] , R d ) the space of continuous functions with ﬁnite p -variation.

Original: In Deﬁnition B.4, denote by l d the linear interpolation in R d .

Original: B Additional Technical Background B.1 Background on metrics and topologies on the space of càdlàg functions B.1.1 Skorokhod-type topologies

Original: The Skorokhod J 1 topology on D ([0 , 1] , R d ) is induced by the following metric.

Original: Deﬁnition B.1 (Skorokhod distance) .

Original: The Skorokhod distance on D ([0 , 1] , R d ) , the space of càdlàg functions, is deﬁned by

Original: [Equation]

Original: where the inf is taken on all increasing bijections λ from [0 , 1] to itself.

Original: Another important topology on D ([0 , 1] , R d ) is the SM 1 topology deﬁned as follows.

Original: Deﬁnition B.2 (Skorokhod SM 1 topology) .

Original: The Skorokhod SM 1 topology on D ([0 , 1] , R d ) , the space of càdlàg functions, is deﬁned by the metric

Original: [Equation]

Original: where the inf is taken on all pairs ( λ i , γ i ) ∈ C ([0 , 1] , [0 , 1] × R d ) such that ( λ i , γ i )(0)

Original: = (0 , X i (0)) , ( λ i , γ i )(1) =

Original: (1 , X i (1)) and γ i ( t ) ∈

Original: [ X i ( λ i ( t ) − ) , X i ( λ i ( t ))] , i = 1 , 2 .

Original: It is equivalent to the metric on the graph of functions in D , with discontinuities connected by the straight segments.

Original: Recall that for 1 ≤ p < ∞ , the p -variation of u : [0 , 1] → R d is given by

Original: [Equation]

Original: and the subspace of D ([0 , 1] , R d ) with ﬁnite p -variation is denoted by D p -var ([0 , 1] , R d ) .

Original: Let us deﬁne the following p -variation generalisations of the Skorokhod topology.

Original: Deﬁnition B.3 (Skorokhod-type p -variation) .

Original: [Equation]

Original: where the inf is taken on all increasing bijections λ from [0 , 1] to itself.

Original: B.1.2 Generalised Skorokhod topologies with interpolations

Original: For discontinuous càdlàg functions, one can interpolate jumps with path functions.

Original: Deﬁnition B.4 (Path function) .

Original: A path function on R d is a map φ : J → C ([0 , 1] , R d ) , where

Original: J ⊂ R d × R d , for which φ ( x, y )(0)

Original: =

Original: x and φ ( x, y )(1) =

Original: y for all ( x, y ) ∈ J .

Original: Deﬁnition B.5.

Original: A pair ( X, φ ) is called admissible if all the jumps of X are in the domain of deﬁniton

Original: J of φ , i.e. ( X ( t − ) , X ( t ))

Original: ∈ J for all jump times t of X .

Original: Denote by D ([0 , 1] , R d ) the space of admissible pairs, modulo the equivalence ( X 1 , φ 1 ) ∼ ( X 2 , φ 2 ) if X 1 = X 2 and φ 1 ( X 1 ( t − ) , X 1 ( t )) is a reparametrization of φ 2 ( X 1 ( t − ) , X 1 ( t )) for all jump times t of X 1 .

Original: Deﬁnition B.6.

Original: For admissible pair ( X, φ ) ∈ D ([0 , 1] , R d ) , we now construct a continuous path

Original: X φ,δ as follows.

Original: • Given a sequence r 1 , r 2 , . . .

Original: > 0 with r :=

Original: (cid:80)

Original: j r j < ∞ , Let τ : [0 , 1] → [0 , 1 + r ] given by τ ( t )

Original: =

Original: t + (cid:88)

Original: k δr k 1 { t k ≤ t

Original: Deﬁne

Original: an intermediate process ˆ X ∈ C ([0 , 1 + δr ] , R d ) ,

Original: ˆ X ( t ) =

Original:  

Original: X ( s ) if t = τ ( s ) for some s ∈ [0 , 1] φ ( X ( t k

Original: [Equation]

Original: X ( t k ))

Original: [Equation]

Original: ) δr

Original: [Equation]

Original: (cid:19) if t

Original: [Equation]

Original: t k − ) , τ ( t k )) for some k.

Original: • Finally, let X φ,δ ( t )

Original: =

Original: ˆ X ( t (1 + δr )) .

Original: We will drop the superscript δ if δ = 1 .

Original: Deﬁnition B.7.

Original: The (pseudo)metric α p -var on D ([0 , 1] , R d ) is deﬁned by

Original: [Equation]

Original: independent of the choice of the series (cid:80) ∞ k

Original: =1 r k .

Original: Denote by D p -var ([0 , 1] , R d ) the subspace of

Original: D ([0 , 1] , R d ) with ﬁnite α p -var distance to 0 .

Original: Conventionally, write

Original: [Equation]

Original: Remark B.1.

Original: If φ is the linear path function, then α ∞ induces the SM 1 topology on the space

Original: D ([0 , 1] , R d ) .

Original: B.2 Background on rough differential equations (RDEs)

Original: Deﬁnition B.8 (Forward RDE) .

Original: We say that X is a solution to the forward RDE d X t

Original: = b ( X ) − t d W t if

Original: [Equation]

Original: where the integral above denotes a limit of Riemann-Stieltjes sums with b ( X ( s − )) evaluated at the left limit points of the partition intervals:

Original: [Equation]

Original: where the P are partitions of [0 , t ] into intervals, and |P| is the size of the longest interval.

Original: Remark B.2.

Original: We make the following two observations.

Original: • If W has ﬁnitely many jumps at times 0 < t 1 < . . .

Original: < t n ≤ 1 , then the forward solution of d X t = b ( X ) − t d W t , X 0 = x can be obtained by solving the canonical RDE on each of the intervals on which W is continuous,

Original: i

Original: [Equation]

Original: e., [0 , t 1 ) , [ t 1 , t 2 ) , . . .

Original: , [ t n , 1) , and requiring that at jump times t k , k = 1 , . . .

Original: , , n :

Original: [Equation]

Original: • If in (7), we write X ( m ) t := x ( m ) (cid:98) mt (cid:99) and V ( m ) t =

Original: [Equation]

Original: tm (cid:99) /m , then the ﬁrst equation of (7) is nothing but the forward RDE

Original: [Equation]

Original: B.2.1 Young’s integral and Marcus SDE

Original: For the purpose of deﬁning Young’s integral for driving functions with ﬁnite q -variation, q ≥ 1 , we ﬁx x ∈ C p -var ([0 , 1] , R d ) and y ∈ C q -var ([0 , 1] , R e × d ) with θ = 1 p + 1 q > 1 .

Original: Deﬁnition B.9 (Young’s integral [You36]) .

Original: There exists a sequence x n ∈ C 1 -var ([0 , T ] , R d ) such that x n → x in C p -var ([0 , T ] , R d ) and a sequence y n ∈ C 1 -var ([0 , T ] , R e × d ) such that y n → y in

Original: C q -var ([0 , T ] , R e × d ) .

Original: For every s < t , the limit of (cid:82) t s y n ( u ) dx n ( u ) exists, which we denote by (cid:82) t s y ( u ) dx ( u ) the Young’s integral of y against x on the interval [ s, t ] .

Original: The Young’s integral does not depend on the choices of sequences x n and y n , and we have

Original: [Equation]

Original: We record the following Young-Lóeve estimate, which will be useful for our purposes.

Original: Proposition B.1 ([FV10] Theorem 6.8) .

Original: Let x ∈ C p -var ([0 , T ] , R d ) and

Original: y ∈ C q -var ([0 , T ] , R e × d ) with 1 p + 1 q > 1 .

Original: [-The integral path t → (cid:82) t 
-]
Revised: [-The integral path t → (cid:82) t 
-]

Original: y ( u ) dx ( u ) is continuous with a ﬁnite p -variation and

Original: [Equation]

Original: Solutions to Marcus rough differential equation (RDE) are deﬁned via Young’s integrals and the linear path function deﬁned in Deﬁnition B.6.

Original: Deﬁnition B.10.

Original: The solution of a Marcus RDE

Original: [Equation]

Original: is obtained by

Original: • ﬁrst solving the continuous RDE d ˜ X

Original: = b ( ˜ X )

Original: d W φ, 1 , with φ = l d being the linear path function on R d ; • then the càdlàg solution path X [0 , 1] → R d is given by X ( t ) =

Original: ˜ X ( τ ( t )) .

Original: Recall that τ : [0 , 1] → [0 , 1 + r ] given by τ ( t ) =

Original: t + (cid:80) k

Original: δr k 1 { t k ≤ t } .

Original: B.3 Example of a Marcus RDE

Original: Here is an example of the solution of a rough differential equation in the Marcus sense [CFKM20].

Original: Let θ > 0 and W ( m ) :

Original: [0 , 1] → R be the deterministic process which are 0 on [0 , 1 / 2] , θ on [1 / 2 + 1 /m, 1] and linear on [1 / 2 , 1 / 2 + 1 /m ] .

Original: Denote by X ( m ) t =

Original: [Equation]

Original: x

Original: [Equation]

Original: m ) t , y ( m ) t ) the solution to the ordinary differential equation

Original: [Equation]

Original: (cid:32)

Original: [Equation]

Original: y ( m ) t x ( m ) t

Original: [Equation]

Original: ( m ) t

Original: [Equation]

Original: x

Original: [Equation]

Original: m )

Original: y

Original: [Equation]

Original: m

Original: [Equation]

Original: (cid:33)

Original: [Equation]

Original: 10 (cid:19)

Original: It is not hard to see that that X ( m ) t

Original: = (cos W ( m ) t , sin W ( m ) t ) .

Original: Therefore, we have that pointwise

Original: W ( m ) t → W t

Original: = θ 1 t ∈ ( 12 , 1] and X ( m ) t → X

Original: [Equation]

Original: = (cid:18) x t

Original: [Equation]

Original: (cid:19)

Original: [Equation]

Original: 10 (cid:19)

Original: [Equation]

Original: t

Original: [Equation]

Original: [Equation]

Original: (cid:19) )

Original: [Equation] t ∈ ( 12 , 1] . If θ (cid:54) = 0 , ( X t ) t ∈ [0 , 1] fails to be the right-hand side of the limiting forward RDE since (cid:90) [Equation]
Revised: Institute), and the European Research Council Starting Grant DYNASTY – 101039676.

Original: s

Original: [Equation]

Original: Effectively, ( X t ) t ∈ [0 , 1] is the solution to the Marcus RDE in the limit.

Original: C Homogenization: The Rigorous Version and the Proof of Theorem 4.

Original: When talking about convergence of solutions to RDEs, we need to specify the sense of integration.

Original: In view of the example in Subsection B.3, it is not enough to look at the solution X as an element of D ([0 , 1] , R d ) – it has to be coupled with jumps of the driving function.

Original: Let us consider the driver-solution space D ([0 , 1] , R r + d ) and introduce a new path function on R r + d .

Original: Deﬁnition C.1.

Original: Consider b ∈ C ( R d , R d × r ) , a ∈ C ( R d , R d ) .

Original: For x ∈ R d and Φ ∈ C 1 -var ([0 , 1] , R r ) , let π a,b

Original: [ x ; Φ] ∈ C 1 -var ([0 , 1] , R d ) denote the solution Π of the equation dΠ = a (Π)d t + b (Π)dΦ , Π(0)

Original: =

Original: x.

Original: For the coefﬁcient b , a pair of admissible points ( w 1 , x 1 ) , ( w 2 , x 2 ) ∈

Original: J a,b ⊂ R r + d is in the space

Original: J a,b

Original: = { (cid:0) ( w 1 , x 1 ) , ( w 2 , x 2 ) (cid:1) : w 1 , w 2 ∈ R d , π b [ x 1 ; l d ( w 1 , w 2 )](1) = x 2 } .

Original: We deﬁne the path function φ a,b on J a,b by φ a,b (( w 1 , x 1 ) , ( w 2 , x 2 ))

Original: = (cid:0) l d ( w 1 , w 2 )( t ) , π b [ x 1 ; l d ( w 1 , w 2 )]( t ) (cid:1) .

Original: Next, we ﬁrst state the precise assumptions for Theorem 4.1 in the main paper.

Original: The required topology for the coefﬁcients a m , b m in (9) is as follows.

Original: For ˜ γ > 0 , n 1 , n 2 ∈ N + , denote by C ˜ γ ( R n 1 , R n 2 ) the space of functions f : R n 1 → R n 2 such that

Original: [Equation]

Original: In particular, if ˜ γ > 1 , we have (cid:107) f (cid:107) ∞ ≤ (cid:107) f (cid:107) C ˜ γ and (cid:107) f (cid:107) Lip ≤ (cid:107) f (cid:107) C ˜ γ .

Original: Assumption 1.

Original: [-Suppose that a ∈ C γ 1 ( R d , R d ) , b ∈ C γ 
-]
Revised: [-Suppose that a ∈ C γ 1 ( R d , R d ) , b ∈ C γ 
-]

Original: ( R d , R d × r ) for some γ 1 > 1 , γ 2 > α .

Original: In addition, we assume that

Original: [Equation]

Original: Assumption 1 requires that the difference between the gradients of coefﬁcients (cid:107)∇ ( a m − a ) (cid:107) ∞ goes to zero and also their Hölder constant |∇

Original: [Equation]

Original: y

Original: [Equation]

Original: goes to zero uniformly on x, y ∈ R m for some (cid:15) > 0 .

Original: This translates to the assumptions that the loss is second differentiable and its Hessian is (cid:15) -Hölder, which are reasonable assumptions.

Original: Similarly, for the coefﬁcients appearing before the α -stable process (for α ∈ (1 , 2) ), it is required that (cid:107)∇ ( b m − b ) (cid:107) ∞ goes to zero and sup x,y ∈ R

Original: [Table]

Original: goes to zero for some (cid:15) (cid:48) >

Original: α − 1 .

Original: In the setup for MPGD (8) where a m = − ˆ R ( x, S n ) = a (independent of m ) and the b m is − µ diag(x) or σI , it is easy to see that these assumptions are satisﬁed.

Original: In terms of how general the assumption is, we note that Assumption 1 can cover more general situations; e.g., the case where is an empirical loss (taking m = n , the number of training samples) and a is the population loss, and therefore our framework can be adapted for analysis in other settings as well.

Original: Theorem 4.1 is an extension of Theorem 4.1 in [GM21] (with the (cid:15) there equal 1 /m ) to the case where the coefﬁcients a , b are dependent on the scaling parameter m .

Original: In particular, the proof of

Original: Theorem 4.1 in [GM21] relies on verifying the hypotheses of Theorem 2.6 in [CFKM20] for the particular example of the observable and Thaler map constructed for the MPGD (8).

Original: We now prove a more general theorem that extends Theorem 2.6 in [CFKM20] and includes Theorem 4.1 as a special case, stating all the needed assumptions.

Original: Applying this theorem together with the hypotheses veriﬁcation in the proof of Theorem 4.1 in [GM21] completes the proof of Theorem 4.1.

Original: Theorem C.1 (A more general version of Theorem 4.1) .

Original: Let α ∈ (1 , 2) , γ 1 > 1 and γ 2 > α .

Original: Let v ∈ L ∞ ( Y, R d ) be Hölder such that (cid:82) v d µ = 0 , where µ is the unique ergodic invariant probability by ∂ j =

Original: ( ∂/∂ j 1 x 1 ) · · · ( ∂/∂ j n x n ) .

Original: measure of T : Y → Y .

Original: We focus on the case where T exhibits superdiffusive behavior, which means that there exists a r -dimensional Lévy process L and

Original: W n ( t ) = n − 1 /α (cid:98) nt (cid:99)− 1 (cid:88)

Original: j = v ◦ T j (d) → L t in D ([0 , 1] , R r ) under the SM 1 topology (Deﬁnition B.2) as n → ∞ .

Original: If, in addition,

Original: • lim n →∞ a n = a in C γ 1 ( R d , R d ) for some a ∈ C γ 1 ( R d , R d ) , • lim n →∞ b n = b in C γ 2 ( R d , R d × r ) for some b ∈ C γ 2 ( R d , R d × r ) ; • lim n →∞ x n = x , • (cid:107) W n (cid:107) p -var is tight for all p > α and (cid:80) t | W n ( t ) − W n ( t − ) | 2 → 0 a.s., where the sum is taken over all jump times of W n ,

Original: then for the forward RDE d X n = a n ( X n ) − d V n + b n ( X n ) − d W n , X n (0) =

Original: x n where V n ( t ) = (cid:98) tn (cid:99) n , we have:

Original: [Equation]

Original: [Equation]

Original: ) as n → ∞ for all p > α , where ( X ( t ))

Original: t ≥ 0 is the solution of the Marcus differential equation

Original: [Equation]

Original: X

Original: [Equation]

Original: Remark C.1.

Original: With the choices of v and T in Section 3, the assumptions in Theorem C.1 are satisﬁed (see the proof in [GM21]).

Original: Before proving Theorem C.1, we start by proving the following lemma.

Original: Lemma C.1.

Original: Let X ∈ D

Original: ([0 , 1] , R d ) , p ≥ 1 such that (cid:107) X (cid:107) p -var < ∞ .

Original: Then for any path function φ ,

Original: [Equation]

Original: Proof.

Original: It is obvious from the scaling invariance of of the p -variation that (cid:107) X φ,δ (cid:107) p -var =

Original: (cid:107)

Original: X φ (cid:107) p -var .

Original: For the second inequality

Original: [Equation]

Original: using again the deﬁnition,

Original: [Equation]

Original: where the last equality follows from the fact that if c lies in an interval [ a, b ] then ( c − a ) p +( b − c ) p ≤ ( b − a ) p for p ≥ 1 .

Original: This completes the proof.

Original: As a ﬁrst step for proving Theorem C.1, we aim to prove a deterministic variant of Theorem

Original: C.1.

Original: In order to make the arguments more concise, let us neglect the ﬁrst time-derivative term (with coefﬁcients a and a n , for which the treatment is the same by considering the jump process

Original: V n : t (cid:55)→ (cid:98) nt (cid:99) /n ) and consider the following.

Original: Theorem C.2.

Original: Assume that p ∈ (1 , 2) and { W n } n ≥ 1 is a sequence in D p -var ([0 , 1] , R r ) with ﬁnitely many jumps.

Original: Suppose that γ > p and

Original: • lim n →∞ b n = b in C γ

Original: ( R d , R d × r ) for some b ∈ C γ ( R d , R d × r ) ; • d SM 1 ( W n , W ) → 0 and (cid:80) | W n ( t ) − W n ( t − ) | 2 → 0 , where the sum is taken over all jump times of W n .

Original: Let X n be the solution of the forward RDE d X n = b n ( X n ) − d W n with X (0) =

Original: x n .

Original: Let X be the solution of the Marcus RDE d X = b ( X ) (cid:5) d W, X (0) =

Original: x .

Original: Then it holds that (( W n , X n ) , l r + d ) → (( W, X ) , φ b ) in ( D ([0 , 1] , R r + d ) , α p -var ) as n → ∞ .

Original: C.1 Proof of Theorem C.

Original: We will use the following lemma in the proof of Theorem C.2.

Original: Lemma C.2 ([CFKM20] Lemma 3.6) .

Original: Suppose that W ∈ D ([0 , 1] , R r ) has ﬁnitely many jumps.

Original: Let

Original: X, ˜ X ∈ D ([0 , 1] , R d ) be given by

Original: [Equation]

Original: Then

Original: [Equation]

Original: where K depends on (cid:107) b (cid:107) C γ , (cid:107) W (cid:107) p -var , γ, p , and the sum is over all jump times t of W .

Original: We are now ready to prove Theorem C.2.

Original: Proof of Theorem C.2.

Original: Let ˜ X

Original: n be the solution to the Marcus RDE

Original: [Equation]

Original: The continuity of the solution map for generalised geometric RDEs gives that

Original: [Equation]

Original: Then let X n be the solution to the Marcus RDE

Original: [Equation]

Original: On any subinterval [ s

Original: [Equation]

Original: s 2 ]

Original: [Equation]

Original: we can compare solutions to the Marcus RDE with b n and b by using

Original: Proposition B.1:

Original: [Equation]

Original: where the last inequality is due to

Original: Lemma C

Original: [Equation]

Original: [-If we choose a subdivision 
-]
Revised: [-If we choose a subdivision 
-]

Original: = s 0 < s 1 < . . .

Original: <

Original: s n − 1 < s n = 1 of the interval [0 , t ] such that for any i = 0 , 1 , . . . , n − 1 ,

Original: [Equation]

Original: 1 , then on the interval [0 , s 1 ] ,

Original: [Equation]

Original: Similarly on the interval [ s 1 , s 2 ] ,

Original: [Equation]

Original: It is not hard to obtain by induction that there exists C ( b

Original: [Equation]

Original: [-W ) > 
-]
Revised: [-W ) > 
-]

Original: such that

Original: [Equation]

Original: Therefore

Original: [Equation]

Original: Since b n → b in C γ

Original: ( R d , R d × r ) , (cid:107) b n (cid:107) Lip and (cid:107) b n (cid:107) ∞ are uniformly bounded independent of n .

Original: Then it follows from Lemma C.2 that

Original: [Equation]

Original: Then the same argument in [CFKM20] gives that for any p (cid:48) > p ,

Original: [Equation]

Original: Recall also from the [CFKM20] that

Original: [Equation]

Original: Finally Theorem C.2 follows from combining (17), (18), (19) and (20).

Original: This completes the proof.

Original: C.2 Proof of Theorem C.

Original: Proof of Theorem C.1.

Original: If we write V n ( t ) = (cid:98) tn (cid:99) /n , the assumptions of Theorem C.1 and the same treatment in [CFKM20] implies that up to subtracting a subsequence, ( V n , W n ) → (id , L) a.s. in α p -var for any p > α .

Original: Then it follows from Theorem C.2 that for p (cid:48) >

Original: p and along each subsequencial limit of ( V n , W n ) as n → ∞ ,

Original: [Equation]

Original: D

Original: [Equation]

Original: α p (cid:48) -var ) .

Original: Therefore (( W n , X n ) , l r + d ) (d) → ((L , X ) , φ a,b ) in ( D ([0 , 1] , R r + d ) , α p -var ) for any p > α .

Original: This completes the proof.

Original: D Generalization Bound: Proof of Theorem 4.

Original: To begin with, we need a geometric regularity assumption over the trajectory of the multiscale perturbed gradient ﬂow, which is common for random fractal processes given as solutions to stochastic differential equations; see [H¸SKM21].

Original: This assumption ensures that the box-counting (Minkowski) dimension coincides with the Hausdorff dimension of the trajectory.

Original: For any x ∈ R d , let ( X t ) t ∈ [0 , 1] be a solution to the stochastic differential equation (12) started from x , and let A be the inﬁnitesimal generator of ( X t ) t ∈ [0 , 1] (let us take it for granted that it exists) deﬁned by

Original: [Equation]

Original: any u ∈

Original: [Equation]

Original: c

Original: [Equation]

Original: R

Original: [Equation]

Original: [Equation]

Original: Let q : R m × R m → C be the symbol of A such that A u ( x ) = − (cid:82) R m e i (cid:104) ξ,x (cid:105) q ( x, ξ )ˆ u ( ξ )d ξ , (informally saying, − q ( x, D ) u ( x ) ), where ˆ u ( ξ )

Original: = (cid:82) R m e − i (cid:104) ξ,x (cid:105) u

Original: ( x )d x is the Fourier transform of u .

Original: Assumption 2.

Original: For almost every X , there exists a ﬁnite Borel measure µ on X and ρ > 0 such that

Original: C ρ := inf 0 <r<ρ,x ∈X µ ( B r ( x ))

Original: r α µ ( X ) > 0 for µ -almost every x .

Original: If ( X t ) t ≥ 0 is a solution with X 0 = 0 to d X t = b (cid:5) dL

Original: αt for ﬁxed b ∈ R d × d and a symmetric ddimensional Lévy process L α , then ( X t ) t ≥ 0 is also a α -stable Lévy process with symbol independent of x :

Original: [Equation]

Original: We also need the following statistical regularity assumption, as discussed in the paragraph before

Original: Theorem 4.2.

Original: Assumption 3.

Original: There exists x (cid:63) ∈ R d such that a ( x ∗ ) = 0 .

Original: Let ( X ∗ t ) t ≥ 0 to the solution of d X t = b ( x ∗ ) (cid:5) dL

Original: αt and write ˜ q ( x, ξ ) :

Original: = q ( x, ξ ) − ψ X ∗ ( ξ ) .

Original: For almost-every S n :

Original: • b ( x (cid:63) ) is positive deﬁnite; • | ∂ j x ˜ q ( x, ξ ) | ≤ Φ j ( x )(1 + κ 0 | ξ | α ) with j ∈ N m 0 , | j | ≤ m + 1 for some Φ j ∈ L 1 ( R m ); • q ( x, 0)

Original: [-= 
-]
Revised: [-= 
-]

Original: and (cid:107) Φ 0 (cid:107) ∞ < ∞ .

Original: Some remarks on Assumption 3 are now in place.

Original: [-It is natural to assume the existence of x ∗ such that a ( x ∗ ) = 
-]
Revised: [-It is natural to assume the existence of x ∗ such that a ( x ∗ ) = 
-]

Original: (since a = −∇ ˆ R ( x, S n ) in our MPGD setup).

Original: For the ﬁrst point, the positive deﬁniteness of b ( x ∗ ) can be satisﬁed by simply choosing b to be the identity map.

Original: For the second point, recall that ˜ q ( x, ξ ) =

Original: q ( x, ξ ) − ψ x ∗ ( ξ ) and we require that: | ∂ j x ˜ q ( x, ξ ) | ≤ Φ j

Original: ( x )(1 + κ 0 | ξ | α ) with j ∈ N m 0 , | j | ≤ m + 1 for some Φ j ∈ L 1 ( R m ) .

Original: Here X ∗ is an α -stable Lévy process, and its characteristic function ψ x ∗ ( ξ ) is given by | ξ T b ( x ∗ ) | α (see Eq.

Original: Moreover, q ( x, ξ ) is nothing but (4) with the b, Σ , µ depending on x .

Original: It is not hard to see that the above assumption holds if the b, Σ , µ depend smoothly on x .

Original: The third point is equivalent to saying that the solution to the SDE exists almost surely on inﬁnite time interval (see [Sch98b]), which is the case for the perturbed gradient ﬂow with respect to second differentiable loss.

Original: With all these ingredients in place, we now prove Theorem 4.2.

Original: Proof.

Original: We observe that the ellipticity condition holds since b ( x (cid:63) ) is positive deﬁnite.:

Original: [Equation]

Original: One can also check that α := inf { λ ≥ 0 : lim | ξ |→∞ | ψ X ∗ ( ξ ) | | ξ | λ = 0 } .

Original: Under Assumption 3, it follows from [Sch98a, Theorem 4] that dim H ( X ) ≤ α almost surely, where dim H denotes the Hausdorff dimension.

Original: Then, under Assumption 2, the result follows from [H¸SKM21, Theorem 1] and [H¸SKM21, Corollary 1].

Original: This completes the proof.

Original: E Implicit Regularization: Proof of Theorem 4.

Original: In this section, we provide proof to Theorem E.

Original: [-Let r 1 = 1 , r 
-]
Revised: [-Let r 1 = 1 , r 
-]

Original: =

Original: d as assumed in Theorem 4.3.

Original: Upon the rescalings µ = µ 0 (cid:15) and σ = σ 0 (cid:15) with (cid:15) > 0 a small parameter, the MPGD recursion Eq.

Original: (8) becomes:

Original: [Equation]

Original: 21) for k = 0 , 1 , 2 , . .

Original: where

Original: [Equation]

Original: with the y ( i ) satisfying y ( i ) k +1 =

Original: f ( i ) (

Original: y ( i ) k ) for i = 1 , 2 .

Original: Consider the following hierarchy of recursive equations.

Original: For k = 0 , 1 , 2 , . . .

Original: [Equation]

Original: [Equation]

Original: with the initial conditions x ( m )

Original: [Equation]

Original: Informally, the processes x ( m ) , φ ( m ) and ϕ ( m ) constitute the zeroth-, ﬁrst- and second-order terms in a pathwise Taylor expansion of x ( m ) about (cid:15) = 0 , i.e., for k = 0 , 1 , 2 , . . .

Original: , x ( m ) k

Original: [Equation]

Original: m

Original: [Equation]

Original: + (cid:15)φ ( m ) k

Original: [Equation]

Original: (cid:15) 2 ϕ ( m ) k

Original: + O ( (cid:15) 3 ) , as (cid:15)

Original: → 0 .

Original: Now we are going to compute a second-order Taylor expansion of ˆ R ( x ( m ) k ) about (cid:15) = 0 .

Original: This is the content of the following lemma, which is adapted from Theorem 9 in [LEHM21] to our setting.

Original: Lemma E.1.

Original: Under the assumptions of Theorem 4.3, we have:

Original: [Equation]

Original: as (cid:15) → 0 , where the ( x ( m ) k , φ ( m ) , ϕ ( m ) ) satisfy Eq.

Original: We now prove Theorem 4.3.

Original: Proof of Theorem 4.3.

Original: Let us ﬁx the m and S n .

Original: To prove Theorem 4.3, we need to compute

Original: E ˆ R ( x ( m ) k ) , where the expectation is with respect to the randomness in the y ( i ) 0 .

Original: Since, E [ v 1 ( y (1) k )] =

Original: E [ v 2 ( y (2) k )] = 0 for all k by assumption, we have E φ ( m ) k

Original: = 0 for all k , and applying Lemma E.1:

Original: [Equation]

Original: It remains to compute the E ϕ ( m ) k and E (cid:104)

Original: [Equation]

Original: φ (

Original: [Equation]

Original: ) k )

Original: [Equation]

Original: ∇ 2 ˆ R ( x ( m

Original: [Equation]

Original: k ) φ ( m ) k (cid:105) in the above expansion.

Original: Iterating Eq.

Original: 25, we obtain

Original: [Equation]

Original: for k = 1 , 2 , . . . .

Original: Similarly, iterating Eq.

Original: [Equation]

Original: we obtain:

Original: [Equation]

Original: Now

Original: [Equation]

Original: we compute: ∂g

Original: [Equation]

Original: for all l , and

Original: [Equation]

Original: Substituting the above formula into Eq.

Original: [-(31) and then taking expectation, we obtain, using the fact that E v 1 ( y (1) i − 1 ) = 
-]
Revised: [-(31) and then taking expectation, we obtain, using the fact that E v 1 ( y (1) i − 1 ) = 
-]

Original: for all i :

Original: [Equation]

Original: with the l th component of:

Original: [Equation]

Original: Now

Original: [Equation]

Original: for k =

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: one can compute, using the assumption that y (1)0 and y (2)0 are independent (and thus E [ v 1 ( y (1) i 1 − 1 )[ v 2 ] r ( y (2)

Original: i 2 − 1 )]

Original: = 0 for all r, i 1 , i 2 ),

Original: [Equation]

Original: where the Φ ( m ) i :=

Original: [Equation]

Original: ) , with the empty product taken to be the identity by convention.

Original: The l th component of λ ( m ) k can thus be written as:

Original: [Equation]

Original: where the C ( m ) i − 1 are covariance matrices with the ( p, q ) -entry of E [[ φ ( m ) i − 1 ] p [ φ ( m ) i − 1 ] q ] whose expression is given in Eq.

Original: Lastly, we compute:

Original: [Equation]

Original: Substituting (36) and (39) into Eq.

Original: (29), we arrive at Eq.

Original: (16) in Theorem 4.3.

Original: F Additional Empirical Results and Details

Original: In this section, we provide additional empirical results, as well as the details and additional results for the experiments considered in the main paper, to strengthen the support for our theory.

Original: F.1 Electrocardiogram (ECG) Classiﬁcation

Original: We consider the Electrocardiogram (ECG) binary classiﬁcation task that aims to discriminate between normal and abnormal heart beats of a patient that has severe congestive heart failure [GAG + 00].

Original: We use 500 sequences of length 140 for training, and 4000 sequences for testing.

Original: We use a fully connected shallow neural network of width 32 with sigmoid activation and train for 1000 epochs, with the binary cross-entropy as the loss.

Original: We choose η = 0 .

Original: For the MPGD and the Gaussian

Original: [Table]

Original: scheme, we inject the perturbations in the ﬁrst 500 epochs instead

Original: [Table]

Original: and stop injecting after that to allow the algorithm to converge.

Original: We use the perturbation level µ = σ = 0 .

Original: The experimental setup is implemented in PyTorch, and all experiments are run on Google Colab.

Original: Table 3 shows the average test accuracy (evaluated for 5 models that are trained with different seed values) for this task.

Original: We see that overall MPGD improves the test accuracy when compared to the vanilla GD and Gaussian perturbations.

Original: In fact, the vanilla GD gets stuck in the loss landscape.

Original: While

Original: Gaussian perturbations can help to mobilize the GD iterates in the landscape, MPGD is more effective in steering the iterates to achieve higher test accuracy (see Figure 3), which is the main goal of the learning task.

Original: This illustrates that adding the perturbations of MPGD can help improve the outcome of the optimization process in situations where the vanilla GD and Gaussian perturbations fail to make meaningful progress.

Original: Note that the higher values of the perturbation levels used in the MPGD schemes lead to instabilities and ﬂuctuations of large magnitudes in the earlier epochs.

Original: These are necessary to boost the test accuracy; see the jump in the improvement of the test accuracy after epoch 500 in Figure 3.

Original: The instabilities and ﬂuctuations could be reduced if lower perturbation levels were used, but this would lower the test accuracy (which would still be higher than the baseline and the Gaussian results) obtained.

Original: Again, we emphasize that we are not going after competitive test performance here, but rather demonstrating the effectiveness of MPGD in improving the test performance in situations when other training schemes fail to make substantial progress in optimization.

Original: F.2 Details and Additional Results for the Airfoil Self-Noise Prediction Task

Original: The experimental setup is implemented in PyTorch, and all experiments are run on 4x NVIDIA

Original: Tesla T4 GPUs with 16 GB VRAM belonging to an internal SLURM cluster.

Original: Table 4 reports the

Original: [Table]

Original: [Table]

Original: [Table]

Original: sample standard deviation for the results obtained in Table 1.

Original: Table 5 shows the corresponding results when different values of β are used instead, illustrating the trade-offs induced by the selection of the stability parameter γ and the skewness parameter β for this particular setting.

Original: Here, we see that using the MPGD variant (11) with γ = 0 .

Original: 6 and β = 0 .

Original: 5 (see Table 1) leads to the lowest RMSE gap.

Original: F.3 Details for the CIFAR-10 Classiﬁcation Task

Original: We use the implementation of ResNet18 and modify the implementation of the full-batch GD training provided at https://github.com/JonasGeiping/fullbatchtraining to set up MPGD and

Original: GD with Gaussian perturbations.

Original: Please refer to Section 5 and [GGP + 21] for the relevant details.

Original: The experimental setup is in PyTorch, and all experiments are run on an NVIDIA A100-SXM4 GPU with 40 GB VRAM belonging to an internal SLURM cluster.

Original: Table 2 reports the mean validation accuracies over runs of 5 models trained with different seed values, whereas Table 6 reports the sample standard deviation for the results in Table 2.