E LIMINATING
S HARP
M INIMA
FROM
SGD
WITH
T RUNCATED H EAVY - TAILED N OISE
Anonymous authors
Paper under double-blind review
A BSTRACT
The empirical success of deep learning is often attributed to SGD’s mysterious ability to avoid sharp local minima in the loss landscape, as sharp minima are known to lead to poor generalization.
Recently, evidence of heavy-tailed gradient noise was reported in many deep learning tasks, and it was shown in (¸Sim¸sekli et al., 2019a;b) that SGD can escape sharp local minima under the presence of such heavy-tailed gradient noise, providing a partial solution to the mystery.
In this work, we analyze a popular variant of SGD where gradients are truncated above a ﬁxed threshold.
We show that it achieves a stronger notion of avoiding sharp minima: it can effectively eliminate sharp local minima entirely from its training trajectory.
We characterize the dynamics of truncated SGD driven by heavy-tailed noises.
First, we show that the truncation threshold and width of the attraction ﬁeld dictate the order of the ﬁrst exit time from the associated local minimum.
Moreover, when the objective function satisﬁes appropriate structural conditions, we prove that as the learning rate decreases the dynamics of the heavy-tailed truncated SGD closely resemble those of a continuous-time Markov chain that never visits any sharp minima.
Real data experiments on deep learning conﬁrm our theoretical prediction that heavy-tailed SGD with gradient clipping ﬁnds a "ﬂatter" local minima and achieves better generalization.
I NTRODUCTION
Stochastic gradient descent (SGD) and its variants have seen unprecedented empirical successes in training deep neural networks.
The training of deep neural networks is typically posed as a non-convex optimization problem, and even without explicit regularization the solutions obtained by SGD often perform surprisingly well on test data.
Such an unexpected generalization performance of SGD in deep neural networks are often attributed to SGD’s ability to avoid sharp local minima 1 in the loss landscape, which tends to lead to poor generalization (Hochreiter & Schmidhuber, 1997; Keskar et al., 2016;
Li et al., 2018b; Jiang et al., 2019); see Appendix C.2 for more details.
Despite signiﬁcant efforts to explain such phenomena theoretically, understanding how SGD manages to avoid sharp local minima and end up with ﬂat local minima within a realistic training time still remains as a central mystery of deep learning.
Recently, the heavy-tailed dynamics of SGD received signiﬁcant attention, and it was suggested that the heavy tails in the stochastic gradients may be a key ingredient that facilitates SGD’s escape from sharp local minima: for example, ¸Sim¸sekli et al.
(2019a;b) report the empirical evidence of heavy-tails in stochastic gradient noise in popular deep learning architectures (see also (Hodgkinson & Mahoney, 2020;
Srinivasan et al., 2021;
Garg et al., 2021)) and show that
SGD can escape sharp local minima in polynomial time under the presence of the heavy-tailed gradient noise.
More speciﬁcally, they view heavy-tailed SGDs as discrete approximations of Lévy driven Langevin equations and argue that the amount of time SGD trajectory spends in each local minimum is proportional to the width of the associated minimum according to the metastability theory (Pavlyukevich, 2007;
Imkeller et al., 2010a;b) for such heavy-tailed processes.
In this paper, we study the global dynamics and long-run behavior of heavy-tailed SGD and its practical variant in depth.
In particular, we consider an adaptive version of SGD, where the stochastic gradient is truncated above a ﬁxed threshold.
Such truncation scheme is often called gradient clipping and employed as default in various contexts (Engstrom et al., 2020; Merity et al., 2018; Graves, 2013;
[Figure]
Pascanu et al., 2013; Zhang et al., 2020; Gorbunov et al., 2020).
We uncover a rich mathematical structure in the global dynamics of SGD under this scheme and prove that the long-run behavior of such SGD is fundamentally different from that of the pure form of SGD: in particular, under a suitable structural condition on the geometry of the loss landscape, gradient clipping completely eliminates sharp minima from the trajectory of SGDs .
This provides a critical insight into how heavy-tailed dynamics of SGD can be utilized to ﬁnd a local minimum that generalizes better.
Figure 1 (Left, Middle) clearly illustrates these points with the histograms of the sample trajectories of
SGDs.
Note ﬁrst that SGDs with light-tailed gradient noise—(c) and (d) of Figure 1 (Left, Middle)— never manages to escape a (sharp) minimum regardless of gradient clipping.
In contrast, SGDs with heavy-tailed gradient noise—(a) and (b) of Figure 1 (Left, Middle)—easily escapes from local minima.
Moreover, there is a clear difference between SGDs with gradient clipping and without gradient clipping.
In (a) of Figure 1 (Left), SGD without gradient clipping spends a signiﬁcant amount of time at each of all four local minima ( { m 1 , m 2 , m 3 , m 4 } ), although it spends more time around the wide ones ( { m 2 , m 4 } ) than the sharp ones ( { m 1 , m 3 } ).
On the other hand, in (b) of
Figure 1 (Left), SGD with gradient clipping not only escapes from local minima but also avoids sharp minima ( { m 1 , m 3 } ) almost completely.
This means that if we stop training SGD at an arbitrary time point, it is almost guaranteed that it won’t be at a sharp minimum, effectively eliminating sharp minima from its training trajectories.
We also propose a novel computational strategy that takes advantage of our newly discovered global dynamics of the heavy-tailed SGD.
While the evidence of heavy tails were reported in many deep learning tasks (¸Sim¸sekli et al., 2019b;a; Garg et al., 2021;
Gurbuzbalaban et al., 2020; Hodgkinson & Mahoney, 2020; Nguyen et al., 2019; Mahoney & Martin, 2019; Srinivasan et al., 2021; Zhang et al., 2020), there seem to be plenty of deep learning contexts where the stochastic gradient noises are light-tailed (Panigrahi et al., 2019) as well.
Guided by our new theory, we propose an algorithm that injects heavy-tails to SGD by inﬂating the tail distribution of the gradient noise and facilitating the discovery of a local minimum that generalizes better.
Our experiments with image classiﬁcation tasks, reported in Tables 1 and 2, illustrate that the tail-inﬂation strategy we propose here can indeed improve the generalization performance of the SGD as predicted by our theory.
The rest of the paper is organized as follows.
Section 2 formulates the problem setting and characterizes the global dynamics of the SGD driven by heavy-tailed noises.
Section 3 presents numerical experiments that conﬁrm our theory.
Section 4 proposes a new algorithm that artiﬁcially injects heavy tailed gradient noise in actual deep learning tasks and demonstrate the improved performance.
Technical Contributions: 1) We rigorously characterize the global behavior of the heavy-tailed
SGD with gradient clipping.
We ﬁrst focus on the case where the loss function is in R 1 with some simplifying assumptions on its geometry.
Even with such assumptions, our theorem involves substantial technical challenges since the traditional tools for analyzing SGD fail in our context due to the adaptive nature of its dynamics and non-Gaussian distributional assumptions.
For example, while the unclipped pure SGD can be analyzed by partitioning its trajectory at arrival times of large noises (as in Pavlyukevich (2005) and Imkeller et al.
(2010a))
, such an approach falls short in our context.
Instead, we developed a set of delicate arguments for dealing with SGD’s (near) regeneration structure and the return times to the local minima, as well as controlling the probability of atypical scenarios that would not arise in the unclipped case.
Moreover, as evidenced by our R d results in Appendix G, the approach developed here is critical in extending the analysis to general loss landscapes.
We propose a novel computational strategy for improving the generalization performance of SGD by carefully injecting heavy-tailed noise.
We test the proposed algorithm with deep learning tasks and demonstrate its superiority with an ablation study.
This also suggests that the key phenomenon we characterize in our theory— elimination of sharp local minima—manifests in real-world tasks.
T HEORETICAL RESULTS
This section characterizes the global dynamics of SGD with gradient clipping when applied to a non-convex objective function f .
In Section 2.1 and 2.2, we make the following assumptions for the sake of the simplicity of analysis.
However, as our multidimensional result in Section 2.3 and the experiments in Section 3 and 4 suggest, the gist of the phenomena we analyze—elimination of sharp local minima—persists in general contexts where the domain of f is multi-dimensional, and the stationary points are not necessarily strict local optima separated from one another.
Assumption 1.
Let f : R → R be a C 2 function.
There exist a positive real L > 0 , a positive integer n min and an ordered sequence of real numbers m 1 , s 1 , m 2 , s 2 , · · · , s n min − 1 , m n min such that (1) − L < m 1 < s 1 < m 2 < s 2 < · · · < s n min − 1 < m n min <
L ; (2) f (cid:48)
( x ) =
iff x ∈ { m 1 , s 1 , · · · , s n min − 1 , m n min } ; (3) For any x ∈ { m 1 , m 2 , · · · , m n min } , f (cid:48)(cid:48)
( x ) > 0 ; (4) For any x ∈ { s 1 , s 2 , · · · , s n min − 1 } , f (cid:48)(cid:48)
( x ) < 0 .
As illustrated in Figure 2 (Left), the assumption above requires that f has ﬁnitely many local minima (to be speciﬁc, the count is n min ), all of which contained in some compact domain [ − L, L ] .
Moreover, the points s 1 , · · · , s n min − 1 naturally partition the entire real line into different regions Ω i =
( s i − 1 , s i ) (here we adopt the convention that s
=
−∞ , s n min = + ∞ ).
We call each region Ω i the attraction ﬁeld of the local minimum m i , as the gradient ﬂow in Ω i always points to m i .
Throughout the optimization procedure, given any location x ∈ R we assume that we have access to the noisy estimator f (cid:48)
( x ) − Z n of the true gradient f (cid:48) ( x ) , and f (cid:48) ( x ) itself is difﬁcult to evaluate.
Speciﬁcally, in this work we are interested in the case where the iid sequence of noises ( Z n ) n ≥ 1 are heavy-tailed.
Typically, the heavy-tailed phenomena are captured by the concept of regular variation: for a measurable function φ : R + (cid:55)→ R + , we say that φ is regularly varying at + ∞ with index β (denoted as φ ∈ RV β ) if lim x →∞ φ ( tx ) /φ
( x ) =
t β for all t > 0 .
For details on the deﬁnition and properties of regularly varying functions, see, for example, chapter 2 of Resnick (2007).
In this paper, we work with the following distributional assumption on the gradient noise.
Let
[Equation]
Assumption 2.
E Z 1 = 0 .
Furthermore, there exists some α ∈ (1 , ∞ ) such that function H ( x ) is regularly varying (at + ∞ ) with index − α .
Besides, regarding the positive and negative tail for distribution of the noises, we have
[Equation]
where p + and p − are constants in interval (0 , 1) .
Roughly speaking, Assumption 2 means that the shape of the tail for the distribution of noises Z n resembles a polynomial function x − α , which is much heavier than the exponential tail of Gaussian distributions.
Therefore, large values of Z n are much more likely to be observed under Assumptioncompared to the typical Gaussian assumption.
The index α of regular variation encodes the heaviness of the tail—the smaller the heavier—and we are assuming that the left and right tails share the same index α .
The purpose of this simplifying assumption is clarity of presentation, but our R d results in
Appendix G relax such a condition and allow different regular variation indices in different directions.
Our work concerns a popular variant of SGD where the stochastic gradient is truncated.
Speciﬁcally, when updating the SGD iterates with a learning rate η > 0 , rather than using the original noisy gradient descent step η ( f (cid:48)
( X n ) − Z n ) , we will truncate it at a threshold b > 0 and use ϕ b (cid:0) η ( f (cid:48)
( X n ) − Z n ) (cid:1) instead.
Here the truncation operator ϕ · ( · ) is deﬁned as
[Equation]
Besides truncating the stochastic gradient, we also project the SGD into [ − L, L ] at each iteration; recall that L is the constant in Assumption 1.
That is, the main object of our study is the stochastic process { X ηj } j ≥ 0 driven by the following recursion
[Equation]
The projection ϕ L and truncation ϕ b here are common practices in many learning tasks for the purpose of ensuring that the SGD does not explode and drift to inﬁnity.
Besides, the projection also allows us to drop the sophisticated assumptions on the tail behaviors of f that are commonly seen in previous works (see, for instance, the dissipativity conditions in Nguyen et al.
For technical reasons, we make the following assumption about the truncation threshold b > 0 .
Note that this assumption is a very mild one, as it is obviously satisﬁed by (Lebesgue) almost every b > 0 .
Assumption 3.
For each i = 1 , 2 , · · · , n min , min {| s i − m i | , | s i − 1 − m i |} /b is not an integer.
2. F IRST EXIT TIMES
Denote the SGD’s ﬁrst exit time from the attraction ﬁeld Ω i with σ i ( η ) =
∆ min { n ≥ 0 : X ηn / ∈ Ω i } .
In this section, we prove that σ i ( η, x ) converges to an exponential distribution when scaled properly.
To characterize such a scaling, we ﬁrst introduce a few concepts.
For each attraction ﬁeld Ω i , deﬁne (note that (cid:100) x (cid:101) =
min { n ∈ Z : n ≥ x } , (cid:98) x (cid:99) =
max { n
[Equation]
x
[Equation] [Equation]
Note that l ∗ i ’s in fact depend on the the value of gradient clipping threshold b even though this dependency is not highlighted by the notation.
Here r i can be interpreted as the radius or the effective width of the attraction ﬁeld, and l ∗ i is the minimum number of jumps required to escape Ω i when starting from m i .
Indeed, the gradient clipping threshold b dictates that no single SGD step can travel more than b , and to exit Ω i when starting from m i we can see that at least (cid:100)
r
i /b (cid:101) steps are required.
We can interpret l ∗ i as the minimum effort required to exit Ω i .
In this sense, l ∗ i is an indicator of the width of the attraction ﬁeld Ω i .
Theorem 1 states that l ∗ i dictates the order of magnitude of the ﬁrst exit time as well as where the iterates X ηn land on at the ﬁrst exit time.
For each Ω i , deﬁne a scaling function λ
i ( η ) =
∆ H (1 /η )
((1 /η ) H (1 /η ))
l ∗ i − 1 .
To stress the initial condition, we write P x for the probability law when conditioning on X η 0 = x , or simply write X ηn ( x ) .
Theorem 1.
Under Assumptions 1-3, there exist constants q i > 0 ∀ i ∈ { 1 , 2 , · · · , n min } and q i,j ≥ 0 ∀ j ∈ { 1 , 2 , · · · , n min } \ { i } such that
(i) Suppose that x ∈ Ω k for some k ∈ { 1 , 2 , · · · , n min } .
Under P x , the scaled ﬁrst exit time q k λ k ( η ) σ k ( η ) converges in distribution to Exp (1) as η ↓ 0 .
(ii) For k, l ∈ { 1 , 2 , · · · , n min } such that k (cid:54) =
l , we have lim η
→ 0 P x ( X η σ k ( η ) ∈ Ω l ) =
q k,l /q k .
The proof and discussion are deferred to Appendix E.
The constants q i , q i,j are explicitly identiﬁed in terms of the gradient ﬂows perturbed by Pareto jumps in Section C of Appendix.
We note here that
Theorem 1 implies (i) for X ηn to escape the current attraction ﬁeld, say Ω i , it takes O (cid:0) 1 /λ i ( η ) (cid:1) time, and (ii) the destination is most likely to be reachable within l ∗ i jumps from m i .
ss 2 − L
L mmm 3 0.2 ← 0.6 → ← 0.9 → 0.
ΩΩΩmmml 2,1 =
l 1,2 =
l 3,2 =
l 2,3 = b=0.
mmml 2,1 =
l 1,2 =
l 3,2 =
l 2,3 =3 × b=0.
E LIMINATION OF S MALL A TTRACTION F IELDS
Under proper structural assumptions on the geometry of f , the sharp minima of f can be effectively eliminated from the trajectory of heavy-tailed SGD, facilitating the discovery of ﬂat minima.
This is somewhat surprising given that gradient clipping mechanism makes the SGD iterates move slower .
The intuition behind this is that for narrow basins, applying gradient clipping has virtually no effect on the order of exit time; whereas for a wide basin that requires multiple jumps to escape under the clipping scheme, the clipping of gradients signiﬁcantly slows down the escape and makes SGD stay longer.
In other words, gradient clipping only makes SGDs stay longer in the wider (better) basins.
Now, we introduce a few new concepts.
Similar to the the minimum number of jumps l ∗ i deﬁned in (3), we deﬁne the following as the minimum number of jumps to reach Ω j from m i for any j (cid:54) =
i :
[Equation]
Recall that Theorem 1 dictates that X ηn is most likely to move out of the current attraction ﬁeld, say Ω i , to somewhere else after O (cid:0) 1 /λ i ( η ) (cid:1) time steps, and the destination is most likely to be reachable within l ∗ i jumps from m i .
Therefore, the transitions from Ω i to Ω j can be considered typical if Ω j can be reached from m i with l ∗ i jumps—that is, l i,j = l ∗ i .
Now we deﬁne the following directed graph that only includes these typical transitions.
Deﬁnition 1 (Typical Transition Graph) .
Given a function f satisfying Assumption 1 and gradient clipping threshold b > 0 satisfying Assumption 3, a directed graph G =
( V, E ) is the corresponding typical transition graph if (1) V = { m 1 , · · · , m n min } ; (2) An edge ( m i → m j ) is in E
iff l i,j = l ∗ i .
Naturally, the typical transition graph G can be decomposed into different communication classes
G 1 , · · · , G K that are mutually exclusive by considering the equivalence relation associated with the existence of the (two-way) paths between i and j .
More speciﬁcally, for i (cid:54) =
j , we say that
i and j communicate if and only if there exists a path ( m i , m k 1 , · · · , m k n , m j ) as well as a path ( m j , m k (cid:48) 1 , · · · , m k (cid:48) n (cid:48) , m i ) in G ; in other words, by travelling through edges on G , m i can be reached from m j and m j can be reached from m i .
We say that a communication class G is absorbing if there does not exist any edge ( m i → m
j ) ∈
E such that m i ∈ G and m j / ∈ G .
Otherwise, we say that G is transient .
In the case that all m i ’s communicate with each other on graph G , we say G is irreducible .
See Figure 2 (Middle) for the illustration of an irreducible case.
When G is irreducible, we deﬁne the set of largest attraction ﬁelds
M large = ∆ { m i : i = 1 , 2 , · · · , n min , l ∗ i =
l large } where l large = max j l ∗ j ; recall that l ∗ i characterizes the width of Ω i .
Deﬁne the longest time scale λ large ( η ) =
H (1 /η )
(( H (1 /η ) /η ))
l large − 1 .
Note that this corresponds exactly to the order of the ﬁrst exit time of the largest attraction ﬁelds; see Theorem 1.
The following theorem is the main result of this paper.
Theorem 2.
Let Assumptions 1-3 hold and assume that the graph G is irreducible .
For any t > 0 , β > 1 + ( α − 1) l large and x ∈ [ − L, L ] ,
[Equation]
in probability as η → 0 .
The proof is deferred to Appendix F.
Here we brieﬂy discuss the implication of the result.
Suppose that we terminate the training after a reasonably long time, say, (cid:98) t/η β (cid:99) iterations.
Then the random variable that converges to zero in eq.
(5) is exactly the proportion of time that X ηn spent in the attraction ﬁelds that are not wide.
Therefore, by truncating the gradient noise of the heavy-tailed
SGD, we can effectively eliminate small attraction ﬁelds from its training trajectory.
In other words, it is almost guaranteed that SGD is in one of the widest attraction ﬁelds after sufﬁciently long training.
Theorem 2 is merely a manifestation of the global dynamics of heavy-tailed SGD.
The main messages of the next theorem are: (a) under clipped heavy-tailed noises, the dynamics of X ηn for small η closely resemble that of a continuous-time Markov chain; (b) this chain only visits local minima of the largest attraction ﬁelds of f , thus minima in small attraction ﬁelds are completely avoided.
Theorem 3.
Let x ∈ Ω i for some i = 1 , 2 , · · · , n min .
If Assumptions 1-3 hold and G is irreducible, then there exist a continuous-time Markov chain Y on M large and a random mapping π satisfying
• π ( m ) ≡ m if m ∈ M large ; • π ( m ) is a random variable that only takes value in M large if m / ∈ M large
such that the scaled process { X η (cid:98) t/λ large ( η ) (cid:99) ( x ) : t ≥ 0 } converges to process { Y t ( π ( m i )) : t ≥ 0 } in the sense of ﬁnite-dimensional distributions: for any positive integer k and any 0 < t 1 < t 2 < · · · < t k , the random vector (cid:16) X η (cid:98) t 1 /λ large ( η ) (cid:99) ( x ) , · · · , X η (cid:98) t k /λ large ( η ) (cid:99) ( x ) (cid:17) converges in distribution to (cid:16) Y t 1 ( π ( m i )) , · · · , Y t k ( π ( m i )) (cid:17) as η ↓ 0 .
In section D of Appendix, we detail the proof, the exact parametrization of the generator matrix of process Y , and the distribution of random mapping π ( · ) .
Here we add some remarks.
Intuitively speaking, this result tells us that, regardless of where we initialize the SGD iterates, the dynamics of the clipped heavy-tailed SGD converge to a continuous-time Markov chain avoiding any local minima that is not in the largest attraction ﬁelds.
Second, under small learning rate η > 0 , if X ηn ( x ) is initialized at x ∈ Ω i where Ω i is NOT a largest attraction ﬁeld, then SGD will quickly escape Ω i and arrive at some Ω j that is indeed a largest one—i.e., m j ∈ M large ; such a transition is so quick that, under time scaling λ large ( η ) , it is almost instantaneous as if X ηn ( x ) is actually initialized randomly at some of the largest attraction ﬁelds.
This randomness is compressed in the random mapping π .
In
Section B, we discuss how our characterization in Theorem 3 are more general and applicable in the machine learning context than metastability results cited in (¸Sim¸sekli et al., 2019b); in Section F we see that the regularization effect of truncated heavy-tailed noises described in Theorem 3 is of great generality and can still be observed locally when the irreducibility condition is removed.
2. R d E XTENSIONS OF THE T HEORETICAL R ESULTS
We focused on the R 1 case so far, for the clarity of the exposition.
In this section, we informally reiterate that the same effect under truncated heavy-tailed noises persists in high-dimensions.
Rigorous statements are provided in Appendix G. We consider a setting similar to those in Imkeller et al.
(2010a) and analyze the ﬁrst exit time σ ( η, x )
=
min { k ≥ 0 :
X ηk ( x ) / ∈ G} from an open, bounded domain G with smooth boundary.
For some f : R d (cid:55)→ R , the SGD iterates X η k ( x ) =
X ηk − 1 ( x ) − ϕ b (cid:0) ∇ f ( X ηk − 1 ( x ))
+ ηZ k (cid:1) are subject to the standard L 2 norm clipping with threshold b > 0 , iid noises Z n with heavy tails that resemble 1 /x
α
with α > 1 , and initial condition X η 0 ( x ) = x .
Assume that the origin 0 is the only attractor in G for the ODE ˙ x ( t ) =
−∇ f
( x ( t )) .
Also, let l ∗G be the minimum number of jumps required for the ODE x ( t ) to escape from G provided that x (0)
=
and the L 2 norm of all jumps are less than b .
The following informal version of Theorem G.2 states that, under the presence of heavy-tailed noises and gradient clipping, the ﬁrst exit time from a region in R d is of order O
(1 /η 1+( α − 1) l ∗G ) .
Therefore, the order of ﬁrst exit times from different regions in R d are still dictated by the geometric characterization l ∗G , i.e. the minimum number of jumps required for escape, and those with largest l ∗G may dominate the SGD trajectory as η ↓ 0 .
Furthermore, we extend the analysis to the generalized case where the distribution of noises Z n exhibits strong preference of certain directions and has different heavy-tailed indices α along different directions.
We give the details of the R d results in Section G and provide a proof in Section H.
Theorem 4 (Informal) .
Under certain regularity conditions, for Lebesgue almost every b > 0 , there exist q > 0 and λ ( η ) that is regularly varying w.r.t η with index 1 + ( α − 1) l ∗G such that
[Equation]
S IMULATION E XPERIMENTS
We empirically demonstrate that, ( a ) as indicated by Theorem 1, the minimum jump number deﬁned in (3) accurately characterizes the ﬁrst exit times of the SGDs with clipped heavy-tailed gradient noises; ( b ) sharp minima can be effectively eliminated from such SGD; and ( c ) these properties are exclusive to heavy-tails.
Under light-tailed noises, SGDs are trapped in sharp minima for a long time.
The test function f ∈ C 2 ( R ) is the same as in Fig.
1 (Left,e).
m
and m 3 are sharp minima in narrow attraction ﬁelds, while m 2 and m 4 are ﬂatter and located in larger attraction ﬁelds.
Heavy-tailed noises have tail index α = 1 . 2 , and light-tailed noises are N (0 , 1) .
See Appendix A for details.
First, we compare the ﬁrst exit time of heavy-tailed SGD (when initialized at -0.7) from Ω 2 = ( − 1 . 3 , 0 . 2) under 3 different clipping mechanism: (1) b = 0 .
28 , where the minimum jump number required to escape is l ∗ = 3 ; (2) b = 0 .
5 , where l ∗ = 2 ; (3) no gradient clipping, where l ∗ =obviously.
According to Theorem 1, the ﬁrst exit times for the aforementioned 3 clipping mechanism are of order (1 /η ) 1 . 6 , (1 /η ) 1 . 4 and (1 /η ) 1 . 2 respectively.
These theoretical predictions are accurate as demonstrated in Figure 1 (Right).
Next, we investigate the global dynamics of heavy-tailed SGD.
We compared the clipped case (with b = 0 . 5 ) against the case without clipping.
Figure 1 (Left, a, b) show the histograms of the empirical distributions of SGD, and Figure 1(Middle, a,b) plots the SGD trajectories.
Without gradient clipping, X n still visits the two sharp minima m 1 , m 3 .
Under gradient clipping, the time spent at m 1 , m 3 is almost completely eliminated and is negligible compared to the time X n spent at m 2 , m 4 in larger attraction ﬁelds.
This matches the predictions of Theorems 2-3: the elimination of sharp minima with truncated heavy-tailed noises.
We stress that the said properties are exclusive to heavy-tailed SGD.
As shown in Figure 1(Left,c,d) and Figure 1(Middle, c,d), light-tailed SGD are easily trapped at sharp minima for extremely long time.
Figure 3 illustrates the same phenomena in R 2 , where f has several saddle points and inﬁnitely many local minima—the local minima of Ω 2 form a line segment, which is an uncountably inﬁnite set.
Under clipping threshold b , attraction ﬁelds Ω 1 and Ω 2 are the larger ones since the escape from them requires at least two jumps.
This suggests that the theoretical results from Section 2 hold under more general contexts than Assumptions 1-3.
In the next section, we provide experimental evidence that suggests that truncated heavy-tailed noise improves the generalization of SGD in deep learning.
H EAVY - TAILED SGD IN D EEP L EARNING : A N A BLATION S TUDY
In this section, we verify our theoretical results and demonstrate the effectiveness of clipped heavytailed noise in training deep neural networks.
Contrary to the report in (¸Sim¸sekli et al., 2019a), heavytailed noise may not be ubiquitous in image classiﬁcation tasks.
For instance, the non-Gaussianity assumption on SGD noise is disputed by experiments in (Panigrahi et al., 2019) for ResNet (see (He et al., 2016)).
For tasks considered in this section, the gradient noise is not heavy-tailed when models are randomly initialized (see Appendix A).
Motivated by the absence of heavy-tailed noises in image classiﬁcation, we make the SGD noise heavy-tailed .
Let θ be the current model weight during training, g SB ( θ ) be the typical small-batch gradient, and g GD ( θ ) be the true (deterministic) gradient evaluated on the entire training dataset.
Then by evaluating g SB ( θ ) − g GD ( θ ) we obtain a sample of the gradient noise.
Due to the prohibitive cost of evaluating g GD ( θ ) , we instead use g SB ( θ ) − g LB ( θ ) as its approximation where g LB denotes the gradient evaluated on a larger batch.
This is justiﬁed by the unbiasedness in E LB [ g LB ( θ )]
=
g GD ( θ ) .
For some heavy-tailed random variable Z , by multiplying Z with SGD noise, we obtain the following perturbed gradient:
[Equation]
1  3        (a)    4            b (b) 0.
1e
Out    (c) gradclipno clip    gradient clipping (d) 0.0 0.5 1.0 1.5 2.0 2.5 3.
[Figure]
where SB and SB ∗ are two mini batches that may or may not be identical.
We use the following update recursion under gradient clipping threshold b : X ηk +
=
X ηk − ϕ b ( ηg heavy ( X ηk )) where ϕ b is the truncation operator.
We consider two different implementations: in our method 1 (labeled as “our 1” in Table 1), SB and SB ∗ are chosen independently, while in our method 2 (labeled as “our 2” in
Table 1), we use the same batch for SB and SB ∗ .
In summary, by simply multiplying gradient noise with heavy-tailed random variables, we inject heavy-tailed noise into the optimization procedure.
We conduct an ablation study and benchmark the proposed clipped heavy-tailed methods against the following optimization methods.
LB : large-batch SGD with X ηk +
=
X ηk − ηg LB ( X ηk ) ; SB : small-batch SGD with X ηk +
= X ηk − ηg SB ( X ηk ) ; SB + Clip : the update recursion is X ηk +
=
X ηk − ϕ b ( ηg SB ( X ηk )) ; SB + Noise : Our method 2 WITHOUT the gradient clipping mechanism, leading to the update recursion X ηk +
=
X ηk − ηg heavy ( X ηk ) .
The experiment setting and choice of hyperparameters are adapted from (Zhu et al., 2019).
We consider three different tasks: (1) LeNet (LeCun et al., 1990) on corrupted FashionMNIST (Xiao et al., 2017), (2) VGG11 (Simonyan & Zisserman, 2014) on SVHN (Netzer et al., 2011), (3) VGG on CIFAR10 (Krizhevsky et al., 2009) (see Appendix A for details).
Here we highlight a few points: First, within the same task, for all the 6 candidate methods will use the same η , batch size, training iteration, and (when needed) the same clipping threshold b and heavy-tailed RV Z for a fair comparison; the training duration is long enough so that LB and SB have attained 100% training accuracy and close-to-0 training loss long before the end of training (the exception here is “ SB +
Noise ” method; see Appendix A for the details); Second, to facilitate convergence to local minima for our methods 1 and 2 , we remove heavy-tailed noise for last ﬁnal 5,000 iterations and run LB instead 2 .
Table 1 shows that in all 3 tasks both our method 1 and our method 2 attain better test accuracy than the other candidate methods.
Meanwhile, both methods exhibit similar test performance, implying that the implementation of the heavy-tailed method may not be a the deciding factor.
We also report the expected sharpness metric E ν ∼N ( 0 ,δ 2 I )
| L ( θ ∗ + ν ) − L ( θ ∗ )
| used in Zhu et al.
; Neyshabur et al.
(2017b) where N ( 0 , δ 2 I ) is a Gaussian distribution, θ ∗ is the trained model weight and L is training loss.
In our experiment, we use δ = 0 .
and the expectation is evaluated by averaging over 100 samples.
We conduct 5 replications for each experiment scenario and report the averaged
[Table] [Table]
Our
[Table]
performance in Table 1.
Smaller sharpness of our methods 1 and 2 conﬁrms that they encourage minimizers with a “ﬂatter” geometry, thus attaining better test performances.
The ablation study in Table 1 shows that both heavy-tailed noise and gradient clipping are necessary to ﬁnd a ﬂat minima and hence achieve better generalization, which is predicted by our analyses.
SB and SB + Clip achieve similar inferior performances, conﬁrming that clipping does not help when noise is light-tailed.
SB + Noise injects heavy-tailed noise without gradient clipping, which achieves an inferior performance.
This poor performance—even after extensive parameter tuning and engineering (see Appendix A for more details)—demonstrates the difﬁculty on the optimization front, especially when heavy-tailed noise is present yet little effort is put into controlling the highly volatile gradient noises.
This is aligned with the observations in Zhang et al.
Gorbunov et al.
(2020) where adaptive gradient clipping methods are proposed to improve convergence of SGD in the presence of heavy-tailed noises.
This conﬁrms that gradient clipping is crucial for heavy-tailed SGD.
Lastly, Table 2 shows that even in the more sophisticated settings with training techniques such as data augmentations and scheduled learning rates, truncated heavy-tailed SGD still manages to consistently ﬁnd solutions with better test performance.
For experiment details, see Section A in the
Appendix and the Table A.3 where we also report the sharpness of the obtained solutions.
C ONCLUSION
We characterized the global dynamics of SGD with truncated heavy-tailed gradient noise and illustrated our theoretical results with numerical experiments.
Our characterizations provide key insights into the global dynamics of SGD and reveal the strong regularization effects of truncated heavy-tailed noises.
For the sake of analysis and presentation, we made some simplifying assumptions.
However, as indicated by our numerical experiments, it is very likely that the same overall principle—elimination of sharp minima—persists in high-dimensional optimization landscapes that are much more sophisticated than our assumptions.
We aim to address this limitation in future works and develop the theories of truncated heavy-tailed SGD at its full level of generality.
Also, the conditions—irreducibility and symmetry of graph G —we impose on the geometry of f in Theoremare likely to be stronger than necessary for eliminating sharp minima.
In view of these, a natural next step is to investigate how the structures of state-of-the-art deep neural nets lend themselves to the
SGD’s global dynamics under general conditions.
A D ETAILS OF N UMERICAL E XPERIMENTS A. D ETAILS OF THE R 1 SIMULATION EXPERIMENT
The function f used in the experiments is f ( x ) =
( x + 1 . 6)( x + 1 . 3) 2 ( x − 0 .
2) 2 ( x − 0 .
7) 2 ( x − 1 . 6) (cid:0) 0 .
| 1 .
65 − x | (cid:1) 0 .
{+· (cid:16) 1 + 0 . + 4( x − 0 . 5)(cid:17)(cid:16) 1 + 0 . 1 + 
+}
( x + 1 . 5)(cid:17)(cid:16) 1 −4 exp( − 5( x + 0 .
8)( x + 0 .
8)) (cid:17) .
(A.1) [Figure] As shown in Figure A.1, the four isolated local minimizers of f are m 1 = − 1 . 51 , s 1 = − 1 .
3 , m 2 = − 0 .
66 , s 2 = 0 .
2 , m 3 = 0 .
49 , s 3 = 0 .
7 , m 4 = 1 .
32 , and in our experiment we restrict the iterates on [ − L, L ] with L = 1 . 6 .
The heavy-tailed noises we used in the experiment were Z n = 0 .
1 U n W n where W n were sampled from Pareto Type II distribution (aka Lomax distribution) with shape parameter α = 1 . 2 , and the signs U n were iid RVs such that P ( U n = 1) =
P ( U n = − 1) = 1 / 2 .
In the ﬁrst exit time experiment, we tested three different settings: (a) b = 0 .
28 (so that l ∗ = 3 ); (b) b = 0 .
5 (so that l ∗ = 2 ); (c) no gradient clipping (so that l ∗ 2 = 1 ).
For the ﬁrst case, we tested learning rates { 0 . 1 , 0 . 05 , 0 .
001 } , while for the other two cases, we tested learning rates { 0 . 1 , 0 . 03 , 0 . 01 , 0 . 003 , 0 .
For each case, we ran the simulation 20 times and plotted the average of the 20 exit times.
Lastly, to prevent excessively long running time of the experiment, the simulation was terminated when the iteration number reached 5 × 10 7 .
This threshold was reached only in the setting with η = 0 .
001 , b = 0 .
−11 X n −11 X n −11 X n −11 X n −11 X n 0.
[Figure]
[Figure]
1e −11 X n −11 X n −11 X n −11 X n −11 X n −11 X n 0.
[Figure]
[Figure] [Figure] [Figure] n
[Figure]
−
[Figure]
1 X n
Next, we present extra sample paths of SGD when applied to function f in eq.
(A.1) in Figure A. and A.3.
The blue curve on the right side of each plot shows f rotated by 90 degrees, and
the
−11 X n −11 X n −11 X n −11 X n −11 X n 0.
1.0 Iteration number n 1e −11 X n −11 X n −11 X n −11 X n −11 X n −11 X n 0.
1.0 Iteration number n 1e −11 X n dashed lines indicate the locations of local minima.
For better readability of the ﬁgures, we plotted
X n for every 5,000 iterations.
To generate these plots, we initialized the SGD iterates at 0 .
3 (so that it is in Ω 3 = (0 . 2 , 0 . 7) ) and ﬁxed the learning rate as η = 0 .
Again, we tested both with gradient clipping (with b = 0 . 5 ) and without gradient clipping.
Moreover, we also tested light-tailed noises where we use N (0 , 1) as the distribution for noises Z n .
For each sample path of X n , we run 10 , 000 , 000 iterations.
In the left plots of Figure A.2, one can see that with clipped heavy-tailed stochastic gradients, the SGD iterates almost always stay around the wide attraction ﬁelds, and the sharp minima are almost completely eliminated from the trajectories of SGD.
In comparison, in the right plots of Figure A.2 one can see that without gradient clipping, the heavy-tailed noises will drive
SGD to spend substantial amount of time in all the different local minima, including the sharp ones.
Lastly, from Figures A.3, one can see that under light-tailed noises and small learning rates, SGD cannot escape a sharp minima once trapped there.
A. D ETAILS OF THE R d SIMULATION EXPERIMENT
As illustrated in the contour plot in Figure 3 (a), the function f in this experiment is a modiﬁed version of Himmelblau function, a commonly used test function for optimization algorithm.
The modiﬁcations serve two purposes.
First, as shown in Figure 3 (b), for the modiﬁed function the four attraction ﬁelds Ω 1 , Ω 2 , Ω 3 , Ω 4 have different sizes; in particular, under gradient clipping threshold b = 2 . 15 , from the local minimizers of Ω 1 and Ω 2 (indicated by red dots in the corresponding area) at least two jumps are required to escape from the attraction ﬁeld, while from the local minimizer in Ω 3 or Ω 4 it is possible to escape with one jump.
Therefore, for the minimum jump number required to escape, we have l ∗ 1 =
l ∗ 2 = 2 > l ∗ 3 =
l ∗ 4 = 1 in this case.
Second, for the modiﬁed test function f , the local minimizer in Ω 2 is not a single point but a connected line segment, which is indicated by the dark line in bottom-left region in Figure 3 (a) and the red line segment in in Figure 3 (b).
Therefore, the modiﬁcation allows us to test the heavy-tailed SGD methods on a more general loss landscape.
Now we describe the construction of the test function f .
Let h be the Himmelblau function with expression h ( x, y )
= ( x 2 + y − 11)
+ ( x + y 2 − 7) 2 .
Next, deﬁne the following transformation for coordinates: φ ( x, y ) =
(cid:16) x (exp( c 0 ( x − c x ) + 1)) , y (exp( c 0 ( x − c x ) + 1) (cid:17) .
Let the composition be h φ ( x, y )
=
h (cid:16) φ ( x − a x , y ) (cid:17) .
To create the connected region of local minimizers, deﬁne the following locally “cut” version of h φ :
[Equation]
In other words, by taking minimum of the original h φ and a polynomial function w.r.t.
y around the original local minimizer of Ω 2 , we obtain a function h ∗ that attains local minimum on an entire line segment with y = a y .
Lastly, the test function we use in the experiment is f = 0 .
1 h ∗ , with a x = 1 . 5 , a y = − 2 .
9 , b [Equation]
= − 5 . 5 , b r = − 0 .
5 , b y = 2 . 0 , c 0 = 0 .
4 , c 1 = 12 .
In the experiment, we initialize the SGD iterates X k at X 0 = (2 . 9 , 1 . 0) , which is very close to the local minimizer in the small attraction ﬁeld Ω 3 .
For both the clipped and unclipped SGD, we perform updates for 3 × 10 7 steps, under learning rate 5 × 10 − 4 and heavy-tailed noise Z k = 0 .
75 W k where the iid samples W k are isotropic and the law of (cid:107) W k (cid:107) , the size of the noise, is Pareto(1.2).
For clipped SGD, we use threshold b = 2 . 15 .
To prevent the iterates from drifting to inﬁnity, after each update X k is projected back to the L 2 ball centered at origin with radius 4 .
2 whenever X k leaves this ball.
[Figure]
In Figure A.4, we use the 3 × 10 7 steps of SGD iterates (for both the clipped and unclipped case) to create heat maps showing locations of SGD iterates.
From this ﬁgure, two points can be made clear: ﬁrst, the heavy-tailed SGD does spend much less time at the two small attraction ﬁelds when gradient clipping is applied; second, in Ω 2 (the bottom-left attraction ﬁeld)
the SGD iterates frequent the entire connected region of local minima instead of a certain point on this line segment.
A. D ETAILS OF THE ABLATION STUDY
[Figure]
[Figure]
[Table]
We ﬁrst mention that the all experiments using neural networks are conducted on Nvidia GeForce
GTX 1080 Ti.
For the ablation study, the experiments and scripts are adapted from the ones in Zhu et al.
In Figure A. [Table] we display the gradient noise distribution in the three tasks of the ablation study after the model is randomly initialized.
The experiment setting and choice of hyperparameters are mostly adapted from the experiment in
Zhu et al.
We consider three different tasks: (1) training LeNet on corrupted FashionMNIST dataset; speciﬁcally, we use a 1200-sample subset of the original FashionMNIST training dataset, and for 200 samples points in the training set we randomly assign a label instead of using the correct ones; (2) VGG11 on SVHN dataset, where we use a 25000-sample subset of the training dataset; (3)
VGG11 on CIFAR10, where we use the entire training set.
For all tasks we use the entire test dataset when evaluating test accuracy.
The heavy-tailed multipliers Z n used in this experiment, whenever heavy-tailed noise is needed, are
Z n = cW n where W n are iid Pareto( α ) RVs.
For each task, we ﬁrst randomly initialize each model, 3 https://github.com/uuujf/SGDNoise [Table] [Table] CIFAR10,
[Table]
FashionMNIST, LeNet
[Table]
[Table]
[Table]
, VGG
[Table]
[Table]
and then run the 6 candidate methods in parallel starting from the same randomly initialized model weights for a fair comparison.
The hyperparameters in training for each task are listed in Table A.1.
The same set of hyperparameters is used for all methods in the same task.
Whenever gradient clipping scheme is applied, we clip the gradient if its L 2 norm exceeds the threshold given in Table A.1.
The exception here is the “ SB + Noise ” method: we use learning rate η = 0 .
005 ; for FashionMNIST task we train for 100, iterations and the heavy-tailed noise is removed for the ﬁnal 50,000 iterations; for SVHN and
CIFAR10 tasks, we train for 150,000 iterations and heavy-tailed noise is removed for the last 70, iterations.
Besides, for this method we always clip the model weights if its L ∞ norm exceeds 1.
The reason for the extra tuning and extended training in “ SB + Noise ” method is that, without the said modiﬁcations, in all three tasks we observed that the model weights quickly drift to inﬁnity and explodes; even with the weight clipping implemented, the model performance stays at random level with no signs of improvements if we do not tune down learning rate.
[-5 × 102 . 8 × 
-]
[-5 × 
 [Table]-] In Table A.2, we also report the sharpness of solutions under different shaprness metrics.
First, the
PAC-Bayes Sharpness metric (see equation (53) in Jiang et al.
(2019)) is deﬁned as 1 /σ 2 where σ is equal to the smallest δ that induces a 0.1 expected sharpness, and reﬂects the sharpness/ﬂatness parameter used in studies on generalization gaps under the PAC-Bayes framework (see Neyshabur et al.
(2017a)).
Besides, the Maximal Sharpness metric (see equation (54) in Jiang et al.
(2019)) is deﬁned as 1 /σ 2 where σ is equal to the smallest radius δ that makes max (cid:107) ν
(cid:107) ∞
≤ δ | L ( θ ∗ + ν ) − L ( θ ∗ )
| ≥ 0 . 1 , and metrics of form max (cid:107) ν (cid:107)≤ δ | L ( θ ∗
+ ν
) − L ( θ ∗ ) | can be considered as a proxy for the spectral norm of the Hessian at the solution (see Dinh et al.
It worth noticing that, for all three sharpness metrics, the smaller the value is the "ﬂatter" the loss landscape is around the solution.
Lastly, for evaluation of the PAC-Bayes Sharpness and Maximal Sharpness metrics, we conduct binary search as in Algorithm 2 of Jiang et al.
(2019) with (cid:15) d = 0 .
01 , (cid:15) σ = 0 , M 1 = 10 and M 2 = 100 ; in our setting we always evaluate the training loss using one sweep of the entire training set, so M 3 is a case-speciﬁc and is equal to the number of batches of the training set under the batch size for the task at hand.
In Figure A.6, we plot the test accuracy of our method against that of the SGD for all 5 replications and 3 tasks.
A. D ETAILS OF CIFAR10/100 EXPERIMENTS WITH DATA AUGMENTATION
For both methods, we train the model for 300 epochs and set the initial learning rate as 0.1. In our method, the training can be partitioned into two phases.
In the ﬁrst phase (the ﬁrst 200 epochs), the learning rate is kept at a constant.
In the second phase, for every 30 epoch we reduce the learning rate by half.
Also, an L 2 weight decaying with coefﬁcient 5 × 10 − 4 is enforced.
As for parameters for heavy-tailed noises in eq.
(6), we use c = 0 .
5 and α = 1 .
in the ﬁrst phase, and in the second phase we remove heavy-tailed noise and use SB to update weights.
In both methods for the small-batch direction g SB the batch size is 128, while for g LB we evaluate the gradient on a large sample batch of size 1,024.
Under the epoch number 300 and batch size 128, the count of total iterations performed during training is 1 .
17 × 10 5 .
To augment the dataset, random horizontal ﬂipping and cropping with padding size 4 is applied for each training batch.
Lastly, gradient clipping scheme is applied for both methods, and we ﬁx b = 0 .
In other words, when the learning rate is η (note that due to the scheduling of learning rates, η will be changing throughout the training), the gradient is clipped if its L 2 norm is larger than b/η .
The scripts are adapted from the ones in https://github.com/chengyangfu/pytorch-vgg-cifar10.
These results are presented in Table 2.
Furthermore, in Table A.3 we see that our truncated heavytailed method also manages to ﬁnd solutions with a ﬂatter geometry.
A. D ISCUSSION ON G RADIENT N OISE D ISTRIBUTIONS IN THE E XPERIMENTS
In this subsection, we (i) present the empirical evidence that supports our characterization of the baseline model—i.e., the absence of heavy tails in the gradient noise—and (ii) clarify that the emergence of heavy-tails in the *stationary distribution* of SGD (argued in Hodgkinson & Mahoney (2020); Gurbuzbalaban et al.
(2020)) does not contradict the observed absence of heavy tails in the *gradient noise* of SGD.
This allows us to study the impact of truncated heavy-tails in the gradient noise separately from the choice of hyper-parameters.
We start with our observations on the noise distributions. In view of the well-established wisdom in heavy-tail literature that there is no single perfect tail estimator, we analyzed the stochastic gradient noise with four different methods: QQ plot, empirical mean residual life (EMRL), Hill plot, and PLFIT (Clauset et al., 2009). We applied these estimation/diagnostic tools (i) at the beginning of the training, (ii) halfway through the training, (iii) at the end of the training.
Throughout all our experiments, we consistently observe strong evidence that the gradient noises are light-tailed, and even if (against all odds) the noises were from a heavy-tailed distribution, the tail index should be far greater (hence, the resulting tail is much lighter) than the heavy-tails we inject (or the popular alpha stable assumption), and hence, the point we make with our tail-inﬂation experiment is still valid.
We summarize the results as follows.
First, QQ plots below (Fig.
A.7-A.21) clearly show that the tails in noise distribution are always much lighter than the Pareto distributions with α = 2 or even 10.
Therefore, the typical power-law assumption, especially the alpha-stable distributions in ¸Sim¸sekli et al.
(2019b) (with α ∈ (0 , 2) ), seems far from the distribution of the actual data we obtained in the image classiﬁcation tasks.
In fact, the tail of the noise distributions seems to be between that of lognormal and normal distributions, implying that it is lighter than any power-law distribution.
[Figure]
Normal
[Figure]
[Equation]
Lognormal
Normal
Pareto
[Figure]
[Equation]
[Figure]
[Equation]
Normal
Pareto, α =
Pareto, α =
Pareto, α =
Lognormal
Normal
[Figure]
Pareto
[Figure]
Pareto, α =
Lognormal
Normal
[Figure]
Pareto
[Figure]
Lognormal
Normal
Pareto
[Figure]
α =
Pareto
[Figure]
α =
Lognormal
Normal
[Figure]
Pareto
[Figure] α = Lognormal Normal
Pareto
[Figure]
Pareto
[Figure]
α
[Figure]
Lognormal
Normal
Next, we plotted the empirical mean residual life (EMRL) of the gradient noise distributions in
Fig.
A.22-A.26.
It is well known that the mean residual life blows up to inﬁnity if and only if the distribution is heavy-tailed (more precisely, long-tailed).
However, from the ﬁgures, one can see that none of the EMRL exhibits such a pattern in any case tested in our experiments.
Instead, we see clear downward trends, which strongly suggest light tails, in all the tested cases.
Pareto
[Figure]
α
[Figure]
Pareto,
[Figure]
Pareto, α
[Figure]
Lognormal
Normal
[Figure]
Lognormal
Normal
Lognormal
Normal
[Figure]
[Figure]
[Figure]
Lognormal
Normal
[Figure]
Lognormal
Normal
[Figure]
Normal
If we assume a power-law tail, the Hill estimator is a classical tail index estimator with a long history in extreme value theory literature.
A critical algorithmic parameter of the Hill estimator is the number of order statistics used in the estimation and the Hill plot is a popular exploratory tool for investigating the Hill estimators with different numbers of order statistics.
Although it is well known that Hill estimators and Hill plots are fallible if the power-law assumption is not satisﬁed, (and hence, it is not suited for deciding whether a given set of samples are from light-tailed distribution or heavy-tailed distribution; in particular, the method will return some power-law tail index
α whether
[Figure]
[Figure]
[Figure]
[Figure]
or not the samples are from a heavy-tailed distribution or a light-tailed one)
we present the Hill plot to see what would be the estimated tail indices if the gradient noises hypothetically followed a power law.
In the Hill plots shown in Fig.
A.27-A.31, we presented the rescaled version (altHill) of the Hill plots (see Chapter 4.4 in (Resnick, 2007)) for the following reason.
Hill estimator is a consistent estimator of the power-law index when the proportion of the samples used approaches and altHill plots allow us to scrutinize the estimated indices under a small proportion of samples.
In particular, the points around the red dashed lines correspond to estimation using the top 1% of the samples.
We can see that most Hill plots stay well above 10 for the most part and almost never drop below 2.
This strongly suggests that even if the gradient noises are from a heavy-tailed distribution, it is likely to have a very high power-law index (implying relatively lighter tails), and hence, we cannot expect to observe a prominent heavy-tailed behavior from them.
[Figure]
[Figure]
[Figure] [Figure] A popular data-driven approach with statistical guarantees (again, under the assumption that the samples are indeed from a heavy-tailed distribution) to select the number of order statistics in the Hill plot is PLFIT (Clauset et al., 2009).
We estimated the power-law indices using the python implementation (Alstott et al., 2014) of PLFIT.
The numbers are presented in Table A.4.
All the estimations are at least 5 for all cases tested in our experiments and most of the time, the estimation is above 10.
Again, this means that even under the hypothetical assumption (against what the QQ plots and EMRLs suggest) that the gradient noises were from a heavy-tailed distribution, the tail indices of the gradient noises should be large and hence, the gradient noises in our experiments have much lighter tails than any α -stable distribution (which requires α < 2 ) or the heavy-tailed noises we injected during tail inﬂation experiments ( α = 1 . 4 ).
In summary extensive statistical analyses above suggest the absence of heavy tails in the gradient noises in our experiments. Therefore, [Figure] [Figure]
the characterization of the vanilla SGD as the light-tailed (or at least lighter than the inﬂated tails) benchmark in our experiments is valid, and our ablation study is well grounded. [Table] Next we compare our work to recent literature on heavy-tailed phenomena in stationary distribution of SGD; see, for instance, Hodgkinson & Mahoney (2020); Gurbuzbalaban et al. To be speciﬁc, Hodgkinson & Mahoney (2020)
; Gurbuzbalaban et al.
(2020) show that heavy tails can arise in the stationary distribution of SGD through multiplicative dynamics and the tail index of the resulting stationary distribution can be characterized by the learning rates and the magnitude of noises (through batch size).
However, the results in Hodgkinson & Mahoney (2020); Gurbuzbalaban et al.
(2020) do not imply the existence of heavy tails in the gradient noises.
In both papers (as well as other works in the literature), the “heaviness” of the tail of the gradient noise (e.g., power-law index α in heavy-tailed cases) is ﬁxed in the model (same as in our setting) and not entangled with the learning rate or batch size.
For example, B k in (5) of Hodgkinson & Mahoney (2020) corresponds to the gradient noise, and its tail index doesn’t depend on the learning rate or the batch size.
In particular, the change of learning rate does not induce heavy tails in B k .
On the other hand, we focus on the impact of the heavy-tails *in the gradient noise* and the truncation scheme for (any) ﬁxed batch size and small learning rates on the global dynamics of SGD.
In view of this, it should be clear that our analysis can be decoupled from the choice of batch size or the impact of the learning rate on SGD’s stationary distribution.
Therefore, our observation and the design of the experiments are compatible with the aforementioned references.
B I MPLICATIONS OF THE THEORETICAL RESULTS
Systematic control of the exit times from attraction ﬁelds: In light of the wide minima folklore, one may want to ﬁnd techniques to modify the sojourn time of SGD at each attraction ﬁeld.
Theoremsuggests that the order of the ﬁrst exit time (w.r.t.
learning rate η ) is directly controlled by the gradient clipping threshold b .
Recall that for an attraction ﬁeld with minimum jump number l ∗ , Theoremtells us the exit time from this attraction ﬁeld is roughly of order (1 /η )
+( α − 1) l ∗ .
Given the width of the attraction ﬁeld, its minimum jump number l ∗ is dictated by gradient clipping threshold b .
Therefore, gradient clipping provides us with a very systematic method to control the exit time of each attraction ﬁeld.
For instance, given clipping threshold b , the exit time from an attraction ﬁeld with width less than b is of order (1 /η )
α , while the exit time from one larger than b is at least (1 /η )
α − 1 , which dominates the exit time from smaller ones.
The role of structural properties of G and f : Recall that in order for Theorem 3 to apply, the irreducibility of G is required.
Along with the choice of b , the geometry of function f is a deciding factor of the irreducibility.
For instance, we say that G is symmetric if for any attraction ﬁeld Ω i such that i
= 2 , 3 , · · · , n min − 1 (so that Ω i is not the leftmost or rightmost one at the boundary), we have q i,i − 1 > 0 , q i,i +1 > 0 .
One can see that G is symmetric if and only if, for any i
= 2 , 3 , · · · , n min − 1 , | s i − m i | ∨ | m i − s i − 1 | < l ∗ i b , and symmetry is a sufﬁcient condition for the irreducibility of G .
The graph illustrated in Figure 2(Middle) is symmetric, while the one in Figure 2(Right) is not.
As the name suggests, in the R 1 case the symmetry of G is more likely to hold if the shape of attraction ﬁelds in f is also nearly symmetric around its local minimum.
If not, the symmetry (as well as irreducibility) of G can be violated as illustrated in Figure 2, especially when a small gradient clipping threshold b is used.
Generally speaking, our results imply that, even with the truncated heavy-tailed noises, the function f needs to satisfy certain regularity conditions to ensure that SGD iterates avoid undesirable minima.
This is consistent with the observations in Li et al.
(2018b): the deep neural nets that are more trainable with SGD tend to have a much more regular structure in terms of the number and shape of local minima.
Heavy-tailed SGD without gradient clipping: It is worth mentioning that our results also characterize the dynamics of heavy-tailed SGDs without gradient clipping.
For instance, since the reﬂection operation at ± L restricts the iterates on the compact set [ − L, L ] , if we use a truncation threshold b that is large than 2 L , then any SGD update that moves larger than b will deﬁnitely be reﬂected at ± L .
Therefore, the dynamics are identical to that of the following iterates without gradient clipping:
[Equation]
The next result follows immediately from Theorem 1 and F.2.
Corollary B.1.
There exist constants q i > 0 ∀ i , q i,j > 0 ∀ j (cid:54) =
i such that the following claims hold for any i and any x ∈ Ω i .
1) Under P x , q i H (1 /η )
σ i ( η ) converges in distribution to an Exponential random variable with rate 1 as η ↓ 0 ; 2)
For any j = 1 , 2 , · · · , n min such that j (cid:54)
=
i ,
[Equation]
i . 3) Let Y be a continuous-time Markov chain on { m 1 , · · · , m n min } with generator matrix Q parametrized by Q i,i = − q
i , Q i,j = q i,j .
Then X η, unclipped
[Equation]
t/H (
[Equation]
η )
[Equation]
( x ) → Y t ( m i ) as η ↓in the sense of ﬁnite-dimensional distributions.
At ﬁrst glance, Corollary B.1 may seem similar to the results in ¸Sim¸sekli et al. (2019a) and Pavlyukevich (2007).
However, the object studied in ¸Sim¸sekli et al. (2019a);
Pavlyukevich (2007) is different: they study the following Langevin-type stochastic differential equation (SDE) driven by an α -stable
Lévy process L t with scaling factor η > 0 :
[Equation]
In particular, Pavlyukevich (2007) studies the metastability of Y ηt and concludes that as η ↓ 0 , the ﬁrst exit time and global dynamics of Y ηt admit a similar characterization as described in our Theorem
B.1.
Then Theorem 4 in ¸Sim¸sekli et al.
(2019a) argues that when the learning rate η is sufﬁciently small, the distribution of the ﬁrst exit time of the SGD X ηn and that of the Lévy-driven Langevin SDE
Y ηt are similar.
However, the analysis of ¸Sim¸sekli et al.
(2019a) hinges critically on the assumption that L αt is symmetric and α -stable.
While such an assumption is convenient for their analysis, it is a strong assumption.
It implies that the gradient noise distribution belongs to a very speciﬁc parametric family and excludes all the other heavy-tailed distributions.
In particular, the assumption precludes analysis of any heavy tails with ﬁnite variance.
On the contrary, our work directly analyzes the SGD
X ηn and reveals the heavy-tailed SGD dynamics at a much greater level of generality.
Speciﬁcally, we allow the noise to have general regularly varying distributions with arbitrary tail index—which includes α -stable distributions as a (very) special case—and extend the characterization of global dynamics of heavy-tailed SGD to the adaptive versions of SGD where gradient clipping is applied.
C R EMARKS ON S HARPNESS AND G ENERALIZATION P ERFORMANCES C. G EOMETRIC C HARACTERIZATION l ∗ AND E XISTING S HARPNESS M ETRICS
Throughout the paper, we have been using the term sharp minima when describing the elimination effect of truncated heavy-tailed SGD, while appealing to l ∗ , the minimum number of jumps requires for escape deﬁned in eq.
(3), when rigorously presenting our theoretical results about dynamics of truncated heavy-tailed SGD.
Despite the possible ambiguity of its use in the main paper, the terminology sharpness is meant to familiarize our rather technically involved notion of how heavytailed SGD behave.
To resolve the potential confusions, in this section we provide a detailed discussion on the relationship between our geometric characterization l ∗ and the existing sharpness metrics.
In light of recent discovery that all local minima might be global minima for over-parametrized deep neural nets (see Li et al.
(2018a)),
one popular explanation for the generalization gap between different local minima (for instance, between solutions found by GD and SGD) is that the geometry around the solution is closely related to its performance in the test setting.
After the seminal work by Keskar et al.
(2016), a myriad of attempts have been made to provide empirical or theoretical evidences for the link between the sharpness of a local minimum and its generalization performance;
See, for instance, Xie et al.
; Jiang et al.
; He et al.
; Zhou et al.
In summary, there exist at least four different classes of sharpness metrics among the current literature:
(a) spectral norm of the Hessian at the local minimum one surrogates of which is the maximal sharpness type of metrics; a variant used in Xie et al.
based on corresponding large deviation theory is the eigenvalue of the Hessian along certain escape directions; (b) expected sharpness type of metrics that evaluate the general ﬂuctuation of the loss function within a L p ball of centered at the local minimum (see Zhu et al.
; (c) complexity metrics based on PAC-Bayes theory (see, for example, Neyshabur et al.
(2017a)); (d) geometric property of a domain or the entire attraction ﬁeld instead of the local minimum itself; see the mass of a Radon measure m ( W ) over the domain W in Theorem 1 in Zhou et al.
Under such taxonomy, the l ∗ characterization proposed in this paper falls into category (d).
Indeed, it describes the minimum effort required for escaping the domain rather than the geometry merely around a certain neighborhood of the local minimum.
Here we have two remarks on the deﬁnition of l ∗ .
First, for a given loss landscape, this quantity l ∗ is dictated by the gradient clipping threshold b > 0 .
In fact, this concept is tailored for the gradient clipping case, as in the unclipped case with heavy-tailed SGD, for any attraction ﬁeld Ω i we always have l ∗ i = 1 .
Second, it is analogous to the term m ( W ) in Zhou et al.
(2020) in the sense that it reﬂects the volume of the attraction ﬁeld (when compared to a given threshold b ).
Therefore, we stress that a more precise interpretation of l ∗ , as well as other metrics in class (d), is that they characterize how wide or narrow each minimum (attraction ﬁeld) is, and a more a clear description of our theoretical results Theorem 1-3 is that truncated heavy-tailed SGD effectively avoids all the narrow minima when learning rate is small enough.
While one can intuitively see that wide minima are more likely to be ﬂat ones, we acknowledge the existence of counterexamples where a sharp minimum lies in a wide attraction ﬁeld.
Nevertheless, as the variety of deﬁnitions for sharpness keeps growing in existing literature, it becomes rather unlikely for one geometric characterization to always agree with (or be equivalent to) other existing approaches.
More importantly, the aim of this paper is not to establish l ∗ as the orthodox geometric property when studying generalization gap.
Instead, as demonstrated in Table 1, A.2 and A.3, in modern deep learning tasks the truncated heavy-tailed method (which prefers solutions with high l ∗ as indicated by our theoretical results) obtains solutions that generalize better and exhibit ﬂatter geometry when evaluated under different sharpness metrics.
This observation is well aligned with the recent large-scale empirical study in Jiang et al.
, suggesting that in typical training setting it is beneﬁcial to ﬁnd wider minima (characterized by l ∗ ) in order to achieve a ﬂatter geometry and better generalization performance.
C. O N THE R ELATIONSHIP B ETWEEN S HARPNESS AND G ENERALIZATION
In principle, a sharp minimum does not necessarily lead to poor generalization in the sense that it is possible to construct pathological counterexamples in theory or in speciﬁc experiment settings; see, for example, Dinh et al.
; Neyshabur et al. (2017a).
However, the loss geometry that arises in practice seems to exhibit a strong correlation between the sharpness and the test error.
For instance, sharpness-aware optimization methods (Foret et al., 2020; Kwon et al., 2021) improve generalization across various tasks and achieve state-of-the-art performance on the CIFAR dataset.
Also, one of the authors of the aforementioned paper Neyshabur et al.
(2017a) continued investigating different complexity measures and conducted much larger scale experiments in Jiang et al.
Among more than 40 complexity measures from theoretical and empirical studies in the literature tested in Jiang et al.
, the sharpness metrics are the top ones for predicting the generalization performance.
In our own experiments, we also show that sharpness and test accuracy are highly correlated.
D S TABILITY - DRIVEN A NALYSES ON SGD
Wu et al.
(2018) takes the perspective of linear stability and establishes conditions for SGD to be attracted to or avoid certain solutions based on learning rate, batch size, and the concept of nonuniformity of local minima. Inspired by (Wu et al., 2018), (Jastrzebski et al., 2020) analyzes the trajectory-wise stability of SGD and found that a “break-even point” partitions the training procedure into two phases: the implicit regularization effects in SGD due to a larger learning rate becomes visible in the second phase after this “break-even point”. Similarly, (Cohen et al., 2021) reports that typical GD trajectories in standard image classiﬁcation tasks can be partitioned into two phases: in the ﬁrst “progressive sharpening” phase the sharpness of the Hessian monotonically increases, while in the second phase we observe the “edge of stability” regime where the sharpness of Hessian hovers slightly above the critical value 2 /η and the training loss slowly decrease in an oscillating fashion. Compared to the aforementioned stability analyses, one major difference of our heavy-tailed regime is that, even under gradient clipping, the trajectory cannot be partitioned into such “phases” and the traditional sense of stability around certain local minima is nulliﬁed by the constant basin hopping and exploration behaviors under heavy-tailed noises.
In particular, the polynomial order of the exit time in heavy-tailed SGD dictates that, when compared to exponentially long exit time in vanilla
SGD (see Xie et al.
, the iterates won’t become “stabilized” and keep staying around a certain region in a typical training procedure.
Instead, our work characterizes the metastability of the truncated heavy-tailed SGD (i.e. constantly transitioning between different wide minima) that cannot be observed in GD or light-tailed SGD.
E P ROOF OF T HEOREM
This section proves Theorem 1.
We ﬁrst start with providing the deﬁnitions of q i and q i,j that appear in the statement of the theorem.
Let Leb + denote the Lebesgue measure restricted on [0 , ∞ ) , and deﬁne a (Borel) measure ν α on R \{ 0 } as follows:
[Equation]
where α , p − , and p + are constants in Assumption 2.
Deﬁne a Borel measure µ i on R l ∗ i × (cid:16) R + (cid:17) l ∗ i −as the product measure
[Equation]
We also deﬁne mappings h i as follows.
For a real sequence w = ( w 1 , w 2 , · · · , w l ∗ i ) and a positive real number sequence t = ( t (cid:48) j ) l ∗ i j =2 , deﬁne t 1 = t (cid:48) 1 = 0 and t j = t (cid:48)
+ t (cid:48) 2 + · · · + t (cid:48)
j for j = 2 , · · · , l ∗ i .
Now we deﬁne a path ˆ x : [0 , ∞ )
(cid:55)→ R as the solution to the following ODE with jumps:
[Equation]
Now we deﬁne the mappings
h
i : R l ∗ i × (cid:16) R + (cid:17) l ∗ i − 1 (cid:55)→ R as h i ( w , t ) =
ˆ x ( t l ∗ i ) .
It is easy to see that h
i ’s are continuous mappings.
With these mappings, we deﬁne the following sets:
[Equation]
Lastly, the constant q i and q i,j are deﬁned as follows:
[Equation]
Before we move on to the proof of Theorem 1, we add a few remarks regarding the intuition behind it.
• Suppose that X ηn is started at the i th local minimum m i of f and consider the behavior of X ηn over the time period H 1 (cid:44) { 1 , . .
, (cid:100) t/η (cid:101)} for a sufﬁciently large t .
The heavy-tailed large deviations theory Rhee et al.
(2019) and a heuristic application of the contraction principle implies that the path of X η (cid:100) n/η (cid:101) over this period will converge to the gradient ﬂow of f , and the event that X η (cid:100) n/η (cid:101) escapes Ω i within this period is a (heavy-tailed) rare event.
This means that the probability of such an event is of order (1 /η )
( α − 1) l ∗ i .
Moreover, whenever it happens, it is almost always because X η n is shaken by exactly l ∗ i large gradient noises of size O (1 /η ) , which translates to l ∗ i jumps in X η (cid:100) n/η (cid:101) ’s path, while the rest of its path closely resemble the deterministic gradient ﬂow.
Moreover, conditional on the event that X ηn fails to escape from Ω i within this period, the endpoint of the path is most likely to be close to the local minima, i.e., X η (cid:100) t/η (cid:101)
≈ m i .
This suggests that over the next time period H 2 (cid:44) {(cid:100) t/η (cid:101) + 1 , (cid:100) t/η (cid:101) + 2 , . . .
, 2 (cid:100) t/η (cid:101)} of length (cid:100) t/η (cid:101) , X ηn will behave similarly to its behavior over the ﬁrst period H 1 .
The same argument applies to the subsequent periods H 3 , H 4 , . . .
as well.
Therefore, over each time period of length (cid:100) t/η (cid:101) , there is (1 /η )
( α − 1) l ∗ i probability of exit.
In view of this, the exit time should be of order (1 /η )
+( α − 1) l ∗ i and resemble an exponential distribution when scaled properly.
• Part (i) of Theorem 1 builds on this intuition and rigorously prove that the ﬁrst exit time is indeed roughly of order
/λ i ( η ) ≈
(1 /η )
+( α − 1) l ∗ i and resembles an exponential distribution.
• Given this, one would expect that X ησ i ( η ) , the location of SGD right at the time of exit, will hardly ever be farther than l ∗ i b away from m i : the length of each update is clipped by b , and there will most likely be only l ∗ i large SGD steps during this successful attempt.
Indeed, from the deﬁnition of q i,j ’s above, one can see that q i,j > 0 if and only if inf
y ∈ Ω
j | y − m i | < l ∗ i b .
Summarizing the three bullet points here, we see that the minimum number of jumps l ∗ i dictates how heavy-tailed SGD escapes an attraction ﬁeld, where the SGD lands on upon its exit, and when the exit occurs.
Now we are ready to start proving Theorem 1.
First, note that Assumption 1 implies the following:
• There exist c 0 > 0 , (cid:15) 0 ∈ (0 , 1) such that for any x ∈ { m 1 , s 1 , · · · , s n min − 1 ,
[Equation]
n min } , | y − x | < (cid:15) 0 ,
[Equation]
and for any y ∈ [ − L, L ] such that | y − x | ≥ (cid:15) 0 for all x ∈ { m 1 , s 1 , · · · , s n min [Equation] 1 , m n min } , we have
[Equation]
• There exist constants L ∈ (0 , ∞ ) , M ∈ (0 , ∞ ) such that | m
| <
L, | m n min | < L , and (for any x ∈ [ − L, L ] )
[Equation]
Recall that for c > 0 , the truncation operator was deﬁned as
[Equation]
and the SGD iterates were deﬁned as
[Equation]
Here η > 0 is the learning rate (step length) and b > 0 is the gradient clipping threshold.
Also, recall that for any k ∈ [ n min ] , we let
[Equation]
to be the time that X ηn exists from the k − th attraction ﬁeld Ω j .
Meanwhile, given the gradient clipping threshold b , recall that
[Equation]
Intuitively speaking, l ∗ i tells us the minimum number of jumps with size no larger than b required in order to escape the attraction ﬁeld if we start from the local minimum of this attraction ﬁeld Ω i .
Lastly, recall the deﬁnition of H ( · ) =
P ( | Z 1 | > · ) and
[Equation]
The proof of Theorem 1 hinges on the following two lemmas that characterize the behavior of X ηn in two different phases respectively.
Let k ∈ [ n min ] and x ∈ Ω k .
We consider the SGD iterates initialized at X η 0 = x .
In the ﬁrst phase, the SGD iterates return to [ m k − 2 (cid:15), m k + 2 (cid:15) ] , a small neighborhood of the local minimizer in attraction ﬁeld Ω k ; in other words, it ends at
[Equation]
During this phase, we show that for all learning rate η that is sufﬁciently small, it is almost always the case that X ηn would quickly return to [ m k − 2 (cid:15), m k + 2 (cid:15) ] , and it never leaves Ω k before T ( k ) return .
Lemma E.1.
There exists some c ∈ (0 , ∞ ) such that for any k ∈ [ n min ] , the following claim holds for all (cid:15) > 0 small enough:
[Equation]
During the second phase, X ηn starts from somewhere in [ m k − 2 (cid:15), m k + 2 (cid:15) ] and tries to escape from Ω k , meaning that the phase ends at σ k ( η ) .
During this phase, we show that the distributions of the ﬁrst exit time σ k ( η ) and the location X ησ k ( η ) do converge to the ones described in Theorem 1 as learning rate η tends to 0 .
Lemma E.2.
There exist constants q i > 0 ∀ i ∈ [ n min ] and q i,j ≥ 0 ∀ j ∈ [ n min ] with i, j (cid:54) =
i such that the following claim holds: given any C > 0 , u > 0 and any k, l ∈ [ n min ] with k (cid:54)
=
l , we have
[Equation]
for all (cid:15) > 0 that are sufﬁciently small.
Now we are ready to show Theorem 1.
Proof of Theorem 1.
Fix some k ∈ [ n min ] and x ∈ Ω k ∩ [ − L, L ] .
Let q k and q k,l be the constants in
Lemma E.2.
We ﬁrst prove the weak convergence claim in Theorem 1(i).
Arbitrarily choose some u > 0 and
C ∈ (0 , 1) .
It sufﬁces to show that
[Equation]
Recall the deﬁnition of the stopping time T ( k ) return in eq.
(E.15).
Deﬁne event
[Equation]
where c < ∞ is the constant in Lemma E.1.
First, since x ∈ Ω k = ( s k − 1 , s k ) , it holds for all (cid:15) >
small enough that x ∈
( s k − 1 + (cid:15), s k − (cid:15) ) .
Next, one can ﬁnd some (cid:15) > 0 such that
• (Due to Lemma E
[Equation]
• (Due to eq.
(E.16)eq.
(E.17) and strong Markov property)
For all η sufﬁciently small,
[Equation]
> (1 − C ) u
(cid:12) (cid:12) (cid:12)
A k ( η, (cid:15) ) (cid:17) ≤ C + exp (cid:0) − (1 − C ) u (cid:1) ,
P x (cid:16) q k λ k ( η ) (cid:0) σ ( η ) − T ( k ) return ( η, (cid:15) ) (cid:1) > u (cid:12)(cid:12)(cid:12) A k ( η, (cid:15) )
(cid:17) ≥ − C + exp (cid:0) − (1 + C ) u (cid:1) .
Fix such (cid:15) .
Lastly, for this ﬁxed (cid:15) , due to λ k ∈ RV − 1 − l ∗ k ( α − 1) ( η ) and α > 1 , we have q k λ k ( η ) · c log(1 /(cid:15) )
η < Cu for all η sufﬁciently small.
In summary, for all η sufﬁciently small, we have
P x ( q k λ k ( η ) σ k ( η ) > u ) ≤ P x (cid:0) ( A k ( η, (cid:15) ))
c (cid:1) + P x (cid:16)(cid:8) q k λ k ( η ) σ k ( η ) > u (cid:9) ∩ A k ( η, (cid:15) )
(cid:17) ≤ C + P x (cid:16)(cid:8) q k λ k ( η ) σ k ( η ) > u (cid:9) ∩ A k ( η, (cid:15) )
(cid:17) =
C + P x (cid:16) q k λ k ( η ) (cid:0) σ ( η ) − T ( k ) return ( η, (cid:15) ) (cid:1) >
u − q k λ k ( η ) T ( k ) return ( η, (cid:15) )
(cid:12)(cid:12)(cid:12) A k ( η, (cid:15) ) (cid:17) · P x ( A k ( η, (cid:15) ))
≤ C + P x (cid:16) q k λ k ( η ) (cid:0) σ ( η ) − T ( k ) return ( η, (cid:15) ) (cid:1) > (1 − C ) u
(cid:12)(cid:12)(cid:12) A k ( η, (cid:15) )
(cid:17) ≤ 2 C + exp (cid:0) − (1 − C ) u (cid:1) and
[Equation]
so this concludes the proof for Theorem 1(i). In order to prove claims in Theorem 1(ii)
[Equation]
we ﬁrst observe that on event A k ( η, (cid:15) )
we must have σ ( η ) > T ( k ) return ( η, (cid:15) ) .
Next, arbitrarily choose some C ∈ (0 , 1) , and note that it sufﬁces to show that
P x ( X η σ ( η ) ∈ Ω l ) ∈ (cid:16) (1 − C ) q k,l
[Equation]
C q
[Equation]
, C
[Equation] q k,l [Equation] (cid:17) holds for all η sufﬁciently small.
Again, we can ﬁnd (cid:15) > 0 such that
• (Due to Lemma E 1) P x (cid:0) ( A k ( η, (cid:15) ))
c (cid:1) ≤ C ∀ η sufﬁciently small ; • (Due to eq.
(E.18)eq.
(E.19) and strong Markov property)
For all η sufﬁciently small, [Equation]
In summary, for all η sufﬁciently small, we have
[Equation]
q
[Equation]
and
[Equation]
=
[Equation]
k and this concludes the proof.
The rest of this section is devoted to the proofs of Lemma E.1 and Lemma E.2.
Speciﬁcally, Lemma
E.1 is an immediate Corollary of Lemma E.13, the proof of which will be provided below.
The proof of Lemma E.2 can be found at the end of this section.
E. P ROOFS OF L EMMA E.1, E.
The following three lemmas will be applied repeatedly throughout this section.
The proofs are straightforward but provided in Section I for the sake of completeness.
Lemma E.3.
Given two real functions a : R + (cid:55)→ R + , b : R + (cid:55)→ R + such that a ( (cid:15) ) ↓ 0 , b ( (cid:15) ) ↓ 0 as (cid:15) ↓ 0 , and a family of geometric RVs { U ( (cid:15) ) : (cid:15) > 0 } with success rate a ( (cid:15) ) (namely, P ( U ( (cid:15) ) > k ) =
(1 − a ( (cid:15) ))
k for k ∈ N ), for any c > 1 , there exists (cid:15) 0 > 0 such that for any (cid:15) ∈ (0 , (cid:15) 0 ) , [Equation] c · a ( (cid:15) ) b ( (cid:15) )
[Equation]
Lemma E.4.
Given two real functions a : R + (cid:55)→ R + , b : R + (cid:55)→ R + such that a ( (cid:15) ) ↓ 0 , b ( (cid:15) ) ↓and
[Equation]
as (cid:15) ↓ 0 , and a family of geometric RVs { U ( (cid:15) ) : (cid:15) > 0 } with success rate a ( (cid:15) ) (namely, P ( U ( (cid:15) ) > k )
= (1 − a ( (cid:15) ))
k for k ∈ N ), for any c > 1 there exists some (cid:15) 0 > 0 such that for any (cid:15) ∈ (0 , (cid:15) 0 ) , [Equation] (cid:15)
[Equation] (cid:15) )) [Equation]
[Equation] (cid:15) ) /b
( (cid:15) )
Lemma E.5.
Suppose that a function g : E (cid:55)→ R (where E is an open set in R d ) is g ∈ C 2 ( E ) and (cid:13)(cid:13) ∇ 2 g ( · ) (cid:13)(cid:13) ≤ C on its domain E for some constant C < ∞ .
For a ﬁnite integer n , a sequence of vectors { z 1 , · · · , z n } in R d , and vectors x, (cid:101) x ∈ E, η > 0 , consider two sequences { x k } k =0 ,...,n and { (cid:101) x k } k =0 ,...,n constructed as follows:
[Equation]
If we have that the line segment from x k to (cid:101) x k is contained in E and η (cid:107)
z 1 + · · · + z k (cid:107) + (cid:107) x − (cid:101) x (cid:107) ≤
[Equation]
c for all k = 1 , 2 , · · · , n for some (cid:101) c > 0 , then
[Equation]
To facilitate the analysis below, we introduce some additional notations.
First, we will group the noises Z n based on a threshold level δ > 0 : let us deﬁne
[Equation]
The former are viewed as small noises while the latter will be referred to as large noises or large jumps .
Furthermore, for any j ≥ 1 , deﬁne the j th arrival time and size of large jumps as
[Equation]
Next, for any (cid:15) > 0 , let Ω i ( (cid:15) ) =
[ m i − (cid:15), m i + (cid:15) ] be an (cid:15) − neighborhood of the local minimum m i , and S i ( (cid:15) )
=
[ s i − (cid:15), s i + (cid:15) ] be an (cid:15) − neighborhood of the local maximum s i .
For most part of this section, we will zoom in on one of the local minima m i and its attraction ﬁeld Ω i =
( s i − 1 , s i ) .
Without loss of generality, we assume m i = 0 , and denote the attraction ﬁeld as Ω = ( s − , s + ) .
(If m i happens to be the local minimum at the left or right boundary, then the attraction ﬁeld is [ − L, s + ) or ( s − , L ] where the SGD iterates will be reﬂected at ± L .)
Henceforth we will drop the dependency on notation i when referring to this speciﬁc attraction ﬁeld until the very end of this section.
Throughout the proof, the following (deterministic) dynamic systems will be used frequently as benchmark processes to indicate the most likely location of the SGD iterates.
Speciﬁcally, given any x ∈ Ω , we use X n ( x ) to indicate that the starting point is x , namely X 0 ( x ) = x .
Similarly, consider the following ODE x η ( t ; x ) as
[Equation]
[Equation]
When we use update rate η = 1 , we will drop the dependency of η and simply use x ( t ; x ) to denote the process.
Based on Assumption 3, we know the existence of some constant ¯ (cid:15) ∈ (0 , (cid:15) 0 ) (note that (cid:15) 0 is the constant in eq.
(E.8))
such that
[Equation]
[Equation]
[Equation]
E.
[Equation]
(E
[Equation]
[Equation]
Here r can be understood as the effective radius of the said attraction ﬁeld.
Also, we ﬁx such ¯ (cid:15) small enough so that (let c L − = − f (cid:48)
( − L ) , c L + = − f (cid:48) ( − L ) ), we have [Equation]
[Equation]
Similar to the deﬁnition of ODE x η , let us consider the following construction of ODE ˜ x
η that can be understood as x η perturbed by l ∗ shocks.
Speciﬁcally, consider a sequence or real numbers
= t 1 < t 2 < t 3 < · · · < t l ∗ and real numbers w 1 , · · · , w l ∗ where | w j | ≤ b for each j .
Let t
=
( t 1 , · · · , t l ∗ ) , w = ( w 1 , · · · , w l ∗ ) .
Based on these two sequences and rate η > 0 , deﬁne ˜ x η ( t ; x ) as
[Equation]
Again, when η = 1 we drop the notational dependency on η and use ˜ x to denote the process.
Now from eq.
(E.28)eq.
(E.29) one can easily see the following fact: there exist constants ¯ t, ¯ δ > 0 such that ˜ x ( t l ∗ , 0; t , w ) / ∈ Ω (note that the starting point is 0 , the local minimum) only if (under the condition that | w j | ≤ b ∀ j )
[Equation]
The intuition is as follows: if the inter-arrival time between any of the l ∗ jumps is too long, then the path of ˜ x
η ( t ; x ) will drift back to the local minimum m i so that the remaining l ∗ − 1 shocks (whose sizes are bounded by b ) cannot overcome the radius r which is strictly larger than ( l ∗ − 1) b ; similarly, if size of any of the shocks is too small, then since all other jumps have sizes bounded by b , the shock created by the l ∗ i jumps will be smaller than ( l ∗ − 1) b
+ 100¯ (cid:15) , which is strictly less than r .
We ﬁx these constants ¯ t, ¯ δ throughout the analysis, and stress again that their values are dictated by the geometry of the function f , thus do not vary with the accuracy parameters (cid:15) and δ mentioned earlier.
In particular, choose ¯ δ such that ¯ δ < ¯ (cid:15) .
In our analysis below, (cid:15) > 0 will be a variable representing the level of accuracy in our analysis.
For instance, for small (cid:15) , the chance that SGD iterates will visit somewhere that is (cid:15) − close to s − or s + (namely, the boundary of the attraction ﬁled) should be small.
Consider some (cid:15) ∈ (0 , (cid:15) 0 ) where (cid:15) 0 is the constant in Assumption 1.
Due to eq.
(E.8)eq.
(E.9), one can see the existence of some g 0 > 0 , c 1 < ∞ such that
• | f (cid:48) ( x ) | ≥ g 0 for any x ∈ Ω such that | x − s − | > (cid:15) 0 , || x − s + | > (cid:15) 0 ; • Let ˆ t ODE ( x, η ) =
∆ min { t ≥ 0 : x η ( t, x ) ∈ [ − (cid:15), (cid:15) ] } be the time that the ODE returns to a (cid:15) − neighborhood of local minimum of Ω when starting from x .
As proved in Lemma 3.5 of Pavlyukevich (2005), for any x ∈ Ω such that | x − s − | > (cid:15), [Equation] x [Equation]
s
[Equation]
| >
[Equation]
, we have
[Equation]
and we deﬁne the function
[Equation]
In short, given any accuracy level (cid:15) , the results above give us an upper bound for how fast the ODE would return to a neighborhood of the local minimum, if the starting point is not too close to the boundary of this attraction ﬁeld Ω .
For the ﬁrst few technical results established below, we show that, without large jumps, the SGD iterates
X ηn ( x ) are unlikely to show signiﬁcant deviation from the deterministic gradient descent process y η n ( x ) deﬁned as
[Equation]
We are ready to state the ﬁrst lemma, where we bound the distance between the gradient descent iterates y ηn ( y ) and the ODE x η ( t, x ) when the initial conditions x, y are close enough.
Lemma E.6.
The following claim holds for all η > 0 : for any t > 0 , we have
[Equation]
where M ∈ (0 , ∞ ) is the constant in eq.
(E.10) .
Proof.
Deﬁne a continuous-time process y η ( s ; y ) =
∆ y η (cid:98) s (cid:99) ( y ) , and note that
[Equation]
Therefore, if we deﬁne function
[Equation]
from the fact | f (cid:48) ( · ) | ≤ M , one can see that | b ( u ) | ≤ ηM + | x − y | for any u ∈ [0 , 1) and | b (1) | ≤ 2 ηM + | x − y | .
In case that s > 1 , from the display above and the fact | f (cid:48)(cid:48) ( · ) | ≤ M , we now have
[Equation]
From Gronwall’s inequality (see Theorem 68, Chapter V of Protter (2005), where we let function α ( u ) be α ( u ) = | b ( u + 1) | ), we have
[Equation]
This concludes the proof.
Now we consider an extension of the previous Lemma in the following sense: we add perturbations to the gradient descent process and ODE, and show that, when both perturbed by l ∗ similar perturbations, the ODE and gradient descent process should still stay close enough.
Analogous to the deﬁnition of the perturbed ODE (cid:101) x
η in eq.
(E.32)-eq.
(E.34),
we can construct a process (cid:101)
Y η as a perturbed gradient descent process as follows.
For a sequence of integers
= t 1 < t 2 < · · · < t l ∗ (let t =
( t j ) j ≥ 1 ) and a sequence of real numbers (cid:101) w 1 , · · · , (cid:101) w l ∗ (let (cid:101)
w =
( (cid:101) w j ) j ≥ 1 ) and y ∈ R , deﬁne (for all n = 1 , 2 , · · · , t l ∗ ) the perturbed gradient descent iterates with gradient clipping at b and reﬂection at ± L as
[Equation]
w ))
+ l ∗ (cid:88)
j = 1 { n = t j } (cid:101) w j (cid:1)(cid:17) (E.41) with initial condition (cid:101)
y η 0 ( y ; t , (cid:101) w ) =
ϕ L (cid:0) y + ϕ b ( (cid:101) w 1 ) (cid:1) .
Corollary E.7.
Given any (cid:15) > 0 , the following claim holds for all sufﬁciently small η > 0 : for any x, y ∈ Ω , and sequence of integers t =
( t j ) l ∗ j =1 and any two sequence of real numbers w = ( w j ) l ∗ j =1 , (cid:101)
w =
( (cid:101) w j ) l ∗ j ≥ 1 such that
• | x − y | < (cid:15) ; • t 1 = 0 , and t j − t j − 1 ≤ 2¯ t/η for all j ≥ 1 where ¯ t is the constant in eq.
(E.35) ; • | w j − (cid:101) w j | < (cid:15) for all j ≥ 1 ;
then we have
[Equation]
where the constant ¯ ρ = (3 exp( ηM ¯ t ) + 3) l ∗ .
Proof.
Throughout this proof, ﬁx some η ∈ (0 , (cid:15)/ 2 M ) .
We will show that for any η in the range the claim would hold.
First, on interval [0 , t 2 ) , from Lemma E.6, one can see that (since 2 Mη < (cid:15) )
[Equation]
The at t = t 2 , by considering the difference between w 2 and (cid:101) w 2 , and the possible change due to one more gradient descent step (which is bounded by ηM < (cid:15) ), we have
[Equation]
Now we proceed inductively.
For any j = 2 , 3 , · · · , l ∗ − 1 , assume that
[Equation]
Then by focusing on interval [ t j , t j +1 ] and using Lemma E.6 again, one can show that
[Equation]
This concludes the proof.
In the next few results, we show that the same can be said for gradient descent iterates (cid:101) y n and the
SGD iterates X n .
Speciﬁcally, our ﬁrst goal is to show that before any large jump (see the deﬁnition in E.21), it is unlikely that the gradient descent process y ηn would deviate too far from X ηn .
Deﬁne the event
[Equation]
and recall that arrival times T ηj ( δ ) are deﬁned in eq.
(E.22).
As a building block, we ﬁrst study the case when the starting point x is close to the reﬂection boundary − L .
The takeaway from the next result is that the reﬂection operator hardly comes into play, since the
SGD iterates would most likely quickly move to somewhere far enough from ± L ; besides, throughout this procedure the SGD iterates would most likely stay pretty close to the corresponding deterministic gradient descent process.
Lemma E.8.
Given (cid:15) ∈ (0 , ¯ (cid:15)/ 9) , it holds for any sufﬁciently small (cid:15), δ, η > 0 that, if x ∈ [ − L, − L + ¯ (cid:15) ] and ρ 0 ( | x − y | + 9 (cid:15) ) <
¯ (cid:15) , then on event A ( n, η, (cid:15), δ ) we have
[Equation]
where (cid:101) T η escape ( x ) =
∆ min { n ≥ 0 : X ηn ( x ) >
− L + ¯ (cid:15) } and ρ 0 = ∆ exp (cid:16) 2 M ¯ (cid:15) 0 .
9 c L − (cid:17) is a constant that does not vary with our choice of (cid:15), δ, η .
Proof.
For any k < T η 1 ( δ ) , we know that Z k = Z ≤ δ k (thus η | Z k | < δ ).
Also, recall that | f (cid:48) ( x ) | ≤ M for any x ∈ Ω i .
Therefore, as long as η and δ are small enough, we will have that
[Equation]
so the gradient clipping operator in eq.
(E.12) has no effect when k < T η 1 ( δ ) , and in fact the only possible time for the gradient clipping trick to work is at T ηj ( δ ) .
Therefore, we can safely rewrite the
SGD update as
[Equation]
where each R k ≥
and it represents the push caused by reﬂection at − L .
First, choose (cid:15) small enough so that 9 (cid:15) < ¯ (cid:15) .
Next, based on eq.
(E.31)
we have the following lower bound:
[Equation]
Let (cid:101) t 0 ( x, (cid:15)
[Equation]
= ∆ min { n ≥ 0 :
X
[Equation]
( x ) ≥ − L + 2 (cid:15) } .
Due to the inequality above, we know that
[Equation]
One the other hand, given the current choice of (cid:15) , if we choose η and δ small enough, then using the same argument leading to eq.
(E.43), we will have
[Equation]
if (cid:101) t 0 ( x, (cid:15) ) ≥ 1 (namely x < − L + 2 (cid:15) ).
Let us inspect the two scenarios separately.
First, assume (cid:101) t 0 ( x, (cid:15) ) ≥ 1 .
For the deterministic gradient descent process y ηn ( y ) , we have the following bounds:
[Equation]
This gives us
[Equation]
, (cid:15) ) ∧ ( T η 1 ( δ ) − 1) . At time k = (cid:101) t 0 ( x, (cid:15) ) , due to previous bound on (cid:101) t 0 ( x, (cid:15) ) , we know that | X η (cid:101) t 0 ( x,(cid:15) )
( x ) − y η (cid:101) t 0 ( x,(cid:15) )
( y ) | ≤ | x − y | + 7 (cid:15) .
If n ∧ ( T η 1 ( δ ) − 1) ≤ (cid:101) t 0 ( x, (cid:15) ) then we have already shown the desired claim.
Otherwise, starting from time (cid:101) t 0 ( x, (cid:15) ) , due to the deﬁnition of event A ( n, η, (cid:15), δ ) in eq.
(E.42), we know
that the
SGD iterates X ηn ( x ) will not touch the boundary − L afterwards.
Therefore, by directly applying
Lemma E.5, and notice that | f (cid:48)(cid:48)
( x ) | ≤ M for any x ∈ [ − L, L ] and, we have
[Equation]
Indeed, it sufﬁces to use Lemma E.5 for the next (cid:100) 2¯ (cid:15)/ (0 . 9 ηc L − ) (cid:101) steps to show that | X ηk ( x ) − y ηk
( y ) | should be smaller than (cid:15) for the next (cid:100) 2¯ (cid:15)/ (0 .
9 ηc L − ) (cid:101) steps, while y ηk ( y ) will reach some where in ( − L + 2¯ (cid:15), − L + 3¯ (cid:15) ) within (cid:100) 2¯ (cid:15)/ (0 . 9 ηc L − ) (cid:101) steps so we must have
[Equation]
Lastly, in the case (cid:101) t 0 ( x, (cid:15) )
=
(which means x ≥ − L + (cid:15) ), we can use Lemma E.5 directly as we did above and establish the same bound.
This concludes the proof.
Obviously, a similar result can be shown if x is in the rightmost attraction ﬁeld ( s n min − 1 , L ] and the approach is identical.
We omit the details here.
In the next Lemma, we consider the scenario where the starting point x is far enough from the boundaries.
Lemma E.9.
Given any (cid:15) > 0 , the following holds for all sufﬁciently small η > 0 : for any x, y ∈ Ω and positive integer n such that | x − L | > 2 (cid:15), | x + L | > 2 (cid:15), | x − s − | > 2 (cid:15), | x − s + | > 2 (cid:15) and | x − y | < (cid:15) 2exp( ηMn ) , on event A [Equation] n
[Equation]
η,
[Equation]
we have
[Equation]
Proof.
For sufﬁciently small η , we will have that the (deterministic) gradient descent iterates | y ηn | is monotonically decreasing in n , which ensures that y ηn always stays in the range that are at least (cid:15) − away from ± L or s − , s + .
We now show that the claim holds for any such η .
On event (cid:110) | y ηj ( y ) | ∈ Ω , | X ηj ( x ) | ∈
Ω ∀ j = 1 , 2 , · · · , n ∧ ( T η 1 ( δ ) − 1) (cid:111) , we are able to apply
Lemma E.5 inductively for any m ∈ [ n ] and obtain that
[Equation]
and conclude the proof.
The reason to apply the Lemma inductively for m = 1 , 2 , · · · , n , instead of directly at step n , is to ensure that SGD iterates X ηn would not hit the boundary ± L (so the reﬂection operator would not come into play on the time interval we are currently interested in), thus ensuring that Lemma E.5 is applicable.
Similar to the extension from Lemma E.6 to Corollary E.7, we can extend Lemma E.9 to show that, if we consider the a gradient descent process that is only perturbed by large noises, then it should stay pretty close to the SGD iterates X ηn .
To be speciﬁc, let
[Equation]
be a gradient descent process (with gradient clipping at threshold b ) that is only shocked by large noises in ( Z n ) n ≥ 1 .
The next corollary can be shown by an approach that is identical to Corollary E.
(namely, inductively repeating Lemma E.9 at each jump time) so we omit the details here.
Corollary E.10.
Given any (cid:15) > 0 , the following holds for any sufﬁciently small η > 0 :
For any | x | < 2 (cid:15) , on event
A 0 ( (cid:15), η, δ ) ∩ B 0 ( (cid:15), η, δ ) , we have
[Equation]
where
[Equation]
and (cid:101) ρ ∈ (0 , ∞ ) is a constant that does not vary with η, δ, (cid:15) .
The next two results shows that the type of events A ( n, η, (cid:15), δ ) deﬁned in eq.
(E.42) is indeed very likely to occur, especially for small (cid:15) .
For clarity of the presentation, we introduce the following deﬁnitions that are slightly more general than the small and large jumps deﬁned in eq.
(E.20)eq.
(E.21) (for any c > 0 )
[Equation]
Lemma E.11.
Deﬁne functions u ( η ) =
δ/η 1 − ∆ , v ( η ) =
(cid:15)η (cid:101) ∆ with (cid:15), δ > 0 .
If real numbers ∆ , (cid:101) ∆ , β, (cid:15), δ and positive integers j, N are such that the following conditions hold:
[Equation]
all η > 0 sufﬁciently small ,
[Equation]
E
[Equation]
[Equation]
then
[Equation]
as η ↓ 0 .
Proof.
From the stated range of the parameters, we know that
[Equation]
so we are able to ﬁnd γ ∈ (0 , 1) small enough such that
[Equation]
Fix such γ ∈ (0 , 1) for the rest of the proof, and let n ( η ) (cid:44) (cid:100) (1 /η ) β (cid:101) , I (cid:44) # (cid:110) i ∈ [ n ( η )] : | Z ≤ u ( η ) i | > u ( η ) 1 − γ (cid:111) .
Then
[Equation]
Note that since | Z ≤ u ( η ) i |
[Equation]
η
[Equation]
2 η
[Equation]
where the last inequality follows from eq.
(E.52).
First
[Equation]
since E Z 1 =
[Equation]
we have
[Equation]
Therefore
[Equation]
for all η > 0 that are sufﬁciently small,
[Equation]
− ∆)(1 − 2 γ ) ≤ 2 η ( α − 1)(1 − ∆)(1 − 2 γ ) − β ≤ (1 /η ) (1 − ∆)(1 − 2 γ ) due to eq.
(E.53) ≤ v ( η ) 4 η due to (cid:101)
∆ / 2 ≤ ∆ in eq. (E.50) and 1 − γ < 1 .
If we let Y n = Z ≤ u ( η ) 1 − γ n − E Z ≤ u ( η ) 1 − γ n and plug the bound above back into eq. (E.55), then (for all η > 0 that are sufﬁciently small) [Equation] E
[Equation]
56) where the last inequality is obtained from Bernstein’s inequality.
Note that from Karamata’s theorem,
[Equation]
Now note that
• In case that α < 2 , for all η > 0 that are sufﬁciently small, we have (using eq.
(E.50))
2 (cid:0) n ( η ) − i (cid:1) E | Y 1 | 2 ≤ (1 /η ) β
+(2 − α )(1 − ∆)
[Equation]
(1 /η ) 2(1 − ∆)
⇒
[Equation]
[Equation]
∆ [Equation] [Equation] ∆ ; • In case that α ≥ 2 , for all η > 0 that are sufﬁciently small, 2 (cid:0) n (
[Equation]
[Equation]
(cid:1) E | Y 1 | 2 < 1 /η β + ∆
and we know that β
+ ∆2 < 2 − 2 (cid:101) ∆ due to 2 − β > 2∆ and 2 (cid:101)
∆ ≤ ∆ (see eq.
(E.48)- eq.
(E.50)); • Since γ > 0 and 2 (cid:101) ∆ ≤ ∆ , we know that
[Equation]
Therefore
[Equation]
it is easy to see that the R.H.S. of eq.
(E.56) decays at a geometric rate as η tends to zero, hence o ( η N ) .
On the other hand,
[Equation]
which is regularly varying w
[Equation] r [Equation] t. η with index (cid:0) α (1 − γ )(1 − ∆) − β (cid:1) j .
Therefore, for all η >sufﬁciently small,
II
) ≤
[Equation]
(cid:0)
[Equation]
η N
due to eq.
(E.54) .
Collecting results above, we have established that
[Equation]
The conclusion of the lemma now follows from Etemadi’s theorem.
Now consider the following setting.
Let us ﬁx some positive integer N and β ∈ (1 , 2 ∧ α ) .
Then we can ﬁnd some positive integer j such that ( α − β ) j > N .
Meanwhile, given any (cid:15) > 0 , we will have (cid:15) − jδ ≥ (cid:15)/ 2 for all δ > 0 sufﬁciently small.
Therefore, by applying Lemma E.11 with ∆ =
(cid:101) ∆ =(hence u ( η ) =
δ/η, v ( η ) =
(cid:15) ) and β, j, N, (cid:15), δ as described here, we immediately get the following result.
Lemma E.12.
Given any β ∈ (1 , α ∧ 2) , (cid:15) > 0 , and N > 0 , the following holds for any sufﬁciently small δ > 0 :
[Equation]
as η ↓ 0 .
Using results and arguments above, we are able to illustrate the typical behavior of the SGD iterates
X ηn in the following two scenarios.
First, we show that, when starting from most parts in the attraction ﬁeld Ω , the SGD iterates X ηn will most likely return to the neighborhood of the local minimum within a short period of time without exiting Ω .
Given that there are only ﬁnitely many attraction ﬁelds on f , it is easy to see that the key technical tool Lemma E.1 follows immediately from the next result.
Lemma E.13.
For sufﬁciently small (cid:15) > 0 , the following claim holds:
[Equation]
where the stopping time involved is deﬁned as
[Equation]
the function ˆ t ( (cid:15) ) is deﬁned in eq.
(E.38) , and the function ρ ( · ) is deﬁned as ρ ( (cid:15) )
=
[Equation]
Proof.
Throughout, we only consider (cid:15) small enough so that Lemma E.8 could hold.
Also, ﬁx some
N > 0 , ∆ α ∈ (0 , α − 1) and β ∈ (1 , α ) .
Let σ ( x, η ) =
∆ min { n ≥ 0 : X ηn / ∈ Ω } .
Without loss of generality, we assume Ω =
[ − L, s + ) and x < 0 (so reﬂection at − L ) is a possibility.
Any other case can be addressed similarly as shown below.
From Lemma E.4 and the regular varying nature of H ( · ) , we have, for any (cid:15), δ > 0 ,
[Equation]
for any sufﬁciently small η .
Let (cid:101) T η escape ( x ) be the stopping time deﬁned in Lemma E.8.
From eq.
(E.44),eq.
(E.45),eq. (E.57) and
Lemma E.12, we know that
[Equation]
for all sufﬁciently small η .
Next, we focus on x ∈ Ω such that | x − s − | ∧ | x − s + | > (cid:15) and x ≥ − L + ¯ (cid:15) .
We start by considering the time it took for the (deterministic) gradient descent process y ηn ( x ) to return to [ − 1 .
5 (cid:15), 1 . 5 (cid:15) ] .
From the deﬁnition of ˆ t ( (cid:15) ) in eq.
(E.38) and Lemma E.6, we know that for η small enough such that η exp(2 M ˆ t ( (cid:15) ))
< 0 .
5 (cid:15) , we have
[Equation]
Now consider event A ( (cid:100) (1 /η ) β (cid:101) , η,
[Equation]
4exp [Equation] 2 M ˆ t ( (cid:15) )) , δ ) (see deﬁnition in eq.
(E.42)).
From Lemma
E.12, we know that for any sufﬁciently small δ , we have
[Equation]
Combining this result with eq.
(E.57)eq.
(E.59) and Lemma E.9, we get sup
[Equation]
for any sufﬁciently small η .
To conclude the proof, we only to combine strong Markov property (at (cid:101) T η escape ) with bounds in eq.
(E.58)eq.
(E.61).
In the next result, we show that, once entering a (cid:15) − small neighborhood of the local minimum, the
SGD iterates will most likely stay there until the next large jump.
Lemma E.14.
Given N 0 > 0 , the following claim holds for any (cid:15), δ > 0 that are sufﬁciently small:
[Equation]
as η ↓ 0 .
Proof.
Fix (cid:15) small enough such that 3 (cid:15) < (cid:15)
(see Assumption 1 for the constant (cid:15) 0 ).
Also, ﬁx some ∆ α ∈ (0 , 1) , β ∈ (1 , α ) , N > α + ∆ α − β + N 0 .
Due to Lemma E.12, for any δ sufﬁciently small, we will have
[Equation]
Fix such δ > 0 .
We now show that the desired claim is true for the chosen (cid:15), δ .
First of all, from Lemma E.3, we know the existence of some θ > 0 such that
[Equation]
E [Equation] [Equation] Next, let us zoom in on the ﬁrst (cid:100) (1 /η ) β (cid:101) SGD iterates.
For any η small enough, we will have y ηn ( x ) ∈ [ − 2 (cid:15), 2 (cid:15) ] for any n ≥ 1 and y η (cid:100) (1 /η ) β (cid:101) ( x ) ∈ [ − (cid:15), (cid:15) ] given x ∈ [ − 2 (cid:15), 2 (cid:15) ] .
From now on we only consider such η .
Due to Lemma E.9, we know that on event (cid:8) max j =1 , 2 , ··· , (cid:100) (1 /η ) (cid:101) η | Z ≤ ,δ+ · · · + Z ≤ ,δ j | > (cid:15) exp(2 M ) (cid:9) , we have
[Equation]
x ) |
[Equation]
(cid:15)
[Equation]
n ≤ (cid:100)
[Equation]
/η β (cid:101) ∧
( T
[Equation]
( δ ) − 1) and on event (cid:8) max j =1 , 2 , [Equation]
, (cid:100) (1 /η ) (cid:101)
[Equation] [Equation] M ) (cid:9) ∩ { T η 1 ( δ ) > (cid:100) (1 /η ) β (cid:101)} , we have X ηT η 1 ( δ ) ( x ) ∈
[ − 2 (cid:15), 2 (cid:15) ] .
Now by repeating the same argument inductively for (cid:100) 1 /η α +∆ α − β (cid:101) times, we can show that on event {∀ i = 1 , 2 , · · · , (cid:100)
[Equation]
we have | X ηn ( x ) | ≤ 3 (cid:15) ∀ n ≤ 1 /η α +∆ α ∧ ( T η 1 ( δ ) − 1) .
To conclude the proof, we only need to combine this fact with eq.
(E.62).
We introduce a few concepts that will be crucial in the analysis below.
Recall the deﬁnition of perturbed ODE (cid:101) x η in eq.
(E.32)-eq.
(E.34) (note that we will drop the notational dependency on learning rate η when we choose η = 1 ).
Consider the deﬁnition of the following two mappings from where w = ( w 1 , · · · , w l ∗ ) is a sequence of real numbers and t =
( t 1 , t 2 , · · · , t l ∗ ) with 0 =
t 1 < t 2 < t 3 < · · · as
[Equation]
Next, deﬁne sets (for any (cid:15) ∈ ( − ¯ (cid:15), ¯ (cid:15) ) )
[Equation]
We add a few remarks about the two types of sets deﬁned above.
• Intuitively speaking, E ( (cid:15) ) contains all the perturbations (with times and sizes) that can send the ODE out of the current attraction ﬁeld (allowing for some error with size (cid:15) ); • From the deﬁnition of ¯ t, ¯ δ in eq.
(E.35)eq.
(E.36) and Corollary
E.7, one can
easily see that for a ﬁxed (cid:15) ∈ ( − ¯ (cid:15), ¯ (cid:15) ) , [Equation] • Lastly, E ( (cid:15) ) are open sets due to f ∈ C 2 .
Use Leb + to denote the Lebesgue measure restricted on [0 , ∞ ) , and deﬁne (Borel) measure ν α with density on R \{ 0 } :
[Equation]
where α > 1 is the regular variation index for the distribution of Z 1 and p − , p + ∈ (0 , 1) are constants in Assumption 2.
Now we can deﬁne a Borel measure µ on R l ∗ × (cid:16) R + (cid:17) l ∗ − 1 as product measure
[Equation]
Due to remarks above, one can see that for (cid:15) ∈ ( − ¯ (cid:15), ¯ (cid:15) ) , we have µ ( E ( (cid:15) )) <
∞ .
We are now ready to analyze a speciﬁc type of noise Z n .
Deﬁnition E.1.
For any n ≥ 1 and any (cid:15) ∈ ( − ¯ (cid:15), ¯ (cid:15) ) , δ ∈ (0 , b ∧ ¯ δ ) , η > 0 , we say that the jump Z n has ( (cid:15), δ, η ) -overﬂow if
• η | Z n | > δ ; •
In the set { n + 1 , · · · , n + 2 (cid:100) l ∗ ¯ t/η (cid:101)} , there are at least ( l ∗ − 1) elements (ordered as n < t 2 < t 3 < · · · < t l ∗ ) such that η | Z t i | > δ for any i = 2 , · · · , l ∗ • Let t 1 = n and t (cid:48)
i =
t i − t
i − 1 for any i = 2 , · · · , l ∗ , w i = ηZ
i for any i = 1 , · · · , l ∗ , for real sequence w = ( w 1 , w 2 , · · · , w l ∗ ) and a sequence of positive number t =
( η ( t i − n ))
l ∗ i =2 , we have ( w
[Equation]
E ( [Equation]
Moreover, if Z n has ( (cid:15), δ, η ) − overﬂow, then we call h ( w , t ) as its ( (cid:15), δ, η ) − overﬂow endpoint .
Due to the iid nature of ( Z j ) j ≥ 1 , let us consider an iid sequence ( V j ) j ≥ 0 where the sequence has the same law of Z 1 .
Note that for any ﬁxed n ≥ 1 , the probability that Z n has ( (cid:15), δ, η ) - overﬂow is equal to the probability that V 0 has ( (cid:15), δ, η ) -overﬂow.
More speciﬁcally, we know that P ( η | V 0 | > δ )
=
H ( δ/η ) , and now we focus on conditional probability admitting the following form:
[Equation]
For any open interval A = ( a 1 , a 2 ) such that A ∩ [ s − + ¯ (cid:15), s + − ¯ (cid:15) ]
= ∅ , we also deﬁne p ( (cid:15), δ, η ; A ) =
P (cid:16) V 0 has ( (cid:15), δ, η ) -overﬂow
and the endpoint is in A (cid:12)(cid:12)(cid:12) η | V 0 | > δ (cid:17) .
[Equation]
Lemma E.15.
For any (cid:15) ∈ ( − ¯ (cid:15), ¯ (cid:15) ) , δ ∈ (0 , b
∧ ¯ δ ) , and any open interval A =
( a 1 , a 2 ) such that | a 1 | ∧ | a 2 | > r − ¯ (cid:15) and | a 1 | (cid:54) =
L, | a 2 | (cid:54) =
L , we have
[Equation]
where µ is the measure deﬁned in eq.
(E.65) , and p ( · , · , · ; A ) is the conditional probability deﬁned in eq.
(E.67) .
Proof.
Let us start by ﬁxing some notations.
Let T 1 = 0 , and deﬁne stopping times T j = min { n >
T j − 1 : η | V n | > δ } and inter-arrival times T (cid:48)
j = T j − T
j − 1 for any j ≥ 1 , and large jump W j = V T j for any j ≥ 0 .
Note that: ﬁrst of all, the pair ( T (cid:48)
i , W i ) is independent of ( T (cid:48)
j , W j ) whenever i (cid:54) =
j ; besides, W j and T (cid:48)
j are independent for all j ≥ 1 .
Deﬁne the following sequence (of random elements) w = ( w 1 , · · · , w l ∗ ) and t =
( t 1 , · · · , t l ∗ ) by w j = ηW j , t j = ηT j .
If V 0 has ( (cid:15), δ, η ) -overﬂow, then the following two events must occur:
• T (cid:48)
j ≤ 2¯ t/η for any j = 2 , · · · , l ∗ ; • η | W j | > ¯ δ for any j = 1 , 2 , · · · , l ∗ ; • ( w , t )
∈ E ( (cid:15) )
Therefore, for sufﬁciently small η , we now have
[Equation]
E
[Equation]
where Q η,δ is the Borel-measurable probability measure on R l ∗ × (cid:16) R + (cid:17) l ∗ − 1 induced by a sequence of independent random variables ( W ↑ 1 ( η, δ ) , · · · , W ↑ l ∗ ( η, δ ) , T ↑ 2 ( η, δ ) , · · · , T ↑ l ∗ ( η, δ ))
such that
• For any i
= 1 , · · · , l ∗ , the distribution of W ↑ i ( η, δ ) follows from P (cid:16) ηW 1 ∈ · (cid:12)(cid:12)(cid:12) η | W l ∗ | > δ (cid:17) ;
• For any i = 2 , · · · , l ∗ , the distribution of T ↑ i ( η, δ ) follows from P (cid:16) ηT 1 ∈ · (cid:12)(cid:12)(cid:12) ηT 1 ≤ 2¯ t (cid:17) ; • Q η,δ ( · ) =
P (cid:16) ( ηW ↑ 1 ( η, δ ) , · · · , ηW ↑ l ∗ ( η, δ ) , ηT ↑ 2 ( η, δ ) , · · · , η (cid:80) l ∗ j =2 T ↑ j ( η, δ ))
∈ · (cid:17) .
Now we study the weak convergence of W ↑ 1 , T ↑ 1 :
• Due to the regularly varying nature of distribution of Z (hence for W 1 ) we know
that
for
any
x > δ
[Equation]
therefore, W ↑ 1 ( η, δ ) weakly converges to a (randomly signed) Pareto RV that admits the density
[Equation]
as η ↓
• For any x ∈ [0 , 2¯ t ] , since lim η ↓ 0 (cid:98) x/η (cid:99) H ( δ/η ) = 0 , it is easy to show that
[Equation]
therefore, we have (for any x ∈ (0 , 2¯ t ] )
[Equation]
as η ↓ 0 , which implies that T ↑ 1 converges weakly to a uniform RV on [0 , 2¯ t ] .
Let us denote the weak limit of measure Q η,δ as µ δ, 2¯ t .
In the discussion before the Lemma we have shown that, for any ( w , t ) ∈
E ( (cid:15) ) (with δ ∈ (0 , ¯ δ ) )
, we have | w i | ≥ ¯ δ and | t (cid:48)
i | ≤ 2¯ t ; since we require δ < ¯ δ , by deﬁnition of measures µ and µ δ, 2¯ t we have
[Equation]
For simplicity of notations, we let E ( (cid:15), A ) =
∆ E ( (cid:15) ) ∩ h − 1 ( A ) .
By deﬁnition of the set E ( (cid:15) ) , we have (recall that A is an open interval ( a 1 , a 2 ) that does not overlap with [ s − + ¯ (cid:15), s + − ¯ (cid:15) ] )
[Equation]
where F ( (cid:15), a 1 , a 2 ) =
∆ (cid:0) ( −∞ , s − − (cid:15) ) ∪ ( s + + (cid:15), ∞ ) (cid:1) ∩ ( a 1 , a 2 ) .
Meanwhile, it is easy to see that h is a continuous mapping, hence
[Equation]
Fix some s with s (cid:54) = ± L, | s | > ( l ∗ − 1) b
+ ¯ (cid:15) .
For any ﬁxed real numbers t 2 , · · · , t l ∗ − 1 , w 1 , · · · , w l ∗ , if h ( w 1 , · · · , w l ∗ , t 2 , · · · , t l ∗ − 1 , t ) =
s , then since (cid:101) x
( t l ∗ − 1 , 0; w 1 , · · · , w l ∗ , t 2 , · · · , t l ∗ − 1 , t ) ∈ [ s − b, s + b ] , due to Assumption 1 (in particular, there is no point x on this interval with | f (cid:48) ( x ) | ≤ c), there exists at most one possible t that makes
h ( w 1 , · · · , w l ∗ , t 2 , · · · , t l ∗ − 1 , t ) =
s .
Therefore, let W ∗ j be iid RVs from law ν
α,δ deﬁned above, and ( T ∗ , (cid:48)
j ) j ≥ 2 be iid RVs from Unif [0 , ¯2 t ] , T ∗ 0 = 0 , T ∗ k
= (cid:80) kj =2 T ∗ , (cid:48)
j .
By conditioning on all W ∗ j and all T ∗ , (cid:48) 2 , · · · , T ∗ , (cid:48)
l ∗ − 1 , we must have
[Equation]
which implies
[Equation]
hence
[Equation]
By Portmanteau theorem (see Theorem 2.1 of Billingsley (2013))
we have
[Equation]
Collecting the results we have and using eq.
(E.68) [Equation] we can see that
[Equation]
Fix some κ > 1 .
From Lemma E.4 and the regularly varying nature of function H , we get
[Equation]
Due to the arbitrariness of κ > 1 , we have established that
[Equation]
The lower bound can be shown by an argument symmetric to the one for upper bound.
The following result is an immediate corollary of Lemma E.15.
Corollary E.16.
For any (cid:15) ∈ ( − ¯ (cid:15), ¯ (cid:15) ) , δ ∈ (0 , b
∧ ¯ δ ) , we have
[Equation]
where µ is the measure deﬁned in eq.
(E.65) , and p ( · , · , · ) is the conditional probability deﬁned in eq.
(E.66) .
Deﬁne the following stopping times:
[Equation]
E. [Equation] σ indicate the time that the iterates escape the current attraction ﬁeld, while R denotes the time the
SGD iterates return to a small neighborhood of the local minimum after ﬁrst exit from this small neighborhood.
In the next few results, we study the probability of several atypical scenarios when
SGD iterates make attempts to escape Ω or return to local minimum after the attempt fails.
First, we show that, when starting from the local minimum, it is very unlikely to escape with less than l ∗ big jumps.
Lemma E.17.
Given (cid:15) ∈ (0 , ¯ (cid:15) ) , N > 0 , the following claim holds for any sufﬁciently small δ > 0 :
[Equation]
as η ↓ 0 .
Proof.
Based on the given (cid:15) > 0 , ﬁx some (cid:101) (cid:15) =
[Equation]
4exp(2 M ˆ t ( (cid:15) )) .
Recall the deﬁnition of ˆ t ( (cid:15) ) in eq.
(E.38).
First, using Lemma E.14, we know that for sufﬁciently small δ , we have
[Equation]
where
[Equation]
Deﬁne event
[Equation] [Equation]
From Lemma E.12, we know that for sufﬁciently small δ > 0 ,
[Equation]
A × 2 ( (cid:101) (cid:15), δ, η ) (cid:17) =
o ( η N ) .
(E.73)
From now on, we only consider such δ that eq.
(E.72)eq.
(E.73) hold.
On event (cid:16) A × 1 ∪ A × 2 (cid:17) c ∩ { σ ( η ) <
R ( (cid:15), η ) } ∩ { σ ( η ) > T η 1 ( δ ) } , we must have σ ( η ) > T η 1 ( δ ) and
T η 2 ( δ ) ∧ σ ( η ) − T η 1 ( δ ) < 2ˆ t ( (cid:15) )
/η.
Otherwise, due to Lemma E.6 and E.9, we know that at step (cid:101) t = T η 1 ( δ ) + (cid:98) ˆ t ( (cid:15) )
/η
(cid:99) , we have | X η (cid:101) t | < 2 (cid:15), and | X ηn | ≤ ¯ (cid:15) ∀ n ≤ (cid:101) t for any sufﬁciently small η .
By repeating this argument inductively, we obtain the following result: deﬁne
[Equation]
then on event (cid:16) A × 1 ∪ A × 2 (cid:17) c ∩ { σ ( η ) <
R ( (cid:15), η ) , σ ( η ) < T ηl ∗ ( δ ) } , we must have
[Equation]
< [Equation] ˆ [Equation]
( (cid:15) [Equation]
[Equation] [Equation]
J.
(E.74) Furthermore, using this bound and Lemma E.9, we know that on event (cid:16) A × 1 ∪ A × 2 (cid:17) c ∩ { σ ( η ) < R ( (cid:15), η ) , σ ( η ) < T
ηl ∗ ( δ ) } , • | X ηT ηj ( δ ) | ≤ | X ηT ηj − 1 ( δ ) | + b + (cid:15) + ¯ (cid:15) for all j = 2 , 3 , J − 1 , • | X ησ ( η ) | ≤ | X ηT ηJ − 1 ( δ ) | + (cid:15) + ¯ (cid:15)
However, this implies | X η σ ( η ) | ≤ l ∗ (¯ (cid:15) + (cid:15) )
+ ( l ∗ − 1) b < r and contradicts the deﬁnition of σ ( η ) .
In summary,
[Equation]
The following two results follow immediately from the proof above, especially the inductive argument leading to bound eq.
(E.74), and we state them without repeating the deatils of the proof.
Corollary E.18.
Given (cid:15) ∈ (0 , ¯ (cid:15) ) , N > 0 , the following claim holds for any sufﬁciently small δ > 0 : sup x ∈ [ − 2 (cid:15), 2 (cid:15) ]
[Equation]
[Equation]
= o
[Equation]
N ) as η ↓ 0 .
Corollary E.19.
Given (cid:15) ∈ (0 , ¯ (cid:15) ) , N > 0 , the following claim holds for any sufﬁciently small δ > 0 :
[Equation]
as η ↓ 0 .
In the next result, we show that, if the inter-arrival time between some large jumps are too long, or some large jumps are still not large enough , then it is very unlikely that the SGD iterates could escape at the time of l ∗ − th large jump (or even get close enough to the boundary of the attraction ﬁeld).
Lemma E.20.
Given (cid:15) ∈ (0 , ¯ (cid:15) ) and any N > 0 , the following holds for all δ > 0 that are sufﬁciently small:
[Equation]
where
[Equation]
[Equation]
Proof.
Let A × 1 , A × 2 be the events deﬁned in the proof of Lemma E.17.
Based on the given (cid:15) > 0 , ﬁx some (cid:101) (cid:15) =
[Equation]
Let J = min { j = 2 , 3 , · · · : T ηj ( δ ) − T ηj − 1 ( δ ) > ¯ t/η } .
On event (cid:16) A × 1 ( (cid:15), δ, η ) ∪ A × 2 ( (cid:101) (cid:15), δ, η ) (cid:17) c ∩{ J ≤ l ∗ } , from Lemma E.6 and E.9 and the deﬁnition of constant ¯ t , we know that
• | X ηT ηj ( δ ) | ≤ | X ηT ηj − 1 ( δ )
| + b + (cid:15) + ¯ (cid:15) for all j = 2 , 3 , J − 1 ; • | X ηT ηJ ( δ ) | ≤ 2¯ (cid:15) • | X ηn | <
s − ¯ (cid:15) ∀ n ≤ T η J ( δ )
Now starting from step T ηJ ( δ ) , by using Lemma E.6 and E.9 again one can see that
• | X ηT ηj ( δ ) | ≤ | X ηT ηj − 1 ( δ )
| + b + (cid:15) + ¯ (cid:15) for all j = J + 1 , · · · , l ∗
Combining these results, we have that | X ηT ηl ∗ | < r − ¯ (cid:15) on event (cid:16)
A × 1 ( (cid:15), δ, η ) ∪ A × 2 ( (cid:101) (cid:15), δ, η ) (cid:17) c ∩ { J ≤ l ∗ } .
Next, deﬁne J (cid:48)
=
min { j = 1 , 2 , · · · ; η | W ηj ( δ ) | ≤ ¯ δ } .
Similarly, on event (cid:16) A × 1 ( (cid:15), δ, η ) ∪
A × 2 ( (cid:101) (cid:15), δ, η ) (cid:17) c ∩ { J > l ∗ } ∩ { J (cid:48) ≤ l ∗ } , using Lemma E.6 and E.9 again one can see that
• | X ηT ηj ( δ ) | ≤ | X ηT ηj − 1 ( δ )
| + b + (cid:15) + ¯ (cid:15) for all j = 1 , 2 , · · · , l ∗ , j (cid:54) =
J (cid:48) ; • | X η T ηJ ( δ ) | ≤ | X η T ηJ − 1 ( δ ) | + ¯ δ + (cid:15) + ¯ (cid:15) for all j = 1 , 2 , · · · , l ∗ , j (cid:54) =
J (cid:48) .
Since ¯ δ ∈ (0 , ¯ (cid:15) ) , we have | X ηT ηl ∗ | < r − ¯ (cid:15) on this event.
In summary, the following bound
[Equation]
holds for any δ that is sufﬁciently small, which is established in Lemma E.17.
This conclude the proof.
In the next lemma, we show that, starting from the local minimum, it is unlikely that the SGD iterates will be right at the boundary of the attraction ﬁeld after l ∗ large jumps.
Recall that there are n min attraction ﬁelds on f , and excluding s 0 =
−∞ , s n min = ∞ the remaining points s 1 , · · · , s n min − 1 are the boundaries of the attraction ﬁelds.
Lemma E.21.
There exists a function Ψ( · ) : R + (cid:55)→ R + satisfying lim (cid:15) ↓ 0 Ψ( (cid:15) )
=
such that the following claim folds.
Given (cid:15) ∈ (cid:0) 0 , ¯ (cid:15)/ (3¯ ρ + 3 (cid:101) ρ + 9) (cid:1) , we have
[Equation]
for all δ sufﬁciently small, where ¯ ρ and (cid:101) ρ are the constants deﬁned in Corollary E.7 and E.10, and the event is deﬁned as
[Equation]
( δ ) ≤ σ ( η ) ∧ R ( (cid:15), η ) (cid:111)
∩ (cid:110) ∃ k ∈ [ n min − 1] such that X ηT ηl ∗ ( δ ) ∈
[ s k − (cid:15), s k + (cid:15) ] (cid:111) .
Proof.
Let A × 1 , A × 2 be the events deﬁned in the proof of Lemma E.17.
Based on the given (cid:15) > 0 , ﬁx some (cid:101) (cid:15) =
(cid:15) 4exp(2 M ˆ t ( (cid:15) )) .
Fix some N > αl ∗ .
Choose δ small enough so that claim in Lemma E.20 holds for the (cid:15) prescribed.
Using the same arguments in Lemma E.20, we have the following inclusion of events:
[Equation]
Therefore, on event B × 3 ( (cid:15), δ, η ) ∩ (cid:16) A × 1 ( (cid:15), δ, η ) ∪ A × 2 ( (cid:101) (cid:15), δ, η ) (cid:17) c , we can apply Corollary E.7 and E. and conclude that Z T η 1 ( δ ) has ( − ¯ (cid:15), δ, η ) − overﬂow, and its ( − ¯ (cid:15), δ, η ) − overﬂow endpoint lies
[Equation]
for some k ∈ [ n min − 1] .
Using Lemma E.15 and Corollary E.16, we have that (for any sufﬁciently small η )
[Equation]
where ˆ (cid:15) =
3(¯ ρ + (cid:101) ρ + 3) (cid:15) .
Besides, as established in the proof of Lemma E.17, we have
[Equation]
for all sufﬁciently small δ .
In conclusion, we only need to choose Ψ
[Equation]
To conclude the proof, just note that by combining the continuity of measure with the conditional probability argument leading to eq.
(E.69), we can show that lim (cid:15) ↓ 0 Ψ( (cid:15) )
= 0 .
Lastly, we establish the lower bound for the probability of the most likely way for SGD iterates to exit the current attraction ﬁeld: making l ∗ large jumps in a relatively short period of time.
Recall that ¯ (cid:15) is the ﬁxed constant in eq.
(E.26)-eq.
(E.29).
Lemma E.22.
Given (cid:15) ∈ (0 , ¯ (cid:15)/ 3) , it holds for any sufﬁciently small δ > 0 such that
[Equation]
where the event is deﬁned as
[Equation]
and the constant
[Equation]
is strictly positive and does not vary with (cid:15), δ .
Proof.
Let A × 1 , A × 2 be the events deﬁned in the proof of Lemma E.17.
Fix some N such that
N > αl ∗ .
Based on the given (cid:15) > 0 , ﬁx some (cid:101) (cid:15) =
(cid:15) 4exp(2 M ˆ t ( (cid:15) )) .
We only consider δ < M .
Furthermore, choose δ small enough so that eq.
(E.72) and eq.
(E.73) hold for the chosen N and (cid:15) .
Also, we only consider η small enough so that ηM <
b ∧ ¯ (cid:15) .
Due to eq.
(E.26)-eq.
(E.29), we can, without loss of generality, assume that r = s + , and in this case we will have
[Equation]
Under this assumption, we will now focus on providing a lower bound for the following event that describes the exit from the right side of Ω (in other words, by crossing s + )
[Equation]
First, deﬁne event
[Equation]
and observe some facts on event
A
◦
( δ,
[Equation]
[Equation]
[Equation] [Equation] [Equation] • | X ηk | ≤ 3 (cid:15) ∀ n < T η 1 ( δ ) ; (due to A × 1 not occurring) • X ηT η 1 ( δ ) ∈
[ b − 3 (cid:15), b + 3 (cid:15) ] ; (due to W η 1 ≥ 2 b and the effect of gradient clipping at step T η 1 , as well as the fact that X ηT η 1 − 1 ∈ [ − 3 (cid:15), 3 (cid:15) ] from the previous bullet point) • Due to | f (cid:48) ( · ) | ≤ M and δ < M , one can see that (for any n ≥ 1 )
[Equation]
this provides an upper bound for the change in SGD iterates at each step
[Equation]
and gives us
[Equation]
where we also used T
[Equation]
• Therefore, at the arrival time of the second large jump, we must have X η T η 2 ( δ ) ≥ 2 b − 3 (cid:15) − ¯ (cid:15) ; • By repeating the argument above inductively, we can show that
for all j = 1 , 2 , · · · , l ∗ )
[Equation]
In particular, we know that X ηn ∈ Ω for any n < T ηl ∗ (so the exit does not occur before T ηl ∗ ), and at the arrival of the l ∗ − th large jump, we have (using 3 (cid:15) < ¯ (cid:15) )
[Equation]
In summary, we have shown that
[Equation]
To conclude the proof, just notice that (for sufﬁciently small η )
[Equation]
due to eq.
(E.72) and eq.
[Equation]
due to Lemma E.
[Equation]
for all η sufﬁciently small, due to H ∈ RV − α
[Equation]
In order to present the main result of this section, we need to take into account the loss landscape outside of the current attraction ﬁeld Ω .
Recall that there are n min attraction ﬁelds on f .
For all the attraction ﬁelds different from Ω , we call them ( (cid:101) Ω k ) n min − 1 k
= where, for each k ∈ [ n min − 1] , the attraction ﬁeld (cid:101)
Ω k
=
( s − k , s + k ) with the corresponding local minimum located at (cid:101) m k .
Also, recall that σ ( η ) is the ﬁrst time X ηn exits from Ω .
Building upon these concepts, we can deﬁne a stopping time
[Equation]
as the ﬁrst time the SGD iterates visit a minimizer in an attraction ﬁeld that is different from Ω .
Besides, let index J σ ( η ) be such that
[Equation]
In other words, it is the label of the attraction ﬁeld that X η n escapes to.
Lastly, deﬁne
[Equation]
For deﬁnitions of the measure µ , set E , and mapping h , see eq.
(E.64) and eq.
(E.65).
Now we are ready to state Proposition E.23, the most important technical tool in this section.
In eq.
(E.80) and eq.
(E.81)
, we provide upper and lower bounds for the joint distribution of ﬁrst exit time σ and the label J σ indexing the attraction ﬁeld we escape to; it is worth noticing that the claims hold uniformly for all u > C .
In eq.
(E.82) and eq.
(E.83)
, we provide upper and lower bounds for the joint distribution of when we ﬁrst visit a different local minimum (which is equal to τ ) and which one we visit (indicated by X ητ ).
The similarity between eq.
(E.80) eq.
(E.81) and eq.
(E.82)eq.
(E.83) suggests a strong correlation between the behavior of the SGD iterates at time σ ( η ) and that of time τ ( η, (cid:15) ) , and this is corroborated by eq.
(E.84): we show that it is almost always the case that τ is very close to σ , and on the short time interval [ σ ( η ) , τ ( η, (cid:15) )]
the SGD iterates stay within the same attraction ﬁeld.
Proposition E.23.
Given C > 0 and some k (cid:48) ∈ [ n min − 1] , the following claims hold for all (cid:15) >
that is sufﬁciently small:
[Equation]
Before presenting the proof to Proposition E.23, we make some preparations.
First, we introduce stopping times (for all k ≥ 1 )
[Equation]
with the convention that τ 0 ( (cid:15), δ, η ) = (cid:101) τ 0 ( (cid:15), δ, η ) = 0 .
The intuitive interpretation is as follows.
For the ﬁxed (cid:15) we treat [ − 2 (cid:15), 2 (cid:15) ] as a small neighborhood of the local minimum of the attraction ﬁeld Ω .
All the (cid:101)
τ k partitioned the entire timeline into different attempts of escaping Ω .
The interval [ (cid:101) τ
k − 1 , (cid:101)
τ k ] can be viewed as the k − th attempt .
If for σ ( η ) , the ﬁrst exit time deﬁned in eq.
(E.70), we have σ ( η ) > (cid:101)
τ k , then we consider the k − th attempt of escape as a failure because the SGD iterates returned to this small neighborhood of the local minimum again without exiting the attraction ﬁeld.
On the other hand, the stopping times τ k − 1 indicate the arrival time of the ﬁrst large jump during the k − th attempt .
The proviso that (cid:101) τ k ≥ τ k − 1 can be interpreted, intuitively, as that an attempt is considered failed only if, after some signiﬁcant efforts to exit (for instance, a large jump) has been observed, the SGD iterates still returned to the small neighborhood [ − 2 (cid:15), 2 (cid:15) ] .
Regarding the notations, we add a remark that when there is no ambiguity we will drop the dependency on (cid:15), δ, η and simply write τ k , (cid:101)
τ k .
To facilitate the characterization of events during each attempt , we introduce the following deﬁnitions.
First, for all k ≥ 1 , let
[Equation]
be the number of large jumps during the k − th attempt .
Two implications of this deﬁnition:
• First, for any k with σ ( η ) < (cid:101)
τ k , we have j k = 0 .
Note that this proposition concerns the dynamics of SGD up until σ ( η ) , the ﬁrst time the SGD iterates escaped from Ω , so there is no need to consider an attempt that is after σ ( η ) , and we will not do so in the analysis below; • Besides, the random variable j k is measurable w.r.t.
F (cid:101)
τ k ∧ σ ( η ) , the stopped σ − algebra generated by the stopping time (cid:101)
τ k ∧ σ ( η ) .
Furthermore, for each k = 1 , 2 , · · · , let
T
[Equation] [Equation]
[Equation] [Equation]
[Equation]
with the convention T k, 0 ( (cid:15), δ, η ) = (cid:101)
τ k − 1 ( (cid:15), δ, η ) .
Note that for any k ≥ 1 , j ≥ 1 , T k,j is a stopping time.
Besides, from the deﬁnition of j k , one can see that
[Equation]
and the sequences (cid:0) T k,j (cid:1)
j k j =1 and (cid:0) W k,j (cid:1)
j k j =1 are the arrival times and sizes of large jumps during the k − th attempt, respectively.
Again, when there is no ambiguity we will drop the dependency on (cid:15), δ, η and simply write T k,j and W k,j .
In order to prove Proposition E.23, we analyze the most likely scenario that the exit from Ω would happen.
Speciﬁcally, we will introduce a series of events with superscript × or ◦ , where × indicates that the event is atypical or unlikely to happen and ◦ means that it is a typical event and is likely to be observed before the ﬁrst exit from the attraction ﬁeld Ω .
Besides, the subscript k indicates that the event in discussion concerns the dynamics of SGD during the k − th attempt.
Our goal is to show that for some event
A × ( (cid:15), δ, η ) its probability becomes sufﬁciently small as learning rate η tends to 0, so the escape from Ω almost always occurs in the manner described by ( A × ( (cid:15), δ, η )) c .
In particular, the deﬁnition of this atypical scenario A × involves the union of some atypical events A × k , B × k that occur in the k − th attempt.
In other words, the intuition of A × is that something abnormal happened during one of the attempts before the ﬁnal exit.
Here is one more comment for the general naming convention of these events.
Events with label A often describe the “efforts” made in an attempt to get out of Ω (such as large noises), while those with label B concern how the SGD iterates return to [ − 2 (cid:15), 2 (cid:15) ] (and how this attempt fails).
For instance,
A × k discusses the unlikely scenario before T k,l ∗ , the arrival of the l ∗ − th large jump in this attempt, while B × k in general discusses the abnormal cases after T k,l ∗ and before the return to [ − 2 (cid:15), 2 (cid:15) ] .
On the other hand, A ◦ k describes a successful escape during k − th attempt, while B ◦ k means that during this attempt the iterates return to without spending too much time.
Now we proceed and provide a formal deﬁnition and analysis of the aforementioned series of events.
As building blocks, we inspect the process ( X ηn ) n ≥ 1 at a even ﬁner granularity, and bound the probability of some events ( A × k,i ) i ≥ 0 , ( B × k,i ) i ≥ 1 detailing several cases that are unlikely to occur during the escape from or return to local minimum in the k − th attempt.
First, for each k ≥ 1 , deﬁne the event
[Equation]
Intuitively speaking, the event characterizes the atypical scenario where, during the k − th attempt, there is some large ﬂuctuations (compared to (cid:101) (cid:15) ) between any of the ﬁrst l ∗ large jumps (or the ﬁrst j k large jumps in case that j k < l ∗ ).
Similarly, consider event (for all k ≥ 1 )
[Equation]
that describes the atypical case where the exit occurs during the k − th attempt with less than l ∗ large jumps.
Next, for all k ≥ 1 we have another atypical event (note that from eq.
(E.85) we can see that, for any j ≥ 1 , j k ≥ j implies T k,j ≤
[Equation] [Equation]
∧
[Equation]
representing the case where we have at least l ∗ large noises during the k − th attempt, but for some of the large noise (from the 2nd to the l ∗ -th), the inter-arrival time is unusually long.
Moving on, we consider the following events (deﬁned for all k ≥ 1 )
[Equation]
that describes the atypical case where the k − th attempt failed but the return to the small neighborhood [ − 2 (cid:15), 2 (cid:15) ] took unusually long time.
The following event also concerns the scenario where there are at least l ∗ large noises during the k − th attempt:
[Equation]
speciﬁcally, it describes the atypical case where, during this attempt, right after the l ∗ − large noise the SGD iterate is far enough from the local minimum yet some of the large noises are not that large .
Lastly, by deﬁning events
[Equation]
we analyze an atypical case where the SGD iterates arrive at somewhere too close to the boundaries of Ω at the arrival time of the l ∗ large noise during this attempt.
As an amalgamation of these atypical scenarios, we let
[Equation]
Also, we analyze the probability of some events ( B × k ) k ≥ 1 that concern the SGD dynamics after the l ∗ − th large noise during the k − th attempt.
Let us deﬁne
[Equation]
where ρ ( · ) is the function in Lemma E.13.
From the deﬁnition of B × k , in particular the inclusion of
B × k, 2 , one can see that the intuitive interpretation of event B × k is that the SGD iterates did not return to local minimum efﬁciently (or simply escaped from the attraction ﬁeld) after the l ∗ − th large noise during the k − th attempt.
In comparison, the following events will characterize what would typically happen during each attempt:
[Equation]
Intuitively speaking, A ◦ k tells us that the exit happened right at T k,l ∗ , the arrival time of the l ∗ − th large noise during the k − th attempt
[Equation]
and B ◦ k tells us that the ﬁrst exit from Ω did not occur during the k − th attempt
[Equation]
and the SGD iterates returned to local minimum rather efﬁciently.
All the preparations above allow use to deﬁne
[Equation]
We need the next lemma in the proof of Proposition E.23.
As mentioned earlier, the takeaway is that
A × is indeed atypical in the sense that we will almost always observe ( A × ) c .
Lemma E.24.
Given any C > 0 , the following claim holds for all (cid:15) > 0 , δ > 0 sufﬁciently small:
[Equation]
Proof.
We ﬁx some parameters for the proof.
First, with out loss of generality we only consider
C ∈ (0 , 1) , and we ﬁx some N > αl ∗ .
Next we discuss the valid range of (cid:15) for the claim to hold.
We only consider (cid:15) > 0 such that
[Equation]
where ¯ ρ and (cid:101) ρ are the constants in Corollary E.7 and Corollary E.10 respectively, and (cid:15) 0 is the constant in eq.
(E.8).
Moreover, recall function Ψ in Lemma E.21 and the constant c ∗ > 0 in Lemma
E.22.
Due to lim (cid:15) ↓
Ψ
(cid:15) )
= 0 , it holds for all (cid:15) small enough such that
[Equation]
In our proof we only consider (cid:15) small enough so the inequality above holds, and the claim in Lemma
E.13 holds.
Now we specify the valid range of parameter δ that will be used below:
• For all sufﬁciently small δ > 0 , the claim in Lemma E.14 will hold for the prescribed (cid:15) and with N 0 = N ; •
For all sufﬁciently small δ > 0 , the claims in Lemma E.17, Corollary E.18, Corollary E. and Lemma E.20 will hold with the prescribed (cid:15) and N ; • For all sufﬁciently small δ > 0 , the inequalities in Lemma E.21 and E.22 will hold for the (cid:15) we ﬁxed at the beginning.
We show that the claim holds for any (cid:15), δ small enough to satisfy the conditions above.
First, recall that
[Equation]
Due to our choice of δ stated earlier and Lemma E.12, there exists some η 0 > 0 such that for all η ∈ (0 , η 0 ) ,
P (cid:0) A
[Equation]
(E
[Equation]
Similarly, recall that A × k, 1 ( (cid:15), δ, η ) =
∆ (cid:110) σ ( η ) < (cid:101) τ k , j k < l ∗ (cid:111) .
Let us temporarily focus on the ﬁrst attempt (namely the case k = 1 ).
From Lemma E.17 and our choice of (cid:15) and δ , we know the existence of some η 1 > 0 such that
[Equation]
Next, for A × k, 2 = ∆ (cid:110) j k ≥ l ∗ , ∃ j = 2 , 3 , · · · , l ∗ s.t.
T k
[Equation]
j − T k,j − 1 > 2ˆ t ( (cid:15) ) /η
(cid:111) , from Corollary E. and our choice of δ at the beginning, we have the existence of some η 2 > 0 such that
[Equation]
Moving on, for A × k,
[Equation]
[Equation]
) /η
[Equation]
, due to Corollary E.19 and our choice of (cid:15), δ , we have the existence of some η 3 > 0 such that
[Equation]
As for A × k, 4 = ∆
[Equation]
≥
[Equation]
, | X
[Equation]
| [Equation] − ¯ (cid:15), ∃ j = 1 , 2 , · · · , l ∗ s.t.
η | W k,j | ≤ ¯ δ
[Equation]
, from Lemma E.20, one can see the existence of η 4 > 0 such that
[Equation]
Lastly, for A × k, 5 =
[Equation]
(cid:110)
j
[Equation]
≥ l
[Equation]
, T k,l
[Equation]
η )
[Equation]
, X ηT
[Equation] [Equation]
[Equation]
− (cid:15), s j + (cid:15) ] (cid:111) , from Lemma
E.21 we see the existence of η 5 > 0 such that
[Equation]
Recall that A × k ( (cid:15), δ, η )
[Equation]
[Equation]
Also, for deﬁnitions of B × k , A ◦ k , B ◦ k , see eq.
(E.93),eq.
(E.94),eq.
(E.95) respectively.
Our next goal is to establish bounds regarding the probabilities of these events.
First, if we consider the event (cid:84) kj =
( A × j ∪ B × j ) c ∩ B ◦ j , then the inclusion of the ( B ◦ j ) kj =1 implies that during the ﬁrst k attempts the SGD iterates have never left the attraction ﬁeld, so
[Equation]
Next, note that
[Equation]
) c ∩ B ◦ j (cid:1) =
P x (cid:0) B × k, 1 | k − 1 (cid:92) j
= ( A × j ∪ B × j ) c ∩ B ◦ j (cid:1) P x (cid:16) B × k, 2 (cid:12)(cid:12)(cid:12) (cid:0) k − 1 (cid:92) j
= ( A × j ∪ B × j ) c ∩ B ◦ j (cid:1)
∩ B × k, 1 (cid:17) ≤ P x (cid:0) j k ≥ l ∗ , T k,j − T k,j − 1 ≤ 2ˆ t ( (cid:15) )
η ∀ j = 2 , 3 , · · · , l ∗ | k − 1 (cid:92) j
= ( A × j ∪ B × j ) c ∩ B ◦ j (cid:1) · P x (cid:16) B × k, 2 (cid:12)(cid:12)(cid:12) (cid:0) k − 1 (cid:92) j
= ( A × j ∪ B × j ) c ∩ B ◦ j (cid:1)
∩ B × k, 1 (cid:17) .
From the deﬁnition of the events A × j , B × j , B ◦ j , one can see that (cid:84) k − 1 j =1 ( A × j ∪ B × j ) c
∩ B ◦ j ∈ F (cid:101)
τ
k − 1 ∧ σ ( η ) , and on this event (cid:84) k − 1 j =1 ( A × j ∪ B × j ) c
∩ B ◦ j we have σ ( η ) > (cid:101) τ
k − 1 .
So by applying strong Markov property at stopping time (cid:101)
τ k − 1 ∧ σ ( η ) , we have
P x (cid:0) B × k | k − 1 (cid:92) j
=
( A × j ∪ B × j ) c
∩ B ◦ j (cid:1) ≤ P (cid:16) T η j ( δ ) − T η j − 1 ( δ ) ≤
2ˆ t ( (cid:15) ) /η
∀ j ∈ [ l ∗ − 1] (cid:17) · P x (cid:16) B × k, 2 (cid:12) (cid:12)(cid:12) (cid:0) k − 1 (cid:92) j
= ( A × j ∪ B × j ) c ∩ B ◦ j (cid:1)
∩ B × k, 1 (cid:17) ≤ 2 (cid:16) H ( δ/η )ˆ t ( (cid:15) ) /η
(cid:17) l ∗ − 1 · P x (cid:16) B × k, 2 (cid:12)(cid:12)(cid:12) (cid:0) k − 1 (cid:92) j
= ( A
[Equation]
c
[Equation]
(cid:1)
[Equation]
(cid:17) (for all η sufﬁciently small due to Lemma E.4)
[Equation] [Equation]
) η
[Equation]
(cid:0)
[Equation]
(cid:92) j
[Equation]
( A
[Equation] [Equation]
(cid:1)
[Equation]
B
[Equation] [Equation]
(cid:17) for all sufﬁciently small η , due to H ∈ RV − α ( η ) .
Meanwhile, note that
• (cid:0) (cid:84) k − 1 j =1 ( A × j ∪ B × j ) c
∩ B ◦ j (cid:1) ∩ B × k, 1 ∈ F T k,l ∗ ; • on this event (cid:0) (cid:84)
k − 1 j =1 ( A × j ∪ B × j ) c
∩ B ◦ j (cid:1) ∩ B × k, 1 we have σ ( η ) ∧ (cid:101)
τ k > T k,l ∗ and X ηT k
,l ∗ ∈ [ s
+ (cid:15), s + − (cid:15) ] .
Therefore, using Lemma E.13 and strong Markov property again (at stopping time T k,l ∗ ), we know the following inequality holds for all η sufﬁciently small:
[Equation]
Therefore
[Equation]
we know the existence of some η 6 > 0 such that
[Equation]
Similarly
[Equation]
we can bound conditional probabilities of the form P x
[Equation]
To be speciﬁc, recall that A × k = ∪ 5 i =
A × k,i .
By combining eq.
(E.98)-eq.
(E.103) with Markov property, we know the existence of some η 7 > 0 such that
[Equation]
On the other hand, a lower bound can be established for conditional probability involving A ◦ k , the event deﬁned in eq.
(E.94) describing the exit from Ω during an attempt with exactly l ∗ large noises.
Using Lemma E.22 and Markov property of ( X ηn ) n ≥ 1 , one can see the existence of some η 8 >such that
[Equation]
In order to apply the bounds eq.
(E.104)-eq.
(E.106), we make use of the following inclusion relationship:
[Equation]
To see why this is true, let us consider a decomposition of the event on the L.H.S. of eq.
(E.107).
As mentioned above, on event (cid:84) k − 1 j =
( A
× j
∪ B ×
j
) c ∩ B ◦
j we know that σ ( η ) > (cid:101)
τ
k − 1 , so the k − th attempt occurred and there are only three possibilities on this event:
• j k < l ∗ ; • j k ≥ l ∗ , X ηT k,l ∗ / ∈ Ω ; • j k ≥ l ∗ , X ηT k,l ∗ ∈ Ω .
Let us partition the said event accordingly and analyze them one by one.
• On (cid:0) (cid:84) k − 1 j =1 ( A × j ∪ B × j ) c
∩ B ◦ j (cid:1) ∩ ( A × k ∪ B × k ) c
∩ { j k < l ∗ } , due to the exclusion of A × k (especially A × k, 1 and A × k, 3 ), we can see that if j k < l ∗ , then we must have σ ( η ) > (cid:101)
τ k and (cid:101)
τ k − T k, 1 ≤ 2 l ∗ ˆ t ( (cid:15) ) /η .
Therefore, (cid:0) k − 1 (cid:92) j
= ( A × j ∪ B × j ) c ∩ B ◦ j (cid:1) ∩ ( A × k ∪ B × k ) c
∩ { j k < l ∗ } ⊆
B ◦ k .
• On (cid:0) (cid:84) k − 1 j =1 ( A × j ∪ B × j ) c
∩ B ◦ j (cid:1) ∩ ( A × k ∪
B × k ) c
∩ { j k ≥ l ∗ , X ηT k,l ∗ / ∈ Ω } , then the exclusion of A × k, 2 implies that T k,j − T k,j − 1 ≤ 2ˆ t ( (cid:15) ) /η
for all j = 2 , · · · , l ∗ , and the exclusion of A × k, 5 tells us that if X ηT k,l ∗ / ∈ Ω , then we have X ηT k,l ∗ / ∈ [ s − − (cid:15), s + + (cid:15) ] .
In summary, (cid:0) k − 1 (cid:92) j
=
( A × j ∪ B × j ) c
∩ B ◦ j (cid:1) ∩ ( A × k ∪
B × k ) c
∩ { j k ≥ l ∗ , X ηT k,l ∗ / ∈ Ω } ⊆
A ◦ k . • On (cid:0) (cid:84) k − 1 j =1 ( A × j ∪ B × j ) c
∩ B ◦ j (cid:1) ∩ ( A × k ∪
B × k ) c
∩ { j k ≥ l ∗ , X ηT k,l ∗ ∈ Ω } , the exclusion of A × k, 2 again implies that T k,j − T k,j − 1 ≤ 2ˆ t ( (cid:15) ) /η
for all j = 2 , · · · , l ∗ , hence T k,l ∗ − T k, 1 ≤ 2 l ∗ ˆ t ( (cid:15) ) /η .
Similarly, the exclusion of A × k, 5 tells us that if X ηT k,l ∗ ∈ Ω , then we have X ηT k,l ∗ ∈ [ s − + (cid:15), s + − (cid:15) ] .
Now since B × k did not occur (see the deﬁnition in eq.
(E.93)), we must have σ ( η ) > (cid:101)
τ k and (cid:101) τ
k − T k,l ∗ ≤ ρ ( (cid:15) )
/η , hence (cid:101)
τ k − T k, 1 ≤ 2 l ∗ ˆ t ( (cid:15) )
+ ρ ( (cid:15)
[Equation]
η .
Therefore,
[Equation]
Collecting results above, we have eq.
(E.107).
Now we discuss some of its implications.
First, from eq.
(E.107) we can immediately get that
[Equation]
Next, recall the deﬁnitions of A ◦ k in eq.
(E.94) and B ◦ k in eq.
(E.95), and one can see that A ◦ k and B ◦ k are mutually exclusive, since the former implies that the ﬁrst exit occurs during the k − th attempt while the latter implies that this attempt fails.
This fact and eq.
(E.108) allow us to conclude that k (cid:92)
[Equation]
Now we use the results obtained so far to bound the probability of
[Equation]
[Table]
This allows us to apply eq.
(E.104)-eq.
(E.106) and conclude that (here we only consider η < min { η i : i ∈ [8] } ),
[Equation]
c
[Equation]
for sufﬁciently small η , due to H ∈ RV − α ( η ) and our choice of N > αl ∗ ≤ 3Ψ( (cid:15) )
c ∗ < C for all η small enough such that 5 η α < Ψ( (cid:15) ) .
The last inequality follows from our choice of (cid:15) in eq.
(E.97).
This concludes the proof.
Having established Lemma E.24, we return to Proposition E.23 and give a proof.
Recall that, aside from the attraction ﬁeld Ω = ( s − , s + ) , there are n min − 1 other attraction ﬁelds (cid:101)
Ω k
=
( s − k , s + k ) (for each k ∈ [ n min − 1] ).
Besides, the function λ ( · ) and constants ν Ω , ν Ω k are deﬁned in eq.
(E.77)- eq. (E.79).
Proof of Proposition E.23.
We ﬁx some parameters for the proof.
First, with out loss of generality we only need to consider C ∈ (0 , 1) .
Next we discuss the valid range of (cid:15) for the claim to hold.
We only consider (cid:15) > 0 such that
[Equation]
where ¯ ρ and (cid:101) ρ are the constants in Corollary E.7 and Corollary E.10 respectively, and (cid:15) 0 is the constant in eq.
(E.8).
Due to continuity of measure µ , it holds for all (cid:15) small enough such that (let ˆ (cid:15) =
3(¯ ρ + (cid:101) ρ + 3) (cid:15) )
[Equation]
In our proof we only consider (cid:15) small enough so the inequality above holds, and the claims in Lemma
E.13 hold.
Moreover, we only consider (cid:15) and δ small enough so that Lemma E.24 hold and we have
[Equation]
We show that the desired claims hold for all (cid:15), δ sufﬁciently small that satisfy conditions above.
First, in order to show eq.
(E.84), we deﬁne event
[Equation]
that X ηn /
[Equation]
Since λ ∈ RV − 1 − l ∗ ( α − 1) and α > 1 , for the (cid:15) we ﬁxed at the beginning of this proof, ρ ( (cid:15) ) is a ﬁxed constant as well (the function ρ is deﬁned in Lemma E.13) and we have lim η ↓ 0 λ ( η ) ρ ( (cid:15) ) /η
= 0 .
Next, the occurrence of (cid:0) A × ( (cid:15), δ, η ) (cid:1) c (in particular, the exclusion of all the A × k, 5 deﬁned in eq.
(E.91))
, we know that X
[Equation]
− (cid:15), s + J σ + (cid:15) ] (recall that for any k ∈ [ n min − 1] , we have (cid:101)
Ω
j = ( s − j , s + j ) ; for deﬁnition of J σ see eq.
(E.76)).
Meanwhile, for all η sufﬁciently small, we have (cid:15)/λ ( η ) > ρ ( (cid:15) ) /η .
Therefore, using Lemma E.13 we can see that (for all η sufﬁciently small)
[Equation]
Lastly, observe that
[Equation]
) − σ ( η ) (cid:1) ≥ C or ∃ n = σ ( η ) + 1 , ·
τ ( η
(cid:15) )
such that X ηn /
∈
(cid:101)
Ω
J
σ
( η ) (cid:111)(cid:17)
[Equation]
( η ) (cid:0) τ ( η, (cid:15) ) − σ ( η ) (cid:1) ≥ C or ∃ n = σ ( η ) + 1 , · · · , τ ( η, (cid:15) ) such that X ηn / ∈ (cid:101)
Ω J σ ( η ) (cid:111)(cid:17) + P x
( A × )
so by combining eq.
(E.115)
with eq.
(E.116), we can obtain eq.
(E.84).
Moving on, we discuss the upper bounds eq.
(E.80) and eq.
(E.82).
Recall that the ﬁxed constant k (cid:48) ∈
[ n min − 1] is prescribed in the description of this proposition.
Let us observe some facts on event ( A × ( (cid:15), η, δ ))
c ∩ { J σ ( η ) =
k (cid:48) } : If we let J ( (cid:15), δ, η )
= ∆ sup { k ≥ 0 : (cid:101)
τ k < σ ( η ) } be the number of attempts it took to escape, and
[Equation]
then for all η sufﬁciently small, we must have J ≤ J ↑ on event ( A × ( (cid:15), η, δ ))
c ∩ { J σ ( η ) =
k (cid:48) } .
To see this via a proof of contradiction, let us assume that, for some arbitrary positive integer j , there exists some sample path on ( A × ) c
∩ { J σ ( η ) =
k (cid:48) } such that J ↑ =
j < J .
Then from the deﬁnition of ( A × ) c , in particular the exclusion of event A × j, 0 (see the deﬁnition in eq.
(E.86)), for all sufﬁciently small η , we are able to apply Corollary E.10 and E.7 and conclude that X ηT j,l ∗ / ∈ Ω : indeed, using
Corollary E.10 and E.7 we can show that the distance between X ηT j,l ∗ and the perturbed ODE
[Equation]
is strictly less than 3(¯ ρ + (cid:101) ρ + 3) (cid:15) ; on the other hand, the deﬁnition of (cid:16) 3(¯ ρ + (cid:101) ρ + 3) (cid:15), δ, η (cid:17) − overﬂow implies that
[Equation]
Therefore, we must have X η T j,l ∗ / ∈ Ω , which contradicts our assumption j = J ↑ < J .
In summary, we have shown that, on ( A × ) c ∩ { J σ = k (cid:48) } , we have J ↑ ( (cid:15), δ, η ) ≥ J ( (cid:15), δ, η ) .
Similarly, if we consider
[Equation]
then by the same argument above we can show that J ↓ ( (cid:15), δ, η ) ≤ J ( (cid:15), δ, η ) .
Now consider the following decomposition of events.
• On { J ↓ < J ↑ } , we know that for the ﬁrst k such that T k, 1 has (cid:0) −
3(¯ ρ
+ (cid:101) ρ + 3) (cid:15), δ, η (cid:1) − overﬂow, it does not have (cid:0) 3(¯ ρ + (cid:101) ρ + 3) (cid:15), δ, η (cid:1) − overﬂow.
Now we analyze the probability that Z 0 does not have (ˆ (cid:15), δ, η ) − overﬂow conditioning on that it does have ( − ˆ (cid:15), δ, η ) − overﬂow (recall that we let ˆ (cid:15)
=
3(¯ ρ + (cid:101) ρ + 3) ).
Using Lemma E.15 and the bound eq.
(E.112), we know that for all η sufﬁciently small,
[Equation]
≤
[Equation]
s
[Equation]
≤
[Equation]
• On ( A × ) c
∩ { J σ = k (cid:48) } ∩ { J ↑ =
J ↓ } , due to J ↑ =
J ↓ =
J we know that T J ( (cid:15),δ,η ) , 1 is the ﬁrst among all T k, 1 to have (ˆ (cid:15), δ, η ) − overﬂow.
Moreover, due to { J σ = k (cid:48) } and using Corollary E.10 and E.7 again as we did above, we know that the overﬂow endpoint of T J ( (cid:15),δ,η ) , 1 is in ( s − k (cid:48)
− ˆ (cid:15), s + k (cid:48)
+ ˆ (cid:15) )
(recall that (cid:101)
Ω k (cid:48)
=
( s − k (cid:48) , s + k (cid:48) ) ).
In summary, for any n ≥( A × ) c ∩ { J σ = k (cid:48) } ∩ { J ↑ =
J ↓ > n } ⊆
( A × ) c ∩ { J ↑ > n } ∩ (cid:110) T J ↑ , 1 has overﬂow endpoint in ( s − k (cid:48) − ˆ (cid:15), s + k (cid:48)
+ ˆ (cid:15) )
(cid:111)
so using Lemma E. we obtain that (for all η sufﬁciently small)
[Equation]
[Equation]
• On the other hand, on ( A × ) c , if T J ↓ , 1 has overﬂow endpoint in ( s − k (cid:48)
+ 2ˆ (cid:15), s + k (cid:48)
− 2ˆ (cid:15) ) , then from Deﬁnition E.1 we know that T J ↓ , 1 also has (ˆ (cid:15), δ, η ) − overﬂow, hence J ↓
=
J ↑ =
J .
Moreover, using Corollary E.10 and E.7 again, we know that X ηT J ↓ ,l ∗ ∈ ( s − k (cid:48) , s + k (cid:48) )
so
J σ = k (cid:48) .
In summary, for any n ≥ 0 ,
[Equation]
has overﬂow endpoint in
[Equation]
so using Lemma E.15, we obtain that (for all η sufﬁciently small)
[Equation]
uniformly for any
n
= 0 , 1 , due to eq.
(E.114).
Besides
[Equation]
the following claim holds on event ( A × ) c .
• From eq.
(E.109), the deﬁnition of B ◦ k as well as the deﬁnition of event
A ◦ k
(see eq.
(E.94)), one can see that for any j = 1 , 2 , · · · , J , we have
[Equation]
• Now if we turn to the interval ( (cid:101)
τ
j − 1 , T j, 1 ] (the time between the start of the j − th attempt and the arrival of the ﬁrst large noise during this attempt) for each j = 1 , 2 , · · · , J , and the following sequence constructed by concatenating these intervals
[Equation]
then the discussion above have shown that, for
[Equation]
Meanwhile, from the deﬁnition of overﬂow we know that the probability that Z 1 has (cid:16)
[Equation]
− overﬂow is equal to
[Equation]
• Therefore, if, within the duration of each attempt, we split the attempt into two parts at the arrival time of the ﬁrst large jump ( T k, 1 ) k ≥ 1 at each attempt, and deﬁne (here the subscript before or after indicates that we are counting the steps before or after the ﬁrst large jump in an attempt)
[Equation]
then we have σ ( η ) =
I before + I after .
Moreover, the discussion above implies that
[Equation]
} and on event ( A × ) c .
Deﬁne
[Table]
[Equation]
Using results above to bound I before and I after separately on event ( A × )
c
[Equation]
we can show that (for all η sufﬁciently small and any u > 0 )
[Equation]
to eq
[Equation] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where the last inequality follows from eq.
(E.117) and eq. (E.118).
Now let us analyze the probability terms on the last row of the display above.
For the ﬁrst term, let a ( η ) =
H ( δ/η ) p (cid:16) 3(¯ ρ + (cid:101) ρ +3) (cid:15), δ, η (cid:17) .
Due to Lemma E.15, we have (recall that ν
Ω = µ ( E (0)) )
[Equation]
Combining this with eq. (E.110), one can see that for all η sufﬁciently small,
[Equation]
u > 0 .
Next, let b ( η, u ) =
P (cid:16) a ( η ) Geom (cid:0) a ( η ) (cid:1) > (1 − C ) 2 u (cid:17) =
P (cid:16) Geom (cid:0) a ( η ) (cid:1) >
[Equation]
1 −
[Equation] [Equation]
a ( η ) (cid:17) .
For g ( y ) = log(1 − y ) , we know the existence of some y 0 > 0 such that for all y ∈ (0 , y 0 ) , we have log(1 − y ) ≤ − (1 − C ) y .
So one can see that for all η sufﬁciently small,
[Equation]
uniformly for all u > 0 .
For the second probability term, if we only consider u ≥ C , then
[Equation]
Using H ∈ RV − α ( η ) with α >
[Equation]
we get
[Equation]
due to the nature of the Geometric random variable U 1 .
Besides, due to H ∈ RV − α ( η ) with α >and Lemma E.15, it is easy to show that
[Equation]
Combining these results with Slutsky’s theorem
[Equation]
we now obtain
[Equation]
Therefore,
[Equation]
Plugging eq.
(E.121) and eq. (E.122) back into eq.
(E.120), we can establish the upper bound in eq.
(E.80).
To show eq.
(E.82), note that for event
[Equation]
we have (for deﬁnitions of τ , see eq.
(E.75))
[Equation]
Therefore, for all η sufﬁciently small,
[Equation]
some n ∈ [
[Equation] [Equation]
uniformly for all u ≥ C , due to eq.
(E.115), eq.
(E.84) and eq.
(E.120).
The lower bound can be shown by an almost identical approach.
In particular, analogous to eq.
(E.120), we can show that (for any u > 0 )
[Equation]
due to P ( E \ F ) ≥ P ( E ) − P ( F ) and eq.
(E.115)eq. (E.117)eq.
(E.119) where
[Equation]
Using the similar argument leading to eq.
(E.121), we are able to show eq.
(E.81), eq.
(E.83) and conclude the proof.
Recall that σ i ( η ) =
min { n ≥ 0 : X n / ∈ Ω i } and that value of constants q i , q i,j are speciﬁed via eq.
(E.2)-eq.
(E.7).
Deﬁne
[Equation]
The following result is simply a restatement of Proposition E.23 under the new system of notations.
Despite the reiteration, we still state it here because this is the version that will be used to prove
Lemma E.2, which is the key tool for establishing Theorem 1, as well as many other results in Section
F.
Proposition E.25.
Given C > 0 and i, j ∈ [ n min ] such that i (cid:54) =
j , the following claims hold for all (cid:15) > 0 that are sufﬁciently small:
[Equation]
[Table]
Concluding this section
[Table]
we apply Proposition E.25 and prove Lemma E.2.
Proof of Lemma E.2.
Fix some C ∈ (0 , 1) , u > 0 , and some k, l ∈ [ n min ] with k (cid:54)
=
l .
Let q i , q i,j be the constants deﬁned in eq.
(E.7).
Fix some C 0 ∈ (cid:0) 0 , Cn min ∧ q k n min C (cid:1) .
Using Proposition E.25, we know that for all (cid:15) sufﬁciently small, we have
[Equation]
Summing up the inequality above over all j ∈ [ n min ] , we can obtain eq.
(E.16).
The lower bound eq.
(E.17) can be established using an identical approach.
In order to show eq.
(E.19), note that we can ﬁnd C 1 ∈ (0 , u ) sufﬁciently small so that
[Equation]
Fix such C 1 .
From Proposition E.25, we also know that for all (cid:15) small enough, we have
[Equation]
Then using P x (cid:16)
X ησ k ( η ) ∈
Ω l (cid:17) ≥ P (cid:16) q k λ k ( η ) σ k ( η ) > u, X ησ k ( η ) ∈
Ω l (cid:17) we conclude the proof for eq.
(E.19).
Moving on, we show eq.
(E.18) in the following way.
Note that we can ﬁnd C 2 ∈ (0 , u ) small enough so that
[Equation]
Fix such C 2 .
Since eq.
(E.17) has been established already, we can ﬁnd some u 2 > 0 such that for all (cid:15) small enough,
[Equation]
Fix such u 2 .
Meanwhile, ﬁx some C 3 ∈ (0 , C 2 ∧ u 2 ) .
From Proposition E.25 we know that for all (cid:15) sufﬁciently small,
[Equation]
Lastly, observe the following decomposition of events (for any x ∈ Ω k )
[Equation]
Combining this bound with eq.
(E.125)-eq.
(E.128), we complete the proof.
F P ROOFS FOR S ECTION 2.
In this section, we show that gradient clipping scheme effectively partitions the entire optimization landscape of f into different regions based on the radius r i and minimum jump number l ∗ i for each attraction ﬁeld Ω i .
Furthermore, when staying in each region, the behavior of SGD iterates closely resembles a Markov chain that only visits wider attraction ﬁelds in this region.
We exclude the trivial case where n min =
and there is only one attraction ﬁeld.
This structure is as follows.
First we present some key lemmas that can be used to prove the Theorem 2-3 in the main paper.
Then we devote the rest of the section to establish those lemmas.
In order to prove Theorem 2, we will make use of the following lemma, where we show that the type of claim in
Theorem 2 is indeed valid if we look at a much shorter time interval.
Then when we move onto the proof of Theorem 2, it sufﬁces to partition the entire horizon into pieces of these short time intervals, on each of which we analyze the dynamics of SGD respectively.
Lemma F.1.
Assume the graph G is irreducible , and let (cid:15) > 0 , δ > 0 be any positive real numbers.
For the following random variables (indexed by η )
[Equation]
the following claim holds for any sufﬁciently small t > 0 :
[Equation]
Proof of Theorem 2.
It sufﬁces to show that for any t > 0 , κ >
+ ( α − 1) l large ,
[Equation]
> 0 , δ
[Equation] [Equation]
) , we have
[Equation]
for
[Equation]
Let us ﬁx some (cid:15) > 0 , δ ∈ (0 , (cid:15) ) .
First, let
[Equation]
The regularly varying nature of H implies that λ large ( η ) ∈ RV 1+ l large ( α − 1) ( η ) .
Since κ > 1 + l large ( α − 1) , we know that lim η ↓ 0 N ( η ) = ∞ .
Next, due to Lemma F.1, we can ﬁnd t 0 > 0 and ¯ η > 0 such that for any η ∈ (0 , ¯ η )
[Equation]
For any k ≥ 1 , deﬁne
[Equation]
It is clear from its deﬁnition that V k stands for the proportion of time that the SGD iterates are outside of large attraction ﬁelds on the interval [( k − 1) (cid:98)
[Equation]
(cid:99) , k
[Equation]
t
[Equation]
λ large
[Equation]
η ) (cid:99) ] .
From eq.
(F.2) and Markov property, one can see that for any η ∈ (0 , ¯ η )
[Equation]
uniformly for all k ≥ 1 .
Now deﬁne K ( η ) =
∆ # { n = 1 , 2 , · · · , N ( η ) :
V k ( η ) > (cid:15) } .
By a simple stochastic dominance argument, we have
[Equation]
Meanwhile, strong law of large numbers implies the existence of some ¯ η 1 > 0 such that
P ( Binomial ( N ( η ) ,δ ) N ( η ) > 2 δ ) < δ for all η ∈ (0 , ¯ η 1 ) , thus sup
[Equation]
L
P
[Equation]
K
[Equation]
η ) /N
[Equation]
η ) > 2 δ ) ≤ δ ∀ η ∈ (0 , ¯ η 1 ∧ ¯ η ) .
Lastly, from the deﬁnition of K ( η ) and N ( η ) , we know that for all the N ( η ) intervals [( k − 1) (cid:98)
[Equation]
] with k ∈ [ N ( η )] , only on K ( η ) of them did the SGD iterates spent more then (cid:15) proportion of time outside of the large attraction ﬁelds, hence
[Equation]
In summary, we now have
[Equation]
for all η ∈ (0 , ¯ η 1 ∧ ¯ η ) .
This concludes the proof.
When introducing Theorem 3 in the main paper, we stated that the results of eliminating sharp minima can be extended to the more general reducible case.
Here we present the corresponding theoretical results in Theorem F.2 and F.3 below.
The main message can be summarized as follows: (a) SGD with truncated and heavy-tailed noise naturally partitions the entire training landscape into different regions; b dynamics closely resemble that of
Markov chain that only visits local minima; (3) In particular, any sharp minima within each region is almost completely avoided by SGD.
When the typical transition graph (see Deﬁnition 1 in the main paper) is not irreducible, there will be multiple communication classes on the graph.
Suppose that there are K communication classes
G 1 , · · · , G K .
From now on, we zoom in on a speciﬁc communication class G ∈ { G 1 , · · · , G K } .
For this communication class G , deﬁne l ∗ G = ∆ max { l ∗ i : i = 1 , 2 , · · · , n min ; m i ∈ G } .
For each local minimum m i ∈ G , we call its attraction ﬁeld Ω i a large attraction ﬁeld if l ∗ i =
l ∗ G , and a small attraction ﬁeld if l ∗ i < l ∗ G .
We have thus classiﬁed all m i in G into two groups: the ones in large attraction ﬁelds m large, · · · , m large i G and the ones in small attraction ﬁelds m small, · · · , m small i (cid:48) G .
Also, deﬁne a scaling function λ G associated with G as λ G ( η ) =
∆ H (1 /η ) (cid:16) H (1 /η ) η (cid:17)
[Equation]
∗ G − 1 .
Theorem F.2.
Under Assumptions 1-3, if G is absorbing , then there exists a continuous-time Markov chain Y on { m large, · · · , m large
i G } such that for any x ∈ Ω i , | x | ≤ L (where i ∈ { 1 , 2 , · · · , n min } ) with m i ∈ G , and
[Equation]
m
[Equation]
) as η ↓in the sense of ﬁnite-dimensional distributions, where π G is a random mapping satisfying (1) π G ( m ) ≡ m if m ∈ { m large, · · · , m large i G }
[Equation]
(2) π G ( m ) is a random variable that only takes value in { m large, · · · , m large
i G } if m ∈ { m
[Equation]
We stress that Theorem 3 in the main paper follows immediately from Theorem F.2 above.
Next, to state the corresponding result for a transient communication class G , we introduce a couple of extra deﬁnitions.
We consider a version of X ηn that is killed when X ηn leaves G .
Deﬁne stopping time
[Equation]
as the ﬁrst time the SGD iterates leave all attraction ﬁelds in G , and we use a cemetery state † to construct the following process X † ,ηn as a version of X ηn with killing at τ G :
[Equation]
Theorem F.3.
Under Assumptions 1-3, if G is transient , then there exists a continuous-time Markov chain Y with killing that has state space { m large, · · · , m large i G , † } (we say the Markov chain Y is killed when it enters the absorbing cemetery state † ) such that for any x ∈ Ω i , | x | ≤ L (where i ∈ { 1 , 2 , · · · , n min } ) with m i ∈ G , and
[Equation]
in the sense of ﬁnite-dimensional distributions, where π G is a random mapping satisfying (1) π G ( m ) ≡ m if m ∈ { m large, · · · , m large i G } ; (2) π G ( m ) is a random variable that only takes value in { m large, · · · ,
m large
i
G
, † } if m ∈ {
m small
, · · · , m small i (cid:48) G } .
To show Theorem F.2 and F.3, we introduce the following concepts.
First, we consider the case where the SGD iterates X ηn is initialized on the communication class G and G is absorbing.
For some
∆ > 0 , η > 0 , deﬁne (let B ( u, v ) =
∆ [ u − v, u + v ] )
[Equation]
F.6)
[Equation]
∆
[Equation]
F.7)
[Equation]
F.8)
[Equation]
F.9)
[Equation] [Equation]
Intuitively speaking, at each τ Gk the SGD iterates visits a minimizer that is not in a small attraction ﬁeld on G , and we use I Gk to mark the label of that large attraction ﬁeld.
Stopping time σ Gk is the ﬁrst time that SGD visits a minimizer that is different from the one visited at τ Gk , and τ Gk +1 is the ﬁrst time that a minimizer not in a small attraction ﬁeld of G is visited again since σ Gk (and including σ Gk ).
It is worth mentioning that, under this deﬁnition, we could have I
Gk = I
Gk +1 for any k ≥ 0 .
Meanwhile, deﬁne the following process that only keeps track of the updates on the labels ( I Gk ) k ≥ 0 instead of the information of the entire trajectory of ( X ηn ) n ≥ 0 :
[Equation]
In other words, when n < τ G 0 we simply let ˆ X η, ∆ n = 0 , otherwise it is equal to the latest “marker” for the last visited wide minimum up until step n .
This marker process ˆ X jumps between the different minimizers of the large attractions in G .
In particular, if for some n we have X ηn ∈ B ( m j , 2∆) for some j with m j ∈ G large , then we must have ˆ X η, ∆ n = m j , which implies that, in this case, ˆ X η, ∆ n indeed indicates the location of X ηn .
Note that results in Theorem F.2 and F.3 concern a scaled version of X η .
Here we also deﬁne the corresponding scaled version of the processes
[Equation]
14) a mapping T ∗ ( n, η ) =
∆ nλ G ( η ) that translates a step n to the corresponding timestamp for the scaled processes, and the following series of scaled stopping times
[Equation]
(cid:1) .
[Equation]
Before presenting the proof of Theorem F.2 and F.3, we make several preparations.
First, our proof is inspired by ideas in Pavlyukevich (2005) and here we provide a brieﬁng.
At any time t > 0 , if we can show that X ∗ ,η t is almost always in set (cid:83) i : m
i ∈
G large B ( m i , 2∆) (so the SGD iterates is almost always close to a minimizer in a large attraction ﬁeld), then the marker process ˆ X ∗ ,η, ∆ t is a pretty accurate indicator of the location of X ∗ ,η t , so it sufﬁces to show that the marker process ˆ X ∗ ,η, ∆ t converges to a continuous-time Markov chain Y .
Second, we construct the limiting process Y and the random mapping π G before utilizing them in Theorem F.2 and F.3.
As an important building block for this purpose, we start by considering the following discrete time Markov chain (DTMC) on the entire graph G =
( V, E ) .
Let P DTMC be a transition matrix with P DTMC ( m i , m j )
= µ i ( E i,j ) /µ
i ( E i ) for all j (cid:54) = i , and Y DTMC =
( Y DTMCj )
j ≥ 0 be the DTMC induced by the said transition matrix.
Let
[Equation]
be the ﬁrst time this DTMC visits a large attraction ﬁeld on the communication class G , or escapes from G .
Lastly, deﬁne (for any j such that m j / ∈ G small )
[Equation]
as the probability that the ﬁrst large attraction ﬁeld on G visited by Y DTMC is m j when initialized at m i .
We add a comment regarding the stopping times T DTMCG and probabilities p i,j deﬁned above.
In the case that G is absorbing, we have Y DTMCj ( m i )
∈ G for all j ≥
if m i ∈ G .
Therefore, in this case, given any i with m i ∈ G , we must have
[Equation]
On the contrary, when G is transient we may have (cid:80) j : m j
[Equation]
i
[Equation]
<
[Equation]
and
[Equation]
j
[Equation]
m
[Equation]
G p i
[Equation]
>
[Equation]
Lastly, whether G is absorbing or transient, we always have p i,j = 1 { i = j } if m i ∈
G large .
Next, consider the following deﬁnition of (continuous-time) jump processes.
Deﬁnition F.1.
A continuous-time process Y t on R is a (cid:16) ( U j ) j ≥ 0 , ( V j ) j ≥ 0 (cid:17) jump process if
[Equation]
where ( U j ) j ≥ 0 is a sequence of non-negative random variables such that U j > 0 ∀ j ≥ 1 almost surely, and ( V j ) j ≥ 0 is a sequence of random variables in R .
Obviously, the deﬁnition above implies that Y t =
V j for any t ∈ [ U j , U j +1 ) .
Now we are ready to construct the limiting continuous-time Markov chain Y .
To begin with, we address the case where G is absorbing.
For any m (cid:48) ∈ G large , let Y ( m (cid:48) ) be a (cid:0) ( S k ) k ≥ 0 , ( W k ) k ≥ 0 (cid:1) - jump process where S
= 0 , W
= m (cid:48) and (for all k ≥ 0 and i, j with m i ∈
G large , m j / ∈ G small )
[Equation]
where
[Equation]
and p k,j is deﬁned in eq.
(F.17).
In other words, conditioning on W k
= m i , the time until next jump
S k +1 and the jump location W k +1 are independent, where S k +1 is Exp ( q i ) and
W k +
= m j with probability q i,j /q i .
First, it is easy to see that Y is a continuous-time Markov chain.
Second, under this deﬁnition Y is allowed to have some dummy jumps where W k = W k +1 : in this case the process
Y t does not move to a different minimizer after the k + 1 -th jump, and by inspecting the path of Y we cannot tell that this dummy jump has occurred.
As a result, that generator Q of this Markov chain admits the form (for all i (cid:54) =
j with m i , m j ∈ G large )
[Equation]
Moreover, deﬁne the following random function π G ( · ) such that for any m i ∈ G ,
[Equation]
By Y ( π G ( m i ))
we refer to the version of the Markov chain Y where we randomly initialize W
= π G ( m i ) .
The following lemma is the key tool for proving Theorem F.2.
Lemma F.4.
Assume that the communication class G is absorbing.
Given any m i ∈ G , x ∈ Ω i , ﬁnitely many real numbers ( t l ) k (cid:48) l =1 such that 0 < t 1 < t 2 < · · · < t k (cid:48) , and a sequence of strictly positive real numbers ( η n ) n ≥ 1 with lim n → 0 η n = 0 , there exists a sequence of strictly positive real numbers (∆ n ) n ≥ 1 with lim n ∆ n = 0 such that
• As n tends to ∞ ,
[Equation]
• For all k ∈
[ k (cid:48) ] ,
[Equation]
Now we address the case where G is transient, let † be a real number such that † / ∈ [ − L, L ] , and we use † as the cemetery state since the processes X ηn or X ∗ ,η t are restricted on [ − L, L ] .
Recall the deﬁnition of τ G deﬁned in eq.
(F.4).
Analogous to the process X † in eq.
(F.5)
, we can also deﬁne
X † , ∗ ,η t = (cid:26) X ∗ ,η t if t < T ∗ ( τ G ( η ) , η ) † otherwise , ˆ X † , ∗ ,η, ∆ t = (cid:40) ˆ X ∗ ,η, ∆ t if t < T ∗ ( τ G ( η ) , η ) † otherwise , .
[Equation]
F
[Equation]
Next, analogous to τ G , consider the stopping time
[Equation]
When G is transient, due to the construction of Y we know that τ YG < ∞ almost surely.
The introduction of τ YG allows us to deﬁne
[Equation]
The following Lemma will be used to prove Theorem F.3.
Lemma F.5.
Assume that the communication class G is transient.
Given any m i ∈ G , x ∈ Ω i , ﬁnitely many real numbers ( t l ) k (cid:48) l =1 such that 0 < t 1 < t 2 < · · · < t k (cid:48) , and a sequence of strictly positive real numbers ( η n ) n ≥ 1 with lim n → 0 η n = 0 , there exists a sequence of strictly positive real numbers (∆ n ) n ≥ 1 with lim n ∆ n = 0 such that
• As n tends to ∞ , (cid:0) ˆ X † , ∗ ,η n , ∆ n t
[Equation]
x )
[Equation]
· · · , ˆ X †
[Equation]
∗ ,η
[Equation]
• For all k ∈
[ k (cid:48) ] , lim n →∞ P x (cid:16) X † , ∗ ,η n t k / ∈ (cid:91) j : m j ∈ G large B ( m j , ∆ n ) and X † , ∗ ,η n t k
[Equation]
Proof of Theorem F.2 and F.3.
We ﬁrst address the case where G is absorbing.
Arbitrarily choose some ∆ > 0 , a sequence of strictly positive real numbers ( η n ) n ≥ 1 with lim n η n = 0 , a positive integer k (cid:48) , a series of real numbers ( t j ) k (cid:48)
j =1 with 0 < t 1 < · · · < t k (cid:48) , and a sequence ( w j ) k (cid:48)
j =1 with w j ∈ G large for all j ∈ [ k (cid:48) ] .
It sufﬁces to show that
[Equation]
Using Lemma F.4, we can ﬁnd a sequence of strictly positive real numbers (∆ n ) n ≥ 1 with lim n ∆ n = 0 such that eq.
(F.23) and eq.
(F.24) hold.
From the weak convergence in eq.
(F.23), we only need to show
[Equation]
For all n large enough, we have 2∆ n < ∆ .
For such large n , observe that
[Equation]
(due to deﬁnition of the marker process ˆ X , see eq.
(F.6)-eq.
(F.11) and eq.
(F.12)-eq.
(F.14))
[Equation]
and by applying eq.
(F.24)
we conclude the proof for Theorem F.2.
The proof of Theorem F.3 is almost identical, with the only modiﬁcation being that we apply Lemma
F.5 instead of Lemma F.4.
In doing so, we are able to ﬁnd a sequence of (∆ n ) n ≥ 1 with lim n ∆ n =such that eq.
(F.27) and eq.
(F.28) hold.
Given the weak convergence claim in eq.
(F.27), it sufﬁces to show that
[Equation]
If X † , ∗ ,η n t k = † , we must have ˆ X † , ∗ ,η n , ∆ n t k = † as well.
Therefore, for all n large enough so that 3∆
[Equation]
<
[Equation] [Equation]
x (cid:16)
[Equation]
n
[Equation]
η n
[Equation]
n
[Equation]
X
[Equation]
(cid:54)
[Equation] [Equation]
[Equation]
j
[Equation]
(cid:54)
[Equation]
j
[Equation]
(cid:54)
[Equation]
Apply eq.
(F.28)
and we conclude the proof.
F. P ROOF OF L EMMA F.
First, we introduce another dichotomy for small and large noises.
For any (cid:101) γ > 0 and any learning rate η > 0 , we say that a noise Z n is small if
[Equation]
and we say Z n is large otherwise.
For this new classiﬁcation of small and large noises, we introduce the following notations and deﬁnitions:
[Equation]
Similar to Lemma E.12, the following result is a direct application of Lemma E.11, and shows that it is rather unlikely to observe large perturbation that are caused only by small noises.
Speciﬁcally, since α > 1 we can always ﬁnd
[Equation]
[Equation]
[Equation]
Now in Lemma E.11, if we let ∆ =
(cid:101) γ, (cid:101) ∆ =
∆ / 2 and (cid:15) =
δ = 1 (in other words, u ( η ) = 1 /η 1 − (cid:101) γ , v ( η ) =
η (cid:101) γ/ 2 ), then for any positive integer j the condition eq.
(E.52) is satisﬁed, allowing us to draw the following conclusion immediately as a corollary from Lemma E.11.
Lemma F.6.
Given N > 0 and
[Equation]
we have (as η ↓ 0)
[Equation]
The ﬂavor of the next lemma is similar to that of Lemma E.13.
Speciﬁcally, we show that, with high probability, the SGD iterates would quickly return to the local minimum as long as they start from somewhere that are not too close the boundary of an attraction ﬁeld (namely, the points s 1 , s 2 , · · · , s n min ).
To this end, we consider a reﬁnement of function ˆ t ( · ) deﬁned in eq.
(E.38).
For any i = 1 , 2 , · · · , n min , any x ∈ Ω i and any η > 0 , γ ∈ (0 , 1) , we can deﬁne the return time to η γ − neighborhood for the ODE x η as
[Equation]
Given the bound in eq.
(E.37) (which is stated for a speciﬁc attraction ﬁeld) and the fact that there only exists ﬁnitely many attraction ﬁelds, we know the existence of some c 2 < ∞ such that for any i = 1 , 2 , · · · , n min , any η > 0 , any γ ∈ (0 , 1) and any x ∈ Ω i such that | x − s i | ∨ | x − s
i − 1 | > η γ , we have
[Equation]
and deﬁne function t ↑ as
[Equation]
Lastly, deﬁne the following stopping time for any i
= 1 , 2 , · · · , n min , any x ∈ Ω i and any ∆ >
[Equation]
where we adopt the notation B ( u, v ) =
∆ [ u − v, u + v ] for the v − neighborhood around point u .
Lemma F.7.
Given
[Equation]
and any i = 1 , 2 , · · · , n min , any ∆ > 0 , we have
[Equation]
= 1 .
Proof.
Throughout this proof, we only consider η small enough such that
[Equation]
The condition above holds for all η > 0 sufﬁciently small because β > 1 , 2 (cid:101) γ < 1 , and γ < (cid:101) γ/ 4 .
Also, ﬁx some β ∈ (cid:0) 1 , (2 − 2 (cid:101) γ ) ∧ ( α − α (cid:101) γ ) (cid:1)
Deﬁne the following events
[Equation]
A
[Equation]
≤
[Equation]
} (see eq.
(F.31) for deﬁnition of the stopping time involved) and ﬁx some N > 0 .
From Lemma F.6, we see that (for all sufﬁciently small η )
P
[Equation]
≤ η N .
Besides, using Lemma E.4 together with the fact that β < α (1 − (cid:101) γ ) , we know the existence of some constant θ > 0 such that
[Equation]
for all small η .
Now we focus on the behavior of the SGD iterates on event (cid:16)
A × 1 ( η ) ∩ A × 2 ( η ) (cid:17) c .
Let us arbitrarily choose some x ∈ Ω i such that | x − s i | ∨ | x − s i − 1 | > 2 η γ .
First, from Lemma E.6 and eq.
(F.32), we know that
[Equation]
Next, from the deﬁnition of the function t ↑ ( · ) and eq.
(F.33)
[Equation]
we know that for
T
[Equation]
,return
[Equation]
34) we have
[Equation] [Equation]
Furthermore, on event (cid:16) A × 1 ( η ) ∩ A × 2 ( η ) (cid:17) c , due to Lemma E.5 and eq.
(F.32), we have that
[Equation]
(1 /η )
/η.
Combining this with eq.
(F.35), eq.
(F.36), we can conclude that (recall that due to eq.
(F.32) we have 2 η γ < ∆ / 2 )
[Equation]
X
[Equation]
on event (cid:16) A × 1 ( η ) ∩ A × 2 ( η ) (cid:17) c .
Therefore,
[Equation]
This concludes the proof.
The takeaway of the next lemma is that, almost always, the SGD iterates will quickly escape from the neighborhood of any s i , the boundaries of each attraction ﬁelds.
Lemma F.8.
Given any γ ∈ (0 , 1) , t > 0 , we have
[Equation]
Proof.
We only consider η small enough so that
[Equation]
Also, the claim is trivial if x / ∈ ∪ j B ( s j ,
[Equation]
η
[Equation]
) , so without loss of generality we only consider the case where there is some j ∈ [ n min ] and x ∈ [ − L, L ] , x ∈ B ( s j , 2 η γ ) .
Let us deﬁne stopping times
[Equation]
and the following two events
[Equation]
First, using Lemma E.3 and the regularly varying nature of H ( · ) =
P ( | Z 1 | > · ) , we know the existence of some θ > 0 such that
[Equation]
for all η > 0 sufﬁciently small.
Next, by deﬁnition of T γ , one can see that (for any η ∈ (0 , 1) )
[Equation]
Again, due to H ∈ RV − α and 1 − γ > 0 , we know the existence of some θ 1 > 0 such that
[Equation]
for all η > 0 sufﬁciently small.
To conclude the proof, we only need to note the following fact on event (cid:0)
A × 1 ( η ) ∪ A × 2 ( η ) (cid:1) c .
There are only two possibilities on this event: T γ escape ≤ T γ − 1 , or
T γ escape ≥ T γ .
Now we analyze the two cases respectively.
• On (cid:0)
A × 1 ( η ) ∪
A × 2 ( η ) (cid:1) c ∩ { T γ escape ≤ T γ − 1 } , we must have T γ escape < T γ ≤ t/H (1 /η ) .
• On (cid:0)
A × 1 ( η ) ∪
A × 2 ( η ) (cid:1) c ∩ { T γ escape ≥ T γ } , we know that at n = T γ − 1 , there exists an integer j ∈ { 1 , 2 , · · · , n min − 1 } such that X ηn ∈ B ( s j , 2 η γ
Now since ηM <
η γ and η | Z T γ | > 5 η γ , we must have
[Equation]
On the other hand, the exclusion of event A × 2 ( η ) tells us that | X ηT γ − X ηT
[Equation]
1 |
[Equation]
2 η 1+ γ 2 .
Due to eq.
(F.37), we then have X ηn / ∈ ∪ i B ( s i , 2 η γ ) .
In summary, (cid:0) A × 1 ( η ) ∪ A × 2 ( η ) (cid:1) c ⊆ { T γ return ≤ t/H (1 /η ) } and this conclude the proof.
In the next lemma, we analyze the number of transitions needed to visit a certain local minimizer in the loss landscape.
In general, we focus on a communication class G and, for now, assume it is absorbing.
Next, we introduce the following concepts to record the transitions between different local minimum.
To be speciﬁc, for any η > 0 and any ∆ > 0 small enough so that B ( m j , ∆) ∩ Ω cj = ∅ for all j , deﬁne
[Equation]
As mentioned earlier, the next goal is to analyze the transitions between attraction ﬁelds it takes to visit m j when starting from m i when m i , m j ∈ G .
Deﬁne
[Equation]
Lemma F.9.
Assume that G is an absorbing communication class on the graph G .
Then there exists some constant p > 0 such that for any i with m i ∈ G , any (cid:15) > 0 , and any ∆ > 0 ,
[Equation]
hold for all η > 0 sufﬁciently small.
Proof.
The claim is trivial if, for the initial condition, we have x ∈ B ( m i , 2∆) .
Next, let us observe the following facts.
• Deﬁne (recall the deﬁnitions of measure µ i and sets E i , E i,j in eq.
(E.1)eq.
E.
eq.
[Equation] [Equation]
[Equation]
• From the deﬁnition of J ( j ) and the fact that there are only ﬁnitely many attraction ﬁelds we can see that p ∗ > 0 .
Moreover, G being a communication class implies that
[Equation]
Indeed, if i < j , then since G is a communication class and there are some m
i ∈
G with i <
j , we will at least have µ j ( E j,j − 1 ) > 0 , so | J ( j ) − i | ≤ | i − j | − 1 ; the case that i > j can be approached analogously.
• Now from the deﬁnition of J ( j ) and Proposition E.25, together with the previous bullet point, we know that for all η sufﬁciently small,
[Equation]
uniformly for all k ≥ 0 .
Meanwhile, since p ∗ > 0 , we are able to ﬁx some δ > 0 small enough such that
[Equation]
• On the other hand, for any (cid:101) j with m (cid:101) j / ∈ G , by deﬁnition of the typical transition graph we must have µ
j ( E j, (cid:101) j ) = 0 for any j with m j ∈ G .
Then due to Proposition E.25 again, one can see that for all η > 0 that is sufﬁciently small,
[Equation]
uniformly for all k ≥ 0 .
• Repeat this argument for n min times, and we can see that for all η sufﬁciently small
[Equation]
uniformly for all k ≥ 1 .
• Lastly, to apply the bounds established above, we will make use of the following expression of several probabilities.
For any j with m j ∈ G and x ∈ B ( m j , 2∆) and any u = 1 , 2 , · · · ,
[Equation]
In summary, now we can see that (for sufﬁciently small η )
[Equation]
such that
[Equation]
G
[Equation]
[ n min
[Equation]
such that m I v
[Equation]
and
[Equation]
uniformly for all u = 1 , 2 , · · · .
To conclude the proof, it sufﬁces to set p = ( p ∗ 2 ) n min .
The proof above can be easily adapted to the case when the communication class G is transient.
Deﬁne
[Equation]
Lemma F.10.
Assume that G is a transient communication class on the graph G .
Then there exists some constant p > 0 such that for any i with m i ∈ G and any ∆ ∈ (0 , ¯ (cid:15)/ 3) ,
[Equation]
hold for all η > 0 sufﬁciently small.
Proof.
The structure of this proof is analogous to that of Lemma F.9.
Again, the claim is trivial if, for the initial condition, we have x ∈ B ( m i , 2∆) .
Next, let us observe the following facts
• Deﬁne (recall the deﬁnitions of measure µ i and sets E i , E i,j in eq.
(E.
1)eq
(E.5)eq
(E.6))
[Equation]
• From the deﬁnition of J ( j ) and the fact that there are only ﬁnitely many attraction ﬁelds we can see that p ∗ > 0 .
Moreover, G being a communication class implies that
[Equation]
Indeed, if i < j , then since G is a communication class and there are some m
i ∈
G with i <
j , we will at least have µ j ( E j,j − 1 ) > 0 , so | J ( j ) − i | ≤ | i − j | − 1 ; the case that i > j can be approached analogously.
• Now from the deﬁnition of J ( j ) and Proposition E.25, together with the previous bullet point, we know that for all η sufﬁciently small,
[Equation]
uniformly for all k ≥ 0 .
• Repeat this argument for n min times, and we can see that for all η sufﬁciently small
[Equation]
uniformly for all k ≥ 1 .
• Lastly, for any j (cid:54)
=
i with m j ∈ G and x ∈ B ( m j , 2∆) and any u = 1 , 2 , · · · ,
[Equation]
In summary, now we can see that (for suffciently small η )
[Equation]
uniformly for all u = 1 , 2 , · · · .
To conclude the proof, it sufﬁces to set p = ( p ∗ 2 ) n min .
We are now ready to prove Lemma F.1, which, as demonstrated earlier, is the key tool in proof of
Theorem 2.
Proof of Lemma F.1.
The claim is trivial if l large = 1 , so we focus on the case where l large ≥ 2 .
Fix some
[Equation]
Let q ∗ = max j µ j ( E j (0)) .
We show that for any t ∈ (0 , δ 4 q ∗ ) the claim is true.
Now we only consider ∆ ∈ (0 , ¯ (cid:15)/ 3) and
η small enough so that ηM ≤ η γ and η γ < ∆ .
Consider the following stopping times
[Equation]
First, from Lemma F.8, we know that
[Equation]
for all η sufﬁciently small.
Besides, by combining Lemma F.7 with Markov property (applied at
T γ escape ), we have
[Equation]
for all η sufﬁciently small.
Therefore, for all η sufﬁciently small,
[Equation]
[ − L,L ]
[Equation]
Let J be the unique index such that X ηT γ return ∈ Ω J .
Our next goal is to show that, almost always, the
SGD iterates will visit the local minimum at some large attraction ﬁelds.
Therefore, without loss of generality, we can assume that m J / ∈ M large , and deﬁne
[Equation]
and introduce the following deﬁnitions:
[Equation]
In other words
[Equation]
the sequence of stopping times ( τ k ) k ≥ 1 is the time that, starting from T
γ return
, the SGD iterates visited a local minimum that is different from the one visited at τ k − 1 , and ( J k ) k ≥ 0 records the label of the visited local minima.
The random variable K is the number of transitions required to visit a minimizer in a large attraction ﬁeld.
From Lemma F.9, we know the existence of some p ∗ >such that (for all η sufﬁciently small)
[Equation]
where Geom ( a ) is a Geometric random variable with success rate a ∈ (0 , 1) .
Therefore, one can ﬁnd integer N ( δ ) such that (for all sufﬁciently small η )
[Equation]
L
[Equation]
Next, given results in Proposition E.25 and the fact that there are only ﬁnitely many attraction ﬁelds, one can ﬁnd a real number u ( δ ) such that (for all sufﬁciently small η )
[Equation]
uniformly for all k = 1 , 2 , · · · , N ( δ ) .
From eq.
(F.44), eq.
(F.45),eq.
(F.46), we now have sup x ∈ [ − L,L ]
[Equation]
≤
[Equation]
for any sufﬁciently small η .
To conclude the proof we just observe the following facts.
First, due to
H ∈ RV − α and l large ≥ 2 , we have
[Equation]
Therefore, for sufﬁciently small η , we will have (note that (cid:15), δ are ﬁxed constants in this proof, so
N ( δ ) , u ( δ ) are also ﬁxed)
[Equation]
Second, recall that we ﬁxed some t ∈ (0 , δ 4 q ∗ ) where q ∗ = max j µ j ( E j ) .
Also, choose some C >small enough so that
[Equation]
From Proposition E.23 and the fact that there are only ﬁnitely many attraction ﬁelds, there exists some ¯ η 0 > 0 such that for any η ∈ (0 , ¯ η 0 ) and any ∆ > 0 sufﬁciently small,
[Equation]
i
[Equation]
Combine this bound with Markov property (applied at τ K ), and we obtain that
[Equation]
x
[Equation]
− L,L
[Equation]
for all η sufﬁciently small.
Together with eq.
(F.47)eq.
(F.48), we have shown that sup x ∈ [ − L,L ]
P x (cid:16) V
[Equation]
( η,
[Equation]
t ) > (cid:15) (cid:17) ≤ 5 δ holds for all η sufﬁciently small.
F. P ROOF OF L EMMA F.4, F.
We shall return to the discussion about the dynamics of SGD iterates on a communication class G .
Recall that
[Equation]
If X ηn is initialized at some sharp minimum on G , then we are interested in the behavior of X ηn at the ﬁrst visit to some large attraction ﬁelds on G .
Deﬁne
[Equation]
Not only is this deﬁnition of T G analogous to the one for T DTMCG in eq.
(F.16), but, as illustrated in the next lemma, T G also behaves similarly as T G on a communication class G in the following sense: the probabilities p i,j deﬁned in eq.
(F.17) govern the dynamics regarding which large attraction ﬁeld on G is the ﬁrst one to be visited.
Besides, T G is usually rather small, meaning that the SGD iterates would efﬁciently arrive at a large attraction ﬁeld on G or simply escape from G .
Lemma F.11.
Given any θ ∈ (0 , ( α − 1) / 2) , (cid:15) ∈ (0 , 1) , i, j ∈ [ n min ] such that m i ∈
G small , m j / ∈
G large , the following claims hold for all ∆ > 0 that is sufﬁciently small:
[Equation]
Proof.
For G small (cid:54) = ∅ to hold (and the discussion to be meaningful), we must have l ∗ G ≥ 2 .
Throughout the proof, we assume this is the case.
Besides, we require that ∆ ∈ (0 , ¯ (cid:15)/ 3) so we have
[Equation]
] and the 3∆ -neighborhood of each local minimum will not intersect with each other.
In this proof we will only consider ∆ in this range.
From Lemma F.9 (if G is absorbing) or Lemma F.10 (if G is transient), we know the existence of some integer N ( (cid:15) ) such that for (see the deﬁnition of I k in eq.
(F.39)-eq.
(F.42))
[Equation]
we have
[Equation]
for all η sufﬁciently small.
Fix such N ( (cid:15) ) .
Next, from Proposition E.25, we can ﬁnd u
( (cid:15) ) ∈ (0 , ∞ ) and ¯∆ ∈ (0 , ¯ (cid:15)/ 3) such that for all ∆ ∈ (0 , ¯∆) , we have
[Equation]
for all η sufﬁciently small.
Fix such u ( (cid:15) ) and ¯∆ .
Now note that on the event
[Equation]
due to the choice of θ ∈ (0 , ( α − 1) / 2) and H ∈ RV − α , we have (when η ∈
[Equation] [Equation]
For any η sufﬁciently small, we will have N ( (cid:15) ) u
( (cid:15) )
η 2 θ λ G ( η ) <
η θ λ G ( η ) .
In summary, we have established that for all ∆ ∈ (0 , ¯∆) ,
[Equation]
Next, let
[Equation]
We can see that S ( (cid:15) ) contains all the possible transition path for Y DTMC where the state m j is visited within the ﬁrst N ( (cid:15) ) steps.
Obviously, | S ( (cid:15) ) | < ∞ .
Let (cid:15)
= (cid:15)/ | S ( (cid:15) )
| .
If we are able to show the existence of some ¯∆ 1 > 0 such that for all ∆ ∈ (0 , ¯∆ 1 ) , the following claim holds for any ( m (cid:48) k ) N
[Equation]
) k
=
[Equation]
∈ S
[Equation]
(cid:15) ) :
lim sup η ↓sup x ∈ B ( m i , 2∆) (cid:12)(cid:12)(cid:12) P x (cid:16) m I k
= m (cid:48)
k ∀ k ∈
[ N ( (cid:15) )] (cid:17) − P (cid:16) Y DTMCk ( m i ) = m (cid:48)
k ∀ k ∈
[ N ( (cid:15) )] (cid:17)(cid:12)(cid:12)(cid:12) < (cid:15) 1 , (F.51)
then we must have (for all ∆ ∈ (0 , ¯∆ ∧ ¯∆ 1 ) )
lim sup η ↓sup x ∈ B ( m i , 2∆) (cid:12)(cid:12)(cid:12) P x (cid:16) X ηT G ∈ B ( m j , 2∆) (cid:17) − p
i,j (cid:12)(cid:12)(cid:12) =
lim sup η ↓sup x ∈ B ( m i , 2∆) (cid:12)(cid:12)(cid:12) P x (cid:16) X ηT G ∈ B ( m j , 2∆) , T G ≤ N ( (cid:15) ) (cid:17) + P x (cid:16) X ηT G ∈ B ( m j , 2∆) , T G > N ( (cid:15) ) (cid:17) − P (cid:16)
Y DTMCT
DTMCG ( m i ) =
m j , T DTMCG ≤ N ( (cid:15) ) (cid:17) − P (cid:16) Y DTMCT
DTMCG ( m i ) =
m j , T DTMCG > N ( (cid:15) ) (cid:17)(cid:12)(cid:12)(cid:12) ≤ lim sup η ↓sup x ∈ B ( m i , 2∆) (cid:12)(cid:12)(cid:12) P x (cid:16) X ηT G ∈ B ( m j , 2∆) , T G ≤ N ( (cid:15) ) (cid:17) − P (cid:16) Y DTMC T DTMC G ( m i ) =
m j , T DTMC G ≤ N ( (cid:15) )
(cid:17)(cid:12)(cid:12) (cid:12) + lim sup η ↓sup x ∈ B ( m i , 2∆)
P x ( T G > N ( (cid:15) ))
+ P ( T DTMCG ( m i ) >
N ( (cid:15) ))
≤| S ( (cid:15) ) | (cid:15) 1 + lim sup η ↓sup x ∈ B ( m i , 2∆)
P x ( T G > N ( (cid:15) ))
+ P ( T DTMCG ( m i ) >
N ( (cid:15) )) ≤ 3 (cid:15).
To show that eq.
(F.51) is true, we ﬁx some ( m (cid:48) k )
N ( (cid:15) )
k =1 ∈ S ( (cid:15) ) and let ( k (cid:48)
( k ))
N ( (cid:15) )
k
=1 be the sequence with m k (cid:48)
( k ) = m (cid:48)
k for each k ∈
[ N ( (cid:15) )] .
From the deﬁnition of Y DTMC we have (let k (cid:48) (0) = i )
P
[Equation]
Y DTMCk
[Equation]
m i )
[Equation]
m (cid:48)
k ∀ k ∈
[ N ( (cid:15) )] (cid:17) =
N ( (cid:15) ) − 1 (cid:89) k
= µ k (cid:48) ( k ) ( E k (cid:48) ( k ) , k (cid:48)
( k +1) )
µ k (cid:48)
( k ) ( E k (cid:48) ( k ) ) .
On the other hand, using Proposition E.25, we know that for any arbitrarily chosen (cid:15) (cid:48) ∈ (0 , 1) , we have
[Equation]
k
[Equation]
(cid:48)
[Equation] [Equation]
(cid:48)
[Equation]
(cid:48)
[Equation] [Equation]
µ k (cid:48) (
[Equation]
( E
[Equation]
(cid:48) (
[Equation]
· (1 − (cid:15) (cid:48) )
[Equation]
for all ∆ > 0 sufﬁciently small.
The arbitrariness of (cid:15) (cid:48) > 0 , together with | S ( (cid:15) ) | < ∞ , allows us to see the existence of some ¯∆ 1 > 0 such that with ∆ ∈ (0 , ¯∆ 1 ) , eq.
(F.51) holds for any ( m (cid:48) k ) N ( (cid:15) )
k =1 ∈ S ( (cid:15) ) .
To conclude the proof, observe that
[Equation]
due to eq.
(F.50).
Recall that continuous-time process X ∗ ,η is the scaled version of X η deﬁned in eq.
(F.13), and the mapping T ∗ ( n, η ) =
∆ nλ G ( η ) returns the timestamp t for X ∗ ,η t corresponding to the unscaled step n on the time horizon of X ηn .
As an inverse mapping of T ∗ , we deﬁne the mapping N ∗ ( t, η ) = (cid:98) t/λ G ( η ) (cid:99) that maps the scaled timestamp t back to the step number n for the unscaled process X η .
In the next lemma, we show that, provided that X ∗ ,η stays on a communication class G before some time t , the scaled process X ∗ ,η t is almost always in the largest attraction ﬁelds of a communication class G .
Lemma F.12.
Let G be a communication class on the graph G .
Given any (cid:15) 1 > 0 , t > 0 and any x ∈ Ω i with m i ∈ G , the following claim holds for all ∆ > 0 small enough:
[Equation]
Proof.
Let ∆ ∈ (0 , ¯ (cid:15)/ 3) for the constant ¯ (cid:15) in eq.
(E.28)eq.
(E.29), so we are certain that each
B ( m i , 2∆) lies entirely in Ω i and would not intersect with each other since
B ( m i , 3∆) ∩ Ω ci = ∅ ∀ i ∈ [ n min ] .
Besides, with (cid:15) = ∆ / 3 , we know the existence of some δ > 0 such that claims in Lemma E.14 would hold of the chosen (cid:15), δ .
Fix such δ for the entirety of this proof.
Lastly, ﬁx some
[Equation]
The blueprint of this proof is as follows.
We will deﬁne a sequence of stopping times ( N j ) 6 j = such that the corresponding scaled timestamps T ∗ j = T ∗ ( N j , η ) gradually approach t .
By analyzing the behavior of X ∗ ,η on a time interval [ t − ∆ t , t ] that is very close to t (in particular, on the aforementioned stopping times T ∗ j ), we are able to establish the properties of a series of events
A 1 ⊇ A 2 ⊇ A 3 .
Moreover, we will show that A 3 ⊆ { X ∗ ,η, ∆ t ∈ (cid:83) i : m i ∈
G large B ( m i , 3∆) } , so the properties about events A, A 2 , A 3 can be used to bound the probability of the target event.
Arbitrarily choose some ∆ t ∈ (0 , t ) .
To proceed, let N 0 = ∆ N ∗ ( t − ∆ t , η ) be the stopping time corresponding to timestamp t − ∆ t for the scaled process.
Using Lemma F.8, we know that for stopping time N 1 = ∆ min { n ≥ N 0 : X ηn / ∈ ∪ j B ( s j , 2 η γ ) } , we have
[Equation]
Next, let N 2 = ∆ min { n ≥ N 1 : X ηn ∈ ∪ j B ( m j , 2∆) } .
From Lemma F.7 and H ∈ RV − α
(so that log(1 /η ) /η
=
o ( H (1 /η )) , we have
[Equation]
Collecting results above, we have
[Equation]
Now note the following fact on the event { N 2 − N 0 < ∆ t / 2 H (1 /η ) } .
The deﬁnition of the mapping
T ∗ implies that, for any pair of positive integers n 1 ≤ n 2 , we have T ∗ ( n 2 , η ) − T ∗ ( n 1 , η ) =
( n 2 − n 1 ) λ G ( η ) ≤ ( n 2 − n 1 ) · H (1 /η ) .
Therefore, on { N 2 − N 0 ≤ ∆ t / 2 H (1 /η ) } we have
[Equation]
Besides, let T ∗ 2 =
T ∗ ( N 2 , η ) .
Now we can see that for event
[Figure]
[Equation]
we have
[Equation]
Meanwhile, from eq.
(F.52) we obtain that
[Equation]
Moving on, we consider the following stopping times
[Equation]
Using Lemma F.
[Equation]
we have
[Equation]
Meanwhile, on event A 1 ∩ (cid:110) N 3 − N 2 ≤ ∆ t / 4 λ G ( η ) (cid:111) ∩ (cid:110) X
[Equation]
∈ (cid:83) k : m k ∈ G Ω k ∀ s ∈ [0 , t ] (cid:111) , we have
T ∗ 3 − T ∗ 2 ≤ ∆ t / 4 , hence T ∗ 3 ∈
[ t − ∆ t , t − ∆ t / 4] .
In summary,
[Equation]
Moreover, on event (cid:110) X
[Equation]
∈ (cid:83)
[Equation] [Equation]
] (cid:111) , if we let J 3 be label of the local minimum visited at T ∗ 3 such that J 3 =
j ⇐⇒ X ∗ ,ηT ∗ 3 ∈ B ( m j , 2∆) , then we must have m J 3 ∈ G large .
Meanwhile, consider the following stopping times
[Equation]
From Proposition E.25, we know that
[Equation]
55) where q ∗ = max j µ j ( E j ) .
Now we deﬁne the event
A
[Equation]
Using eq.
(F.53)-eq.
(F.55)
, we get
[Equation]
Furthermore, on event A 2 , due to T ∗ 3 ∈ [ t − ∆ t , t − ∆ t / 4] as established above, we must have
[Equation]
Now let us focus on a timestamp T ∗ 4 = t − ∆ t / 8 H (1 /η )
λ G ( η ) and N 4 = N ∗ ( T ∗ 4 , η ) .
Obviously, T ∗ 4 > T ∗on event A 2 .
Next, deﬁne
[Equation]
Using Lemma F.7 and F.8 again as we did above when obtaining eq.
(F.52), we can show that
[Equation]
On the other hand, on event A 2 ∩ (cid:110) N 5 − N 4 ≤ ∆ t / 16 H (1 /η )
(cid:111)
we must have
• T ∗ 5 − T ∗ 4 ≤ ∆ t / 16 H (1 /η )
λ G ( η ) , so T ∗ 5 ∈
[ t − ∆ t / 8 H (1 /η )
λ G ( η ) , t − ∆ t / 16 H (1 /η )
λ G ( η )]
• X ∗ ,η T ∗ 5 ∈ Ω J 3 , due to T σ − T ∗ 3 > ∆ t .
This implies that for event (cid:101)
[Equation]
[Equation]
T
[Equation] [Equation]
we have A 2 ∩ (cid:110) N 5 − N 4 ≤ ∆ t / 16 H (1 /η )
(cid:111) ⊆
(cid:101) A .
Lastly, observe that
• From Lemma E.4, we know that for N 6 ( δ ) =
∆ min { n > N 5 : η | Z n | > δ } we have
[Equation]
• As stated at the beginning of the proof, our choice of δ allows us to apply Lemma E.14 and show that
[Equation]
• Combining the two bullet points above, we get lim sup
[Equation] [Equation]
such that
[Equation] [Equation]
On the other hand,
[Equation]
In summary, for event
[Equation]
we have A 3 ⊆ { X ∗ ,η t ∈ (cid:83) k : m k ∈ G large B ( m k , 3∆) } .
Besides, due to eq.
(F.56)eq.
(F.57)eq.
(F.58), we get
[Equation]
L
[Equation]
Remember that δ, (cid:15) 1 , q ∗ are ﬁxed constants while ∆ t can be made arbitrarily small, so by driving ∆ t to 0 we can conclude the proof.
Recall the deﬁnition of jump processes in Deﬁnition F.1.
Central to the proof of Lemma F.4, the next result provides a set of sufﬁcient conditions for the convergence of a sequence of such jump processes in the sense of ﬁnite dimensional distributions.
Lemma F.13.
For a sequence of processes ( Y n ) n ≥ 1 that, for each n ≥ 1 , Y n is a (cid:16) ( U nj )
j ≥ 0 , ( V nj ) j ≥ 0 (cid:17) jump process, and a (cid:16) ( U j ) j ≥ 0 , ( V j ) j ≥ 0 (cid:17) jump process Y , if
• U 0 ≡ 0 ; • ( U n 0 , V n 0 , U n 1 , V n 1 , U n 2 , V n 2 , · · · ) converges in distribution to (0 , V 0 , U 1 , V 1 , U 2 , V 2 , · · · ) as n → ∞ ; • For any x > 0 and any n ≥ 1 ,
[Equation]
• For any x > 0 , lim n →∞
[Equation]
then the ﬁnite dimensional distribution of Y n converges to that of Y in the following sense: for any k ∈ N and any 0 < t 1 < t 2 < · · · < t k < ∞ , the random element ( Y nt 1 , · · · , Y nt k ) converges in distribution to ( Y t 1 , · · · , Y t k ) as n → ∞ .
Proof.
Fix some k ∈ N and 0 < t 1 < t 2 < · · · < t k < ∞ .
For notational simplicity, let t = t k .
Let ( D , d ) be the metric space where D = D
[0 ,t ] , the space of all càdlàg functions in R on the time interval [0 , t ] , and d is the Skorokhod metric deﬁned as
[Equation]
where Λ is the set of all nondecreasing homeomorphism from [0 , t ] onto itself, and I ( s )
= s is the identity mapping.
Also, we arbitrarily choose some (cid:15) ∈ (0 , 1) and some open set A ⊆ R k .
From the assumption, we can ﬁnd integer J ( (cid:15) ) such that P (cid:16) (cid:80) J ( (cid:15) )
j =1 U j ≤ t (cid:17) < (cid:15) , as well as an integer
N ( (cid:15) ) such that, for all n ≥ N ( (cid:15) ) , we have P (cid:16) (cid:80) J ( (cid:15) )
j =1 U nj ≤ t (cid:17) + P (cid:16) U n 0 ≥ t 1 (cid:17) < (cid:15) .
We ﬁx such
J ( (cid:15) ) , N ( (cid:15) )
(we may abuse the notation slightly and simply write J, N when there is no ambiguity).
Using Skorokhod’s representation theorem, we can construct a probability space ( Ω , F , Q ) that supports random variables ( (cid:101) U n 0 , (cid:101) V n 0 , · · · , (cid:101) U nJ , (cid:101) V nJ ) n ≥ 1 and ( (cid:101) U 0 , (cid:101) V 0 , · · · , (cid:101) U J , (cid:101) V J ) and satisﬁes the following conditions:
• L ( U n 0 , V n 0 , · · · , U nJ , V nJ ) =
L ( (cid:101) U n 0 , (cid:101) V n 0 , · · · , (cid:101) U nJ , (cid:101) V nJ ) for all n ≥ 1 ; • L ( U 0 , V 0 , · · · , U J , V J )
=
L ( (cid:101) U 0 , (cid:101) V 0 , · · · , (cid:101) U J , (cid:101) V J ) ; • U nj a
s. −−→ U j and V nj a.s. −−→ V j as n → ∞ for all j ∈ [ J ] .
Therefore, on ( Ω , F , Q ) we can deﬁne the following random elements (taking values in the space of càdlàg functions):
[Equation]
Note that (1) for the ﬁrst jump time of Y ↓ J we have (cid:101) U 0 ≡ 0 , hence Y ↓ J= (cid:101) V 0 ; (2) when deﬁning
Y n, ↓ J we set its value on [0 , (cid:101) U n 0 ) to be (cid:101) V n 0 instead of 0.
Since U nj a.s. −−→ U j and V nj a.s. −−→ V j as n → ∞ for all j ∈
[ J ] , we must have
[Equation]
almost surely, which further implies that Y n, ↓ J s ⇒ Y ↓
J s as n → ∞ on ( D , d ) .
Now from our assumption that, for the jump times U 1 + · · · + U j , we have P ( U 1 + · · · + U j = x ) =
∀ x > 0 , j ≥ 1 , as well as (13.3) in Billingsley (2013), we then obtain
[Equation]
as n → ∞ .
Recall that A is the open set we arbitrarily chose at the beginning of the proof, and (cid:15) >is also chosen arbitrarily.
Now we observe the following facts.
• Using eq.
(F.59), we can see that
[Equation]
• The choice of N ( (cid:15) ) and J ( (cid:15) ) above implies that
[Equation]
Y
[Equation]
[Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
Collecting the two results above, we have established that
[Equation]
From Portmanteau theorem, together with arbitrariness of (cid:15) > 0 and open set A , we can now conclude that ( Y nt 1 , · · · , Y nt k ) converges in distribution to ( Y t 1 , · · · , Y t k ) .
The following lemma concerns the scaled version of the marker process ˆ X ∗ ,η, ∆ deﬁned in eq.
(F.13)-eq.
(F.14).
Obviously, it is a jump process that complies with Deﬁnition F.1.
When there is no ambiguity about the sequences ( η n ) n ≥ 1 and (∆ n ) n ≥ 1 , let ˆ X ( n ) t =
∆ ˆ X ∗ ,η n , ∆ n t .
From eq.
(F.6)-eq.
(F.11) and eq.
(F.15), we know that for any n ≥ 1 , ˆ X ( n ) is a (cid:16)(cid:0)
τ ∗ k ( η n , ∆ n ) − τ ∗ k − 1 ( η n , ∆ n ) (cid:1) k ≥ 0 , (cid:0) m I k ( η n , ∆ n ) (cid:1) k ≥ 0 (cid:17) -jump process (with the convention that τ ∗− 1 = 0 ).
Also, for clarity of the exposition, we let (for all n ≥ 1 , k ≥ 0 )
[Equation]
Lastly, remember that Y is the continuous-time Markov chain deﬁned in eq.
(F.18)-eq.
(F.21) and π G ( · ) is the random mapping deﬁned in eq.
(F.22) that is used to initialize Y .
Besides, Y is a (cid:0) ( S k ) k ≥ 0 , ( W k ) k ≥ 0 (cid:1) jump process under Deﬁnition F.1, with S 0 = 0 and W
= π G ( m i ) (here x ∈ Ω i and X η 0 = x , so i is the index of the attraction ﬁeld where the SGD iterate is initialized).
The following result states that, given a sequence of learning rates ( η n ) n ≥ 1 that tend to 0 , we are able to ﬁnd a sequence of (∆ n ) n ≥ 1 to parametrize ˆ X ( n )
=
ˆ X ∗ ,η n , ∆ n , X ( n ) =
X ∗ ,η n , ∆ n so that they have several useful properties, one of which is that the jump times and locations of ˆ X ( n ) converges in distribuiton to those of Y ( π G ( m i )) .
Lemma F.14.
Assume the communication class G is absorbing.
Given any m i ∈ G , x ∈ Ω i , ﬁnitely many real numbers ( t l ) k (cid:48) l =1 such that 0 < t 1 < t 2 < · · · < t k (cid:48) , and a sequence of strictly positive real numbers ( η n ) n ≥ 1 with lim n → 0 η n = 0 , there exists a sequence of strictly positive real numbers (∆ n ) n ≥ 1 with lim n ∆ n = 0 such that
• Under P x
(so X η 0 = x ), as n tends to ∞ ,
[Equation]
• (Recall the deﬁnition of T k , I k in eq.
(F.39) -eq.
(F.42) )
Given any (cid:15) > 0 , the following claim
holds
for all n sufﬁciently large:
[Equation]
• Given any (cid:15) > 0 , the following claim holds for all n sufﬁciently large,
[Equation]
• For any l ∈ [ k (cid:48) ] ,
[Equation]
where p ∗ > 0 is a constant that does not vary with our choices of η n or ∆ n .
Proof
[Equation]
Let
[Equation]
so from the deﬁnition of q j,k we have q j,k = 1 { j (cid:54) =
k } ν j,k + (cid:80) l : m l ∈ G small ν j,l p l,k .
In order to specify our choice of (∆ n ) n ≥ 1 , we consider a construction of sequences ( ¯ ∆ ( j ))
j ≥ 0 , (¯ η ( j ))
j ≥ 0 as follows.
Fix some θ ∈ (0 , α − 1) / 2) .
Let ¯ ∆ (0) =
¯ η (0) = 1 .
One can see the existence of some ( ¯ ∆ ( j ))
j ≥ 1 , (¯ η ( j ))
j ≥ 1 such that
• ¯ ∆ ( j ) ∈ (cid:0) 0 , ¯ ∆ ( j − 1) / 2 (cid:3) , ¯ η ( j ) ∈ (cid:0) 0 , ¯ η ( j − 1) / 2 (cid:3) for all j ≥ 1 ; • (Due to Lemma F.7) for any j ≥ 1 , η ∈ (0 , ¯ η ( j )] , (remember that x
and i are the ﬁxed constants prescribed in the description of the lemma)
[Equation]
For deﬁnitions of σ Gk , τ Gk , I Gk , (cid:101) I
Gk , see eq.
(F.6)-eq.
(F.11).
• (Due to Lemma F.11) for any j ≥ 1 , η ∈ (0 , ¯ η ( j )] , (cid:12)(cid:12) (cid:12)(cid:12) P x (cid:16) τ ∗ k (cid:0) η, ¯ ∆ ( j ) (cid:1) − σ ∗ k (cid:0) η, ¯ ∆ ( j ) (cid:1) <
η θ
[Equation]
< 1 / 2 j uniformly for all k ≥ 0 and all m i
1 ∈ G small , m i 2 ∈ G large .
Also, by deﬁnition of σ ∗ and τ ∗ , we must have
[Equation]
all k ≥ 0 and m i
∈ G large .
• (Due to Proposition E.25) for any j ≥ 1 , η ∈ (0 , ¯ η ( j
[Equation] [Equation]
iuniformly for all k ≥ 1 , all u > 1 / 2 j , and all m i 1 ∈ G large , m i 2 ∈ G .
• (Due to G being absorbing and, again, Proposition E.25) for any j ≥ 1 , for any j ≥ 1 , η ∈ (0 , ¯ η ( j )] , (Recall the deﬁnition of T k , I k in eq.
(F.39)-eq.
[Equation]
F.42))
[Equation]
64) uniformly for all k ≥ 0 , m i
∈ G
, m i 2 /
∈ G
• (Due to Lemma F.9)
There exists some p ∗ > 0 such that for any j ≥ 1 , for any j ≥ 1 ,
[Equation]
uniformly for all k
≥
u ≥
and m i
1 ∈ G
Due to Lemma F.12)
for any j ≥ 1 , for any j ≥ 1 , η ∈ (0 , ¯ η ( j )]
[Equation]
uniformly for all k ∈ [ k (cid:48) ] .
Fix such ( ¯ ∆ ( j ))
j ≥ 0 , (¯ η ( j ))
j ≥ 0 .
Deﬁne a function J ( · ) : N (cid:55)→ N as
[Equation]
n
[Equation]
with the convention that max ∅ = −∞ .
Lastly, let
[Equation]
Note that, due to lim n η n = 0 , we have lim n J ( n ) = ∞ , hence lim n ∆ n = 0 .
Besides, the deﬁnition of J ( · ) tells us that in case that J ( n ) ≥ 1 (which will hold for all n sufﬁciently large), the claims above holds with η = η n and j = J ( n ) .
In particular, by combining lim n J ( n ) = ∞ with eq.
(F.64)eq.
(F.65)eq.
(F.66) respectively, we have eq.
(F.61)eq.
(F.62)eq.
(F.63).
Now it remains to prove eq.
(F.60).
To this end, it sufﬁces to show that, for any positive integer
K , we have ( S ( n ) 0 , W ( n ) 0 , · · · , S ( n ) K , W ( n ) K ) converges in distribution ( S 0 , W 0 , · · · , S K , W K ) as n tends to inﬁnity.
In particular, note that S 0 = 0 , W 0 =
π G ( m i ) , so W 0 = m j with probability p
i,j if m i ∈ G small , and W 0 ≡ m i if m i ∈
G large .
For clarity of the exposition, we restate some important claims above under the new notational system with (cid:101) S ( n ) k , (cid:102) W ( n ) k , S ( n ) k , W ( n ) k we introduced right above this lemma.
Given any (cid:15) > 0 , the following claims hold for all n sufﬁciently large:
• First of all
[Equation]
• For all k ≥ 0 and all m i 1 ∈ G small , m i 2 ∈ G large
[Equation]
k
[Equation]
• For all k ≥ 0 and all m i 1 ∈ G large ,
[Equation]
• For all k ≥ 0 , all m i 1 ∈ G large , m i 2 ∈ G and all u > (cid:15) ,
[Equation]
• Here is one implication of eq.
(F.68).
Since | G | ≤ n min , we have
[Equation]
for all k ≥ 0 and m i 1 ∈ G small .
• Note that for any m i 1 , m i 2 ∈ G large and any k ≥
[Equation]
[Table] [Table]
[Table] [Table]
[Equation]
On the other hand
[Equation]
by considering the integral on ( u − η θn
[Equation]
∞ )
[Equation]
we get
[Equation]
due to eq.
(F.68) and eq.
(F.70).
Meanwhile,
[Equation]
due to eq.
(F.68), eq.
(F.70) and eq.
(F.71).
• Therefore, for any m i 1 , m i 2 ∈ G large and any k ≥ 0 ,
[Equation]
( n ) k > u, W ( n ) k
+
= m i 2 (cid:12)(cid:12)(cid:12) W ( n ) k
= m i 1 (cid:17) ≤ g ( (cid:15) )
+ exp (cid:0) − (1 − (cid:15) )
q i 1 u (cid:1) 1 { i 2 (cid:54) =
i 1 } ν i 1 ,i 2 + (cid:80) i 3 : m i
3 ∈
G small ν i 1 ,i 3 p i 3 ,iq i≤ g ( (cid:15) )
+ exp (cid:0) − (1 − (cid:15) )
q i 1 u (cid:1) q i 1 ,i 2 q i(F.72) and
P x (cid:16) S ( n ) k +1 − S ( n ) k > u, W ( n ) k
+
= m i 2 (cid:12)(cid:12)(cid:12) W ( n ) k
= m i 1 (cid:17) ≥ − g ( (cid:15) )
+ exp (cid:0) − (1 + (cid:15) )
q i 1 u (cid:1) 1 { i 2 (cid:54) =
i 1 } ν i 1 ,i 2 + (cid:80) i 3 : m i
3 ∈
G small ν
i 1 ,i 3 p i 3 ,iq i≥ − g ( (cid:15) )
+ exp (cid:0) − (1 + (cid:15) )
q i 1 u (cid:1) q i 1 ,i 2 q i (F.73) where q ∗ = max
i q i and g ( (cid:15) )
=
∆ (cid:15) + (cid:15) q ∗ + n min (1 + (cid:15) )
(cid:15) + (cid:15)
+ (cid:15) q ∗ n min + n min ( (cid:15) + (cid:15) q ∗ ) + n min ( n min (cid:15) + (cid:15) + 1)( [Table] [Equation] [Table]
[Equation]
Now we apply the bounds in eq.
(F.67)eq.
(F.72)eq.
(F.73) to establish the weak convergence claim regarding ( S ( n ) 0 , W ( n ) 0 , · · · , S ( n ) K , W ( n ) K ) .
Fix some positive integer K , some strictly positive real numbers ( s k )
Kk =0 , a sequence ( w k )
Kk =0 ∈ (cid:0) G large (cid:1) K +1 with w k
= m i k for each k , and some (cid:15) >such that (cid:15) < min k =0 , 1 , ··· ,K { s k } .
On the one hand, the deﬁnition of the CTMC Y implies that
P (cid:16) S 0 < t 0 , W 0 = w 0 ; S k > s k and W k = w k ∀ k ∈
[ K ] (cid:17) =
P ( π G ( m i ) = w 0 )
K (cid:89) k
= (cid:16) S k > s k and W k = w k (cid:12)(cid:12)(cid:12) W k − 1 = w k − 1 (cid:17) =
(cid:16) 1 { m i ∈
G large , i
= i } + 1 { m i ∈
G small } p i 0 ,i 1 (cid:17) ·
K (cid:89) k
= exp( − q i
k − 1 s k )
q
i k − 1 ,i k q i k −.
On the other hand, using eq.
(F.67)eq.
(F.72)eq.
(F.73), we know that for all n sufﬁciently large,
P x (cid:16) S ( n )< s 0 , W ( n )=
w 0 ; S ( n ) k > s k and W ( n ) k
= w k ∀ k ∈ [ K ] (cid:17) ≥ (1 − (cid:15) )
(cid:16) 1 { m i ∈
G large , i
[Equation]
and
P x (cid:16) S ( n
[Equation]
n )
[Equation]
and
[Equation]
( n )
[Equation]
Since (cid:15) > 0 can be arbitrarily small, we now obtain
[Equation]
and
[Equation] [Equation]
P
[Equation]