Fix some positive integer K , some strictly positive real numbers ( s k )
Kk =0 , a sequence ( w k )
Kk =0 ∈ (cid:0) G large (cid:1) K +1 with w k
= m i k for each k , and some (cid:15) >such that (cid:15) < min k =0 , 1 , ··· ,K { s k } .
On the one hand, the deﬁnition of the CTMC Y implies that
P (cid:16) S 0 < t 0 , W 0 = w 0 ; S k > s k and W k = w k ∀ k ∈
[ K ] (cid:17) =
P ( π G ( m i ) = w 0 )
K (cid:89) k
= (cid:16) S k > s k and W k = w k (cid:12)(cid:12)(cid:12) W k − 1 = w k − 1 (cid:17) =
(cid:16) 1 { m i ∈
G large , i
= i } + 1 { m i ∈
G small } p i 0 ,i 1 (cid:17) ·
K (cid:89) k
= exp( − q i
k − 1 s k )
q
i k − 1 ,i k q i k −.
On the other hand, using eq.
(H.67)eq.
(H.72)eq.
(H.73) , we know that for all n sufﬁciently large,
[Equation] )< s 0 , W ( n )=
w 0 ; S ( n ) k > s k and W ( n ) k
= w k ∀ k ∈ [ K ] (cid:17) ≥ (1 − (cid:15) )
(cid:16) 1 { m i ∈
G large , i {+
+}
= i } + 1 { m i ∈ G small } ( p i 0 ,i 1 − (cid:15) ) (cid:17) ·
K (cid:89) k = (cid:16) − g ( (cid:15) ) + exp( − (1 + (cid:15) ) q i k − 1 s k ) q i k − 1 ,i k q i k −(cid:17) and
P x (cid:16) S ( n )< s 0 , W ( n )=
w 0 ; S ( n ) k > s k and W ( n ) k = w k ∀ k ∈ [ K ] (cid:17) ≤ (cid:16) 1 { m i ∈
{+G large , i 
+}
= i } + 1 { m i ∈ G small } ( p i 0 ,i 1 + (cid:15) ) (cid:17) ·
K (cid:89) k
= (cid:16) g ( (cid:15) ) + exp( − (1 − (cid:15) ) q i k − 1 s k ) q i k − 1 ,i k q i k −(cid:17) .
Since (cid:15) > 0 can be arbitrarily small, we now obtain
[Equation]
= P
[Equation]
s 0 , W
[Equation]
= w 0 ; S k > s k and W k = w k ∀ k ∈ [ K ] (cid:17) , and the arbitrariness of the integer K , the strictly positive real numbers ( s k )
Kk =0 , and the sequence ( w k )
Kk =0 ∈ (cid:0) G large (cid:1)
K +1 allows us to conclude the proof.
To extend the result above to the case where the communication class G is transient, we revisit the deﬁnition of the Y † in eq.
(H.26).
Let ¯ G
=
G large ∪ { † } and let m 0 = † (remember that all the local minimizers of f on [ − L, L ] are m 1 , · · · , m min ).
Meanwhile, using q i and q i,j in eq.
(H.20)eq.
(H.21), we can deﬁne [Equation] [Equation]
[Equation]
and q † 0 = 1 , q † i =
q i ∀
i ≥ 1 .
Next, ﬁx some i with m i ∈ G and x ∈ Ω i .
Deﬁne a sequence of random variables ( S † k ) k ≥ 0 , ( W † k ) k ≥ 0 such that S † k
=
and W
= π G ( m i ) , W † 0 = † 1 { W 0 / ∈
G large } + W 0 1 { W 0 ∈ G large } (see the deﬁnition of random mapping π G in eq.
(H.22))
and (for all k ≥
and i, j with m j , m l ∈ ¯ G )
[Equation]
Then it is easy to see that Y † ( π G ( m i )) deﬁned in eq.
(H.26) is a (cid:0) ( S † k ) k ≥ 0 , ( W † k ) k ≥ 0 (cid:1) jump process.
In particular, from at any state that is not † (namely, any m j with m j ∈ G large ), the probability that
Y † moves to † in the next transition is equal to the chance that, starting from the same state, Y moves to a state that is not in G .
Once entering m 0 = † , the process Y † will only make dummy jumps (with interarrival times being iid Exp(1)): indeed, we have q † 0 = q † 0 , 0 = 1 and q † 0 ,j = 0 for any j ≥ 1 , implying that, given W † k
= m 0 = † , we must have W † k
+
= m 0 = † .
These dummy jumps ensure that Y † is stuck at the cemetery state † after visiting it.
Similarly, we can characterize the jump times and locations of the jump process ˆ X † , ∗ ,η, ∆ (for the deﬁnition, see eq.
(H.25)).
When there is no ambiguity about the sequences ( η n ) n ≥ 1 , (∆ n ) n ≥ 1 , let ˆ X † , ( n )
= ˆ X † , ∗ ,η n , ∆ n and X † , ( n ) =
X ∗ ,η n , ∆ n .
Also, recall that τ G deﬁned in eq.
(H.4) is the step n when X ηn exits the communication class G .
Now let ( E k ) k ≥ 0 be a sequence of iid Exp(1) random variables that is also independent of the noises ( Z k ) k ≥ 1 (so they are independent from the SGD iterates X ηn ).
For all n ≥ 1 , k ≥ 0 , deﬁne (see eq.
(H.6)-eq.
(H.11) and eq.
(H.15) for deﬁnitions of the quantities involved)
[Equation]
otherwise
[Equation]
otherwise
[Equation]
otherwise
[Equation]
† otherwise with the convention that τ ∗− 1 = 0 .
Note that T ∗ ( τ G ( η n ) , η n ) is the scaled timestamp for X ( n ) =
X ∗ ,η corresponding to τ G ( η n ) , hence T ∗ ( τ G ( η n ) , η n ) =
min { t ≥ 0 : X ( n ) / ∈ (cid:83) j : m j ∈ G Ω j } .
One can see that ˆ X † , ( n ) is a (cid:0) ( S † , ( n [Equation] [Equation] [Equation]
[Equation]
jump process.
The next lemma is similar to Lemma
H.14 and discusses the convergence of the jump times and locations of ˆ X † , ( n ) on a communication class G in the transient case.
Lemma H.15.
Assume that the communication class G is transient.
Given any m i ∈ G , x ∈ Ω i , ﬁnitely many real numbers ( t l ) k (cid:48) l =1 such that 0 < t 1 < t 2 < · · · < t k (cid:48) , and a sequence of strictly positive real numbers ( η n ) n ≥ 1 with lim n → 0 η n = 0 , there exists a sequence of strictly positive real numbers (∆ n ) n ≥ 1 with lim n ∆ n = 0 such that
• Under P x
(so X η 0 = x ), as n tends to ∞ , ( S † , ( n ), W † , ( n )
[Equation]
S
[Equation]
• For any l ∈ [ k (cid:48) ] ,
[Equation]
Proof.
Let
[Equation]
In order to specify our choice of (∆ n ) n ≥ 1 , we consider a construction of sequences ( ¯ ∆ ( j ))
j ≥ 0 , (¯ η ( j ))
j ≥ 0 as follows.
Fix some θ ∈ (0 , α − 1) / 2) .
Let ¯ ∆ (0) =
¯ η (0) = 1 .
One can see the existence of some ( ¯ ∆ ( j ))
j ≥ 1 , (¯ η ( j ))
j ≥ 1 such that
• ¯ ∆ ( j ) ∈ (cid:0) 0 , ¯ ∆ ( j − 1) / 2 (cid:3) , ¯ η ( j ) ∈ (cid:0) 0 , ¯ η ( j − 1) / 2 (cid:3) for all j ≥ 1 ; • (Due to Lemma H.7) for any j ≥ 1 , η ∈ (0 , ¯ η ( j )] , (remember that x
and i are the ﬁxed constants prescribed in the description of the lemma)
[Equation]
For deﬁnitions of σ Gk , τ Gk , I Gk , (cid:101) I
Gk , see eq.
(H.6)-eq.
(H.11).
• (Due to Lemma H.11) for any j ≥ 1 , η ∈ (0 , ¯ η ( j )] , (cid:12) (cid:12)(cid:12)(cid:12) P x (cid:16) τ ∗ k (cid:0) η, ¯ ∆ ( j ) (cid:1) − σ ∗ k (cid:0) η, ¯ ∆ [Equation] j )
[Equation] < η θ [Equation]
I
[Equation] η
[Equation]
¯ ∆
[Equation]
(cid:12)(cid:12)(cid:12) < 1 / 2 j uniformly for all k ≥ 0 and all m i 1 ∈ G small , m i 2 / ∈ G small .
Also, by deﬁnition of σ ∗ and τ ∗ , we must have
[Equation]
for all k ≥ 0 and m i 1 ∈ G large .
• (Due to Proposition G.25) for any j ≥ 1 , η ∈ (0 , ¯ η ( j )] ,
[Equation]
uniformly for all k ≥ 1 , all u > 1 / 2 j , and all m i 1 ∈ G large , m i 2 ∈ G .
• (Due to Lemma H.12) for any j ≥ 1 , for any j ≥ 1 , η ∈ (0 , ¯ η ( j )] , P x (cid:16) X ( n ) t k / ∈
(cid:91)
j : m j ∈ G large B ( m j , ¯ ∆ ( j )) , X ( n ) s ∈ (cid:91) j : m j ∈ G Ω j ∀ s ∈ [0 , t k ] (cid:17) < 1 / 2 j (H.78) uniformly for all k ∈ [ k (cid:48)
Fix such ( ¯ ∆ ( j ))
j ≥ 0 , (¯ η ( j ))
j ≥ 0 .
Deﬁne a function J ( · ) : N (cid:55)→ N as
[Equation]
with the convention that max ∅ = −∞ .
Lastly, let
[Equation]
Note that, due to lim n η n = 0 , we have lim n J ( n ) = ∞ , hence lim n ∆ n = 0 .
Besides, since
X † , ( n ) t = X ( n ) t given X ( n ) s ∈ (cid:83) j : m j ∈ G Ω j for all s ∈ [0 , t ] , by combining lim n J ( n ) = ∞ with eq.
(H.78) we obtain eq.
(H.77).
Now it remains to prove eq.
(H.76).
To this end, it sufﬁces to show that, for any positive integer K , we have ( S † , ( n ), W † , ( n ), · · · , S † , ( n ) K , W † , ( n ) K ) converges in distribution ( S † 0 , W † 0 , · · · , S † K , W † K ) as n tends to inﬁnity.
In particular, due to introduction of the dummy jumps, we know that for any k with τ ∗ k ( η n , ∆ n ) ≥ τ G ( η n ) (in other words, ˆ X † , ( n ) has reached state † within the ﬁrst k jumps) we have S † , ( n )
k +1 ∼
Exp (1) and W † , ( n )
k
+1 ≡ † .
Similarly, for any k with S † 0 + · · · + S † k ≤ τ YG , we have
S † k +1 ∼
Exp
(1) and W † k +1 ≡ † .
Therefore, it sufﬁces to show
that, for any positive integer K , any series of strictly positive real numbers ( s k ) Kk =0 , any sequence ( w k ) Kk =0 ∈ (cid:0)
¯ G (cid:1) K +1 such that w j (cid:54) = † for any j < K , indices
i k such that w k = m i k for each k , we have
[Equation]
Fix some ( s k )
Kk =0 , ( w k )
Kk =0 ∈ (cid:0) ¯ G (cid:1) K +1 , and indices ( i k )
Kk
=1 satisfying the conditions above.
Besides, arbitrarily choose some (cid:15) >
so that (cid:15) < min k =0 , ··· ,K s k .
To proceed, we start by translating the inequalities established above under the new system of notations.
• First of all, for all n sufﬁciently large, (remember that x
and i are prescribed constants in the description of this lemma)
[Equation]
• For all k ≥ 0 and all m i 1 ∈ G small , m i 2 ∈ G large , it holds for all n sufﬁciently large that
[Equation]
• On the other hand, for all k ≥ 0 and all m i 1 ∈ G small , it holds for all n sufﬁciently large that
[Equation]
• For all k ≥ 0 and all m i 1 ∈ G large , it holds for all n that
[Equation]
• For all k ≥ 0 , all m i 1 ∈ G large , m i 2 ∈ G and all u > (cid:15) , the following claim holds for all n sufﬁciently large:
[Equation]
• On the other hand, for all k ≥ 0 , all m i 1 ∈ G large , the following claim holds for all n sufﬁciently large:
[Equation]
• Here is one implication of eq.
(H.81)eq.
(H.82).
Since | G | ≤ n min , we have (when n is sufﬁciently large)
[Equation]
for all k ≥ 0 and m i 1 ∈ G small .
• Note that for any m i 1 , m i 2 ∈ G large and any k ≥
[Equation]
i 3 : m i 3 ∈ G small (cid:90) s>
P x (cid:16) S † , ( n ) k +1 − (cid:101) S † , ( n ) k +1 ≥ ( u − s ) ∨ 0 , (cid:102) W † , ( n ) k
{++
+}
= m i 2 (cid:12)(cid:12)(cid:12) (cid:102) W † , ( n ) k
= m i 3 (cid:17) · P x (cid:16)(cid:101) S † , ( n ) k +1 − S † , ( n ) k
= ds, (cid:102) W † , ( n ) k
{++
+}
= m i 3 (cid:12)(cid:12)(cid:12) W † , ( n ) k
= m i 1 (cid:17) .
Fix some i 3 with m i 3 ∈ G small .
Due to eq.
{+(H.86) (cid:90) s ∈ (0 ,u − η θn ] P x (cid:16) S † , ( n ) k +1 − (cid:101) S † , ( n ) k +1 ≥ u − s, (cid:102) W † , ( n ) k +
+}
{+= m i 2 (cid:12)(cid:12)(cid:12) (cid:102) W † , ( n ) k = m i 3 (cid:17) · P x (cid:16)(cid:101) S † , ( n ) k +1 − S † , ( n ) k = ds, (cid:102) W † , ( n ) k +
+}
= m i 3 (cid:12)(cid:12)(cid:12) W † , ( n ) k
= m i 1 (cid:17) ≤ n min (cid:15).
Meanwhile, by considering the integral on ( u − η θn , ∞ ) , we get
[Equation]
due to eq.
(H.81) and eq.
(H.84).
As for the upper bound
[Equation]
due to eq.
(H.81), eq.
(H [Equation] 84) and eq.
(H.86).
• Therefore, for any m i 1 , m i 2 ∈ G large and any k ≥ 0 ,
[Equation]
and
[Equation]
where q ∗ = max i q i and
[Equation]
that
[Equation]
↓ [Equation] • On the other hand, for the case where the marker process ˆ X † , ( n ) jumps to the cemetery state † from some m i 1 ∈ G large , note that
[Equation]
S † , ( n ) k +1 − S † , ( n ) k > u, (cid:102) W † , ( n ) k +1 = † (cid:12) (cid:12)(cid:12) W † , ( n ) k = m i 1 (cid:17) + (cid:88) i 3 : m i 3 ∈
G small (cid:90) s> P x (cid:16) S † , ( n ) k +1 − (cid:101) S † , ( n ) k +1 ≥ ( u − s ) ∨ 0 , (cid:102) W † , ( n ) k +1 = † (cid:12)(cid:12)(cid:12) (cid:102) W † , ( n ) k = m i 3 (cid:17) · P x (cid:16)(cid:101) S † , ( n ) k +1 − S † , ( n ) k = ds, (cid:102) W † , ( n ) k
{++
 = m i 3 (cid:12)(cid:12)(cid:12) W † , ( n ) k = m i 1 (cid:17) . Arguing similarly as we did above by considering the integral on [0 , u − η θn ] and ( u − η θn , ∞ ) separately,+} and using eq.
(H.82) and eq.
(H.85), we then get (for all n sufﬁciently large)
[Equation]
and
[Equation]
For simplicity of presentation, we also let q j,
=
q j, † and p j, 0 = p
j, † .
First of all, remember that we have ﬁxed some series of strictly positive real numbers ( s k ) Kk =0 , some sequence ( w k ) Kk =0 ∈ (cid:0) ¯ G (cid:1) K + such that w j (cid:54) = † for any j < K , and indices i k such that w k
= m i k for each k .
The deﬁnition of the continuous-time Markov chain Y † implies that
[Equation]
On the other hand, using eq.
(H.87)-eq.
(H.90), we know that for all n sufﬁciently large,
[Equation]
[-G large , i 
-]
+ (cid:15) )
q i k − 1 s k ) q
i k − 1 ,i k q i k −(cid:17) and
P x (cid:16) S ( n )< s 0 , W ( n )=
w 0 ; S ( n ) k > s k and W ( n ) k = [Equation] k ∀ k
[Equation]
[ K ] (cid:17) ≤ [Equation] { m i [Equation]
G large , i
= i } + [Equation] m i
[Equation] p i 0 , i [Equation]
[Equation] k
[Equation]
The arbitrariness of (cid:15) > 0 then allows us to establish eq.
(H.79) and conclude the proof.
Now we are ready to prove Lemma H.4 and Lemma H.5.
Proof of Lemma H.4.
From Lemma H.14, one can see the existence of some (∆ n ) n ≥ 1 with lim n ∆ n = 0 such that eq.
(H.60)-eq.
(H.63) hold.
For simplicity of notations, we let ˆ X ( n ) =
ˆ X ∗ ,η n , ∆ n , X ( n ) =
X ∗ ,η n , and let ¯ t = t k (cid:48)
Combine eq.
(H.60) with Lemma H.13, and we immediately get eq.
(H.23).
In order to prove eq.
(H.24), it sufﬁces to show that for any (cid:15) > 0 ,
[Equation]
Fix (cid:15) > 0 , and observe following bound by decomposing the events
[Equation]
+ P x (cid:16) ∃ t ∈ [0 , ¯ t ] such that X ( n ) t
[Equation]
Therefore, given eq.
(H.63), it sufﬁces to prove lim sup n
P x (cid:16) ∃ t ∈ [0 , ¯ t ] such that X ( n ) t / ∈ (cid:91)
j : m j ∈ G Ω j [Equation] ≤ 3 (cid:15).
(H.91)
Let
[Equation]
Building upon this deﬁnition, we deﬁne the following stopping times and marks that only records the hitting time to minimizer in large attraction ﬁelds in G (with convention k ( n ) , large ( − 1)
= − 1 , T ( n ) , large −
[Equation]
Now by deﬁning
[Equation]
we use J ( n ) ( t ) to count the numbers of visits to local minima on G , and J ( n ) large ( t ) for the number of visits to minimizers in the large attraction ﬁelds on G .
J ( n ) ( s, t ) counts the indices k such that at
T ( n ) k a minimizer on G is visited and regarding the hitting time we have T ( n )
k ∈
[ s, t ] .
First of all, the weak convergence result in eq.
(H.60) implies the existence of some positive integer
N ( (cid:15) ) such that
[Equation]
Fix such N ( (cid:15) ) .
Next, from eq.
(H.62), we know the existence of some integer K ( (cid:15) )
such that
[Equation]
Fix such K ( (cid:15) ) as well.
From the results above, we know that for event
[Equation]
we have lim sup n P x (cid:16)(cid:0)
A 1 ( n ) (cid:1) c (cid:17) ≤ 2 (cid:15) .
On the other hand, on event A 1 ( n ) , we must have
[Equation]
Meanwhile, it follows immediately from eq.
(H.61)
that
[Equation]
[Equation]
n
[Equation]
T ( n ) k ] such that X ( n ) t
[Equation]
hence for event
[Equation]
we must have lim sup n P x (cid:16)(cid:0)
A [Equation] n ) (cid:1) c (cid:17) ≤ (cid:15) .
To conclude the proof, note that
[Equation]
so we have established eq.
(H.91).
Proof of Lemma H.5.
From Lemma H.15, one can see the existence of some (∆ n ) n ≥ 1 with lim n ∆ n = 0 such that eq.
(H.76) and eq.
(H.77) hold.
For simplicity of notations, we let ˆ
X † , ( n ) =
ˆ X † , ∗ ,η n , ∆ n , X † , ( n ) =
X † , ∗ ,η n , and let ¯ t = t k (cid:48)
Combine eq.
(H.76) with Lemma H.13, and we immediately get eq.
(H.27).
In order to prove eq.
(H.28), note that
[Equation]
and X
[Equation]
X
[Equation]
so the conclusion of the proof follows directly from eq.
(H.77).
I F IRST E XIT T IME OF T RUNCATED H EAVY - TAILED SGD IN R d I. M AIN R ESULT
The object of interests is the following truncated heavy-tailed SGD iterates
[Equation]
where the initial condition is prescribed by X 0 ( x ) = x , f is a real-valued function on R d , η >is the learning rate, ( Z k ) k ≥ 1 is the sequence of heavy-tailed noises, the standard gradient clipping operator is ϕ b ( v )
= min { 1 , b (cid:107) v (cid:107) } · v with (cid:107)·(cid:107) being L 2 norm.
To ease notations, we also use P x to denote the conditional law on { X η 0 = x } .
Speciﬁcally, we are interested in the ﬁrst exit time of X ηn from a domain G , i.e. the stopping time σ ( η )
= min { n ≥ 0 : X ηn / ∈ G} .
We work with following assumptions.
Assumption I.1.
The region G is connected, bounded and open, and 0 ∈ G .
Assumption I.2.
The function f is smooth, i.e. f ∈ C 2 ( R d ) .
Assumption I.3.
The boundary set ∂ G is a ( n − 1) -dimensional manifold of class C 2 such that the vector ﬁeld n ( · ) of the outer normals on ∂ G exists with
[Equation]
for some constant c 0 > 0 .
Assumption I.4.
For ∇ 2 f ( 0 ) , the Hessian of f ( · ) at point 0 , all the eigenvalues are strictly positive.
For any x ∈ ¯ G , let x t ( x ) be the ODE ﬂow with x 0 ( x ) =
x solving
[Equation]
Assumption I.5.
W.L.O.G., the origin 0 is an attractor of the domain, i.e. ∇ f ( 0 )
=
and 0 is asymptotically stable in G in the sense that
[Equation]
We have the following assumption regarding the heavy-tailed noises ( Z n ) n ≥ 1 .
For any x ∈ R d , x (cid:54) = 0 , deﬁne the following polar transformation
[Equation]
with T r ( x ) = (cid:107) x (cid:107) , T θ ( x ) =
x/ (cid:107) x
(cid:107) .
Also, let O = { 0 }
Assumption I.6.
E Z 1 = 0 .
Besides, there exist a positive integer m ≥ 1 , a sequence 1 < α 1 < α 2 < · · · < α m < ∞ , a sequence of slowly-varying functions ( l 1 , · · · , l m ) , a sequence of probability measures ( S 1 , · · · , S m ) on the unit sphere S d − 1 with support F j = ∆ supp ( S j ) as closed sets on S d −such that
• F i ∩ F j = ∅ for any i (cid:54) =
j ; • For any j ∈ [ m ] , deﬁne the cone E j = ∆ T − 1 θ ( F j ) ∪ O in R d and measure P ( j ) ( · ) =
P ( Z 1 ∈ · ∩ E j ) , we have
[Equation]
[Equation]
in the sense of M ( E j \ O ) .
Here ν α is a Borel measure deﬁned on (0 , ∞ ) satisfying ν α [ t, ∞ ) =
t − α ∀ t > 0 ; • For measure P (0) ( · ) =
∆ P (cid:0) Z 1 ∈ · \ ( ∪ mj =1 E j ) (cid:1) and any α > 0 ,
[Equation]
in the sense of M ( R d \ O ) ; • For any j ∈ [ m ] , the measure S j is absolutely continuous w.r
t. spherical measure on S d − 1 .
A function l : R + (cid:55)→ R + is slowly varying (at ∞ ) if lim t →∞ l ( tx ) /l
( x ) = 1 holds for any x >
For details on M − convergence and regular variation in general metric spaces, see Lindskog et al.
Here we state one implication of the assumption.
For any j = 0 , 1 , · · · , m , let
[Equation]
[Equation] The assumption above immediately implies that, for any j = 1 , · · · , m , H j ( · ) is regularly varying (at ∞ ) with index − α j (denoted as H j ∈ RV − α j ),
namely lim t →∞ H j
( tx ) /H
j ( t ) =
x − α j ∀ x > 0 .
Meanwhile, for any α > 0 , H 0 ( x ) =
o
(1 /x α ) as x → ∞ .
The asymptotically behavior of the ﬁrst exit time hinges on the following geometric characterization of the domain G .
For any integer k ≥ 1 , a sequence of strictly positive real numbers ( t 2 , · · · , t k ) with t 1 = 0 , a sequence of non-zero vectors ( w 1 , · · · , w k ) and some η > 0 , let t ( k ) =
( t 1 , · · · , t k ) , w ( k ) =
( w 1 , · · · , w k ) , and deﬁne jumps ( b ) by w ( k ) =
( w 1 , · · · , w k ) , we deﬁne perturbed ODE path (cid:101) x η as (cid:101) x η (0; t ( k ) , [Equation] ( k
[Equation] = [Equation] b
[Equation]
j with the convention that g ( t − ) =
∆ lim s ↑ t g ( s ) for any function g .
Also, when η = 1 we simply write (cid:101) x .
Now we can assign a cost to each jump w j based on the direction using the following function:
[Equation]
Given any set of perturbations described by t ( k ) ∈ { 0 } × R k − 1 + , w ( k ) ∈ (cid:0) R d \ O (cid:1) k , we identify the destination of the ﬂow as
[Equation]
Besides, deﬁne function I ( w ( k ) )
=
( i 1 , · · · , i m )
if # { i ∈ [ k ] : w i ∈ E j } = i j for all j ∈ [ m ] .
This allows us to deﬁne the following conﬁguration sets
[Equation]
for any ( i 1 , · · · , i m ) ∈ N m , i.e., some set of jumps w ( k ) is said to have conﬁguration ( i 1 , · · · , i m ) or belong to the conﬁguration set A ( i 1 , · · · , i m ) if the number of jumps in cone E j is equal to i j .
We can also deﬁne the cost for each conﬁguration as
[Equation]
Now we can characterize the minimum cost to exit G :
[Equation]
From the boundedness of G , one can see that 0 < J G < ∞ regardless of the actual value of the clipping threshold b > 0 .
We need the following technical assumption regarding the conﬁgurations of jumps that can trigger the exit with the minimum cost.
Assumption I.7.
There exists only one array ( i 1 , · · · , i m ) ∈ N m such that (cid:80) mj
=1 i j ( α j − 1) =
J G and (for k = (cid:80) mj
=
i m )
[Equation]
We use i ∗ =
( i ∗ 1 , · · · , i ∗ m ) to denote the unique conﬁguration in Assumption I.7 and k ∗ =
(cid:80) mj
=
i ∗ j .
The implication is that, for a set of jumps t ( k ) , w ( k ) with any other conﬁguration , one of the following must happen: (i) this set of jumps has a cost strictly higher than J G ; (ii) this set of jumps cannot send the ODE ﬂow out of G .
Meanwhile, we introduce the following concept as the coverage of a certain conﬁguration set.
[Equation]
It is easy to see that, for any conﬁguration ( i 1 , · · · , i m ) with cost J ( i 1 , · · · , i m )
≤ J G , the coverage
G ( i 1 , · · · , i m ) is a closed set, and the following technical assumption holds for (Lebesgue) almost every b > 0 .
Here the distance d ( A, B ) =
inf x ∈ A,y ∈ B (cid:107) x − y (cid:107) for any A, B ⊆ R d , and A ◦ is the interior of the set A .
Assumption I.8.
For any ( i 1 , · · · , i m ) ∈ N m , one of the following must occur:
• d (cid:0) G ( i 1 , · · · , i m ) , G c ) > 0 ; • (cid:0) G ( i 1 , · · · , i m ) ∩ G c (cid:1) ◦ (cid:54) = ∅
Note that with i ∗ and k ∗ deﬁned above, we can deﬁne the following mapping h from r = ( r 1 , · · · , r k ∗ ) , θ = ( θ 1 , · · · , θ k ∗ ) ∈
( S d − 1 ) k ∗ , t = ( t 1 , · · · , t k ∗ ) ∈ { 0 } × R k ∗ + such that
[Equation]
where w = ( w i ) k ∗ i =1 with w i =
r i θ i .
Also, we introduce the concept of type for conﬁguration i ∗ .
Speciﬁcally, deﬁne
[Equation]
m [Equation] and for any j ∈ j ( i ∗ ) , we say that w = ( w 1 , · · · , w k ∗ )
∈
A ( i ∗ ) has type j if
[Equation]
In other words, based on the direction of each jump in w ∈ A ( i ∗ ) we group them into different types.
Note that | j ( i ∗ ) | < ∞ .
Now for any type j ∈ j ( i ∗ ) , deﬁne a (Borel) measure on R k ∗ + × ( S d − 1 ) k ∗ ×
R k [Equation] − 1 + as
[Equation]
where, for any α > 0 , the measure ν α is the Borel measure on (0 , ∞ ) with ν α ( x, ∞ )
= 1 /x
+ α .
Lastly, deﬁne measure µ as µ = (cid:80)
j ∈ j
( i ∗ ) µ j .
As will be established in Lemma K.2, the following technical assumption is also a very moderate one since it holds for (Lebesgue) almost every b >under the current setting.
Assumption I.9.
The set h − 1 (cid:0) ∂ G (cid:1) has zero mass under the measure µ .
Having speciﬁed the problem setting, we are now ready to present Theorem I.1, the main result of this section.
The implication of the theorem is clear: under proper scaling, the ﬁrst exit time σ ( η ) converges in distribution to an Exponential random variable.
Moreover, the scaling λ ( η ) is roughly of order η 1+ J G , implying that the ﬁrst exit time σ ( η ) is roughly of order 1 /η 1+ G as the learning rate η approaches 0 .
Theorem I.1.
Let Assumptions I.1-I.9 hold.
There exists a function λ ( η ) that is regularly varying (as η ↓ 0 ) with index (1 + J G ) such that, for any x ∈ G and any t > 0 ,
[Equation]
where the constant q = µ (cid:0) h − 1 ( G c ) (cid:1) .
The proof is provided in Section J. As a concluding remark, we stress that the order of the ﬁrst exit time is dictated by J G , the minimum cost for exit we introduced above.
Bearing obvious similarity to the ﬁrst exit time analysis of SDE driven by heavy-tailed Lévy processes in Imkeller et al.
(2010a), our result can be viewed as a natural generalization when gradient clipping is applied and heavytailed noises may not always align with a ﬁnite number of lines.
In Imkeller et al.
(2010a), the unclipped setting implies that the escape from the domain G can always be achieved with one big jump.
However, a big perturbation (in noise) along different directions may correspond to different heavy-tailed indices, i.e., induce different costs given our deﬁnition of cost function J ( w ) in eq.
(I.6).
In Theorem 1 of Imkeller et al.
(2010a), we see that the order of the ﬁrst exit time is determined by α 1 , the smallest heavy-tailed index.
The underlying reason is that the exit is almost always trigger by a single big jump with the smallest cost.
Similarly, in our setting where multiple jumps are required for escape due to the clipping mechanism, we see that the order of the ﬁrst exit time is not dictated by the number of jumps or the accumulated distances of the jumps, but the smallest possible accumulated cost deﬁned as the summation of J ( w i ) where w i ’s are the jumps the lead to escape from G .
As detailed in the proof, this is because such jumps with the smallest costs J G dictates the most likely way for exiting G .
I. A S PECIAL C ASE : U NIFORM H EAVY T AIL I NDEX A LONG A LL D IRECTIONS
As stated above, Theorem I.1 deals with general case where the noise distribution is allowed to have different heavy-tailed indices along different directions (see Assumption I.6).
As a special case, it is worth noticing that if a single heavy-tailed index α can be used to describe the tail behavior of noises along any direction (this can be easily guaranteed if the heavy-tailed noise is manually injected into a light-tailed setting), then the minimum cost J G will be equal to l ∗ ( α − 1) where l ∗ is the minimum number of jumps required for escape.
The readers can see that this is a natural extension of our R 1 results, implying that the same strong preference for “wide” minima still hold in R d under truncated heavy-tailed SGD.
To be speciﬁc, we work with the following assumption about the noise distribution.
Assumption I.10.
E Z 1 = 0 .
Besides, there exists some α > 1 , a slowly-varying functions l , a probability measures S on the unit sphere S d − 1 with support supp ( S j ) =
S d − 1 such that
• For the measure ν ( · ) =
P ( Z 1 ∈ · ∩ E j ) , we have
[Equation]
in the sense of M ( R d \ O ) .
Here ν α is a Borel measure deﬁned on (0 , ∞ ) satisfying ν α [ t, ∞ ) =
[Equation]
; • The measure S is absolutely continuous w.r.t.
spherical measure on S d − 1 .
Compared to Assumption I.6, one can easily see that Assumption I.10 is a stronger version with the speciﬁc proviso that a single index α can describe the tail of the noise distribution along any direction in R d .
The ﬁrst exit time results now admit a simpliﬁed form.
In particular, the cost of jumps will degenerate to the count of jumps in the sense that (cid:80) k i = 1 J ( w i ) =
k ( α − 1) for any k ≥ 1 and any ( w 1 , · · · , w k ) ∈ ( R d \ O ) k .
Moreover, we now have J G = l ∗G · ( α − 1) where
[Equation]
and the measure µ can now be expressed as µ = ( (cid:81) l ∗G i =
1 ν α ) × ( (cid:81) l ∗G i =1 S )
× m l ∗G − 1 Leb .
Therefore, the following theorem is merely a restatement of Theorem I.1 in this simpliﬁed setting.
Still, we present the result to highlight the role of the minimum jump number J G in the ﬁrst exit time.
Theorem I.2.
Let Assumptions I.1-I.5, I.10 and I.7-I.9 hold.
There exists a function λ ( η ) that is regularly varying (as η ↓ 0 ) with index 1 + l ∗G ( α − 1) such that, for any x ∈ G and any t > 0 ,
[Equation]
where the constant q = µ (cid:0) h − 1 ( G c ) (cid:1) .
It is clear that ﬁrst exit time since the ﬁrst exit time is roughly of order 1 /η 1+( α − 1) l ∗G .
Therefore, even in the general R d case, the quantity l ∗ G , i.e. the minimum count of jumps to escape from a domain G , induces a hierarchy of ﬁrst exit time as the ﬁrst exit time from the domain with largest l ∗G (requiring most number of jumps to escape) will dominate the ﬁrst exit time from other regions.
Lastly, if we work under the standard setting such as the ones in Zhou et al.
(2020) where local strong convexity of f in the domain G is assumed, the results can be further simpliﬁed and the minimum jump number
l ∗G will be directly tied to the width of the each domain.
Speciﬁcally, the quantity r G = ∆ sup y ∈G (cid:107) y
(cid:107) can be interpreted the effective width or radius of G , and we work with the following assumption.
Assumption I.11.
The function f is strongly convex on the closed ball { y ∈ R
d : (cid:107) y (cid:107) ≤ r G } .
Two consequences follow immediately from this assumption.
First of all, there exists some constant c > 0 such that
[Equation]
As a result, we must have l ∗G ≥ r G /b.
Next, as long as r G /b is not an integer (which holds for Lebesgue almost every b > 0 ), for k = (cid:100) r G /b (cid:101) we can ﬁnd t 2 > 0 , · · · , t k > 0 and w 1 (cid:54) = 0 , · · · , w k (cid:54) =
where (cid:101) x
(cid:0) s, 0 ; (0 , t 2 , · · · , t k ) , ( w 1 , · · · , w k ) (cid:1) / ∈ G for some s > 0 .
In fact, it is worth noticing that
Assumption I.8 now degenerates to the condition that r G /b is not an integer, and we now know that for any such b > 0 , we have l ∗G ≤ (cid:100) r G /b (cid:101) .
In summary, we have established the following result indicating that the strong preference for wider minima under truncated heavy-tailed SGD still persists in R d given proper convexity assumption on f .
Theorem I.3.
Let Assumptions I.1-I.5, I.10-I.11 and I.7-I.9 hold.
There exists a function λ ( η ) that is regularly varying (as η ↓ 0 ) with index 1 + l ∗ ( α − 1) such that, for any x ∈ G and any t > 0 ,
[Equation]
where the constant q = µ (cid:0) h − 1 ( G c ) (cid:1) and l ∗ =
(cid:100) r G /b (cid:101) .
I. R ELAXING THE T ECHNICAL A SSUMPTIONS
In order to achieve the tightest characterization of the ﬁrst exit time, some assumptions introduced above are slightly stronger than the ones in Imkeller et al.
(2010a).
For instance, in Assumption I. we require ∂ G to be of class C 2 while the assumption (A3) in Imkeller et al.
(2010a) only requires it to be a C 1 manifold.
Besides, Assumption I.6 requires that S j , the limiting distribution of the directions of each heavy-tailed component in noises, to be absolutely continuous w.r.t.
the spherical measure.
As will be stressed in Section K.1, these conditions are only imposed to prove Lemma K.2, thus ensuring that Assumption I.9 holds for almost every b > 0 .
Brieﬂy speaking, all the efforts to guarantee Assumption I.9 allows us to conclude that the law of scaled ﬁrst exit time λ ( η ) σ ( η ) converges exactly to that of Exp ( q ) with q = µ (cid:0) h − 1 ( G c ) (cid:1) .
Fortunately, the discussion below will show that, even when we relax all the extra technical assumptions above (hence removing Assumption I.9), the order of the ﬁrst exit time is still dictated by the minimum cost for escape J G , and a similar result about the ﬁrst exit time can be obtained where the distribution of the scaled ﬁrst exit time λ ( η ) σ ( η ) will be asymptotically bounded by two Exponential
RVs.
Speciﬁcally, we reiterate that the ﬁrst half of Assumption I.3, i.e. ∂ G is a differential manifold, will only be applied to prove Lemma K.2.
The second half of the proof, namely the lower bound in eq.
(I.1), only serves the ensure that for any x ∈ G , we have x t ( x ) ∈ G ∀ t ≥
so that the gradient ﬂow starting in G will never leave this domain.
Now let us focus on the open sets G (cid:15) and G (cid:15) for some small (cid:15) > 0 where
[Equation]
Note that ∩ (cid:15) G (cid:15)
=
G , ∪ (cid:15) G (cid:15)
=
G and for any positive reals (cid:15) 1 , (cid:15) 2 such that (cid:15) 1 (cid:54) =
(cid:15) 2 and the two reals are small enough for G (cid:15) 1 and G (cid:15) 2 to be non-empty,
[Equation]
Given eq.
(I.1), as well as the smoothness of the vector ﬁeld ∇ f
(see Assumption I.2) and the fact that G is connected, bounded and open and contains a unique attractor 0 (see Assumption I.1 and I.5), it is easy to see that for all (cid:15) > 0 sufﬁciently small, we will have x t ( ∈ G (cid:15) ∀ t ≥ any x ∈ G (cid:15) and x t ( x ) ∈ G (cid:15) t ≥ any
Meanwhile, in our proof below we will establish eq.
(J.17) (which, again, does not require the
C 2 class assumption on ∂ G or the absolute continuity assumption on the measures S j ).
Note that eq.
(J.17) implies the existence of ∆ > 0 such that µ (cid:0) h − 1 ( G ∆ ) (cid:1) < ∞ .
Therefore, for all but only countably many (cid:15) ∈ (0 , ∆) , we have µ (cid:0) h − 1 ( ∂ G (cid:15) )
(cid:1) =
µ (cid:0) h − 1 ( ∂ G (cid:15) ) (cid:1) = 0 .
In other words, we can ﬁnd a sequence of (cid:15) n with lim n (cid:15) n = 0 such that
[Equation]
Also, for a ﬁxed b > 0 that ensures Assumption I.7 and I.8, one can see that similar conditions will also hold for sets G (cid:15) and G (cid:15) as long as (cid:15) > 0 is sufﬁciently small.
In summary, if we consider ﬁrst exit times
[Equation]
with the obvious bounds that σ n ( η ) ≤ σ ( η ) ≤ σ n ( η ) for all n ≥ 1 , then it sufﬁces to apply
Theorem I.1 directly onto σ n ( η ) , σ n ( η ) .
Note that we did not attempt to establish that ∂ G (cid:15) or ∂ G (cid:15) are differential manifolds.
However, we reiterate that this is not needed since the C 2 manifold condition in Assumption I.3 only serves to ensure that Assumption I.9 holds for almost surely every b > 0 , which we sidestep by picking a proper sequence (cid:15) n to explicitly satisfy the condition.
More formally speaking, consider the following relax ted assumptions.
Assumption I.12.
The boundary set ∂ G is a ( n − 1) -dimensional manifold of class C 1 such that the vector ﬁeld n ( · ) of the outer normals on ∂ G exists with
[Equation]
for some constant c 0 > 0 .
Assumption I.13.
E Z 1 = 0 .
Besides, there exist a positive integer m ≥ 1 , a sequence 0 <
α 1 <
α 2 < · · · < α m < ∞ , a sequence of slowly-varying functions ( l 1 , · · · , l m ) , a sequence of probability measures ( S 1 , · · · , S m ) on the unit sphere S d − 1 with support F j = ∆ supp ( S j ) as closed sets on S d −such that
• F i ∩ F j = ∅ for any i (cid:54) =
j ; • For any j ∈ [ m ] , deﬁne the cone E j = ∆ T − 1 θ ( F j ) ∪ O in R d and measure P ( j ) ( · ) =
P ( Z 1 ∈ · ∩ E j ) , we have
[Equation]
in the sense of M ( E j \ O ) .
Here ν α is a Borel measure deﬁned on (0 , ∞ ) satisfying
ν α
[ t , ∞ )
=
t
− α ∀ t > 0 ; • For measure P (0) ( · ) =
∆ P (cid:0) Z 1 ∈ · \ ( ∪ mj =1 E j ) (cid:1) and any α > 0 , t α · P (0) ◦ T − 1 (cid:0) t · dr × dθ (cid:1) → 0 as t → ∞ in the sense of M ( R d \ O ) .
We say Assumptions I.1-I.2, I.12, I.4-I.5, I.13, I.7-I.8 are the set of relaxed assumptions .
The discussion above implies that the following result is an immediate consequence from Theorem I.1.
Theorem I.4.
Let the relaxed assumptions hold.
There exists a function λ ( η ) that is regularly varying (as η ↓ 0 ) with index (1 + J G ) such that, for any x ∈ G and any t > 0 ,
[Equation]
with q = µ (cid:0) h − 1 ( G c ) (cid:1) , q ↑ = µ (cid:16) h − 1 (cid:0) ( G ) c (cid:1)(cid:17) .
J P ROOF OF T HEOREM I.
This section is devoted to establishing Theorem I.1.
For clarity of the exposition, we break down the proof into several major steps, each contained in a subsection below.
J. P ICKING CONSTANTS ¯ t, ¯ (cid:15), ¯ δ
The goal of this subsection is to ﬁx several important constants that characterize the typical sizes and inter-arrival times of large perturbations in SGD trajectory that can cause the escape from domain
G .
We will abuse the notations slightly when referencing certain constants, and quantities such as c 0 , c 1 , c 2 may not be equal to the ones used in assumptions above.
First, from Assumption I.6 and the deﬁnition of J G , we can deﬁne
[Equation]
and we must have ∞ > l ∗ > k ∗ .
Meanwhile, from Assumption I.8, we can ﬁnd ¯ (cid:15) > 0 such that
[Equation]
for any ( i 1 , . . .
, i m ) ∈
N m with (cid:80) mj
=1 i j ( α j − 1) <
J G , as well as ∃ x ∈ G
( i ∗ ) such that d ( x, G ) > 100 l ∗ ¯ (cid:15).
Furthermore, Assumption I.5 implies that ∇ f ( v ) (cid:54) =
for any v ∈ G\{ 0 } .
From Assumptions I.3, I. and I.5 (and by picking a smaller ¯ (cid:15) > 0 if needed), one can see the existence of some c 0 > 0 , ¯ (cid:15) >such that [Equation] ( x ) , G
[Equation]
(cid:1) ≥ d ( x, G
[Equation]
+
[Equation] t if x s ( x ) ∈ G and d (cid:0) x s ( x ) , G c (cid:1) ≤ ¯ (cid:15) ∀ s ∈ [ [Equation] t ]
[Equation]
Here B ( x, r ) =
{ y ∈ R
d : (cid:107) x − y (cid:107) < r } is the open ball centered at x with radius r > 0 .
One immediate consequence from eq.
(J.5) is that
[Equation]
Besides, the boundedness of domain G and smoothness of f ( · ) imply the existence of some M >such that
[Equation]
where for the matrix norm we use spectral norm.
Recall the deﬁnitions of the conﬁguration sets A ( i 1 , · · · , i m ) , the unique conﬁguration i ∗ and count of jumps k ∗ to trigger the exit with minimum cost (see Assumption I.7 and the remark underneath).
The following three results provide control on the sizes and inter-arrival times of typical jumps that can trigger the escape from G .
The proofs involve analyses of the deterministic dynamical system (cid:101) x t and will be provided in Section K.
In particular, we introduce another concept as type of jumps ( w 1 , · · · , w k ) that is similar conﬁguration ( i 1 , · · · , i m ) and conﬁguration sets A ( i 1 , · · · , i m ) .
For any k ∈ N and any ( w 1 , · · · , w k ) ∈ ( R d \ O ) k , we say that the ( w 1 , · · · , w k ) is of type- j for some j = ( j [Equation]
[Equation]
In other words, if w 1 and w 2 are of the same type j , then they have the same cardinality | w 1 | = | w 2 | = | j | = k for some k ∈ N ; moreover, for any i ∈ [ k ] , the jumps w 1 i and w 2 i are both in the cone
E j i so both jumps point at directions in cone E j i and have the same cost J ( w 1 i )
=
J ( w 2 i ) =
α j i −for some j i ∈ [ m ] .
Deﬁne the set
[Equation]
The accumulated cost for any w ∈ A type j is deﬁned as J type ( j ) =
(cid:80)
| j | i =
( α j i − 1) .
Lastly, deﬁne the skip-one accumulated cost for any type j as
[Equation]
i.e., J ↓ type ( j ) is the highest possible accumulated cost if we remove one element in w for w ∈ A type j .
Lemma J.1.
There exist some ¯ t ∈ (0 , ∞ ) , ¯ δ ∈ (0 , ∞ ) , (cid:15) 0 > 0 such that the following claim holds.
Let k (cid:48) ∈ N and j = ( j 1 , ·
· · , j k (cid:48) ) ∈ { 1 , 2 , · · · , m } k (cid:48) be such that J ↓ type ( j ) <
J G .
For any t ( k (cid:48) )
= ( t 1 , · · · , t k (cid:48) ) ∈ {
[Equation]
× R k (cid:48)
[Equation]
+ ,
[Equation]
j , the following set of conditions
[Equation]
is the necessary condition for inf s ≥
d (cid:0)(cid:101) x ( s, 0 ; t ( k ) , w ( k ) ) , G c (cid:1) ≤ (cid:15) 0 .
Lemma J.2.
Let ¯ t, ¯ δ, (cid:15) 0 be the positive constants prescribed in Lemma J.1.
There exists some (cid:15) 1 >such that the following claim holds.
Let k (cid:48) ∈ N and j = ( j 1 , ·
· · , j k (cid:48) ) ∈ { 1 , 2 , · · · , m } k (cid:48) be such that J ↓ type ( j ) <
J G .
For any x ∈ B ( 0 , (cid:15) 1 ) , any t ( k (cid:48) )
= ( t 1 , · · · , t k (cid:48) )
∈ { 0 } × R k (cid:48) − 1 + , w ( k (cid:48) )
= ( w 1 , · · · , w
[Equation]
type j , the following set of conditions
[Equation]
is the necessary condition for inf s ≥
d (cid:0)(cid:101) x ( s, x ; t ( k ) , w ( k ) ) , G c (cid:1) ≤ (cid:15) 0 / 2 .
Lemma J.3.
There exist some (cid:15) 0 > 0 , δ 0 > 0 such that [-sup s ≥ 
-][Equation]
for all x ∈ B ( 0 , δ 0 ) , all ( i 1 , · · · , i m ) ∈ N m with J ( i 1 , · · · , i m ) <
J G and k = (cid:80) mj
=1 i j , and all w ( k ) ∈
A ( i 1 , · · · , i m ) , t ( k ) ∈ { 0 } × R k − 1 + .
Similar to the ODE path x , we deﬁned the following paths with rate η > 0 :
[Equation]
In other words, ODE ﬂow x t is equivalent to x ηt with rate η = 1 .
The next result gives us an upper bound for the return time to a neighborhood of the local minimum as for the gradient ﬂow.
Lemma J.4.
For τ ηODE ( x, (cid:15) ) =
∆ min { t ≥ 0 : x ηt ( x ) ∈ B ( 0 , (cid:15) ) } , there exists c 1 ∈ (0 , ∞ ) such that for any (cid:15) ∈ (0 , ¯ [Equation]
[Equation]
Proof.
Due to eq.
(J.4), we know that T 0 =
[Equation]
sup
[Equation]
< ∞ and sup
[Equation]
ODE
[Equation]
Furthermore, from eq.
(J.3) one can see that
[Equation]
Now combining these two bounds with a 1 /η time scaling, we have
[Equation]
η for c 1 = max { T 0 + log(¯ (cid:15) ) c 0 , 1 c 0 } .
With Lemma J.1-J.4, we are able to pick some constants to facilitate the analysis below.
By decreasing ¯ (cid:15) and increasing M if needed, we can assume the existence of some ¯ t < ∞ , ¯ δ < ¯ (cid:15) such that
• (Due to Lemma J.1 and J.2)
For any k ∈ N and j ∈ { 1 , 2 , · · · , m } k such that J ↓ type ( j ) <
J G , any x ∈ B ( 0 , 100 l ∗ ¯ (cid:15) ) , any t ( k ) =
( t 1 , · · · , t k ) ∈
[Equation]
0 } × R k
[Equation]
[Equation] [Equation]
[Equation]
A type j with
[Equation]
it holds that
[Equation]
• For any ( i 1 , . .
, i m ) ∈ N m with J ( i 1 , . .
, i m ) <
J G
[Equation]
• There exists some x ∈ G ( i ∗ ) such that
[Equation]
• (Due to Lemma J.3) The
constant ¯ (cid:15) > is sufﬁciently small such that
[Equation]
for all x ∈ B ( 0 , 100 l ∗ ¯ (cid:15) ) , all ( i 1 , · · · , i m ) ∈ N m with J ( i 1 , · · · , i m ) < J G and k = (cid:80) mj
{+=
+} i j , and all w ( k ) ∈
A ( i 1 , · · · , i m ) , t ( k ) ∈ { 0 } × R k − 1 + ; • (Due to Lemma J.4)
The constant ¯ t < ∞ is large enough so that [Equation]
J. B OUNDING THE DISTANCE BETWEEN ODE FLOW AND SGD ITERATES
Moving on, we show that, without large jumps, the SGD iterates
X ηn ( x ) are unlikely to show signiﬁcant deviation from the deterministic gradient descent process y ηn ( x ) deﬁned as
[Equation]
We are ready to state the ﬁrst lemma, where we bound the distance between the gradient descent iterates y ηn ( y ) and the ODE x η ( t, x ) when the initial conditions x, y are close enough.
Lemma J.5.
Given t > 0 and x, y ∈ G such that the line segment between x η ( k, x ) and y ηk ( y ) lies in G for any k = 1 , 2 , · · · , (cid:98) t (cid:99) ,
[Equation]
where M ∈ (0 , ∞ ) is the constant in eq.
(J.8) eq.
(J.9) .
Proof.
Deﬁne a continuous-time process y η ( s ; y ) =
∆ y η (cid:98) s (cid:99) ( y ) , and note that
[Equation]
Therefore
[Equation]
if we deﬁne function
[Equation]
from the fact (cid:107)∇ f ( · ) (cid:107) ≤ M , one can see that (cid:107) b ( u ) (cid:107) ≤ ηM + (cid:107) x − y (cid:107) for any u ∈ [0 , 1) and (cid:107) b (1) (cid:107) ≤ 2 ηM + (cid:107) x − y (cid:107) .
In case that s > 1 , from the display above and the fact (cid:13)(cid:13) ∇ 2 f ( · ) (cid:13)(cid:13) ≤ M , we now have
[Equation]
From Gronwall’s inequality (see Theorem 68, Chapter V of Protter (2005), where we let function α ( u ) be α ( u ) = (cid:107) b ( u + 1) (cid:107) ), we have
[Equation]
This concludes the proof.
Now we consider an extension of the previous Lemma in the following sense: when both perturbed by at most l ∗ similar perturbations, the ODE ﬂow and gradient descent process should still stay close enough.
Analogous to the deﬁnition of the perturbed ODE (cid:101) x
η , we can construct a process (cid:101)
y η as a perturbed gradient descent process as follows.
For some integer 1 ≤ l ≤ l ∗ , a sequence of strictly positive integers ( t 2 , · · · , t l ) (with convention t 1 = 0 and let t ( l ) = ( t 1 , · · · , t l ) ), a sequence of non-zero vectors (cid:101) w ( l ) = ( (cid:101) w 1 , · · · , (cid:101) w l ) and y ∈ R d , deﬁne the perturbed gradient descent iterates with gradient clipping at b as
[Equation]
with initial condition (cid:101) y η 0 ( y ; t ( l ) , (cid:101) w ( l ) ) =
y + ϕ b ( (cid:101) w 1 ) .
Corollary J.6.
For any ﬁxed (cid:15) > 0 , t > 0 , the following claim holds for all η > 0 sufﬁciently small: given any x, y ∈ G , some integer 1 ≤ l ≤ l ∗ , a sequence of strictly positive integers t ( l ) =
( t j ) lj
= (with convention t 1 = 0 ), and any two sequences of vectors w ( l ) = ( w j ) lj
=1 , (cid:101) w ( l ) =
( (cid:101) w j ) lj ≥ 1 such that (let t end =
(cid:80) lj
=1 t j )
• | x − y | < (cid:15) ; • t j ≤ 2 t/η for all j ∈ [ l ] ; • | w j − (cid:101) w j | < (cid:15) for all j ∈ [ l ] ; • The line segment between (cid:101) x
η ( k, x ; t ( l ) , w ( l ) ) and (cid:101) y ηk ( y ; t ( l ) , (cid:101) w ( l ) ) lies in G for any k ≤ t end − 1 ,
then we have
[Equation]
t end ]
[Equation]
where the constant ¯ ρ = (2 exp(2 Mt )
+ 2) l ∗ .
In particular, ¯ ρ = ¯ ρ (¯ t ) is a constant where ¯ t is the constant in eq.
(J.11) .
Proof.
Throughout this proof, ﬁx some η ∈ (0 , (cid:15)/ 2 M ) .
We will show that for any η in the range the claim would hold.
First, on interval [0 , t 2 ) , from Lemma J.5, one can see that (since 2 Mη < (cid:15) ) sup [Equation] t
[Equation]
s, x ; s ( l ) , w ( l ) )
−
[Equation]
y
[Equation] s (cid:99) ( y ; t ( l ) , (cid:101) w ( l ) ) (cid:13)(cid:13)(cid:13) ≤ 2 exp(2 Mt ) · (cid:15). The at t = t 2 , by considering the difference between w 2 and (cid:101) w 2 , and the possible change due to one more gradient descent step (which is bounded by ηM < (cid:15) ), we have
[Equation]
Now we proceed inductively.
For any j = 2 , 3 , · · · , l − 1 , assume that
[Equation]
Then by focusing on interval [ t j , t j +1 ] and using Lemma J.5 again, one can show that sup t ∈ [ t j
[Equation]
t j +1 ]
[Equation] ≤ (2 exp(2 Mt ) + 2) j · (cid:15).
This concludes the proof.
Using an almost identical approach, one can show the following results that bounds the distance between two ODE ﬂows with similar initial values and jumps.
Corollary J.7.
There exists some constant ρ ∗ ∈ (0 , ∞ ) such that the following claim holds: For any ﬁxed (cid:15) > 0 , any x, y ∈ G , any integer 1 ≤ l ≤ l ∗ , a sequence of strictly positive integers t ( l ) =
( t j ) lj
=2 (with convention t 1 = 0 ), and any two sequences of vectors w ( l ) = ( w j ) lj
=1 , (cid:101) w ( l ) =
( (cid:101) w j ) lj ≥ 1 such that (let t end =
(cid:80) lj
=1 t j )
• | x − y | < (cid:15) ; • t j ≤ 2¯ t/η for all j ∈ [ l ] ; • | w j − (cid:101) w j | < (cid:15) for all j ∈ [ l ] ; • The line segment between (cid:101) x
η ( s, x ; t ( l ) , w ( l ) ) and (cid:101) x
η ( s, y ; t ( l ) , (cid:101) w ( l ) ) lies in G for any s < t end ,
then we have sup
[Equation]
In the next few results, we show that a similar type of control can be obtained on the distance between gradient descent iterates (cid:101) y ηn and the SGD iterates X ηn .
Speciﬁcally, our ﬁrst goal is to show that before any large jump, it is unlikely that the (deterministic) gradient descent process y ηn would deviate too far from X ηn .
To facilitate the analysis below, we introduce some additional notations.
First, we will group the noises Z n based on a threshold level δ > 0 : let us deﬁne
[Equation]
The former are viewed as small noises while the latter will be referred to as large noises or large jumps .
Furthermore, for any j ≥ 1 , deﬁne the j th arrival time, size, and direction of large jumps as
[Equation]
The following event
[Equation]
describes a scenario where, before the ﬁrst large jump, not much perturbation has been caused by noises.
Lemma J.8.
For any (cid:15) > 0 , η > 0 , any x, y ∈ G and positive integer n such that (cid:107) x − y (cid:107) < (cid:15) 2exp [Equation] ηMn ) , on event
[Equation]
n,
[Equation]
[Equation]
(cid:110)
the line segment between y ηj ( y ) and X ηj ( x ) lies in G ∀ j = 0 , 1 , · · · , n ∧ ( T η 1 ( δ ) − 1) (cid:111) , we have
[Equation]
Proof.
On the said event, we are able to apply Lemma G.5 and obtain that
[Equation]
and conclude the proof.
Similar to the extension from Lemma J.5 to Corollary J.6, we can extend Lemma J.8 to show that, if we consider the a gradient descent process that is only perturbed by large noises, then it should stay pretty close to the SGD iterates X ηn .
To be speciﬁc, let
[Equation]
be a gradient descent process (with gradient clipping at threshold b ) that is only perturbed by large noises.
The next corollary can be shown by an approach identical to the one for Corollary J.6 (namely, inductively repeating Lemma J.8 at arrival time of each jump) so we omit the details here.
Corollary J.9.
There exists some function (cid:101) ρ : (0 , ∞ ) (cid:55)→ (0 , ∞ ) such that the following claim hold for all (cid:15) > 0 , t > 0 and all sufﬁciently small η > 0 :
For any x, y ∈ G ) with (cid:107) x − y (cid:107) < (cid:15) and any l ∈ [ l ∗ ] , on event
A 0 ( (cid:15), η, δ, l, t ) ∩ B 0 ( (cid:15), η, δ, l, t ) ∩ C 0 ( (cid:15), η, δ, l, t ) , we have
[Equation]
where
[Equation]
∆ { the line segment between Y η j ( y ) and X η j ( x ) lies in G ∀ j = 0 , 1 , · · · , T η l ( δ ) − 1 } .
In particular, (cid:101) ρ
= (cid:101) ρ (¯ t ) is a constant
The next result shows that the type of events A ( n, η, (cid:15), δ ) deﬁned in eq.
(J.29) is indeed very likely to occur, especially for small (cid:15) .
For clarity of the presentation, we introduce the following deﬁnitions that are slightly more general than the small and large jumps deﬁned in eq.
(J.24)eq.
(J.25) (for any c > 0 )
[Equation]
Lemma J.10.
Given any t > 0 , (cid:15) > 0 , and N > 0 , the following holds for any sufﬁciently small δ > 0 :
[Equation]
as η ↓ 0 .
Proof.
For any C ∈ (0 , ∞ ) , δ > 0 , β ∈ (0 , 1) , consider the following decomposition of the small noise Z ≤ δ/η n .
Here we adopt the convention that E
= R d \ ( ∪ mj =1 E j ) .
(For deﬁnitions and properties of the closed cones E 1 , · · · , E m , see Assumption I.6 and the remark underneath.)
[Equation]
Based on this decomposition, we deﬁne the following iid random variable sequences
[Equation]
Meanwhile, deﬁne the projection operators P i : R d (cid:55)→ R with P i ( v 1 , · · · , v d ) = v i .
First, recall that E
Z 1 = 0 .
So for all C sufﬁciently large, we have
[Equation]
Moreover, due to H j ∈ RV − ( α j +1) for all j = 1 , 2 , · · · , m and H 0 ( x ) =
o (1 /x α ) for any α >(for deﬁnition of functions H j , see eq.
(I.5))
, we know that E (cid:13)(cid:13)(cid:13) Z ( j ) n (cid:13)(cid:13)(cid:13) < ∞ for any j = 0 , 1 , · · · , m , implying that for all C large enough, we have (for any j = 0 , 1 , · · · , m )
[Equation]
Fix such C .
Besides, recall that in Assumption [-G.
-]I.6 we have 1 < α 1 < α 2 < · < α m .
Therefore, we are able ﬁx some β ∈
0 , 1) and δ >
so that
[Equation]
[Equation]
J.
[Equation]
N
[Equation]
(J.
[Equation]
(cid:0)
[Equation]
(cid:15)
[Equation]
(J.33)
With the ﬁxed C and δ , note that eventually δ/η > C as η ↓ 0 .
Therefore, for all η sufﬁciently small,
P (cid:16)
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
−
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
max
[Equation]
J
[Equation]
[Equation]
J.35)
We bound the two terms eq.
(J.34) and eq.
(J.35) respectively.
First, observe that
[Equation]
[Equation]
Let (cid:101) z
( j ) i ( k ) =
P i (cid:0)(cid:101) z ( j ) ( k ) (cid:1) and let (cid:101) z
( j ) i be an iid copy.
By deﬁnition, | (cid:101) z
( j ) i | ≤ 2 C .
Then from
Hoeffding’s inequality,
[Equation]
As for term eq.
(J.35),
w.l.o.g. let us ﬁx some j = 0 , 1 , · · · , m and bound
[Equation]
(J
[Equation]
36) =
[Equation]
i
[Equation]
[Equation]
[-Z ( j ) ( n ) > (cid:15) 6 √ m + 
 , I ≥ k (cid:17) (cid:124) (cid:123)(cid:122) (cid:125) (cid:44) (II)-] where I =
∆ # { n = 1 , 2 , · · · , (cid:100) t/η (cid:101) : (cid:101) Z ( j ) ( n ) > 1 /η β } and k is some positive integer such that
[Equation]
Note that we can ﬁnd such k due to our choice of constants in eq.
(J.31)-eq.
(J.33).
For (I) and any i = 0 , 1 , · · · , k − 1 , observe that (the non-negative RVs (cid:101) Z β ( n ) and (cid:101) Z ( n ) are deﬁned at the beginning of the proof)
[Equation]
t/η (cid:101) i · P (cid:16) (cid:100) t/η (cid:101)− i (cid:88) n = (cid:101)
Z ( j ) ( n ) > (cid:15) 12 √ m + 1 · η , (cid:101)
Z
( j ) ( n ) ≤
1 /η β ∀ n ∈
[ (cid:100) t/η (cid:101) − i ] (cid:17) ≤ (cid:100) t/η (cid:101) i · P (cid:16) (cid:100) t/η (cid:101) (cid:88) n = (cid:101)
Z ( j ) β ( n ) > (cid:15) 12 √ m + 1 · η (cid:17) .
Let (cid:102) W ( j ) ( n ) =
∆ (cid:101) Z ( j ) β ( n ) − E (cid:101)
Z
( j ) β ( n ) and let (cid:102) W ( j ) be an iid copy.
Due to eq.
(J.30), for all η sufﬁciently small we will have that (cid:100) t/η (cid:101)| E (cid:101)
Z ( j ) β (1) | < (cid:15) 24 √ m +1 · 1 η .
Therefore, for all η sufﬁciently small, (I) ≤ (cid:100) t/η (cid:101) i · P (cid:16) | (cid:100) t/η (cid:101) (cid:88) n = (cid:102) W ( j ) ( n ) | > (cid:15) 24 √ m + 1 · η (cid:17) .
(J.37)
Observe that | (cid:102) W ( j ) | ≤ 2 /η β and E
[Equation]
var
[Equation]
Note that there are only two possibilities:
• If j (cid:54) =
and α j ≤ 2 , then from Karamata’s theorem, we have E ( (cid:101)
Z ( j ) β ) 2 ∈ RV − β
(2 − α j ) ( η ) and note that β (2 − α j ) < 1 ; • Otherwise, one can ﬁnd some ∆ > 0 such that H j ( x ) =
o (1 /x 2+∆ ) , implying that E ( (cid:101)
Z ( j ) β ) 2 ≤ (cid:82) ∞ 0
yH j ( y ) dy < ∞ regardless of the actual value of η > 0 .
In any case, using Bernstein’s inequality, we can conclude that
[Equation]
On the other hand,
[Equation]
which is upper bounded by a function regularly varying w.r.t.
η with index k (cid:0) βα 1 − 1 (cid:1) > N .
Therefore, (II) =
o ( η N ) and we conclude the proof.
J. A NALYZING DIFFERENT SCENARIOS BEFORE THE FIRST EXIT OR RETURN
Using results and arguments above, we are able to illustrate the typical behavior of the SGD iterates
X ηn in the following two scenarios.
First, we show that, when starting from anywhere in G , the SGD iterates X η n will most likely return to the neighborhood of the 0 within a short period of time without exiting Ω .
Lemma J.11.
For any (cid:15) ∈ (0 , ¯ (cid:15) ) , the following claim holds:
[Equation]
where the stopping time involved is deﬁned as
[Equation]
and the function ρ ( (cid:15) )
=
c 1 + c 1 log(2 /(cid:15) ) with c 1 ∈ (0 , ∞ ) being the constant speciﬁed in Lemma
J.4.
Proof.
Let t
= ρ ( (cid:15) )
=
c 1 + c 1 log(2 /(cid:15) ) and n = (cid:100) t/η (cid:101) and arbitrarily choose some x with d ( x, G c ) > (cid:15).
Also, ﬁx some N > 0 .
Due to Lemma J.10, we are able to pick some δ > 0 such that (note that ηn ≤ 2 t eventually as η ↓ 0 )
[Equation]
First, from Lemma J.5 we know that for any η small enough such that
[Equation]
we must have
[Equation]
Next, using Lemma J.8, we see that on event
[Equation]
, we must have
[Equation]
Combining these facts with the Lemma J.4, we have shown that, for all η ∈ (0 , (cid:15) 4 M exp(2 tM ) ) , on event A ( n, η, (cid:15) 2exp( ηnM ) , δ ) (cid:84) { T η 1 ( δ ) > (cid:100) t/η (cid:101)} we must have X ηn ∈
G ∀ n ≤
T return ( η, (cid:15) ) and T return ( η, (cid:15) ) ≤ ρ ( (cid:15) ) /η .
In other words,
[Equation]
for some n ≤ T
[Equation]
OR T
[Equation]
( η, (cid:15) )
[Equation]
However, our choice of δ at the beginning of the proof implies that
[Equation]
Meanwhile, from Assumption I.6, we see that
[Equation]
for all η sufﬁciently small (due to regularly varying nature of H j and 1 < α 1 < · · · < α m ).
In particular, since H 1 ∈ RV − α 1 , for any ﬁxed ∆ ∈ (0 , α 1 − 1) , we have (for all η sufﬁciently small),
P ( T
[Equation]
[Equation]
≤ (
[Equation] [Equation]
where U ( η ) =
d Geom ( η α 1 +∆ ) .
Lemma G.4 then tells us that, for all η sufﬁciently small, P ( U ( η ) ≤ (cid:100) t/η (cid:101) ) ≤ 2 t η α 1 − 1+∆ , thus implying
[Equation]
and concluding the proof.
In the next result, we show that, once entering a (cid:15) − small neighborhood of the local minimum, the
SGD iterates will most likely stay there until the next large jump.
Lemma J.12.
Given N 0 > 0 , the following claim holds for all (cid:15) ∈ (0 , ¯ (cid:15)/ 3) and all δ > 0 that is sufﬁciently small:
[Equation]
as η ↓ 0 .
Proof.
Let t =
c
+ c 1 log(1 /(cid:15) )
where c 1 ∈ (0 , ∞ ) is the constant speciﬁed in Lemma J.4, and n = (cid:100) t/η (cid:101) .
Fix some β > 1 + α 1 and N > β − 1 + N 0 .
Due to Lemma J.10, the following claim holds for all δ that is sufﬁciently small:
[Equation]
We ﬁx one of such δ .
Therefore, for event
[Equation]
ηnM [Equation] (cid:111) , the iid nature of noises ( Z n ) n ≥ 1 implies that
[Equation]
=
o ( η N − β +1 ) as η ↓ 0 .
Meanwhile, Note that T η 1 ( δ ) ≤ T η, (1)( δ ) =
∆ min { n ≥ 0 : η | Z n | > δ, Z n ∈ E 1 } .
Therefore,
P ( T η 1 ( δ ) > 1 /η β ) ≤ P ( T η, (1)( δ ) > 1 /η β ) .
Speciﬁcally, note that T η 1 ( δ ) =
d Geom ( H 1 ( δ/η )) and
H 1 ∈ RV −
α 1 .
Lemma G.3 then tells us the existence of some c > 0 such that
[Equation]
as η ↓ 0 .
In summary, we have established that
[Equation]
Now let us focus on the complementary event ( (cid:101) A )
c ∩ { T η 1 ( δ ) ≤ 1 /η β } .
On this event, there must be some j ≤ (cid:100) 1 /η β (cid:100) t/η (cid:101) (cid:101) such that
+ ( j − 1) (cid:100) t/η (cid:101) ≤ T [Equation]
[Equation]
≤ j (cid:100) t/η (cid:101) .
Let us arbitrarily choose some x ∈ B ( 0 , 2 (cid:15) ) .
If j = 1 , then due to eq.
(J.3), we know that x η ( s, x ) ∈ B ( 0 , 2 (cid:15) ) for all s ≥ 0 .
Also, from Lemma J.5 we know that for any η small enough such that 2 ηM exp(2 tM ) <
(cid:15)/ 2 , we must have
[Equation]
Besides, on event ( (cid:101) A ) c ∪ { T η 1 ( δ ) ≤ 1 /η β } ∩ { j = 1 } , due to Lemma J.8, we must have (cid:107)
y ηk ( x ) − X ηk ( x ) (cid:107) <
(cid:15)/ 2 ∀ k = 0 , 1 , · · · , T η 1 ( δ ) with T η 1 ( δ ) < (cid:100) t/η (cid:101) .
In conclusion, in the case that j = 1 , we must have (cid:107) X ηn ( x ) (cid:107) < 3 (cid:15) for all n < T η 1 ( δ ) .
Now consider the case with j ≥ 2 .
Similarly, due to Lemma J.5 and J.8, we now have that
[Equation]
In particular, due to our choice of t at the beginning of the proof and Lemma J.4, one can see that [Equation] x η ( [Equation]
t/η
[Equation]
, x ) (cid:107) < (cid:15).
Collecting all the results, we now know that
[Equation]
Now it sufﬁces to apply the repeated apply the same arguments above.
For instance, if j = 2 , then by letting x 1 =
X η (cid:100) t/η (cid:101) ( x ) and bounding the gap between trajectory of X η (cid:98)
s (cid:99)
+ (cid:100) t/η (cid:101) ( x ) and x η ( s, x 1 ) using Lemma J.5 and J.8, one can show that (cid:107) X ηn ( x ) (cid:107) < 3 (cid:15) for all n < T η 1 ( δ ) (since in the case of j = 2 , we have T η 1 ( δ ) < 2 (cid:100) t/η (cid:101) ).
Otherwise, we have j ≥ 3 and the same arguments above can be applied to show that
[Equation]
In summary, by proceeding inductively, we establish that for all η ∈ (0 ,
[Equation]
and this concludes the proof.
We introduce a few concepts that will be crucial to the analysis below.
Recall the deﬁnition of ODE with jumps (cid:101) x
(note that we will drop the notational dependency on learning rate η when we choose η = 1 ).
Recall the deﬁnitions of i ∗ , k ∗ (see Assumption I.7 and I.8 and the remarks underneath).
We deﬁne the following mapping h from r = ( r 1 , · · · , r k ∗ ) , θ =
( θ 1 , · · · , θ k ∗ ) ∈ ( S d − 1 ) k ∗ , t = ( t 1 , · · · , t k ∗ ) ∈ { 0 } × R k ∗ + such that
[Equation]
where w = ( w i ) k ∗ i =1 with w i =
r i θ i .
From the continuity of the ODE ﬂow, we see that h is a continuous mapping.
Also, we introduce the concept of type for conﬁguration i ∗ .
Speciﬁcally, deﬁne
[Equation]
and for any j ∈ j ( i ∗ ) , we say that w = ( w 1 , · · · , w k ∗ )
∈
A ( i ∗ ) has type j if w i ∈
E j i ∀ i ∈ [ k ∗ ] .
In other words, based on the direction of each jump in w ∈ A ( i ∗ ) we group them into different types.
Note that | j ( i ∗ ) | < ∞ .
Now for any type j ∈ j ( i ∗ ) , deﬁne a (Borel) measure on R k ∗ + × ( S d − 1 ) k ∗ ×
R k ∗ − 1 + as
[Equation]
where, for any α > 0 , the measure ν α is the Borel measure on (0 , ∞ ) with ν α ( x, ∞ )
= 1 /x α .
We use the concepts above to characterize behavior of large noises in ( Z n ) n ≥ 1 .
Deﬁnition J.1.
For any η, δ > 0 , j ∈ j ( i ∗ ) and any Borel set A ⊆ R d , we say noise Z n is of type- ( A, δ, η, j ) iff
• η (cid:107) Z n (cid:107) > δ ; • For (cid:101) t 1 = n and (cid:101) t i =
∆ min { k > (cid:101) t i − 1 : η (cid:107) Z k (cid:107) > δ } for all i = 2 , 3 , · · · k ∗ , it holds that
(cid:101) t i − (cid:101) t i − 1 < 2 ¯ t/η for all
i = 2 , 3 , · · · , k ∗ ; • For t i = ∆ (cid:101) t
i − (cid:101) t i − 1 (with t 1 = 0 ), w i = ηZ (cid:101) t i , and r = ∆ ( (cid:107) w 1 (cid:107)
[Equation]
it holds that h ( r , θ , t ) ∈ A .
More generally, we say noise Z n is of type- ( A, δ, η ) iff there exists
j ∈ j
( i ∗ ) such that Z n is of type- ( A, δ, η, j ) .
Due to the iid nature of ( Z j ) j ≥ 1 , let us consider an iid sequence ( V j ) j ≥ 0 where the sequence has the same law of Z 1 .
Note that for any ﬁxed n ≥ 1 , the probability that Z n is of type- ( A, δ, η, j ) is equal to the probability that V 0 is of type- ( A, δ, η, j ) .
More speciﬁcally, let H ( x ) =
(cid:80) mj
=
H j ( x ) =
P ( (cid:107) Z 1 (cid:107) > x ) .
Due to Assumption I.6, H ∈ RV − α 1 .
Besides, P ( η (cid:107) V 0 (cid:107) > δ ) =
H ( δ/η ) .
Now we focus on conditional probability admitting the following form:
[Equation]
Lemma J.13.
For any δ > 0 , any j ∈ j ( i ∗ ) , and any Borel set A ⊆ R d such that h ( r 1 , · · · , r k ∗ , θ 1 , · · · , θ k ∗ , t 2 , · · · , t k ∗ ) ∈
A = ⇒ t i <
2¯ t ∀ i = 2 , · · · , k ∗ and r i > δ [Equation] i ∈
[ k ∗ ] , [Equation] J.41) it holds that
[Equation]
Proof.
Let us start by ﬁxing some notations.
Let T 1 = 0 , and deﬁne stopping times T j = min { n >
T j − 1 : η (cid:107) V n (cid:107) > δ } and inter-arrival times T (cid:48)
j = T j − T
j − 1 for any j ≥ 2 (with T (cid:48) 1 = 0 ), and large jump W j = V T j for any j ≥ 0 .
Note that: ﬁrst, the pair ( T (cid:48)
i , W i ) is independent of ( T (cid:48)
j , W j ) for any i (cid:54) =
j ; besides, W j and T (cid:48)
j are independent for all j ≥ 1 .
Deﬁne the following sequence (of random elements) w = ( w 1 , · · · , w l ∗ ) and t =
( t 1 , · · · , t l ∗ ) by w i = ηW i , t i = ηT (cid:48)
i .
and r = ( T r ( w i ))
k ∗ i =1 , θ =
( T θ ( w i ))
k ∗ i =1 where operators T r ( x ) = (cid:107) x
(cid:107) , T θ ( x ) =
x/ (cid:107) x (cid:107) constitute the polar coordinate transform.
By deﬁnition of type- ( A, δ, η, j ) , we know that
• T (cid:48) i ≤ 2¯ t/η for any i = 2 , · · · , k ∗ ; • η (cid:107) W i (cid:107) > δ for any i = 1 , 2 , · · · , k ∗ ; • w i ∈
E j i (i.e. θ i ∈ F i ) for all i ∈ [ k ∗ ] ; • h ( r , θ , t ) ∈
A .
Therefore, (let R i
=
T r ( W i ) , Θ i =
T θ ( W i ) ) p [Equation] A, δ, η, j ) =
[Equation]
where Q η,δ, j is the probability measure on R k ∗ + × ( S d − 1 ) k ∗ × R k ∗ − 1 + induced by a sequence of random elements ( ηR 1 , · · · , ηR k ∗ , Θ 1 , · · · , Θ k ∗ , ηT ↑ 2 ( η ) , · · · , ηT ↑ k ∗ ( η )) such that
• For any i = 1 , · · · , k ∗ , we have R i
=
T r ( W ↑ i ( η )) , Θ i =
T θ ( W ↑ i ( η )) where the distribution of W ↑ i ( η ) follows the law P (cid:16) V 0 ∈ · (cid:12)(cid:12)(cid:12) η (cid:107) V 0 (cid:107) >
δ, V 0 ∈
E j i (cid:17) ; Besides, ( W ↑ i ( η ))
k ∗ i =1 is an independent sequence; • For any i
= 2 , · · · , l ∗ , the distribution of T ↑ i ( η ) follows from P (cid:16) ηT δ 1 ( η ) ∈ · (cid:12)(cid:12)(cid:12) ηT δ 1 ( η ) ≤ 2¯ t (cid:17) (for the deﬁnition of stopping times T δk ( η ) , see eq.
(J.26)); Besides, ( T ↑ i ( η ))
k ∗ i =1 is an independent sequence, and it is independent of ( W ↑ i ( η ))
k ∗ i =1 ; • Q η,δ, j ( · ) =
P (cid:16) ( ηR 1 , · · · , ηR k ∗ , Θ 1 , · · · , Θ k ∗ , ηT ↑ 2 ( η ) , · · · , ηT ↑ k ∗ ( η )) ∈ · (cid:17) .
As for weak convergence of ( ηR 1 , · · · , ηR k ∗ , Θ 1 , · · · , Θ k ∗ , ηT ↑ 2 ( η ) , · · · , ηT ↑ k ∗ ( η )) , observe that
• For any i ∈ [ k ∗ ] , from Assumption I.6 one can see the regularly varying nature of distribution of V 0 on the cone E j i (hence for W ↑ i ( η ) as well)
thus yielding that ( ηR i , Θ i ) ⇒ ( R ∗ i , Θ ∗ i ) as η ↓
where R ∗ i and Θ ∗ i are independent, the law of Θ ∗ i is S j i , and the law of R ∗
i
is the
Pareto distribution with
[Equation]
• For any x ∈ [0 , 2¯ t ] , since lim η ↓ 0 (cid:98) x/η (cid:99) H ( δ/η ) = 0 , it is easy to show that
[Equation]
therefore, we have (for any x ∈ (0 , 2¯ t ] )
[Equation]
as η ↓ 0 , which implies that T ↑ i ( η ) converges weakly to a uniform RV on [0 , 2¯ t ] .
Together with the assumption on set A in eq.
(J.41), we can now see that, if we denote the weak limit of measure Q η,δ, j as µ δ, j , then (the measure µ j is deﬁned in eq.
(J.40))
[Equation]
From the continuity of mapping h , one can see that h − 1 ( ¯ A ) is a closed set and h − 1 ( A ◦ ) is an open set.
Using Portmanteau theorem, we now have
[Equation]
Moving on, we analyze the limit of the other terms in eq.
(J.42).
For any i ∈ [ k ∗ ] , note that
[Equation]
Now due to the regularly varying nature of H j and H ,
[Equation]
On the other hand, from Lemma G.4 and the regularly varying nature of function H , for any ﬁxed κ > 1 , we have
[Equation]
Due to arbitrariness of κ >
[Equation]
we yield that
[Equation]
Collecting all the limits in eq.
(J.44)-eq.
(J.46) and plugging them into eq.
(J.42), we now have established that
[Equation]
[Equation] (cid:1) . To conclude the proof, recall that for any type j ∈ j ( i ∗ ) , we have # { k = 1 , 2 , · · · , k ∗ : j k = j }
= i ∗ j (i.e. for any j ∈ [ m ] , the number of elements in ( j 1 , · · · , j k ∗ ) that are equal to j is exactly i ∗ j ), thus (cid:81) k ∗ i =1 H j i (1 /η ) =
(cid:81) mk =1 (cid:0) H k
(1 /η ) (cid:1) i ∗ k .
More generally, for
[Equation]
we know that p ( A, δ, η ) =
(cid:80) j ∈ j
( i ∗ ) p
( A, δ, η, j ) .
Also, deﬁne measure µ as
[Equation]
[Equation]
The next result follows immediately from Lemma J.13 and the fact that | j ( i ∗ ) | < ∞ .
Recall that J G is deﬁned in eq.
(I.9).
Corollary J.14.
There exists a function (cid:101) λ ( η ) ∈ RV 1+ J G − α 1 ( η ) such that, given any δ > 0 and any
Borel set A ⊆ R d with h ( r
[Equation]
[Equation]
θ
[Equation]
· · · , θ
[Equation]
∗
[Equation]
t [Equation] [Equation] t
[Equation]
2¯ t ∀
i [Equation] [Equation] · , k ∗ and r i > δ ∀ i ∈
[ k ∗ ] ,
[Equation]
49) it holds that
[Equation]
In particular, the regularly varying function (cid:101) λ ( · ) admits the form
[Equation]
Consider the following stopping times
[Equation]
Here σ indicate the ﬁrst time that the iterates exit domain G , while R denotes the time the SGD iterates return to a small neighborhood of the attractor 0 after the ﬁrst large jump.
In the next few results, we study the probability of the different scenarios regarding the ﬁrst exit time σ ( η ) and ﬁrst return time R ( (cid:15), δ, η ) .
To this end, let
[Equation]
R ( [Equation] , δ, η ) } be the count of large jumps before the ﬁrst exit or ﬁrst return.
Furthermore, we introduce the following concept as the accumulated cost of large jumps before the ﬁrst exit or return.
Let J η,δ=
and
[Equation]
where the cost function J ( · ) is deﬁned in eq.
(I.6).
In other words, J η,δk indicates the total cost of the ﬁrst k large jumps, if there are at least k large jumps, before the ﬁrst exit or ﬁrst return.
Similarly, we can also deﬁne a step-wise accumulated cost as
[Equation]
In other words, J ↓ η,δ ( n ) evaluates the total cost of large jumps up until step n .
As a preparation for our analyses below, we ﬁrst discuss the following technical tools.
For any (cid:15) > 0 , let ˆ t ( (cid:15) )
=
c
1 + c 1 log(1 /(cid:15) )
where c 1 is the constant in Lemma J.4.
Besides, deﬁne
[Equation]
where functions ¯ ρ ( · ) , (cid:101) ρ ( · ) are deﬁned in Corollary J.6 and J.9 respectively.
Furthermore, deﬁne an event A × =
A × 1 ( (cid:15), δ, η ) ∪ A × 2 ( (cid:15), δ, η ) where
A
[Equation]
where the positive integer l ∗ is deﬁned in eq.
(J.1).
By deﬁnition of l ∗ , we must have K η ( δ ) <
l ∗ .
Lastly, deﬁne events (for any K ∈ [ l ∗ ] )
[Equation]
Recall [Table] and B
[Table]
K ) above, [Table] [Table] only if j = K = K
[Table]
Lemma J.15.
For any K ∈ [ l ∗ ] , any (cid:15) ∈ (0 , ¯ (cid:15)/ 3) and any η > 0 sufﬁciently small, the following claim holds on event B × 1 ( K ) :
[Equation]
where T =
[Equation]
and X
[Equation]
x
[Equation]
Proof.
We focus on the distances between the following three objects: X ηn ( x ) ,
[Equation]
otherwise; and
[Equation]
First of all, for X (1) =
X ηT η 1 ( δ ) − 1 ( x ) =
(cid:101) Y ηT η 1 ( δ ) − 1 ( x ) , by deﬁnition of event ( A × ) c we have (cid:13)(cid:13) X (1) (cid:13)(cid:13) < 3 (cid:15) < ¯ (cid:15) .
Then due to eq.
(J.20) and the deﬁnition of event B × 1 ( K ) (in particular, the fact that J η,δK < J G ), we know that
[Equation]
In light of the deﬁnition of B × 1 ( K ) (i.e. the upper bound on T ηj +1 ( δ ) − T ηj ( δ ) ), by applying Corollary
J.6, we can show that sup
[Equation]
In particular, by applying Corollary J.6 inductively for all n ≤ T ηK +1 ( δ ) ∧ σ ( η ) ∧ R ( (cid:15), δ, η ) − T η 1 ( δ ) , we know that the line segment between z ( s ) and (cid:101) Y η (cid:98) s (cid:99) + T η 1 ( δ ) ( x ) is in G for all s < T ηK +1 ( δ ) ∧ σ ( η ) ∧
R ( (cid:15), δ, η ) − T η 1 ( δ ) so Corollary J.6 can be further applied to n + 1 and establish the inequality above.
Similarly, due to our choice of ˆ (cid:15) ( (cid:15) ) in eq.
(J.51) and the upper bound on T ηj +1 ( δ ) − T ηj ( δ ) in deﬁnition of event B × 1 ( K ) , by applying Corollary J.9 inductively, we know that for all η sufﬁciently small, sup
[Equation]
holds on event B × 1 ( K ) and this concludes the proof.
Lemma J.16.
For any K ∈ [ l ∗ ] , any (cid:15) > 0 and any η > 0 sufﬁciently small,
[Equation]
Proof.
The deﬁnition of event B × 2 ( K ) ensures that we can deﬁne some j ∗ as the smallest integer in [ K η ( δ )] such that for T ∗ =
∆ T ηj ∗ ( δ ) + (cid:100) 2ˆ t ( (cid:15) ) /η (cid:101) , we have T ηj
[Equation]
( δ )
[Equation]
Analogous to the proof of the previous lemma
[Equation]
we focus on the pair-wise distances between the following three objects: X [Equation] x ) ,
[Equation]
otherwise [Equation] and
[Equation]
with T =
[Equation]
, T
[Equation] , T [Equation] W = (cid:0) W
[Equation] ( δ [Equation]
, W η 2 ( δ ) ,
[Equation]
, W ηj
[Equation]
+1 ( δ ) (cid:1) , and X (1) =
X ηT η 1 ( δ ) .
Again, using Corollary J.6 and J.9, one can see that for all η > 0 sufﬁciently small, we must have sup
[Equation]
on event B × 2 ( K ) .
However, eq.
(J.20) and (cid:13)(cid:13)
X (1) (cid:13)(cid:13) < 3 (cid:15) <
¯ (cid:15) implies that sup
[Equation]
In the meantime, Lemma J.4 and our choice of (cid:101) (cid:15) ( (cid:15) ) in eq.
(J.51) implies that
[Equation]
and
[Equation]
X
[Equation]
thus dictating that R ( (cid:15), δ, η ) < T ∗ and contradicting eq.
(J.54).
This concludes the proof.
Now we are ready to apply the tools above and analyze some atypical scenarios regrading the ﬁrst exit and ﬁrst return time.
In the next result we show that, when starting from the local minimum, it is very unlikely to escape if the total cost of large jumps is less than J G .
Lemma J.17.
For any ﬁxed (cid:15) ∈ (0 , ¯ (cid:15)/ 3) , N > 0 , the following claim holds for any sufﬁciently small δ > 0 :
[Equation]
Proof.
Since the constant (cid:15) > 0 is ﬁxed, one can see that ˆ t =
c 1 + c 1 log(1 /(cid:15) ) is also ﬁxed where cis the constant in Lemma J.4.
Besides, we can ﬁx some (cid:101) (cid:15) > 0 such that
[Equation]
where functions ¯ ρ ( · ) , (cid:101) ρ ( · ) are deﬁned in Corollary J.6 and J.9 respectively.
Let us deﬁne an event A × =
A × 1 ∪ A × 2 where
[Equation]
where the positive integer l ∗ is deﬁned in eq.
(J.1).
By deﬁnition of l ∗ , we must have K η ( δ ) <
l ∗ on event {J ↓ η,δ ( σ ( η )) <
J G } .
Now let us analyze the dynamics of X ηn on event (
A × ) c ∩ {J ↓ η,δ ( σ ( η )) <
J G } .
In particular, we decompose it into two events
[Equation]
Using Lemma J.15 and the fact that B 1 = (cid:0) (cid:83) l ∗ K =1 B × 1 ( K ) ∩ { K η ( δ )
=
K } (cid:1) ∩ {J η,δK η ( δ ) <
J G } , one can see that for any η > 0 sufﬁciently small, we must have
[Equation]
[Table]
on event B 1 .
In other words, on event B 1 we must have R ( (cid:15), δ, η ) < σ ( η ) .
On the other hand, Lemma J.16 the fact that B 2 ⊆ (cid:83) l ∗ K =1 B × 2 ( K ) ∩ { K η ( δ ) =
K } implies that
B 2 = ∅ whenever η > 0 is sufﬁciently small.
In summary, we have established that
[Equation]
Lastly, from Lemma J.10 and J.12, one can see that for any δ > 0 that is sufﬁciently small,
[Equation]
and this concludes the proof.
Using an almost identical approach, we can establish the next two results and conclude that it is also rather unlikely to have scenarios where
• The accumulated cost of large jumps exceeds J G before the ﬁrst return or exit, • Or the ﬁrst return occurs before the ﬁrst exit,
yet it takes rather long for the said event to occur.
Lemma J.18.
Given (cid:15) ∈ (0 , ¯ (cid:15)/ 3) , N > 0 , the following claim holds for any sufﬁciently small δ > 0 : sup
[Equation]
P [Equation] ∃ K ∈ N s.t.
J η,δK ≥ J G and T ηj ( δ ) − T ηj − 1 ( δ ) > 2ˆ t ( (cid:15) )
/η for some j = 2 , 3 , · · · , K (cid:17) =
o ( η N ) as η ↓ 0 where ˆ t ( (cid:15) )
=
c
1 + c 1 log(1 /(cid:15) ) .
Proof.
Given the ﬁxed (cid:15) > 0 , deﬁne ˆ t =
c 1 + c 1 log(1 /(cid:15) ) .
Also, ﬁx (cid:101) (cid:15) > 0 as the largest possible value such that
[Equation]
where functions ¯ ρ ( · ) , (cid:101) ρ ( · ) are deﬁned in Corollary J.6 and J.9 respectively.
Let us deﬁne an event A × =
A × 1 ∪ A × 2 where
[Equation]
where the positive integer l ∗ is deﬁned in eq.
(J.1).
Furthermore, deﬁne event
B = ∆ ( A
[Equation]
) c ∩
[Equation]
∃ K ∈ N s.t. J η,δK ≥ J G and T ηj ( δ ) − T ηj − 1 ( δ ) > 2ˆ t η for some j = 2 , 3 , · · · , K (cid:111) .
Note that on event B , the index
[Equation]
is well deﬁned with 1 ≤ K ∗ < l ∗ (due to the deﬁnition of l ∗ ).
As a consequence, we must have
T ηK ∗ ( δ ) ≤ σ ( η ) ∧ R
(cid:15), δ, η ) .
Now based on the exact value of K ∗ , we can decompose the event B into B = ∪ l ∗ K =0 B ( K ) where B ( K ) = ∆ ( A × ) c ∩ { K ∗
[Equation]
= [Equation] [Equation] [Equation]
η,δK < J G } ∩ {
[Equation]
+1 ( δ ) − T ηj ( δ ) > 2ˆ t η for some j ∈ [ K ] } .
By applying Lemma J.16, we know that for all η > 0 sufﬁciently small, B ( K ) = ∅ . Therefore, for all sufﬁciently small η , sup
[Equation]
Lastly, from Lemma J.10 and J.12, one can see that for any δ > 0 that is sufﬁciently small,
[Equation]
and this concludes the proof.
Lemma J.19.
Given (cid:15) ∈ (0 , ¯ (cid:15)/ 3) , N > 0 , the following claim holds for any sufﬁciently small δ > 0 : sup
[Equation]
where ˆ t ( (cid:15) )
=
c
1 + c 1 log(1 /(cid:15) ) .
Proof.
Given the ﬁxed (cid:15) > 0 , deﬁne ˆ t =
c 1 + c 1 log(1 /(cid:15) ) .
Also, ﬁx (cid:101) (cid:15) > 0 as the largest possible value such that
[Equation]
≤ [Equation] where functions ¯ ρ ( · ) , (cid:101) ρ ( · ) are deﬁned in Corollary J.6 and J.9 respectively.
Let us deﬁne an event A × =
A × 1 ∪ A × 2 where
[Equation]
[Equation]
where the positive integer l ∗ is deﬁned in eq.
(J.1).
By deﬁnition of l ∗ , we must have K η ( δ ) <
l ∗ on event { J η,δK η ( δ ) <
J G } .
Furthermore, deﬁne event
[Equation]
for some j ∈ [ K η ( δ )]
(cid:111) .
Now observe that B ⊆ (cid:83) l ∗ K =1 B ( K ) where
[Equation]
By applying Lemma J.16, we know that for all η > 0 sufﬁciently small, B ( K ) = ∅ .
Therefore, for all sufﬁciently small η , sup
[Equation]
for some j ∈ [ K η
[Equation]
sup
[Equation]
Lastly, from Lemma J.10 and J.12, one can see that for any δ > 0 that is sufﬁciently small,
[Equation]
[Equation]
, and this concludes the proof.
Recall that in Lemma J.17, we have shown that it is rather unlikely to have the ﬁrst exit with accumulated cost of large jumps less than J G .
In the next result, we show that even with large jumps of accumulated cost J G , if some jumps are still not large enough, or the inter-arrival times are too long, then it is still very unlikely for the SGD iterates to even get close the the boundary.
Deﬁne
[Equation]
and let K η,δJ G = ∞ if J η,δK η ( δ ) < J G .
Also, if K η,δJ G < ∞ , let
[Equation]
In other words, T ≥ J G ( η, δ ) is the ﬁrst time that, prior to ﬁrst exit or return, the accumulated cost of large jumps has reached J G .
Regarding the o ( η 1+ J G − α 1 +∆ ) term in the next Lemma, we note that the function (cid:101) λ deﬁned in eq.
(J.50) is regularly varying (as η ↓ 0 ) with index 1 + J G − α 1 .
Therefore, η 1+ J G − α 1 +∆ =
o ( (cid:101) λ ( η )) as η approaches 0 .
Lemma J.20.
There exists some ∆ > 0 such that the following claim holds for all (cid:15) ∈ (0 , ¯ (cid:15)/ 3) and all δ > 0 that is sufﬁciently small:
[Equation]
, δ, η ) ∪ B × 2 , (II) ( (cid:15), δ, η ) (cid:1) = o ( η 1+ J G − α 1 +∆ ) as η ↓ 0 where ˆ t ( (cid:15) )
=
c
1 + c 1 log(1 /(cid:15) ) and
B × 2 , (I) ( (cid:15), δ, η ) = { K η,δJ G < ∞} ∩ {J η,δ K η,δJ G > J G } , B × 2 , (II) ( (cid:15), δ, η ) = { K η,δ
[Equation]
[Equation] ∩ (cid:110) ∃ j = 2 , 3 , · · · , K η,δ
[Equation]
G s.t.
T ηj ( δ )
[Equation]
T ηj
[Equation]
1 ( δ ) > 2¯ t/η or ∃ j = 1 , 2 , · · · , K η,δJ G s.t.
η (cid:13)(cid:13) W ηj ( δ ) (cid:13)(cid:13) ≤ ¯ δ (cid:111) ∩ (cid:8)
min j =0 , 1 , ··· ,T ≥ J G ( η,δ ) d (cid:0) X ηj , G c (cid:1) ≤ ¯ (cid:15) (cid:9) .
Proof.
From Assumption I.8, we know the existence of some ∆ > 0 satisfying the following condition:
For any k ∈ N and any j ∈ { 1 , · · · , m } k such that (cid:80)
k − 1 i =1 ( α j i − 1) <
J G < (cid:80) ki
=1 ( α j i − 1) (note that there are only ﬁnitely many possible choices for such j ), we have J
[Equation]
(cid:88) i
[Equation]
( α
[Equation]
− 1) (J.55)
Fix such ∆ > 0 .
Meanwhile, given the ﬁxed (cid:15) > 0 , we can deﬁne ˆ t =
c 1 + c 1 log(1 /(cid:15) )
and choose (cid:101) (cid:15) > 0 as the largest possible value such that
[Equation]
where functions ¯ ρ ( · ) , (cid:101) ρ ( · ) are deﬁned in Corollary J.6 and J.9 respectively.
We stress that ¯ t is a ﬁxed constant while ˆ t depends on the value of (cid:15) [Equation] and for any sufﬁciently small (cid:15) we will have ˆ t > ¯ t .
Let us deﬁne an event A × =
A × 1 ∪ A × 2 where
[Equation]
where the positive integer l ∗ is deﬁned in eq.
(J.1).
By deﬁnition of l ∗ , we must have K η,δJ G < l ∗ on event { K η,δJ G < ∞} .
First, we analyze the following event
[Equation]
In particular, note that B = (cid:83) l ∗ − 1 K =
B ( K ) where
[Equation]
When K = 0 < 1 , the event B (0) = ∅ by deﬁnition (due to event (cid:110) T ηj +1 ( δ ) − T ηj ( δ ) > 2ˆ tη for some j = 1 , 2 , 3 , · · · , K (cid:111) ).
For K = 1 , · · · , l ∗ , Lemma J.16 implies that B ( K ) = ∅ for all η sufﬁciently small.
In summary, B = ∅ for all η sufﬁciently small.
Now we focus on the following two events
[Equation]
On the one hand, event C can be decomposed as follows.
Let
[Equation]
be the set that contains all the types j such that the accumulated cost reach J G if and only if the last element is kept.
One can see that there are only ﬁnitely many elements in J ↑ .
Now we have
C =
[Equation]
j ∈ J ↑ C ( j ) where (see the deﬁnition in eq.
(J.10))
[Equation]
of type
[Equation]
For any j ∈ J ↑ , observe that (let k j = | j | )
[Equation]
j
[Equation]
W ηi ( δ
[Equation]
∈ E j i ∀ i ∈ [ K η,δJ G ] and T ηi ( δ ) − T ηi
[Equation]
1 ( δ ) ≤ 2ˆ t η for all i = 2 , 3 , · · · , K η
[Equation]
δJ G (cid:17) = (cid:0) k j (cid:89) i =
H j i ( δ/η )
H ( δ/η ) (cid:1) · (cid:0)
[Equation]
The last equality is due to independence of W ηi ( δ ) and T ηi ( δ ) − T ηi − 1 ( δ ) .
The regularly varying natures of functions H j and H imply that
[Equation]
Meanwhile, from Lemma G.4 and the regularly varying nature of function H j and H (in particular, repeating the same calculations that leads to eq.
(J.46))
, we have
[Equation]
Therefore, we have yielded that
[Equation]
where
[Equation]
due to eq.
(J.55).
Therefore, one can see that as η ↓ 0 ,
[Equation]
Furthermore, the discussion above have shown that for all η sufﬁciently small,
[Equation]
From Lemma J.10 and J.12, one can see that for all N > 1 + J G − α 1 + ∆ and all δ > 0 that is sufﬁciently small, sup x : (cid:107) x
(cid:107)≤ 2 (cid:15) P ( A × ) =
o ( η N ) .
Fix such δ > 0 .
In conclusion, we now have that
[Equation] 1 +∆ ) .
On the other hand, on event D , Assumption I.7 shows that (see the deﬁnition in eq.
(J.39))
[Equation]
( δ ) (cid:1) is of type- j for some j ∈ j
( i ∗ ) (cid:9) .
Therefore,
[Equation]
is of type- j for some j ∈ j
[Equation]
i ∗ )
[Equation]
In particular, using Lemma J.15, one can see that when η is sufﬁciently small, on event B × 2 , (II) ( (cid:15), δ, η ) ∩
D we have (cid:107) X ηk (cid:107) < 3 (cid:15) for all k < T η 1 ( δ ) and sup
[Equation]
< 2 (cid:15).
Here z
[Equation]
=
[Equation]
[Equation]
with T =
[Equation]
[Equation]
(cid:0) W η 1 ( δ ) , W η 2 ( δ ) , · · · , W η K η,δJ G ( δ ) (cid:1) , and X (1) =
X ηT η 1 ( δ ) − 1 .
Besides, eq.
(J.17) dictates that, on event B × 2 , (II) ( (cid:15), δ, η ) ∩ D , we must have inf s ≥
d ( z η ( s ) , G c ) ≥ 100 l ∗ ¯ (cid:15), hence
[Equation]
However, this clearly contradicts the deﬁnition of event B × 2 , (II) ( (cid:15), δ, η ) .
In conclusion, we have established that (for all η sufﬁciently small) B × 2 , (II) ( (cid:15), δ, η ) ∩ D = ∅ and
[Equation]
However, as per the argument above, our choice of sufﬁciently small δ > 0 ensures that sup x : (cid:107) x (cid:107)≤ 2 (cid:15) P ( A × ) =
o ( η 1+ J G − α 1 +∆ ) and concludes the proof.
Furthermore, starting from the local minimum 0 , it is unlikely that the SGD iterates will be extremely close to ∂ G when the accumulated cost of large jumps reaches J G .
The next lemma provides an upper bound for the probability of the said scenario.
For any set A ⊆ R d , deﬁne its (cid:15) − enlargement as the following open set A (cid:15)
=
∆ { x : d ( x, A ) < (cid:15) } .
Lemma J.21.
For any (cid:15) ∈ (cid:0) 0 , ¯ (cid:15)/ (3 + 3 ρ ∗ ) (cid:1) and any δ > 0 that is sufﬁciently small,
[Equation]
where B × 3 ( (cid:15), δ, η ) = [Equation] ) < (cid:15) } and Ψ( (cid:15) )
= µ (cid:16) h − 1 (cid:0) ( ∂ G ) (3+3 ρ ∗ ) (cid:15) (cid:1)(cid:17) with h deﬁned in eq.
(J.38) and µ deﬁned in eq.
(J.48) , and ρ ∗ ∈ (0 , ∞ ) is the constant provided in Corollary J.7.
Proof.
Again, given the ﬁxed (cid:15) > 0 , we can deﬁne ˆ t =
c 1 + c 1 log(1 /(cid:15) )
and choose (cid:101) (cid:15) > 0 as the largest possible value such that
[Equation]
where functions ¯ ρ ( · ) , (cid:101) ρ ( · ) are deﬁned in Corollary J.6 and J.9 respectively.
We stress that ¯ t is a ﬁxed constant while ˆ t depends on the value of (cid:15) , and for any sufﬁciently small (cid:15) we will have ˆ t > ¯ t .
Let us deﬁne an event A × =
A × 1 ∪ A × 2 where
[Equation]
where the positive integer l ∗ is deﬁned in eq.
(J.1).
Consider the following decomposition of event
[Equation]
δ ) > 2¯ t/η or ∃ j = 1 , 2 , · · · , K η,δJ G s.t.
η
[Equation]
Using Lemma J.20, we know that for all δ > 0 sufﬁciently small, [Equation] [Equation] P
[Equation]
As a result,
[Equation]
P [Equation] From Lemma J.10 and J.12, one can see that for all N > 1+ J G − α 1 and all δ > 0 that is sufﬁciently small, we have sup x : (cid:107) x
(cid:107)≤ 2 (cid:15) P ( A × ) =
o ( η N ) =
o ( (cid:101) λ ( η )) .
Moving on, we focus on bounding the probability of event C (III) .
In particular, note that
[Equation]
G = J G } ∩ { d ( X η T ≥ J G ( η,δ ) , ∂ G ) < (cid:15) } [Table] (cid:110)
[Table]
j
[Table] [Table] [Table]
η
[Table]
δ [Table] ηj
[Table]
δ
[Table] − [Table]
ηj
[Table] δ [Table] ≤ 2¯ t/η and
[Table] [Table]
η
[Table]
η
[Table]
W ηj
[Table]
(cid:13)(cid:13) ≥ [Table] δ (cid:111) . By applying Lemma J.15 on event C (III) , one can see that when η is sufﬁciently small, on this event we have (cid:107) X ηk (cid:107) < 3 (cid:15) for all k < T η 1 ( δ ) and sup
[Equation]
Here
[Equation]
with T = (cid:0)
[Equation]
, T
[Equation] [Equation]
(cid:0)
[Equation]
[Equation]
[Equation]
, and X (
[Equation]
) = X ηT
[Equation]
[Equation]
) − 1 .
To better control the location of z η ( s ) , we also construct
[Equation]
where the only difference is that we substitute the initial value X (1) with 0 .
Note that on event C (III) , we have (cid:13) (cid:13) X (1) (cid:13) (cid:13) < 3 (cid:15) .
By applying Corollary J.7 (the condition about line segments contained in G is veriﬁed due to eq.
(J.18))
, we have the following bound on event C (III) :
[Equation]
Now recall Deﬁnition J.1.
Combining all the bounds we have obtained so far, we see that, when η is sufﬁciently small, on event C (III) we have
[Equation]
and W is of type- j for some j ∈ j
( i ∗ ) .
It then follows immediately from Corollary [-H.
-]J.14 that
[Equation]
and this concludes the proof.
Lastly, we provide a lower bound for the probability of the most likely way of escape, i.e., due to k ∗ large jumps with accumulated cost J G and relatively short inter-arrival time less (when compared to ¯ t/η ).
We stress that, in the next result, the value of constant c ∗ > 0 would not vary with the choice of parameters (cid:15), δ .
Lemma J.22.
There exists some c ∗ > 0 such that the following claim holds for any (cid:15) ∈ (0 , ¯ (cid:15)/ (4 + 3 ρ ∗ )) and any sufﬁciently small δ > 0 :
[Equation]
where the event is deﬁned as
[Equation]
and ρ ∗ ∈ (0 , ∞ ) is the constant provided in Corollary J. [Equation]
Proof.
First of all, Assumption I.7 implies that
[Equation]
We also stress that K η,δJ G = k ∗ means that σ ( η ) ∧ R ( (cid:15), δ, η ) ≥ T ηk ∗ ( δ ) (i.e. the arrival time of the k ∗ − th large jumps and that the accumulated cost of large jumps W η 1 ( δ ) , · · · , W ηk ∗ ( δ ) is exactly J G .
Due to eq.
(J.19) and eq.
(J.17), there exists some j ∈ j
( i ∗ ) , some (cid:101) θ =
( (cid:101) θ 1 , · · · , (cid:101) θ k ∗ ) with (cid:101) θ
i ∈ F i ∀ i , some (cid:101)
r
=
( (cid:101) r 1 , · · · , (cid:101) r k ∗ ) such that (cid:101) r i ≥ ¯ δ ∀
i , some (cid:101) t
= ( (cid:101) t 1 , · · · , (cid:101) t k ∗ ) such that (cid:101) t 1 = 0 and (cid:101) t i ∈ (0 , ¯ t ) for all i = 2 , · · · , k ∗ such that
[Equation]
where h is the mapping deﬁned in eq.
(J.38).
Let (cid:101) y =
∆ h ( (cid:101) r , (cid:101) θ , (cid:101) t ) .
Moreover, the continuity of mapping h implies the existence of some ∆ > 0 such that ( (cid:101) r , (cid:101) θ , (cid:101) t )
∈
X ⊆
h − 1 (cid:0) B
( (cid:101)
y, ¯ (cid:15) ) (cid:1) where the open domain X is deﬁned as
[Equation]
In particular, the parameter ∆ can be chosen small enough with
[Equation]
so that for any ( r , θ , t ) ∈ X , we have t i > 0 for all i = 2 , · · · , k ∗ and r i > 0 for all i ∈
[ k ∗ ] .
Fix such ∆ > 0 .
Meanwhile, from the deﬁnition of the measure µ ( · ) in eq.
(J.40)eq.
(J.48) and the fact that (cid:101) θ i ∈
F j i where the closed set F j i is the support of the probability measure S j i on the unit sphere S d −(see Assumption I.6), one can see that µ ( X ) > 0 .
Let R =
( R i ) k ∗ i =1 , Θ = (Θ i )
k ∗ i =1 , T =
( T i ) k ∗ i =1 with R i =
η (cid:107) W ηi ( δ ) (cid:107) , Θ i = W ηi ( δ ) / (cid:107) W ηi ( δ ) (cid:107) and T i =
η ( T ηi ( δ ) − T ηi − 1 ( δ )) for all
i ≥ 2 and T 1 = 0 .
Now recall Deﬁnition J.1 and consider the following event
[Equation]
Corollary J.14 then gives the bound
[Equation]
From now on, we ﬁx some c ∗ ∈ (0 , µ ( X )) .
Given the ﬁxed (cid:15) > 0 , we can deﬁne ˆ t =
c 1 + c 1 log(1 /(cid:15) )
and choose (cid:101) (cid:15) > 0 as the largest possible value such that
[Equation]
where functions ¯ ρ ( · ) , (cid:101) ρ ( · ) are deﬁned in Corollary J.6 and J.9 respectively.
We stress that ¯ t is a ﬁxed constant while ˆ t depends on the value of (cid:15) , and for any sufﬁciently small (cid:15) we will have ˆ t > ¯ t .
Let us deﬁne an event A × =
A × 1 ∪ A × 2 where
[Equation]
where the positive integer l ∗ is deﬁned in eq.
(J.1).
Now we focus on the event B \
A × .
On one hand, Lemma J.15 shows that, for all η sufﬁciently small, on this event we have (cid:107) X η k (cid:107) < (cid:15) < ¯ (cid:15) for all
[Table] and sup
[Table]
Here
[Equation]
with T =
[Equation]
, T
[Equation]
[Equation]
) , W [Equation] [Equation] [Equation]
ηk
[Equation]
δ )
[Equation]
and X (1) =
X ηT [Equation]
To better control the location of z η ( s ) , we also construct
[Equation]
where the only difference is that we substitute the initial value X (1) with 0 .
First of all, from eq.
(J.20), we see that
[Equation]
Besides, due to X ⊆ h − 1 (cid:0) B ( (cid:101) y, ¯ (cid:15) ) (cid:1) , we have that d (cid:0) z η 0 (cid:0) (cid:80) k ∗ i =1 T i (cid:1) , G (cid:1) >
99 l ∗ ¯ (cid:15).
Next, using Corollary J.7, we have that sup ∈ [0 , (cid:80) k ∗ i =1 T i ] (cid:107) z η ( s
[Equation]
− z
[Equation]
s ) (cid:107) < 3 ρ ∗ (cid:15) < ¯ (cid:15) , which further implies that
[Equation]
Now we have the following facts regarding the distance between SGD iterates X ηj and the domain G .
First, on event B \ A × , it holds that
[Equation]
implying that σ ( η ) ≥ T ηk ∗ ( δ ) .
Next, if R ( (cid:15), δ, η ) < T ηk ∗ ( δ ) , then there exists some T η 1 ( δ ≤ j < T ηk ∗ ( δ ) such that (cid:13)(cid:13)
X ηj (cid:13)(cid:13) ≤ 2 (cid:15) hence [Equation]
z η
[Equation] − T η 1 ) (cid:107) ≤ 2 (cid:15) + 2 (cid:15) <
¯ (cid:15) < 100 l ∗ ¯ (cid:15) .
However, in light of eq.
(J.20) we know that on event { R ( (cid:15), δ, η ) <
T ηk ∗ ( δ ) } ∩ ( B \
A × ) , we have
[Equation]
and yield a contradiction.
Therefore, on event B \
A × we must have R ( (cid:15), δ, η ) ≥ T ηk ∗ ( δ ) .
Besides, bounding the gap between X ηj , z η ( j ) and z η 0 ( j ) at time j = T ηk ∗ ( δ ) using results above, we can show that
[Equation]
Therefore, we must have σ ( η ) =
T ηk ∗ ( δ ) .
In summary, we have shown that, for all η sufﬁciently small, B \ A × ⊆
A ◦ ( (cid:15), δ, η ) .
Combining all the bounds above, we have
[Equation]
To conclude the proof, it sufﬁces to invoke Lemma J.10 and J.12 with some N > 1 + J G − α 1 .
J. P ROOF OF THE MAIN RESULT
Now we are ready to state the main result and provide upper and lower bounds to the distribution of the scaled ﬁrst exit time from domain G .
In particular, we deﬁne a scaling function
[Equation]
where the function (cid:101) λ ( · ) is deﬁned in eq.
(J.50).
One can easily see that λ ∈ RV 1+ J G ( η ) .
We show that the scaled ﬁrst exit time λ ( η ) σ ( η ) converges in distribution to an exponential random variable as η ↓ 0 , which implies that, in expectation, the ﬁrst exit time is roughly a 1 /η 1+ J G term and its order is dictated by the minimum cost for exit J G .
Proposition J.23.
Given any C ∈ (0 , 1) , u > 0 , the following inequalities hold for all (cid:15) >sufﬁciently small,
[Equation]
where q = µ (cid:0) h − 1 ( G c ) (cid:1) .
Before presenting the proof to Proposition J.23, we make some preparations.
First, we introduce stopping times (for all k ≥ 1 )
[Equation]
with the convention that τ 0 ( (cid:15), δ, η ) = (cid:101) τ 0 ( (cid:15), δ, η ) = 0 .
Intuitively, at each (cid:101)
τ k the SGD iterates have just returned to a small neighborhood of the local minimum 0 .
Due to Markov property of X ηn , the SGD iterates almost regenerate at each (cid:101)
τ k despite the previous trajectory, and the times ( (cid:101) τ k ) k ≥ 1 partitions the entire timeline into different segments that can almost be interpreted as regeneration cycles , where (cid:101)
τ k − 1 can be understood as the starting point of the k − th cycle.
In light of the embedded (informal) regeneration process, one natural approach is to determine the dynamics and probability of making an exit on one (hence every) cycle, and this will be carried out with the help of technical results in the previous section.
It is worth noticing that (cid:101)
τ k are deﬁned under the proviso that (cid:101)
τ k ≥ τ k where τ k is the ﬁrst big jump during the k − th cycle.
Considering results such as Lemma J.12, it is reasonable to expect that the SGD iterates would be trapped at local minimum until a large jump strikes.
Therefore, we deﬁne the (informal) regeneration points (cid:101)
τ k in such a way that a cycle [ (cid:101) τ k − 1 , (cid:101) τ k ) ends only if we have observed at least one large jump already.
Regarding the notations, we add a remark that when there is no ambiguity we will drop the dependency on (cid:15), δ, η and simply write τ k , (cid:101)
τ k .
Speciﬁcally, we are interested in the large jumps and the accumulated cost thereof during each cycle, which deﬁnitely entails some systematic bookkeeping.
For all k ≥ 1 , the random variable
[Equation]
can be understood as the count of large jumps during the k − th cycle.
Here are two remarks on this deﬁnition.
• First, for any k with σ ( η ) < (cid:101)
τ k , we have j k = 0 .
Recall that the object we study here is the ﬁrst exit time σ ( η ) , so there is virtually no need to keep track of the dynamics of the SGD after it leaves the domain G .
• The random variable j k is measurable w.r.t.
F (cid:101)
τ k ∧ σ ( η ) , the stopped σ − algebra generated by the stopping time (cid:101)
τ k ∧ σ ( η ) , which, intuitively, is stating that one should be able to determine the number of large jumps during the k − th cycle when this cycle ends.
Furthermore, for each k = 1 , 2 , · · · , let
[Equation]
[Equation] with the convention T k, 0 ( (cid:15), δ, η ) = (cid:101)
τ k − 1 ( (cid:15), δ, η ) .
Note that for any k ≥ 1 , j ≥ 1 , T k,j is a stopping time.
Besides, for any k with (cid:101) τ k < σ ( η )
[Equation]
and the sequences (cid:0) T k,j (cid:1)
j k j =1 and (cid:0) W k,j (cid:1)
j k j =1 are the arrival times and sizes of large jumps during the k − th cycle, respectively.
Again, when there is no ambiguity we will drop the dependency on (cid:15), δ, η and simply write T k,j and W k,j .
We are now able to keep track of the accumulated cost of jumps during each cycle.
For any k ≥ 1 , i ≥ 1 , we deﬁne
[Equation]
k,i ( (cid:15), δ, η )) .
Moreover, we deﬁne the following index
[Equation]
∞ otherwise with the convention that min ∅ = ∞ .
That is to say, if (cid:101)
τ k ≥ σ ( η ) or J cycle k ( j k ) <
J G , we deﬁne j ≥ J G ( k ) as ∞ ; otherwise j ≥ J G ( k ) is the index for the ﬁrst large jump during the k − th cycle that drives the accumulated cost of large jumps to reach J G .
For clarity of the presentation below, we also deﬁne
[Equation]
In other words, when j ≥ J G ( k ) < ∞ , the random variable X ≥ J G ( k ; (cid:15), δη ) is equal to X ηT k, j ≥ J G ( k ) , the location of the SGD iterates right when the accumulated cost on the k − th cycle reaches J G .
Notation wise, we will drop the dependency on (cid:15), δ, η again and simply write j ≥ J G ( k ) , X ≥ J G ( k ) or J cycle k
( j ) when there is no risk of ambiguity.
As have mentioned above, we want to zoom in on the each cycle and analyze the probability of each possible case.
To be speciﬁc, we want to introduce a series of scenarios, formally deﬁned as several events, that exhaust all the possibilities during a cycle.
The events will be denoted in the form of A ◦ k or B × j largely following the next few rules.
First, we say an event/scenario is atypical if its probability is rather small, and we assign it with superscript × ; otherwise we say the event is typical and add a superscript ◦ .
Besides, the subscript k indicates the cycle that the event concerns.
Lastly, events with label (of type)
A usually describe how the SGD iterates try to escape from G , whereas events with label B focuses on how the SGD iterates return to the local minimum.
Now we proceed and formally deﬁne the said series of events.
First, for each k ≥ 1 , deﬁne the event
A × k, 0 ( (cid:15), δ, η ) = ∆ (cid:110) ∃ i ∈ [ l ∗ ] s.t. max
[Equation]
∪ (cid:110) (cid:13)(cid:13)
X ηj (cid:13)(cid:13) > 3 (cid:15) for some j = (cid:101) τ k − 1 , (cid:101) τ k −
[Equation]
+ 1 , · · · , T k
[Equation]
−
[Equation]
(cid:111) (J.58) with ˆ t ( (cid:15) )
= c
1 + c 1 log(1 /(cid:15) ) and function (cid:101) (cid:15) ( · ) deﬁned in eq.
(J.51).
It is similar to the event A × deﬁned in eq.
(J.52)eq.
(J.53) in the previous section, and its probability will be controlled using a similar approach.
Next, consider event (for all k ≥ 1 )
[Equation]
that describes an atypical case where the exit occurs during the k − th cycle yet the accumulated cost of all large jumps is less than J G .
Another atypical event is deﬁned as
[Equation]
This event describes the case where the accumulated cost of large jumps in a cycle has reached J G but the inter-arrival time between some large jumps are unusually long.
We stress that, by deﬁnition of j ≥ J G ( k ) , we have { j ≥ J G ( k ) < ∞} = { (cid:101) τ k − 1 < σ ( η ) } ∩ { J cycle k ( j
[Equation]
Moving on, we consider the following events (deﬁned for all k ≥ 1 )
[Equation]
that describes another similar case where the k − th cycle ends with return to the local minimum but the inter-arrival time between some large jumps are unusually long.
The next event
[Equation]
[Equation] J.62) describes the case where the accumulated cost on the k − th cycle has hit J G exactly at some point with SGD iterates being rather close to G c yet some large jumps are not large enough when compared to the ﬁxed constant ¯ δ .
Meanwhile, with events
[Equation]
we analyze the case where the SGD iterates reaches somewhere close the the boundary set ∂ G with large jumps of cost equal to J G .
Lastly, deﬁne event
[Equation]
for the case where the accumulated cost of large jumps on the k − th cycle exceeds J G without hitting it.
As an amalgamation of these atypical scenarios
[Equation]
we let
[Equation]
Next, we analyze the probability of some events ( B × k ) k ≥ 1 that concern the behavior of SGD iterates during the k − th cycle after T ≥ J G ( k ) .
Let us deﬁne
[Equation]
where ρ ( · ) is the function in Lemma J.11.
From the deﬁnition of B × k , in particular the inclusion of
B × k, 2 , one can see that the intuitive interpretation of event B × k is that the SGD iterates did not return to local minimum efﬁciently or even escaped from G after large jumps with cost equal to J G .
In comparison, the following events will characterize what would typically happen during each attempt:
[Equation]
Intuitively speaking, A ◦ k tells us that the exit happened right at T ≥ J G ( k ) with accumulated cost of large jumps in the cycle being exactly J G , and B ◦ k requires that that the ﬁrst exit did not occur during the k − th cycle, and the SGD iterates returned to local minimum rather efﬁciently.
It is worth noticing that, by deﬁnition, j ≥ J G ( k ) <
l ∗ if j ≥ J G ( k ) < ∞ .
Our immediate next goal is to analyze the event
[Equation]
In particular, we show that its probability can be made arbitrarily small with proper (cid:15), δ, η , implying that we will almost always observe the event ( A × ) c .
Lemma J.24.
Given any C > 0 , the following claim holds for all (cid:15) > 0 , δ > 0 sufﬁciently small:
[Equation]
Proof.
We start by bounding for the probabilities of all the A × k,i events.
Fix some N > 1 + J G .
For
A × k, 0 ( (cid:15), δ, η ) = (cid:110) ∃ i ∈ [ l ∗ ] s.t.
[Equation]
∪ (cid:110) (cid:13)(cid:13)
X
η j (cid:13)(cid:13) > 3 (cid:15) for some j = (cid:101) τ k
[Equation]
1 , (cid:101) [Equation] k
[Equation]
1 + 1 , · · · , T k [Equation] [Equation] 1 (cid:111) , using Lemma J.10 and J.12, one can show the existence of some (cid:15) 0 ∈ (0 , ¯ (cid:15)/ 3) such that for any (cid:15) ∈ (0 , (cid:15) 0 ) , there is some δ 0 ( (cid:15) ) > 0 with
[Equation]
Next, for A × k, 1 ( (cid:15), δ, η ) =
(cid:110) σ ( η ) < [Equation] τ [Equation]
, J cycle [Equation] ( j k ) < J G (cid:111) , it follows from Lemma J.17 that, for any (cid:15) ∈ (0 , (cid:15) 0 ) , there exists some 0 <
δ 1 ( (cid:15) ) ≤ δ 0 ( (cid:15) )
such that [Equation]
For event A × k, 2 = { j ≥ J G ( k ) < ∞} ∩ {∃ j = 2 , 3 , ·
[Equation] [Equation]
j
[Equation]
s
[Equation]
t
[Equation]
T k
[Equation]
j − T k,j − 1 > 2ˆ t ( (cid:15) ) /η } ,
Lemma J.18 implies that, for any (cid:15) ∈ (0 , (cid:15) 0 ) , there exists some 0 < δ 2 ( (cid:15) ) ≤ δ 1 ( (cid:15) ) such that lim sup
[Equation]
For event A × k,
[Equation]
cycle k
[Equation]
thanks to Lemma J.19, one can see that, for any (cid:15) ∈ (0 , (cid:15) 0 ) , there exists some 0 < δ 3 ( (cid:15) ) ≤ δ 2 ( (cid:15) )
such that
[Equation]
For event
A ×
[Equation]
using Lemma J.20, one can see the existence of some ∆ > 0 such that, for all (cid:15) ∈ (0 , (cid:15) 0 ) , there is some 0 <
δ 4 ( (cid:15) ) ≤ δ 3 ( (cid:15) )
such that
[Equation]
J.74)
As for event A × k, 5 =
{ j ≥ J G ( k ) < ∞ , J cycle k
( j ≥ J G ( k )) =
J G } ∩ (cid:110) d ( X ≥ J G ( k ) , ∂ G ) <
(cid:15) (cid:111) , the plan is to use Lemma J.21 to control its probability.
But before that, we recall the deﬁnition of function Ψ( (cid:15) )
= µ (cid:16) h − 1 (cid:0) ( ∂ G ) (3+3 ρ ∗ ) (cid:15) (cid:1)(cid:17) in Lemma J.21 and make several observations.
First, note that ∂ G is a closed set, so ∩ (cid:15)> 0 h − 1 (cid:0) ( ∂ G ) (cid:15) (cid:1) =
h − 1 ( ∂ G ) for the mapping h in eq.
(J.38).
Then it follows from Assumption I.9 that lim (cid:15) ↓ 0 µ (cid:16) h − 1 (cid:0) ( ∂ G ) (cid:15) (cid:1)(cid:17) = 0 , implying that for (cid:15) > 0 sufﬁciently small, we must have
[Equation]
where C > 0 is the ﬁxed constant in the description of the Lemma, and c ∗ > 0 is the constant in
Lemma J.22.
We stress that both of them would not vary with (cid:15), δ, η .
Combining this with Lemma
J.21, we know the existence of some (cid:15) 5 ∈ (0 , (cid:15) 0 ] such that, for all (cid:15) ∈ (0 , (cid:15) 5 ) , one can ﬁnd some 0 < δ 5 ( (cid:15) ) ≤ δ 4 ( (cid:15) ) with
[Equation]
[Equation]
(J.75)
Lastly, for event
A × k, 6 = { j ≥ J G ( k ) < ∞ , J cycle k
( j ≥ J G ( k )) > J G } , from Lemma J.20 again we know that (let (cid:15)
=
(cid:15) 5 ) for all (cid:15) ∈ (0 , (cid:15) 6 ) , one can ﬁnd some 0 <
δ 6 ( (cid:15) ) ≤ δ 5 ( (cid:15) ) with lim sup η sup (cid:107) x (cid:107)≤ 2 (cid:15)
P x
( A × 1 , 6 ( (cid:15), δ, η ))
[Equation] (J.
[Equation]
Recall that A × k ( (cid:15), [Equation]
[-(cid:83) 6 i =
-][Equation] [Equation] [Equation]
[Equation]
Also, the events B × k , A ◦ k , B ◦ k are deﬁned in eq.
(J.66)eq.
(J.67)eq.
(J.68) respectively.
Before bounding the conditional probability of events B × k , we make several observations.
First, if we consider the event (cid:84) kj =
( A × j ∪ B × j ) c ∩ B ◦ j , the inclusion of the ( B ◦ j ) kj =1 implies that during the ﬁrst k cycles the SGD iterates have never left G , so
[Equation]
Next, note that
[Equation]
We start by analyzing term (I).
From Assumption I.7, one can see that on event B × k, 1 , it holds that j ≥ J G ( k )
=
k ∗ , ( W k, 1 , · · · , W k,k ∗ ) is of type- j for some j ∈ j
( i ∗ ) and T k,j − T k,j − 1 ≤ 2 ˆ t ( (cid:15) )
η ∀ j = 2 , 3 , · · · , k ∗ .
Due to independence of W k,i and T k,i +1 − T k,i , we have
[Equation]
Due to the strong Markov property at stopping time (cid:101) τ k − 1 , Lemma G.4, and the regularly varying nature of functions H i , H ,
[Equation]
In other words, for any (cid:15), δ > 0 , there exists some C ( (cid:15), δ ) < ∞ such that
[Equation]
On the other hand, due to Lemma J.11 and the strong Markov property at stopping time T k,k ∗ , for any (cid:15) ∈ (0 , ¯ (cid:15) ) and any δ > 0 ,
[Equation]
In summary, we have established that for any (cid:15) ∈ (0 , ¯ (cid:15) ) and any δ > 0 ,
[Equation]
Similarly, we can bound conditional probabilities of the form P x (cid:0)
A × k | (cid:84) k − 1 j =1 ( A × j ∪ B × j ) c
∩ B ◦ j (cid:1) .
To be speciﬁc, recall that A × k
= ∪ 6 i =
A × k,i .
Combining eq.
(J.70)-eq.
(J.76) with strong Markov property, we know the for all (cid:15) ∈ (0 , (cid:15) 6 ) and k ≥ 1 , [Equation] [Equation] J.
[Equation]
On the other hand, a lower bound can be established for conditional probability of event A ◦ k , the event deﬁned in eq.
(J.67).
Due to Lemma J.22 and the strong Markov property at (cid:101) τ k − 1 , one can see the existence of some (cid:15) 7 ∈ (0 , (cid:15) 6 ] such that, for any (cid:15) ∈ (0 , (cid:15) 7 ) , there is some 0 < δ 7 ( (cid:15) ) ≤ δ 6 ( (cid:15) )
such that
[Equation]
In order to apply the bounds eq.
(J.77)-eq.
(J.79), we make use of the following inclusion relationship:
[Equation]
To see why this is true, let us consider a decomposition of the event on the L.H.S. of eq.
(J.80).
As mentioned above, on event (cid:84)
k − 1 j =1 ( A × j ∪ B × j ) c
∩ B ◦ j we know that σ ( η ) > (cid:101)
τ k − 1 , implying that for the ﬁrst k − 1 cycles the ﬁrst exit does not occur and leaving us with only three possibilities on this event:
• j ≥ J G ( k ) =
∞ (on { σ ( η ) > (cid:101)
τ k − 1 } , this would happen if and only if J cycle k ( j k ) <
J G ); • j ≥ J G ( k ) < ∞ , X ≥ J G ( k ) / ∈ G ; • j ≥ J G ( k ) < ∞ , X ≥ J G ( k ) ∈ G .
Let us partition the said event accordingly and analyze them one by one.
• On (cid:0) (cid:84) k − 1 j =1 ( A × j ∪ B × j ) c
∩ B ◦ j (cid:1) ∩ ( A × k ∪
B × k ) c
∩ {J cycle k ( j k ) <
J G } , due to the exclusion of A × k (especially A × k, 1 and A × k, 3 ), we can see that if j k < l ∗ , then we must have σ ( η ) > (cid:101)
τ k and (cid:101)
τ k − T k, 1 ≤ 2 l ∗ ˆ t ( (cid:15) ) /η .
In particular, note that on this event we must have l ∗ > j k by deﬁnition of l ∗ in eq.
(J.1).
Therefore, (cid:0) k − 1 (cid:92) j
= ( A × j ∪ B × j ) c ∩ B ◦ j (cid:1) ∩ ( A × k ∪
B × k ) c
∩ { j ≥ J G ( k ) = ∞} ⊆
B ◦ k .
• On (cid:0) (cid:84) k − 1 j =1 ( A × j ∪ B × j ) c
∩ B ◦ j (cid:1) ∩ ( A × k ∪
B × k ) c
∩{ j ≥ J G ( k ) < ∞ , X ≥ J G ( k ) / ∈ G} , the exclusion of A × k, 2 implies that T k,j − T k,j − 1 ≤ 2ˆ t ( (cid:15) ) /η
for all j = 2 , · · · , j ≥ J G ( k ) , and the exclusion of A × k, 6 tells us that J cycle k ( j ≥ J G ( k )) =
J G .
Moreover, the exclusion of A × k, 5 ∪ A × k, 6 tells us that if X ≥ J G ( k ) / ∈ G , we must have d ( X ≥ J G ( k ) , G ) > (cid:15) .
In summary, (cid:0) k − (cid:92) j
= ( A ×
[Equation]
∪ B × j ) c ∩ B ◦ j (cid:1) ∩ ( A ×
k ∪ B × k c
∩ { j ≥ J G ( k ) < ∞ , X ≥ J G k ) / ∈ G} ⊆
A ◦ k • On (cid:0) (cid:84) k − 1 j =1 ( A × j ∪ B × j ) c
∩ B ◦ j (cid:1) ∩ ( A × k ∪
B × k ) c
∩ { j ≥ J G ( k ) < ∞ , X ≥ J G ( k ) ∈ G} , the same
argument
in the previous bullet point can be applied to show that T k,j − T k,j − 1 ≤ 2ˆ t ( (cid:15) ) /η for all j = · · · , j ≥ J G ( k
J cycle
k ( j ≥ J G ( k )) = J G , and d ( X ≥ J G ( k ) , G c ) > (cid:15) .
Now since B × k did not occur, we must have σ ( η ) > (cid:101)
τ k and (cid:101)
τ k − T ≥ J G ( k ) ≤ ρ ( (cid:15) )
/η , hence (cid:101) τ k − T k, 1 ≤ 2 l ∗ ˆ t ( (cid:15) ) + ρ ( (cid:15) ) η .
Therefore, (cid:0)
[Equation]
[Equation] j = ( A × j ∪ B × j ) c ∩ B ◦ j (cid:1) ∩ ( A × k ∪
B × k ) c ∩ { j ≥ J G ( k ) < ∞ , X ≥ J G ( k ) ∈ G} ⊆ B ◦ k .
With eq.
(J.80) established, we can immediately get that (cid:0) k − [Equation] j = ( A
[Equation]
( A
[Equation]
[Equation]
J.
[Equation]
Next, recall the deﬁnitions of A ◦ k in eq.
(J.67) and B ◦ k in eq.
(J.68), and one can see that the two events A ◦ k and B ◦ k are mutually exclusive, since the former implies that the ﬁrst exit occurs during the k − th cycle whereas the latter implies the ﬁrst exit does not occur in the ﬁrst k cycles.
This fact and eq.
(J.81) allow us to conclude that
[Equation]
The last step is to use all the results so far to bound the probability of event
[Equation]
Using eq.
(J.82), we can see that (for any x with (cid:107) x (cid:107) ≤ 2 (cid:15) )
P
[Equation]
A ×
[Equation]
, δ, η )) =
(cid:88) k
[Equation]
P
[Equation]
k
[Equation]
[Equation]
i =
[Equation]
=
[Table]
[Equation]
[Table]
This allows us to apply eq.
(J.77)-eq.
(J.79) and conclude that for all (cid:15) ∈ (0 , (cid:15) 7 ) , all δ ∈ (0 , δ 7 ( (cid:15) )) and all η >sup
[Equation]
when η is sufﬁciently small.
Having established Lemma J.24, we are ready to provide a proof for Proposition J.23.
Proof of Proposition J.23.
Deﬁne sets
[Equation]
where ρ ∗ is the ﬁxed constant in Corollary J.7.
From the continuity of measure, we have
[Equation]
From Assumption I.7 we know q = ∆ µ (cid:0) h − 1 ( G c ) (cid:1) > 0 .
From Assumption I.9 we also have µ (cid:0) h − 1 ( ∂ G ) (cid:1) = 0 , hence µ (cid:0) h − 1 (( G ) c ) (cid:1) =
q as well.
Therefore, together with Lemma J.24, we know the existence of some (cid:15) 1 ∈ (0 , [Equation] 3+3 ρ ∗ ) such that, for any (cid:15) ∈ (0 , (cid:15) 1 ) , there is some 0 <
δ 1 ( (cid:15) ) with
[Equation]
and
[Equation]
Henceforth, we ﬁx some (cid:15) and δ in this range and prove the inequalities in the Proposition.
To proceed, let J ( (cid:15), δ, η ) = ∆ sup { k ≥ 1 : (cid:101)
τ k − 1 < σ ( η ) } .
We now show that on event ( A × ( (cid:15), δ, η )) c , we must have J ≤ J ↑ where (recall Deﬁnition J.1)
[Equation]
with E + ( z ) =
∆ { y ∈ R
d : d ( y, G ) > (3 + 3 ρ ∗ ) z } where ρ ∗ is the ﬁxed constant in Corollary J.7.
To see this via a proof of contradiction [Equation] let us assume that J ↑ =
j < J for some integer j .
Now we make some observations.
• First of all, due to J ↑ =
j < J and eq.
(J.17), on event ( A × [Equation] [Equation] [Equation]
c [Equation] { J ↑ =
j < J } we have
[Equation]
• The deﬁnition of event ( A × ) c , especially the exclusion of event A × j, 0 deﬁned in eq.
(J.58) now allows us to apply Lemma J.15 and show that
[Equation]
Here
[Equation]
with T = (cid:0)
[Equation]
T
[Equation]
[Equation]
[Equation]
and X (1) =
X η − 1+ T j, 1 .
Moreover, the previous bullet point allows us to apply Corollary J.7 and conclude that
[Equation]
where z η ( s ) = (cid:101)
x η (cid:0) s, 0 ; T , W (cid:1) ∀ s ≥ {+
+}
and ρ ∗ is the ﬁxed constant in Corollary J
As a result,
(cid:13)(cid:13)(cid:13)
[Equation]
• However, due to J ↑ =
j , we have d ( z η 0 ( T j,k ∗ − T j, 1 ) , G ) >
(3 + 3 ρ ∗ ) (cid:15) .
This immediately implies that X ηT j,k ∗ / ∈ G and yields the contradiction J ≤ j .
Similarly, one can show that on event ( A × ) c
we must have J ≥ J ↓ for
[Equation]
of type [Equation] with E − ( z ) =
∆ { y ∈ R
d : d ( y, G c ) < (3 + 3 ρ ∗ ) z } .
Besides, the following claim holds on event (
A × ) c
∩ { J < ∞} .
The
deﬁnition of ( A × ) c implies that during the J − th cycle the event A ◦ J occurs
whereas
for any j < J we have B ◦ j .
Therefore , for any k = 1 , 2 , · · · , J , we have
[Equation]
In particular, the coefﬁcient l ∗ on the R.H.S. can be justiﬁed as follows: by deﬁnition of A ◦ k and B ◦ k , we can see that J cycle k ( j k ) ≤ J G for all k ≤ J (with j J = k ∗ due to the occurrence of A ◦ J ); the deﬁnition of l ∗ in eq.
(J.1) then implies that l ∗ > j k for all k ≤ J .
• Now we consider the following set
[Equation]
the can be understood as the concatenation of all steps between any return time (cid:101)
τ k − 1 and the ﬁrst large jump time T k, 1 during the k − th cycle.
Our discussion above implies that, on event ( A × ) c , we have then the discussion above have shown that, for min { n ∈ S ( (cid:15), δ, η ) : Z n is of type- ( E + ( (cid:15) ) , η, δ ) } ≥ T J, 1 .
It is worth noticing that the probability of Z 1 being of type- ( E + ( (cid:15) ) , η, δ ) is (see the deﬁnition in eq.
(J.47))
H ( δ/η ) p (cid:0) E
+ ( (cid:15) ) , δ, η (cid:1) .
Therefore, for the following partition of the timeline
S before ( (cid:15), δ, η ) =
∆ { n ∈ S ( (cid:15), δ, η ) : n ≤ σ ( η ) } , I before ( (cid:15), δ, η ) =
∆ # S before ( (cid:15), δ, η ) ,
S after ( (cid:15), δ, η ) =
∆ { n / ∈ S ( (cid:15), δ, η ) : n ≤ σ ( η ) } , I after ( (cid:15), δ, η ) =
∆ # S after ( (cid:15), δ, η ) , we have σ ( η ) =
I before + I after .
Moreover, on event ( A × ) c , we must have
[Equation]
I before ≤ min { n ∈ S ( (cid:15), δ, η ) : Z n is of type- ( E + ( (cid:15) ) , η, δ ) }} .
Next, deﬁne geometric random variables with the following success rates
[Equation]
Given the results above for bounding I before and I after on event ( A × ) c , we have
[Equation]
For term (I)
[Equation]
recall that λ ( η ) =
H (1 /η ) (cid:101) λ ( η ) .
Corollary J.14 and the regularly varying nature of H ( · ) then imply that
[Equation]
From Lemma G.3, we then have lim sup η ↓ 0 (I) ≤ exp( − (1 − C ) 2 u ) .
For term (II), let C ( (cid:15) ) =
∆ 2 l ∗ ˆ t ( (cid:15) )
+ ρ ( (cid:15) ) and note that
[Equation]
Moreover, since
[Equation]
we know that for any M > 0 , we have (cid:101) λ
( η ) λ ( η ) /η >
M eventually as η ↓ 0 .
Therefore, given any
M > 0 , we have (II) ≤ P (cid:16) U 2 > M/ (cid:101) λ ( η ) (cid:17) for any η sufﬁciently small.
Now from Lemma G.3, lim sup η ↓ 0 (II) ≤ exp( − M ) for any M > 0 , and here we ﬁx some M > 0 such that exp( − M ) < C .
In summary, we have shown
[Equation]
and established the upper bound.
The lower bound can be shown by an almost identical approach.
In particular, since
[Equation]
where U (cid:48) 1 ( (cid:15), δ, η ) ∼ Geom (cid:16) H ( δ/η ) p (cid:0) E − ( (cid:15) ) , δ, η (cid:1)(cid:17) .
The same calculation above for term (I) can be used here to provide a lower bound for lim inf η ↓ 0 P (cid:0) λ ( η ) · U (cid:48) 1 > u (cid:1) and conclude the proof.
Now we are ready to prove Theorem I.1, the main theorem of this section.
Proof of Theorem I.1.
Recall that our choice of scaling function λ ( · ) in eq.
(J.56) is regularly varying (w.r.t.
η ) with index 1 + J G .
Fix some x ∈ G , t > 0 and C ∈ (0 , 1) .
It sufﬁces to show that
[Equation]
First, we are able to pick some (cid:15) ∈ (0 , 1) sufﬁciently small such that d ( x, G c ) > (cid:15) and Proposition
J.23 is applicable.
Due to Lemma J.11, for event
[Equation]
, X ηn ∈ G ∀ n ≤ T return ( η, (cid:15) ) } , we have lim η ↓ 0 P x ( A ) = 1 .
Next, on event A , we have σ ( η ) − T return ( η, (cid:15) ) > 0 and (cid:13)(cid:13)(cid:13)
X ηT return ( η,(cid:15) )
(cid:13)(cid:13)(cid:13) ≤ 2 (cid:15) .
Moreover, by combining Proposition J.23 with the strong Markov property at stopping time
T return ( η, (cid:15) ) , we have
[Equation]
Observe that
[Equation]
Besides, on event A we have T return ( η, (cid:15) )
≤ O (1 /η ) as η ↓ 0 .
Given that λ ( η ) ∈ RV (1+ J G ) ( η ) , we have T return ( η, (cid:15) )
λ ( η ) ≤ Ct on event A for all η sufﬁciently small.
Therefore, by applying the bounds above, we establish that lim sup η ↓ 0 P x ( σ ( η ) λ ( η ) > t ) ≤ 2 C + exp( − (1 − C ) 3 qt ) .
Similarly, in order to show the lower bound, observe that
[Equation]
Taking lim inf on both sides yields lim inf η P x ( σ ( η ) λ ( η ) > t ) ≥ − C + exp( − (1 + C ) qt ) and concludes the proof.
K P ROOF OF T ECHNICAL L EMMAS
Proof of Lemma G.3.
For any (cid:15) > 0 , P
[Equation]
By taking logarithm on both sides, we have
[Equation]
Since lim x → 0 ln(1+ x ) x = 1 , we know that for (cid:15) sufﬁciently small, we will have
[Equation]
By taking exponential on both sides, we conclude the proof.
Proof of Lemma G.4.
To begin with, for any (cid:15) > {+
+} we have
[Equation]
Using bound eq.
(K.1), we know that for (cid:15) sufﬁciently small, P ( U ( (cid:15) ) >
1 /b ( (cid:15) )) ≥ exp( − c · a ( (cid:15) ) /b
( (cid:15) )) .
The upper bound follows from the generic bound 1 − exp( − x ) ≤ x, ∀ x ∈ R with x = c · a ( (cid:15) ) /b
( (cid:15) ) .
Now we move onto the lower bound.
Again, from bound eq.
(K.1), we know that for sufﬁciently small (cid:15) , we will have
[Equation]
Due to the assumption that lim (cid:15) ↓ 0 a ( (cid:15) ) /b
( (cid:15) )
=
and the fact that 1 − exp( − x ) ≥ x √ c for x >sufﬁciently close to 0 , we will have (for (cid:15) small enough) P (cid:16) U ( (cid:15) ) ≤ 1 b ( (cid:15) )
(cid:17) ≥ 1 c · a ( (cid:15) )
b ( (cid:15) ) .
Proof of Lemma G.5.
Let a k = ∆ x k − (cid:101) x k .
Using intermediate value theorem, one can easily see that
[Equation]
The desired bound then follows immediately from Gronwall’s inequality.
Proof of Lemma J.1.
Due to the ﬁniteness of all types j (cid:48) with J type ( j (cid:48) ) <
J G , it sufﬁces to ﬁx one of such j (cid:48) and prove the existence of constants ¯ t, ¯ δ, (cid:15) 0 .
Let ¯ (cid:15) be the constants in eq.
(J.3).
For any r ∈ (0 , ¯ (cid:15) ) , deﬁne the following stopping time
[Equation]
Due to Assumptions I.4 and I.5 as well as continuity in Poincaré Map (see Theorem 12 in Immler & Traut (2019)), for any ﬁxed r >
we know that τ r ( · ) is a continuous function on G .
Note that, by deﬁnition, we have τ r ( x ) = 0 for any x ∈ B ( 0 , r ) .
Due to G being compact and eq.
(J.3), we know the existence of some constant T r ∈ (0 , ∞ ) such that sup x ∈ ¯ G τ r ( x ) ≤ T r and x t ( x ) ∈ B ( 0 , r ) for any x ∈ G , t ≥ T r .
Due to Assumption I.8 and the ﬁniteness of all types j (cid:48) with J type ( j (cid:48) ) < J G , there exists a (cid:15) 0 ∈ (0 , ¯ (cid:15)/ 2) such that
[Equation]
for all j (cid:48) with J type ( j (cid:48) ) < J G , t ∈ { 0 } × R | j (cid:48) |− 1 + , w ∈ A type j (cid:48) .
We ﬁx such (cid:15) 0 > 0 .
Here is one implication that is worth mentioning.
Recall that j is ﬁxed in the description of the Lemma and k (cid:48)
=
| j | .
If
[Equation]
for some t ( k (cid:48) ) ∈ { 0 } × R k (cid:48) − 1 + , w ( t (cid:48) )
∈
A type j , then we must have
[Equation]
To show the existence of some ¯ δ > 0 we appeal to a proof by contradiction.
(For a clean presentation, in t and w we omit the ( k (cid:48) ) term in the superscript since the cardinality is ﬁxed.)
Assume that we can ﬁnd a sequence (cid:0) t n , w n (cid:1) n ≥ 1 such that min (cid:8) (cid:107) w n 1 (cid:107) , · · · , (cid:107) w nk (cid:48) (cid:107) (cid:9) ≤ 1 /n and d (cid:0)(cid:101) x (cid:0) (cid:80) k
(cid:48)
i =1 t ni , 0 ; t n , w n (cid:1) , G c (cid:1) ≤ (cid:15) 0 for any n .
Due to the truncation operator ϕ b ( · ) in the deﬁnition of (cid:101) x t , without loss of generality we can replace all jumps w nj by ϕ b (cid:0) w nj (cid:1) to ensure that w nj is always in a compact set.
Deﬁne
[Equation]
By picking a subsequence when necessary, we can further assume that
• For any i ≥ 1 , w ( n ) j converges to some w ∗
i and y ni converges to some y ∗ i .
In particular, there exists some I ∈
[ k (cid:48) ] such that w ∗ I = 0 .
• Also, since G c is a closed set, we have d ( y ∗ k (cid:48) , G c ) ≤ (cid:15) 0 .
• For any i = 2 , 3 , · · · , k (cid:48) , t ni either converges to some ﬁnite t ∗ i , or lim n t ni = ∞ .
• Note that
[Equation]
for the following reason: for any r ∈ (0 , ¯ (cid:15) ) , our discussion about τ
r ( x ) at the beginning of the proof implies that lim sup n (cid:107) y ni (cid:107) ≤ r .
Let (cid:101) I be the largest index i ∈
[ k (cid:48) ] such that y ∗ i = 0 .
In case that we cannot ﬁnd such index, let (cid:101) I = 0 .
We ﬁrst consider the case where (cid:101) I = 0 .
The bullet points above then imply that, for any i ≥ 1 , we must have lim n t ni = t ∗ i < ∞ .
Now due to boundedness of t ∗ i and the continuity of ODE ﬂow, for t ∗ =
∆ (0 , t ∗ 2 , t ∗ 2 , · · · , t ∗ k (cid:48) ) ,
[Equation]
[Equation]
[Equation] [Equation] [Equation] (cid:48) ) , we have
[Equation]
Now recall that w ∗ I = 0 for some I ∈
[ k (cid:48) ] .
By removing the vacuous (size-zero) jumps in ( t ∗ , w ∗ ) , we now know the existence of some (cid:101) k (cid:48) ∈ N , (cid:101) w ∗ ∈
( R d \ O ) (cid:101) k (cid:48) , (cid:101) t ∗ ∈ { 0 } × R (cid:101) k (cid:48)
+ such that d (cid:0)(cid:101) x [Equation] i =1 (cid:101) t ∗ i , 0 ; (cid:101) t ∗ , (cid:101) w ∗ ) , G c (cid:1) ≤ (cid:15) 0 .
Meanwhile, the condition J ↓ type ( j ) <
J G implies that, for the total cost of jumps in (cid:101)
w ∗ after removal of size-zero jumps, (cid:80) (cid:101) k (cid:48) i =
1 J ( (cid:101) w ∗ j ) < J G .
However, this contradicts eq.
(K.2).
Similarly, in the case with (cid:101) I ≥ 1 , we have that for any i > (cid:101)
I , we must have lim n t
ni
= t ∗ i < ∞ .
From the boundedness of t ∗ i and the continuity of ODE ﬂow, for
[Equation]
5) we have
[Equation]
In particular, if (cid:101) I = 1 , then for the index I with w ∗
I = 0 , we must have that I ≥ (cid:101) I , meaning that there is at least one vacuous jump in w ∗ .
The same argument for the case (cid:101) I = 0 above can lead to the same contradiction with eq.
(K.2).
Otherwise, with (cid:101) I ≥ 2 , we already know that the accumulated cost of all jumps in w ∗ is strictly less than J G , yet we still have d (cid:0)(cid:101) x ( (cid:80) k [Equation] i =
[Equation]
+1 t ∗ i , 0 ; t ∗ , w ∗ ) , G c (cid:1) ≤ (cid:15) 0 .
This contradicts eq.
(K.2) again.
In summary, we have established the existence of the lower bound ¯ δ > 0 on jump sizes.
We ﬁx such ¯ δ > 0 .
The existence of ¯ t < ∞ can be shown by an almost identical argument.
In particular, if such ¯ t < ∞ does not exist, then by picking a subsequence if needed we are able to ﬁnd a converging sequence (cid:0) t n , w n (cid:1) n ≥ 1 such that eq.
(K.6) holds and (cid:101) I ≥ 1 due to inter-arrival time blowing up to inﬁnity.
In particular, since y ∗ 1 = w ∗ 1 and (cid:107) w ∗ 1 (cid:107) ≥ ¯ δ > 0 , we must have (cid:101)
I ≥ 2 .
By considering the same t ∗ , w ∗ pair in equation K.4 and equation K.5, one can see that the accumulated cost of jumps in w is strictly less than J G and yield a contradiction with eq.
(K.2).
Proof of Lemma J.2.
Due to the ﬁniteness of all types j (cid:48) with J type ( j (cid:48) ) <
J G , it sufﬁces to ﬁx one of such j (cid:48) and prove the existence of the positive constant (cid:15) 1 .
To proceed with a proof by contradiction, the assume that such (cid:15) 1 > 0 does not exists.
(For a clean presentation, in t and w we omit the ( k (cid:48) ) term in the superscript since the cardinality is ﬁxed.)
As a result, we are able to pick a sequence ( x n , t n , w n ) n ≥ 1 such that one of the following two cases must occur:
• lim n (cid:107) x n (cid:107) =
For any n ≥ 1 , inf s ≥ 0 d (cid:0)(cid:101) x
( s, x n ; t n , w n ) , G c (cid:1) ≤ (cid:15) 0 /
and there is some j n ∈ [ k (cid:48) ]
such that t nj n ≥ 2¯ t ; • lim n (cid:107) x n (cid:107) = 0 ; For any n ≥ 1 , inf s ≥
d (cid:0)(cid:101) x ( s, x n ; t n , w n ) , G c (cid:1) ≤ (cid:15) 0 / [-
-]2 and there is some j n ∈ [ k (cid:48) ] such that (cid:13)(cid:13) w nj n (cid:13)(cid:13) ≤ ¯ δ/ 2 .
We detail the analysis for the ﬁrst case, as the second case can be addressed by an almost identical argument.
First of all, due to the truncation operator ϕ b ( · ) in the deﬁnition of (cid:101) x t , without loss of generality we can replace all jumps w nj by ϕ b (cid:0) w nj (cid:1) to ensure that w nj is always in a compact set.
Deﬁne
[Equation]
By picking a subsequence if necessary, we can further assume that
• For any i ≥ 1 , w ( n ) j converges to some w ∗ i and y ni converges to some y ∗ i .
• Also, since G c is a closed set, we have d ( y ∗ k (cid:48) , G c ) ≤ (cid:15) 0 / 2 .
• For any i
= 2 , 3 , · · · , k (cid:48) , t ni either converges to some ﬁnite t ∗ i , or lim n t ni = ∞ .
In particular, there is some I ∈
[ k (cid:48) ] such that lim n t nI ≥ 2¯ t .
• Due to eq.
(K.3), lim n t ni = ∞ would imply y ∗ i = 0 .
Let I 1 = max { i ∈
[ k (cid:48) ] : lim n t ni ∈ [2¯ t, ∞ ) } and I
= max { i ∈
[ k (cid:48) ] : lim n t ni = ∞} .
If either of the two sets above is empty, let
the corresponding I
or I 2 be 0.
The discussion above implies that at least one of them must be non-zero, so there are only two possibilities: (i) 0 ≤ I 1 < I 2 ; (ii)
≤
I 2 < I 1 .
We consider each scenario respectively.
First, if 0 ≤ I 1 < I 2 , then lim n y nI
=
y ∗ I 2 = 0 , implying that I 2 < k (cid:48) .
Moreover, for any i = I 2 + 1 , I 2 + 2 , · · · , k (cid:48) , lim n t ni = t ∗ i < ∞ .
Using the boundedness of t ∗ i and the continuity of
ODE ﬂow, for
[Equation]
we have (cid:101)
[Equation]
=
[Equation] [Equation] ∗ , w ∗ ) =
y ∗ k (cid:48) with d ( y ∗ k (cid:48) , G c ) ≤ (cid:15) 0 / 2 .
However, this contradicts equation J.2.
Next, in scenario (ii) with 0 ≤ I 2 <
I 1 , we have lim n t nI 1 =
t ∗ I 1 ≥ 2¯ t .
Moreover, for any i = I 2 + 1 , I 2 + 2 , · · · , k (cid:48) , lim n t ni = t ∗ i < ∞ .
Using the boundedness of t ∗ i and the continuity of
ODE ﬂow, for
[Equation]
we have (cid:101) x
( (cid:80) k (cid:48)
i =
1+ I 2 t ∗ i , 0 ; t ∗ , w ∗ ) =
y ∗ k (cid:48) with d ( y ∗ k (cid:48) , G c ) ≤ (cid:15) 0 / 2 .
However, bmw ∗ is still of type j with J ↓ type ( j ) < J G , and there is some i such that t ∗ i ≥ 2¯ t .
This would contradict Lemma J.1.
In summary, we have established the existence of (cid:15) 1 > 0 such that t j < 2¯ t is a necessary condition for any x ∈ B ( 0 , (cid:15) 1 ) and any t =
( t 1 , · · · , t k (cid:48) ) ∈ { 0 } × R k (cid:48) − 1 + , w [Equation] [Equation] w k
[Equation]
) ∈
A type j with inf s ≥ 0 d (cid:0)(cid:101) x ( s, x ; t ( k ) , w ( k ) ) , G c (cid:1) ≤ (cid:15) 0 / 2 .
As mentioned above, the necessity of (cid:107) w j (cid:107) >
¯ δ/ 2 can be shown in an almost identical way.
We omit the details here and conclude the proof.
Proof of Lemma J.3.
Since there are only ﬁnitely many ( i 1 , · · · , i m ) with J ( i 1 , · · · , i m ) <
J G , it sufﬁces to ﬁx one of such ( i 1 , · · · , i m ) and establish the existence of the required (cid:15) 0 , δ 0 .
(Henceforth, let k = (cid:80) j =
i m .)
Assumption I.8, together with the bound in eq.
(J.6), implies the existence of some (cid:15) 1 ∈ (0 , ¯ (cid:15) )
such that
[Equation]
for any w ( k ) ∈
A ( i 1 , · · · , i m ) , t ( k ) ∈ { 0 } × R k − 1 + .
For (cid:15)
= (cid:15) 1 / 2 , we establish the existence of the prescribed δ 0 via a proof by contradiction.
(Henceforth we drop the notational dependence on ( k ) when referencing sequences t ( k ) , w ( k ) since the cardinality k is ﬁxed.)
Assume the existence of a sequence ( x n ) n ≥ 1 in R d , a sequence of real positives ( s n ) n ≥ 1 , a sequence ( t n ) n ≥
=
( t 1 ,n , · · · , t k,n ) n ≥ 1 in { 0 } × R k − 1 + , and a sequence ( w n ) n ≥
= ( w 1 ,n , · · · , w k,n ) n ≥ 1 in
A ( i 1 , · · · , i m ) such that lim n (cid:107) x n (cid:107) =
and
[Equation]
s
[Equation]
w
[Equation]
[Equation] (cid:1) [Equation] Due to existence of the clipping operator, all jumps w j,n can be replaced by ϕ b ( w j,n ) without loss of generality to ensure that all w j,n are in a compact set.
Also, without loss of generality, all s n can be chosen as
[Equation]
From Assumption eq.
(J.6), one can see that s n must be equal to (cid:80)
j n i =1 t i,n for some j n ∈ [ k ] , i.e., it must be the arrival time of some jump.
Moreover, one can easily see that for
[Equation]
we always have d ( y n , G ) ≤ b so all y n are in a compact set as well due to the boundedness of G .
Therefore, by picking a subsequence when necessary, we can further assume that
• There exists some j ∗ ∈
[ k ] such that s n = (cid:80) j ∗ i =
1 t i,n for all n ≥ 1 , i.e. s n is always the arrival time of the j ∗ − th jump; • For any j ∈ [ j ∗ ] , there exists some y ∗ j such that for y n,j = (cid:101) x
( (cid:80) ji =1 t i,n , x n ; t n , w n ) we have lim n y n,j = y ∗ j ; In particular, for any j < j ∗ we have d ( y ∗ j , G c ) >
(cid:15) 1 and d ( y ∗ j ∗ , G c ) ≤ (cid:15) 0 ; • For any j ∈ [ k ] , there exists some w ∗ j such that lim n w j,n = w ∗ j ; Moreover, note that y ∗ 1 = w ∗ 1 .
• In particular, ( w ∗ 1 , · · · , w ∗ k
) ∈
A ( i 1 , · · · , i m ) ; • For any j ∈ [ k ] , either there exists some t ∗ j < ∞ such that lim n t j,n = t ∗ j , or lim n t j,n = ∞ (in this case we let t ∗ j = ∞ ); in the latter case, due to the same argument in eq.
(K.3), we must have lim n (cid:101) x
( (cid:80) ji =1 t i,n , x i ; t n , w n ) =
Obviously, y ∗ j ∗ (cid:54) = 0 .
Let j ↓ = ∆ max { j = 0 , 1 , · · · , j ∗ : y ∗ j = 0 } with the convention that y ∗ 0 = 0 , y ∗ 0 ,n = x n .
We must have j ↓ <
j ∗ and
• lim n y ∗ j ↓ ,n = y ∗ j ↓ =
• For any j = j ↓ + 1 , j ↓ + 2 , · · · , j ∗ , lim n t j,n = t ∗ j < ∞ .
Now using the continuity of the ODE ﬂow [Equation] we must have that
[Equation]
However, due to d ( y ∗ j ∗ , G c ) ≤ (cid:15) 0 and recall our choice of (cid:15)
= (cid:15) 1 / 2 , for all n sufﬁciently large, we must have
[Equation]
Meanwhile, using Gronwall’s inequality repeatedly and sup j = j ↓ +1 , ··· ,j ∗ ,n ≥ 1 t j,n < ∞ , by substituting the initial condition y ∗ j ↓ ,n with 0 , we have
[Equation]
This implies that, for all n sufﬁciently large,
[Equation]
However, this contradicts eq.
(K.7).
This implies the existence of the required δ 0 and concludes the proof.
K. S UFFICIENT CONDITIONS FOR A SSUMPTION I.
In this section, we show that under a proper set of regularity conditions on the boundary set ∂ G and the distribution of noises Z n , Assumption I.9 will hold for (Lebesgue) almost every b > 0 .
In particular, we stress that the C 2 condition about manifold ∂ G in Assumption I.3, as well as the condition that measures S j are absolutely continuous w.r.t.
the spherical measure σ on S d − 1 , are only used to prove
that µ (cid:0) h − 1 ( ∂ G ) (cid:1) =
and will only be applied in this section.
The key of our argument is the following geometric observation regarding the intersection of
C 2 − manifold ∂ G and ∂B of some ball B .
We stress that, as made evident by the proof, this lemma is essentially based on two assumptions: (I)
As the boundary set of the connected bounded region G , ∂ G is a closed set in R d ; (II) As a ( d − 1) -dimensional manifold, ∂ G is of class C 2 .
Lemma K.1.
Let σ x,b be the spherical measure on the sphere of the open ball B ( x, b ) for any x ∈ R d , b > 0 .
Under Assumptions I.1 and I.3, it holds for (Lebesgue) almost every b > 0 that
[Equation]
Proof.
The fact that ∂ G is a subset of a separable metric space implies the existence of a countable atlas for this manifold.
Therefore, we can ﬁnd a sequence ( U i )
i ≥ 1 that are bounded open subsets of R d − 1 containing 0 , a sequence ( V i )
i ≥ 1 that are open sets in ∂ G (in the metric space induced by Euclidean distance), and a sequence of injective C 2 mapping f
i with f i : R d − 1 (cid:55)→ R d and f i ( U i ) =
V i such
that ∂ G
=
∪
i
V i .
Next, we zoom in on a speciﬁc chart ( U i , V i , f i ) and observe the following facts.
For any x ∈
U i ,
y ∈ V i with f i ( x ) =
y , there exist some orthogonal matrix
Q y ∈ R
d × d such that Q y n ( y ) =
(0 , 0 , · · · , 0 , 1) T where the vector ﬁeld n ( · ) is the outer normal on ∂ G .
Besides, there is some vector a y ∈ R d such that Q y y
+ a y = 0 .
Moreover, there exist an open set on (cid:101)
U x in R d − 1 containing 0 , an open set y ∈
V
y
⊂ V i , an open set (cid:101) V y =
∆ { Q y w + a y : w ∈ V y } , a C 2 function (cid:101) g
y : (cid:101) U x
(cid:55)→ R satisfying (cid:101) g y ( 0 ) = 0 and
[Equation]
In other words, for any given y on this chart we simply rotate the chart to ensure that the tangent space at y after rotation is { ( x 1 , · · · , x d − 1 , 0) :
x i ∈ R ∀ i ∈ [ d − 1] } , and reparametrize the Cdiffeomorphism associated to this chart around y so that the coordinates are simply the projection onto the said tangent space.
In this sense, the (rotated) manifold is also the graph of the C 2 mapping g y .
This allows us to deﬁne (for any y ∈ ∂ G )
[Equation]
where, for any real symmetric ( d − 1) × ( d − 1) matrix [Equation] , λ 1 ( A ) ≥ λ 2 ( A ) ≥ · · · ≥ λ d − 1 ( A ) are the ordered eigenvalues of the matrix, and ∇ 2 g y ( · ) is the Hessian of g y .
It is worth noticing that A ( · ) is a continuous function (on ∂ G ) due to the manifold being of class C 2 .
Restricting our discussion on some ﬁxed chart ( U i , V i , f i ) for now, for any b > 0 , let
A i ( b ) = ∆ { x ∈ U i : for y = f i ( x ) , ∃ j ∈ [ d − 1] such that | λ j (cid:0) ∇ 2 g y ( 0 ) (cid:1) | = 1 /b } .
Note that for any b > 0 , the set
A i ( b ) is a closed set (hence Borel measurable) since the continuity of
A ( · ) implies that ( A i ( b ))
c is an open set on U i .
Furthermore, since m Leb ( U i ) < ∞ , there are at most countably many b > 0 such that m Leb
( A i ( b )) > 0 .
Given the countability of the atlas, we know that
[Equation]
} contains at most countably many elements.
In the rest of this proof, we show that eq.
(K.8) holds for any b > 0 such that b / ∈ B ∗ .
Henceforth, we arbitrarily choose some b > 0 such that b / ∈ B ∗ .
We also arbitrarily choose some x ∈ R d and let B = B ( x, b ) .
To facilitate the discussion, we introduce a concept that is closely related to the set
A i ( b ) : let
[Equation]
and let A G ( b )
= ∆ ∪ i ≥
A G i ( b ) .
Now consider the following decomposition of the sphere ∂ B :
[Equation]
First, note that for any y ∈ B 1 , one of the following four cases has to occur:
• y / ∈
∂
G ; • y ∈
∂ G and the vector y − x lies in T y ∂
G , the tangent space of the manifold ∂ G at y ; • y ∈
∂ G ; the vector y − x is not in T y ∂
G
yet it is not orthogonal to T y ∂ G either, i.e. y − x is not equal to c · n ( y ) for any c ∈ R where n ( y ) is the outer normal at y ; • y ∈
∂ G ; the vectors y − x and n ( y ) are linearly dependent.
Our next goal is to show that, in any of these four cases, we can always ﬁnd a set y ∈
O y that is open on the sphere ∂ B such that σ x,b ( O y ∩ ∂ G )
= 0 .
Note that this is obviously true when y / ∈ ∂ G , since both ∂ G and ∂ B are closed sets in R d .
Now we consider the second case.
If the vector y − x lies in T y ∂
G , then after applying the afﬁne transformation with orthogonal matrix Q y (recall that
Q y n ( y ) =
(0 , · · · , 0 , 1) ), we have
[Equation]
Since Q y y + a y = 0 , we now know that after the afﬁne transformation, the center of the ball B moves to
[Equation]
Without loss of generality, we can assume that Q y x + a y = ( b, [Equation]
In other words, after the afﬁne transformation, the ball becomes (cid:101) B y =
∆ Q y B + a y = B (cid:0) ( b, 0 , 0 , · · · , 0) , b (cid:1) .
Moreover, for any w = ( w 1 , · · · , w d ) ∈ ∂ (cid:101) B y , we must have w 1 ≥ 0 and ( w 1 − b ) 2 + ( w 2 ) 2 + · · · + ( w d ) 2 = b 2 .
Meanwhile, from the deﬁnition of the mapping (cid:101) g y , one can see that there is an open set U y around y such that for any w ∈ U y with w ∈ ∂
G ∩ ∂ B (hence (cid:101)
w = Q y w + a y ∈ (cid:101)
V y ∈
∂ (cid:101) B y ) such that
[Equation]
Let (cid:101) f
( w 1 , · · · , w d − 1 ) =
∆ (cid:101) g 2 y ( w 1 , · · · , w d − 1 ) + w 21 − 2 bw 1 + w 22 + · · · + w 2 d − 1 .
Now we now that (cid:101) f ( 0 ) =
and (cid:101) f
( (cid:101) w ) = 0 for any (cid:101) w ∈ (cid:101)
V y ∈
∂ (cid:101) B y .
Moreover, by deﬁnition we have ∇ (cid:101) g y ( 0 ) =
so ∂∂w 1 (cid:101) f
= − 2 b (cid:54) =
Due to implicit function theorem, we now know the existence of some open set U ∗ in R d containing 0 , some C 2 function g ∗ : R d − 2 (cid:55)→ R such that for any w ∈ (cid:101) V y ∩ ∂ (cid:101) B
[Equation]
Therefore, within some open neighborhood V ∗ y of such y , the set V ∗ y ∩ ∂ B ∩ ∂ G is a submanifold with dimension d − 2 , so we must have
[Equation]
Next, we consider the case where y ∈ ∂ G ∩ B 1 but y − x is neither in T y ∂
G nor orthogonal to
T y ∂ G .
Similar to the construction of the afﬁne transformation with Q y , a y above, one can ﬁnd an orthogonal matrix (cid:101) Q y and a vector (cid:101) a y such that (cid:101)
Q y y + a y = 0 and (cid:101) Q y ( y − x ) =
(0 , 0 , · · · , 0 , b ) .
Let (cid:101) B
= B ((0 , 0 , · · · , − b ) , b ) .
In other words, this time the rotation we constructed ensures that, after rotation, the vector between (cid:101) Q y y + a y =
(on the sphere ∂ (cid:101) B ) and the center (cid:101) Q
y x + (cid:101) a y of the ball (cid:101) B is aligned with the d − th axis.
Moreover, there is an open set y ∈ V alt y ⊂ R d and a C 2 function (cid:101)
g alt y : R d − 1 (cid:55)→ R with ∇ (cid:101)
g alt y ( 0 ) (cid:54) =
such that for any (cid:101) y ∈ (cid:101) V alt y = ∆ (cid:101) Q y V alt y + (cid:101) a y , we have
[Equation]
By applying a change of coordinates if necessary (which can be achieved by multiplying another orthogonal matrix), we can assume without loss of generality that ∇ (cid:101) g alt y ( 0 ) =
(0 , 0 , · · · , c ) for some c (cid:54) = 0 .
Then for any (cid:101) w ∈ V alt y such that w ∈ ∂ B ∩ ∂ G , let w = (cid:101) Q y (cid:101) w + (cid:101) a y and note that we must have
[Equation]
In particular, (cid:101) g alt y ( w 1 , · · · , w d − 1 ) =
cw d − 1 + r ( w ) for some C 2 function r with r ( 0 ) = 0 , ∇ r ( 0 ) = 0 .
As a result, for function (cid:101) f ( w ) =
∆ w 21 + w 22 + · · · + w 2 d − 1 + (cid:0) b + (cid:101)
g alt y ( w 1 , · · · , w d − 1 ) (cid:1) 2 .
we have ∂ ∂w d − 1 (cid:101) f ( 0 )
= 2 bc (cid:54) = 0 .
Using implicit function theorem again, one can see the existence of some open set 0 ∈ V ∗ y ∈ R d and some C 2 function g ∗ : R d − 2 (cid:55)→ R such that for any (cid:101) w ∈ V ∗ y ∩ ∂ G ∩ ∂
B , we have that (for w = ∆ (cid:101) Q y (cid:101)
w
+ (cid:101) a y )
[Equation]
[Equation] [Equation] [Equation]
) , w d = (cid:101) g alt y (
[Equation] [Equation]
· , w d − 1 ) . Again, we have established that within some open neighborhood V ∗ y of such y , the set V ∗ y ∩ ∂ B ∩ ∂ G is a submanifold with dimension d − 2 , so we must have
[Equation]
Lastly, consider the case where y ∈
∂ G ∩ B 2 and the vectors y − x and n ( y ) are linearly dependent.
In other words, the tangent space T y ∂ G is also the tangent space T y ∂ B .
Since y / ∈ B 1 , we know that (cid:101) g y ( w ) =
12 w T A y w + r 1 ( w ) where A y is a real symmetric matrix with no eigenvalue equal to ± 1 /b and r 1 is a C 2 function with | r 1 ( w ) |
= o ( (cid:107) w (cid:107) 2 ) .
On the other hand, for any (cid:101) w in the open set
V y ⊂ ∂ G , if we also have (cid:101) w ∈ ∂ B ∩ ∂ G , then for w = ∆ Q y (cid:101) w + a y we have
[Equation]
Also, note that (cid:113) b [Equation] w
[Equation]
+ r 2 ( w 1 , · · · , w d − 1 ) where r 2 is also a C 2 function with | r 2 ( w ) |
= o ( (cid:107) w (cid:107) 2 ) .
Therefore, for any ( w 1 , · · · , w d − 1 ) satisfying the equation eq.
(K.9), we have
[Equation]
However, for the real symmetric matrix
A y − I d − 1 b , note that none of its eigenvalue is equal to 0, implying the existence of some (cid:15) > 0 such that
[Equation]
For this ﬁxed (cid:15) > 0 , we can also ﬁnd δ > 0 such that
[Equation]
for any w 21 + · · · + w 2 d − 1 < δ.
As a result, the only solution to eq.
(K.9) with w 21 + · · · + w 2 d − 1 < δ is w 1 = w 2 = · · · , w d − 1 = 0 .
In summary, we have shown that there exists some set y ∈ V ∗ y that is open in R d such that V ∗ y ∩ ∂ B ∩ ∂ G = { y } .
Collecting the results we have established so far, we now know that for any y ∈ B 1 (recall that B 1 is an open set on ∂ B ), there is an open set V ∗ y containing y and satisfying σ x,b ( V ∗ y ∩ ∂ B ∩ ∂ G ) = 0 .
In particular, given the open cover ∪ y ∈ B 2 V ∗ y = B 1 , Lindelöf property then allows us to extract a countable open ∪ i ≥ 1 V ∗ y i
= B 1 cover and conclude that
[Equation]
Moving on, we evaluate σ x,b ( B 2 ∩ ∂ G ) .
For any y ∈ ∂ B , one of the four cases has to occur:
• y / ∈
A G ( b ) ; • y ∈
A G ( b ) and the vector y − x lies in T y ∂
G , the tangent space of the manifold ∂ G at y ; • y ∈
A G ( b ) ; the vector y − x is not in T y ∂
G
yet it is not orthogonal to T y ∂ G either, i.e. y − x is not equal to c · n ( y ) for any c ∈ R where n ( y ) is the outer normal at y ;
• y ∈
A G ( b ) ; the vectors y − x and n ( y ) are linearly dependent.
Again, we show that in any of these four cases, there is some set y ∈ V ∗ y open in ∂ B such that σ x,b ( A G ( b ) ∩ V ∗ y )
= 0 .
In the ﬁrst case, the fact that A G ( b ) is closed on ∂ G immediately implies the existence of some V ∗ y such that A G ( b ) ∩ V ∗ y = ∅ .
For the second and third case, this can be shown using exactly the same implicit function argument above.
For the last case where y ∈ A G ( b ) ∩ ∂ B and the vectors y − x and n ( y ) are linearly dependent, from y ∈
A G ( b ) ∩ ∂
B we know that y ∈ V i where ( U i , V i , f i ) is a chart of ∂ G and V
i is open on ∂ G .
Moreover, recall the construction of open set
y ∈
V y ⊂ V i at the beginning of the proof.
It is worth noticing that V
i ∩A G ( b ) =
V i ∩A
G i ( b ) .
Besides, due to the fact that the vectors y − x and n ( y ) are linearly dependent, we know that the tangent space T y ∂ G is also the tangent space T y ∂ B .
Therefore, by deﬁnition of (cid:101) g y and Q y , a y , we know that
Q y y + a y = 0 , and under the afﬁne transformation, the ball becomes B ((0 , 0 , · · · , ± b ) , b ) .
Without loss of generality, we assume it is (cid:101) B = ∆ B ((0 , 0 , · · · , b ) , b ) .
Moreover, since b / ∈ B ∗ , we have that m Leb ( { w ∈ (cid:101) U y : x ∈
A i ( b ) } )
= 0 where, as deﬁned at the beginning of the proof, (cid:101)
U y is the domain of the C 2 mapping (cid:101)
g y , V y is the image of the mapping, and (cid:101) V y =
∆ { Q y w + a y : w ∈ V y } is the image of V y under the afﬁne transformation.
Therefore, for any (cid:101) w ∈
A G i ( b ) ∩ ∂ B , let w = Q y (cid:101)
w + a y
and we must have
[Equation]
Now let
[Equation]
b 2 − w 21 − w 22 − · · · − w 2 d − 1 (cid:1) for some w ∈ (cid:101)
U x (cid:111) and note that y ∈ V ∗ y is an open set on ∂ B .
(Speciﬁcally, note that we simply identify an open set on the transformed sphere ∂ (cid:101) B y , and then perform the inverse transformation to move the set back to the original sphere ∂ B .)
Then it follows immediately from m Leb ( { w ∈ (cid:101)
U y :
x ∈ A i ( b ) } ) =
that σ x,b (cid:0) A G ( b ) ∩ V ∗ y (cid:1)
= 0 .
In summary, for any y ∈ ∂ B , we can ﬁnd a set y ∈ V ∗ y open on ∂ B such that σ x,b (cid:0) A G [Equation] b ) ∩ V ∗ y (cid:1) = 0 .
Lastly, by applying Lindelöf property again, we extract a countable open cover ∪ i ≥ 1 V ∗ y i
= B 2 cover and conclude that
[Equation]
and this concludes the proof.
As a result of Lemma K.1, the following Lemma is essentially built upon three assumptions/facts: (I)
As a boundary set, ∂ G is a closed set in R d ; (II) As a ( d − 1) -dimensional manifold, ∂ G is of class
C 2 ; (III)
The measures S j in Assumption I.6 are absolutely continuous w.r.t.
the spherical measure σ .
Lemma K.2.
Under Assumptions I.1, I.3 and I.6, it holds for (Lebesgue) almost every b > 0 that
[Equation]
Proof.
Fix some b > 0 satisfying the conditions in Lemma K.1.
Recall that µ = (cid:80) j ∈ j
( i ∗ ) µ j (see eq.
(J.40)).
It sufﬁces to show that µ j (cid:16) h − 1 (cid:0) ∂ G (cid:1)(cid:17) =
for some ﬁxed j ∈ j ( i ∗ ) .
In particular, observe that
[Equation]
where the function h ∗ is deﬁned as
[Equation]
Let z = ∆ h ∗ ( r 1 , · · · , r k ∗ − 1 , θ 1 , · · · , θ k ∗ − 1 , t 2 , · · · , t k ∗ ) .
Now by separating the two cases based on whether the truncation operator takes effect or not
[Equation]
we have
[Equation]
For term eq.
(K.11), note that it is equal to (cid:82) 1 (cid:8) T − 1 ( r k ∗ , θ k ∗ ) ∈ ( − z + ∂ G ) ∩ ∂B (0 , b ) (cid:9)
S j k ∗ ( dθ k ∗ ) ×
ν α j k ∗ ( dr k ∗ ) where T − 1 ( r, θ ) =
rθ is the inverse of the polar coordinate transform.
Furthermore, since the set ( − z + ∂ G ) ∩ ∂B (0 , b ) is either empty or is a ( d − 1)
-dimensional C 2 submanifold (w.r.t.
B (0 , b ) when viewed as a d − dimensional manifold).
In other words, it has zero mass under m d Leb .
For the measure ν ∗ =
∆ T − 1 ◦ ( S j k ∗ ×
ν α j k ∗ ) , due to S j being absolutely continuous w.r.t.
σ , it is easy to see that ν ∗ is absolutely continuous w.r.t.
m d Leb .
Therefore, we must have
[Equation]
[Equation]
implying that the integral in eq.
(K.11)
= 0 .
On the other hand, for term eq.
(K.12), we know that (cid:82)
[Equation]
< ∞ .
Besides,
[Equation]
Then it follows immediately from Lemma K.1 and S j being absolutely continuous w.r.t.
the spherical measure σ that the integral in term eq.
(K.12) is equal to 0 .
In summary, we have shown that
[Equation]
for any ( r 1 , · · · , r k ∗ − 1 , θ 1 , · · · , θ k ∗ − 1 , t 2 , · · · , t k ∗ ) .
Plug this result back into eq.
(K.10) and we conclude the proof.
L N OTATIONS
Table L.1 lists the notations used in Section G.
[Table]
[Equation]
[Table]
Truncation threshold of stochastic gradient
[Table]
An accuracy parameter; typically used to denote an (cid:15) − neighborhood of s i , m i
[Table]
A threshold parameter used to deﬁne large noises
[Table]
A constant deﬁned for eq.
(G.28)-eq.
(G.29).
Since ¯ (cid:15) < (cid:15) 0 , in eq.
(G.8) the claim holds for | x − y | < ¯ (cid:15) .
Note that the value of the constant ¯ (cid:15) does not vary with our choice of η, (cid:15), δ .
[Table]
Upper bound of | f (cid:48) | and | f (cid:48)(cid:48) | eq.
(G.10)
[Table]
eq.
(G.10)
[Table]
The open interval ( s − , s + ) ; a simpliﬁed notation for Ω
i
[Table]
[Equation]
[Table]
[Equation]
[Table]
truncation
[Table] [Table]
[Equation]
“small”
[Table]
eq.
(G.20)
[Equation]
[Table]
[Equation]
[Table]
[Equation]
“large” noise eq.
(G.21)
[Equation]
arrival time of j -th large noise eq.
(G.22)
[Equation]
size of j -th large noise eq.
(G.23)
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
ODE
[Table]
perturbed
[Table] [Equation] [Table] {−
[Table]
Effective radius of the attraction ﬁeld Ω .
[Table]
The minimum number of jumps required to escape Ω when starting from its local minimum m = 0 .
[Table]
A mapping deﬁned as h ( w , t ) = (cid:101) x ( t l ∗ , 0; t , w ) .
[Table]
Necessary conditions for h ( w , t ) to be outside of Ω eq.
(G.35)-eq.
(G.36) ˆ t ( (cid:15) )
ˆ t ( (cid:15) )
=
∆ c 1 log(1 /(cid:15) ) .
The quantity ˆ t ( (cid:15) ) /η provides an upper bound for the time it takes x η to return to 2 (cid:15) − neighborhood of local minimum m = 0 when starting from somewhere (cid:15) − away from s − , s + .
See eq.
(G.37).
[Table]
[Equation]
[Table]
[Equation]
[Table]
The probability that, for t = (cid:0) T ηj ( δ ) − 1 (cid:1) l ∗ j =1 and w = (cid:0) ηW ηj ( δ ) (cid:1) l ∗ j =1 , we have ( w , t )
∈ E ( (cid:15) ) conditioning on { T η 1 ( δ ) = 1 } .
Intuitively speaking, it characterizes the probability that the ﬁrst l ∗ large noises alone can drive the ODE out of the attraction ﬁeld.
Deﬁned in eq.
(G.66).
[Table]
The Borel measure on R with density
[Equation]
where p − , p + are constants in Assumption 2 in the main paper.
[Table]
The product measure µ = ( ν α ) l ∗ × ( Leb + ) l ∗ − 1 .
[Table] [Table] min { n ≥ 0 : X η n / ∈ Ω } .
ﬁrst exit time
[Table]
M R ESULTS ABOUT TAIL DISTRIBUTIONS OF NOISES IN OUR NUMERICAL EXPERIMENTS M. QQ PLOTS
QQ plots below clearly show that the tails in noise distribution are always much lighter than the
Pareto distributions with alpha
= 2 or even 10.
In fact, the tail of noise distributions seem to be between that of lognormal and normal distributions, implying that it is lighter than any power-law distribution. [Figure] Normal [Figure]
Pareto
[Figure]
Pareto
[Figure]
Lognormal
Normal
Pareto α
Lognormal Normal Pareto, α [Figure]
Pareto [Figure] α [Figure]
Lognormal
Normal
Pareto
[Figure]
Pareto
[Figure]
Lognormal
Normal Pareto, α = Pareto, α = Pareto, α =
Lognormal Normal , α =
Pareto, α =
Pareto, α =
Lognormal
Normal
Pareto, α = Pareto, α =
Lognormal
Normal
Lognormal
Normal
Pareto
Lognormal
Normal
Pareto, α
Pareto, α =
Lognormal
Normal
[Figure]
Lognormal
Normal
[Figure]
Lognormal
Normal
[Figure]
Lognormal
Normal
Lognormal
M. E MPIRICAL MEAN RESIDUAL LIFE (EMRL) PLOTS
It is well known that the mean residual life blows up to inﬁnity if and only if the distribution is heavy-tailed (more precisely, long-tailed).
However, from the ﬁgures below, one can see that none of the EMRL exhibits such a pattern in any case tested in our experiments.
Instead, we see clear downward trends, which strongly suggests light tails in all cases tested.
[Figure]
[Figure] [Figure]
[Figure]
M. H ILL PLOTS
In the hill plots below, the estimated power-law indices using only the top 1% of samples (on the left hand side of the dashed lines) stay well above 10 for the most part and almost never drop below 2.
This strongly suggests that even if the gradient noises are from a heavy-tailed distribution, it is likely to have a very high power law index (implying relatively lighter tails), and hence, we cannot expect to observe a prominent heavy-tailed behavior from them.
[Figure]
[Figure] [Figure]
[Table]
[Figure]
[Table] [Table]