For this we start from the estimate (E.59) and apply the argument following that, but with more care in tracking the constants: if ν is within π/ 4 of either 0 or π , we can assert
[Equation]
whenever g 1 i (cid:54) = 0 for every i (a set of full measure); and when ν is within π/ 4 of π/ 2 , we can assert
[Equation]
where C i , C (cid:48) i > 0 are absolute constants.
By the triangle inequality, independence, and Lemma G.9, when we consider | E g 2 [ q (cid:48) ( ν, g 1 , g 2 )] | , the term E [ (cid:107) g
i 2 (cid:107) 2 ] is bounded by an absolute constant.
Additionally, by Gauss-Lipschitz concentration and Lemma G.9, we have that simultaneously for all i (cid:107) g i 1 (cid:107) 2 ≤ (cid:107) g 1 (cid:107) 2 ≤ 2 with probability at least 1 − 2 e − cn .
Moreover, since (cid:107) g 1 (cid:107) ∞ ≤ (cid:107) g 1 (cid:107) 2 we also have control of the magnitude of each | g 1 i | on this event, so with probability at least 1 − 2 e − cn
we have for every ν
[Equation]
for absolute constants C, C i > 0 .
If X ∼ N (0 , 1) , we have for any t ≥ 0 that P [ | X | ≥ t ] ≥ 1 − Ct , where C > 0 is an absolute constant; so if X i ∼ i .
i .
d . N (0 , 1) , we have by independence and if t is less than an absolute constant P [ ∀ i, | X i | ≥ t ] ≥ (1 − Ct ) n ≥ 1 − C (cid:48)
nt , where the last inequality uses the numerical inequality e − 2 t ≤ 1 − t ≤ e − t , valid for 0 ≤ t ≤ 12 .
From this expression, we conclude that when 0 ≤ t ≤ cn − 1 / 2 for an absolute constant c > 0 , we have
[Equation]
so choosing in particular t = cn − ( δ +32) for any δ > 0 , we conclude that P [ ∀ i ∈ [ n ] , | g 1 i | ≥ cn − 3 / 2 − δ ] ≥ 1 − C (cid:48)
n − δ .
Consequently, for any δ > 0 we have with probability at least 1 −
C (cid:48) n − δ − 2 e − cn
[Equation]
and since k = (cid:112) 2 /n , this yields | E g 2 [ q (cid:48)
( ν, g 1 , g 2 )]
| ≤ C 1 n 1+ δ + C 2 + C 3 n 5 / 2 + C 4 n 3 / 2 with the same probability.
Consequently we can conclude that for any δ ≥ 3 / 2 , we have
[Equation]
Lemma E.41.
In the notation of Lemma E.13, if d ≥ 1 , there are absolute constants c, c (cid:48) C, C (cid:48) >and an absolute constant K > 0 such that if n ≥ K , there is an event with probability at least 1 − Ce − cn on which
[Equation]
Proof.
Let E denote the event E 0 .
5 , 0 in Lemma E.16; then by that lemma, E has probability at least 1 − Ce − cn as long as n ≥ C (cid:48) , where c, C, C (cid:48) > 0 are absolute constants, and for ( g 1 , g 2 ) ∈ E , one has for all ν ∈ [0 , π ] [Equation] =
[Equation]
This allows us to calculate, for each ν ,
[Equation]
after applying Lemma E.37 and Lyapunov’s inequality and worst-casing constants.
We conclude that with probability at least 1 − Ce − cn
[Equation]
Lemma E.42.
In the notation of Lemma E.13, if d ≥ 1 , there are absolute constants c, c (cid:48) C, C (cid:48) >and an absolute constant K > 0 such that if n ≥ K , there is an event with probability at least 1 − Ce − cn on which
[Equation]
Proof.
The argument is identical to Lemma E.41.
Let E denote the event E 0 .
5 , 0 in Lemma E.16; then by that lemma, E has probability at least 1 − Ce − cn as long as n ≥ C (cid:48) , where c, C, C (cid:48) > 0 are absolute constants, and for ( g 1 , g 2 ) ∈ E , one has for all ν ∈ [0 , π ]
[Equation]
This allows us to calculate, for each ν ,
[Equation]
after applying Lemma E.37 and Lyapunov’s inequality and worst-casing constants.
We conclude that with probability at least 1 − Ce − cn
[Equation]
Lemma E.43.
In the notation of Lemma E.13, if d ≥ 1 , there are absolute constants c, C, C (cid:48) , C (cid:48)(cid:48) > 0 and absolute constants K, K (cid:48) > 0 such that if n ≥ Kd log n and d ≥ K (cid:48) , there is an event with probability at least 1 − Ce − cn − C (cid:48)
n − d on which one has
[Equation]
Proof.
We are going to control the expectation ﬁrst, showing that it is small; then prove that | Ξ 4 | is small uniformly in ν .
Let E denote the event E 0 .
5 , 0 in Lemma E.16; then by that lemma, E has probability at least 1 − Ce − cn as long as n ≥ C (cid:48) , where c, C, C (cid:48) > 0 are absolute constants, and for ( g 1 , g 2 ) ∈ E , one has for all ν ∈ [0 , π ]
[Equation]
Thus, if we write
[Equation]
we have (cid:101) Ξ 4 =
Ξ 4 for all ν whenever ( g 1 , g 2 ) ∈ E , so that for any ν
[Equation]
where the second line uses the triangle inequality and the Schwarz inequality and Lemma E. together with the Lyapunov inequality.
We proceed with analyzing the expectation of (cid:101)
Ξ 4 .
Using the
Schwarz inequality gives
[Equation]
and the checks at and around (E.33) and (E.34) in the proof of Lemma E.15 show that we can apply
Lemma E.30 to obtain
[Equation]
But we have using rotational invariance that E [ σ ( g 11 cos ν + g 21 sin ν )( g 21 cos ν − g 11 sin ν )] = 0 , which implies
[Equation]
from which we conclude
[Equation]
Next, we control the deviations of Ξ 4 with high probability.
By Lemma E.17, there is an event E a with probability at least 1 − e − cn on which (cid:107) ˙ v ν (cid:107) 2 ≤ 4 for every ν ∈ [0 , π ] .
Therefore on the event E b = E ∩ E a , which has probability at least 1 − Ce − cn by a union bound, we have using
Cauchy-Schwarz that for every ν
[Equation]
The coordinates of the random vector v ν (cid:12) ˙ v ν are σ ( g 1 i cos ν
+ g 2 i sin ν )( g 2 i cos ν − g 1 i sin ν ) , and we note
[Equation]
g 1 i ) g 2 i ] = 0 , by rotational invariance.
Moreover, the calculation (E.34) together with Lemmas G.11 and E. demonstrates subexponential moment growth with rate C/n , so Lemma G.2 implies for t ≥
[Equation]
For large enough n , this gives
[Equation]
We turn to the uniformization of this pointwise bound.
The map ν
(cid:55)→ (cid:80) i σ ( g 1 i cos ν
+ g 2 i sin ν )( g 2 i cos ν − g 1 i sin ν ) is continuous, and differentiable at all but ﬁnitely many points of [0 , π ] (following the zero crossings argument in the proof of Lemma E.22) with derivative ν (cid:55)→ (cid:88)
[Equation]
which is evidently integrable using the triangle inequality and Lemma G.11.
In particular, we can write the derivative as (cid:107) ˙ v ν (cid:107) 22 − (cid:107) v 0 (cid:107) 22 .
Thus, by (Cohn, 2013, Theorem 6.3.11), to get a Lipschitz estimate on ν (cid:55)→ (cid:104) v ν , ˙ v ν (cid:105) it sufﬁces to bound the magnitude of the derivative ν
(cid:55)→ (cid:107) ˙ v ν (cid:107) 22 − (cid:107) v 0 (cid:107) 22 .
But this is immediate, since on the event
E b we have |(cid:107) ˙
v ν (cid:107) 22 − (cid:107) v 0 (cid:107) 22 | ≤ 20 .
It thus follows from
Lemma E.48 that with probability at least 1 − Ce − cn − C (cid:48) n − 2 d +1 / 2 we have
[Equation]
E.63)
As long as d ≥ 12 , we have that this probability is at least 1 − Ce − cn − C (cid:48) n − d , and so the triangle inequality and a union bound yield ﬁnally that with probability at least 1 − Ce − cn − C
[Equation]
n − d
[Equation]
Lemma E.44.
In the notation of Lemma E.13, if d ≥ 1 , there are absolute constants c, c (cid:48) , c (cid:48)(cid:48) , C, C (cid:48) , C (cid:48)(cid:48) , C (cid:48)(cid:48)(cid:48) , C (cid:48)(cid:48)(cid:48)(cid:48) > 0 and an absolute constant K > 0 such that if n ≥ Kd log n , there is an event with probability at least 1 − Ce − cn + C (cid:48)
e − d on which one has
[Equation]
Proof.
Fix ν ∈ [0 , π ] .
Let E denote the event E 0 .
5 , 0 in Lemma E.16; then by that lemma, E has probability at least 1 − Ce − cn as long as n ≥ C (cid:48) , where c, C, C (cid:48) > 0 are absolute constants, and for ( g 1 , g 2 ) ∈ E , one has for all ν ∈ [0 , π ]
[Equation]
Thus
[Equation]
if we write
[Equation]
we have (cid:101) Ξ 5 =
Ξ 5 for any ν whenever ( g 1 , g 2 ) ∈ E , so that by the triangle inequality, for any ν
[Equation]
where the second line uses the triangle inequality, and the third line uses the Schwarz inequality and
Lemma E.37 together with the Lyapunov inequality.
So, we can proceed analyzing (cid:101)
Ξ 5 .
First, we aim to apply Lemma E.33 with the choices
[Equation]
since XY =
(cid:101)
Ξ 5 ( ν, · , · ) ; square-integrability of X and Y is evident from the deﬁnition of 1 E , and we have | X | ≤ 1 by Cauchy-Schwarz.
To control Y , we start by noting
[Equation]
where C > 0 is an absolute constant; the ﬁrst inequality is the Minkowski inequality, the second uses the property of E and drops the indicator by nonnegativity, and the third applies Lemma E.29, and discards the n − 1 factor.
For deviations, we start by noting that E [ (cid:107) v ν (cid:107) 22 ] = 1 , and that by
Lemmas G.2 and G.11, we have
[Equation]
It follows that there exists an absolute constant C (cid:48) > 0 such that
[Equation]
putting t = C (cid:48)
(cid:112) d/n and choosing n ≥ ( C (cid:48) /C )
2 d , we have
[Equation]
Moreover, by Lemma E.17, we can run a similar argument on (cid:107) ˙ v ν (cid:107) 22 to get that if n is larger than a constant multiple of d
[Equation]
Next, Taylor expansion with Lagrange remainder of the smooth function x (cid:55)→ x − 1 on the domain x > 0 about the point 1 gives
[Equation]
where ξ lies between 1 and x .
If ( g 1 , g 2 ) ∈ E , then (cid:107) v ν (cid:107) 62 ≥ (1 / 64) , and we can therefore assert
[Equation]
with probability at least 1 − Ce − cn .
Using a union bound together with (E.64) (and changing the constant to C ), we have with probability at least 1 − 2 e − cd − C (cid:48) e − c
[Equation]
n that
[Equation]
Given that n ≥ d , it follows that with the same probability we have
[Equation]
d n, which implies that with probability at least 1 − 2 e − d − C (cid:48)
e − cn , we have
[Equation]
Now, the triangle inequality gives
[Equation]
When ( g 1 , g 2 ) ∈ E , we have (cid:107) v ν (cid:107) 22 ≥ 14 , so, by a union bound, with probability at least 1 − 4 e − d −
C (cid:48)
e − cn
we have
[Equation]
(cid:107) ˙ v ν (cid:107)
[Equation]
v
[Equation]
22 −
[Equation]
(cid:12)(cid:12)
[Equation]
C
[Equation]
d n, and since
[Equation]
[Equation]
another union bound and the measure bound on E let us conclude that with probability at least 1 − 4 e − d − C (cid:48)
e − cn , we have
[Equation]
n.
If we choose n ≥ (1 /c )
( d + log C (cid:48) / 4) , we have 4 e − d + C (cid:48)
e − cn ≤ 8 e − d , so the previous bound occurs with probability at least 1 − 8 e − d .
We can now apply Lemma E.33 to get with probability at least 1 − 8 e − d
[Equation]
Next, we attempt to apply Lemma E.33 again, this time to X = 1 E (cid:104) v 0 , v ν (cid:105) and Y = 1 E ( (cid:107) v 0 (cid:107) 2 (cid:107) v ν (cid:107) 2 ) − 1 .
Using the deﬁnition of E , we have | X | ≤ 4 and (cid:107) Y − 1 (cid:107) L 2 ≤ (cid:107) Y (cid:107)
L 2 + 1 ≤ 5 , where the second bound also leverages the Minkowski inequality; so we need only establish deviations of Y .
Applying again (E.66), and using ( g 1 , g 2 )
∈
E implies (cid:107) v 0 (cid:107) 2 (cid:107) v ν (cid:107) 2 ≥ 14 , we get ( (cid:107)
[Equation]
with probability at least 1 − Ce − cn .
Using Lemma G.11 and (Vershynin, 2018, Theorem 3.1.1), we can assert for any ν ∈ [0 , π ] and any t ≥
[Equation]
which implies that there exists an absolute constant C > 0 such that for any d >
[Equation]
In particular, when n ≥ d [Equation] we can assert that (cid:107) v ν (cid:107) 2 ≤ 1 + C with probability at least 1 − 2 e − d .
By the triangle inequality and a union bound, it follows
[Equation]
with probability at least 1 − 6 e − d .
Then a union bound gives that with probability at least 1 − 6 e − d −
C (cid:48)
e − cn , (E.67) leads to
[Equation]
and using n ≥ d and worst-casing constants implies that with the same probability
[Equation]
Then since ( g 1 , g 2 ) ∈ E = ⇒ Y
=
( (cid:107) v 0 (cid:107) 2 (cid:107) v ν (cid:107) 2 ) − 1 , another union bound gives that with probability at least 1 − 6 e − d − C (cid:48)
e − cn
we have | Y − 1 | ≤ C (cid:112) d/n .
As in the previous step of the reduction, we can choose n ≥ (1 /c )
( d + log C (cid:48) / 6) to get that 6 e − d + C (cid:48)
e − cn ≤ 12 e − d , so that the previous bound occurs with probability at least 1 − 12 e − d .
We can thus apply Lemma E.33, a union bound, and our previous work to get that with probability at least 1 − 20 e − d
[Equation]
Whenever ( g 1 , g 2 ) ∈ E , we have by the triangle inequality, the Schwarz inequality [Equation] and Lemmas E.16 and {+E.
+} that
[Equation]
allowing us to drop the indicator.
We have (cid:104) v 0 , v ν (cid:105) =
(cid:80) i σ ( g 1 i ) σ ( g 1 i cos
ν + g 2 i sin ν ) , which is a sum of independent random variables; following the argument at and around (E.35)
[Equation]
we conclude moreover that these random variables are subexponential with rate C/n
[Equation]
where C > 0 is an absolute constant.
We therefore obtain from Lemma G.2 the tail bound
[Equation]
which, for a suitable choice of absolute constant C (cid:48) > 0 and choosing n ≥ C (cid:48)
d
[Equation]
yields the deviations bounds
[Equation]
d n (cid:35) ≤ 2 e − d .
Taking a ﬁnal union bound (since we assumed throughout that ( g 1 , g 2 ) ∈ E ) gives that with probability at least 1 − Ce − cn + C
[Equation]
e − d , one has
[Equation]
which is sufﬁcient to conclude pointwise concentration as claimed for sufﬁciently large n after we put d = d (cid:48) log n and include extra log n factors in any points where we need to choose n larger than d .
Lemma E.45.
In the notation of Lemma E.13, there are absolute constants c, C, C (cid:48) , C
[Equation]
C
[Equation]
>such that for any δ ≥ 12 , one has
[Equation]
as long as δ ≥ 12 .
Proof.
We will differentiate with respect to ν the function
[Equation]
and construct an event on which f (cid:48) has size poly( n ) .
We need to also differentiate the function E [ f ( · , g 1 )] ; for this we will additionally show that f (cid:48) ( ν, · ) is absolutely integrable over the product [0 , π ] × R n × R n , which allows us to apply Fubini’s theorem to move both the g 1 and g 2 expectations under the ν integral in the ﬁrst-order Taylor expansion we obtain.
In particular, the derivative of E [ f ( · , g 1 )] will in this way be shown to be E [ f (cid:48) ( · , g 1 )] , so that linearity and the triangle inequality imply a poly( n ) magnitude bound for the derivative of E g 2 [Ξ 5 ] − E
[Ξ 5 ] .
Deﬁne
[Equation]
so that, for almost all g 1 ,
[Equation]
For each ﬁxed ( g 1 , g 2 ) and each i , the only obstructions to differentiability of q i in ν arise from the function σ (using smoothness of ψ from Lemma E.31 and the fact that it is constant whenever (cid:107) v ν (cid:107) is small enough that nondifferentiability of (cid:107) · (cid:107) 2 could pose a problem); following the zerocrossings argument of Lemma E.22, q i fails to be differentiable at no more than n points of [0 , π ] , and otherwise has derivative
[Equation]
− ψ [Equation]
sin [Equation] ψ
[Equation] (cid:107) [Equation] (cid:107)(cid:33) .
(E.68) by the chain rule and the product rule.
To conclude absolute continuity of q i ( · , g 1 , g 2 ) , we need to show that q (cid:48)
i is integrable; this follows from Cauchy-Schwarz, the integrability of (cid:107) v 0 (cid:107) 2 , (cid:107) v ν (cid:107) 2 ,
(cid:107) ˙ v ν (cid:107) 2 (Lemma E.17), the triangle inequality, and the Lemma E.31 estimates ψ ≥ 14 , | ψ (cid:48) | ≤ C , | ψ (cid:48)(cid:48) | ≤ C (cid:48) , and | ψ (cid:48)
( x ) /x | ≤ C (cid:48)(cid:48) for any x ∈ R (to see this last estimate, note that | ψ (cid:48) | is bounded on R , and use that ψ is constant whenever x ≤ 14 ).
Then (Cohn, 2013, Theorem 6.3.11) implies that q i ( · , g 1 , g 2 ) is absolutely continuous with a.e. derivative q (cid:48)
i .
Next, we can write
[Equation]
using Lemma E.37 to see that Fubini’s theorem can be applied.
Our aim is now to apply
Lemma E.27, so we need to check its remaining hypotheses.
First, continuity of q i ( ν, · ) follows from continuity of σ , smoothness of ψ , and the fact that the denominator never vanishes.
Joint absolute integrability of q i and q (cid:48)
i follows from our veriﬁcation of absolute integrability of q (cid:48)
i above, which produces a ﬁnal upper bound that does not depend on ν (which is therefore integrable over [0 , π ] as well); the corresponding result for q i follows from Lemma E.37.
Last, we need the growth estimate.
We have from Lemma E.
[Equation]
which is evidently quadratic in | g 2 i | once | g 2 i | ≥ 1 .
Consequently we can apply Lemma E.27 to differentiate f ( · , g 1 ) ; we get at almost all gf
[Equation]
, where (cid:101) g i 2 is the vector g 2 but with its i -th coordinate replaced by − g
1 i cot t , and where ρ is the pdf of a N (0 , 2 /n ) random variable.
The changes in (cid:101) g i 2 drive updates to the terms in q i as follows: we have σ ( g 1 i cos ν
+ g 2 i sin ν ) becoming 0 , and g 2 i cos ν − g 1 i sin ν becoming − g 1 i / sin ν .
Thus, we have
[Equation]
where the notation v it is in use in the Ξ 1 control section and is deﬁned in Lemma E.26, and v i 0 is deﬁned here similarly (the R n − 1 vector which is the projection of v 0 onto all but the i -th coordinates).
Using Lemma E.31, we can further assert
[Equation]
69) where we use that (cid:107) v i 0 (cid:107) 2 ≤ (cid:107) v 0 (cid:107) 2 .
For each ﬁxed g 1 having no coordinates equal to zero, we write
K i = | g 1 i | > 0 ; if 0 ≤ t ≤ π/ 4 or 3 π/ 4 ≤ t ≤ π , we have cos 2 t ≥ 12 , and so
[Equation]
Using Lemma E.36, we have
[Equation]
On the other hand, when π/ 4 ≤ t ≤ 3 π/ 4 , then sin t ≥ 2 − 1 / 2 , and we can assert
[Equation]
We conclude for any t
[Equation]
E.
[Equation]
for absolute constants C, C (cid:48) > 0 , and this upper bound is integrable jointly over t and g 2 .
We have checked previously the joint integrability of the q (cid:48)
i terms when applying Lemma E.27, so we can therefore apply Fubini’s theorem to get g 1 -a.s.
[Equation]
Consequently, to conclude a Lipschitz estimate for f ( · , g 1 ) it sufﬁces to control the quantity under the t integral in the previous expression.
We will start by controlling the second term using Markov’s inequality.
Following (E.69) [Equation] we calculate
[Equation]
where the last line follows from the change of variables
g (cid:55)→ g sin t in the integral.
We can evaluate this integral with Lemma G.11, which gives a bound
[Equation]
, and therefore a bound of C (cid:48) > 0 an absolute constant on the sum over i .
As a byproduct of this estimate, we can assert that the second term is jointly integrable over [0 , π ] × R n × R n , which allows us to apply Fubini’s theorem and obtain the same differentiation result for E [ f ( · , g 1 )] .
Meanwhile, beginning from (E.70), we can write using the triangle inequality
[Equation]
By Gauss-Lipschitz concentration and Lemma G.9, we have that (cid:107) g 1 (cid:107) 2 ≤ (cid:107) g 1 (cid:107) 2 ≤ 2 with probability at least 1 − 2 e − cn , and since (cid:107) g 1 (cid:107) ∞ ≤ (cid:107) g 1 (cid:107) 2 , we conclude with the same probability that | g 1 i | ≤ 2 simultaneously for all i .
Meanwhile, if X ∼ N (0 , 1) , we have for any t ≥ 0 that P [ | X | ≥ t ] ≥ 1 − Ct , where C > 0 is an absolute constant; so if X i ∼ i .
i .
d . N (0 , 1) , we have by independence and if t is less than an absolute constant P [ ∀ i, | X i | ≥ t ] ≥ (1 − Ct ) n ≥ 1 − C (cid:48)
nt , where the last inequality uses the numerical inequality e − 2 t ≤ 1 − t ≤ e − t , valid for 0 ≤ t ≤ 12 .
From this expression, we conclude that when 0 ≤ t ≤ cn − 1 / 2 for an absolute constant c > 0 , we have
[Equation]
so choosing in particular t = cn − ( δ +32) for any δ > 0 , we conclude that P [ ∀ i ∈ [ n ] , | g 1 i | ≥ cn − 3 / 2 − δ ] ≥ 1 − C (cid:48)
n − δ .
Then with probability at least 1 − C
[Equation]
n − δ − 2 e − cn , we have
[Equation]
so as long as δ ≥ 12 , we have
P
[Equation]
i =
[Equation]
(cid:12) ≥ Cn 1+ δ (cid:35) ≤
[Equation]
cn .
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Using the triangle inequality, Cauchy-Schwarz
[Equation]
and Lemma E.
[Equation]
we obtain
[Equation]
(using also the fact that ψ (cid:48) ( x ) = 0 and ψ
[Equation]
( x ) =
whenever x is sufﬁciently near to 0 ); and
[Equation]
from which we conclude
[Equation]
for some absolute constants C, C (cid:48) > 0 .
By Lemma E.17, there is an event E of probability at least 1 − Ce − cn on which we have (cid:107) ˙
v ν (cid:107) 2 ≤ 4 for every ν .
Moreover, we have from the triangle inequality that (cid:107) ˙
v ν (cid:107) 2 ≤ [Equation] g 1 (cid:107)
+ (cid:107) g 2 (cid:107) 2 , which is independent of ν ; and in particular we have
[Equation]
n (cid:88) i = q (cid:48)
i ( ν
[Equation]
which is a polynomial in (cid:107) g 1 (cid:107) 2 and (cid:107) g 2 (cid:107) 2 by the binomial theorem.
Thus, applying independence,
Lemma G.10, Lemma G.11 yields that there is an absolute constant C (cid:48)(cid:48) > 0 such that
[Equation]
Therefore
[Equation]
as in the framework section of the proof of Lemma E.
[Equation]
we can use the inequality
[Equation]
[Equation]
together with the partition
[Equation]
and this last expression can be used to obtain a g 1 event of not much smaller probability 1 − Ce − cn on which the LHS of (E.73), and hence the LHS of (E.72), is controlled by an absolute constant uniformly in ν (in particular, using Markov’s inequality as in the framework section of the proof of
Lemma E.13).
Consequently, one more application of the triangle inequality gives that
[Equation]
as long as δ ≥ 12 .
Lemma E.46.
In the notation of Lemma E.13, if d ≥ 1 , there are absolute constants c, C, C (cid:48) , C (cid:48)(cid:48) > 0 and absolute constants K, K (cid:48) > 0 such that if n ≥ Kd log n and d ≥ K (cid:48) , there is an event with probability at least 1 − Ce − cn − C (cid:48)
n − d on which one has
[Equation]
Proof.
The argument is extremely similar to Lemma E.43, since both terms have small expectations and deviations essentially determinable by the same mean-zero random variable.
We are going to control the expectation ﬁrst, showing that it is small; then prove that | Ξ 6 | is small uniformly in ν .
Let E denote the event E 0 .
5 , 0 in Lemma E.16; then by that lemma, E has probability at least 1 − Ce − cn as long as n ≥ C (cid:48) , where c, C, C (cid:48) > 0 are absolute constants, and for ( g 1 , g 2 ) ∈
E , one has for all ν ∈ [0 , π ]
[Equation]
Thus, if we write
[Equation]
we have (cid:101) Ξ 6 =
Ξ 6 for all ν whenever ( g 1 , g 2 ) ∈ E , so that for any ν
[Equation]
where the second line uses the triangle inequality and the Schwarz inequality and Lemma E. together with the Lyapunov inequality.
We proceed with analyzing the expectation of (cid:101)
Ξ 6 .
Using the
Schwarz inequality gives
[Equation]
and the checks at and around (E.35) in the proof of Lemma E.15 show that we can apply Lemma E. to obtain
[Equation]
But we have using rotational invariance that E [ σ ( g
[Equation]
[Equation] [Equation]
, which implies
[Equation]
from which we conclude for all ν
[Equation]
Next, we control the deviations of Ξ 6 with high probability.
By Lemma E.17, there is an event E a with probability at least 1 − e − cn on which (cid:107) ˙ v ν (cid:107) 2 ≤ 4 for every ν ∈ [0 , π ] .
Therefore on the event E b = E ∩ E a , which has probability at least 1 − Ce − cn by a union bound, we have using
Cauchy-Schwarz that for every ν
[Equation]
Using the high probability deviations bound established in (E.63), it follows that if n is large enough then with probability at least 1 − Ce − cn − C (cid:48) n − 2 d +1 / 2 we have
[Equation]
As long as d ≥ 12 , we have that this probability is at least 1 − Ce − cn − C (cid:48) n − d , and so the triangle inequality yields ﬁnally that with probability at least 1 − Ce − cn − C (cid:48)
n − d
[Equation]
Lemma E.47.
Consider the function
[Equation]
which is the negated numerator of ¨ ϕ .
Then if 0 ≤ ν ≤ π/ 2 , one has a bound
[Equation]
and the lower bound is positive if 0 < ν ≤ π/ 2 .
Proof.
To see that the lower bound is positive under the stated condition, write
[Equation]
the quantity in parentheses is positive in a neighborhood of zero by continuity, and in fact one calculates for its unique zero ν 0 =
π 2 / 249 , and one veriﬁes numerically
that 48 π 2 / 249 > 1 . 9 > π/ 2 .
We conclude that the bound is positive for 0 < ν < 1 . 9 by continuity.
To establish the bound, we employ Taylor expansion of the numerator, which is a smooth function on (0 , π ) with continuous derivatives of all orders on [0 , π ] , in a neighborhood of zero.
In our development in the proof of Lemma E.5, we showed that the analytic function − g ( ν ) =
− (2 π 2 / 3) ν 3 + O ( ν 4 ) near zero, so Taylor’s theorem with Lagrange remainder implies
[Equation]
and so it sufﬁces to get suitable bounds on the fourth derivative of g .
We will develop the bounds rather tediously.
Start by distributing in g to write
[Equation]
Using the Leibniz rule, we have for the fourth derivative g
[Equation]
3)3 ( ν ) (cid:17) + ν (cid:16) g (4)1 ( ν ) + 8 g (3)2 ( ν ) + 36 g (2)3 ( ν ) (cid:17) + (cid:16) g (4)0 ( ν ) + 4 g (3)1 ( ν ) + 12 g (2)2 ( ν ) + 24 g (1)3 ( ν ) (cid:17) .
To calculate these derivatives, we just need to differentiate sin , cos , and their third powers.
Write c ( ν ) =
cos 3 ( ν ) and s ( ν ) = sin 3 ( ν ) ; using the elementary calculations
[Equation]
one can calculate the results
[Equation]
and g
[Equation]
and
[Equation]
and ﬁnally
[Equation]
Plugging back into (E.74) and canceling, we get g
[Equation]
[Equation]
+
[Equation]
[Equation]
Since ν > 0 , we can leverage lower bounds on each h i term.
We have trivially | h 3 | ≤ 1 , so that | ν 3 h 3 ( ν ) | ≤ π 3 / 8 .
We will study νh 1 ( ν ) + h 0 ( ν ) together to get a better bound.
We have
[Equation]
[Equation]
[Equation]
[Equation] 75) using ν ≤ π/ 2 and cos ≥ 0 on this domain.
We will show that the RHS of the ﬁnal inequality, denoted q , is a decreasing function of ν , and is therefore lower bounded by its value at ν = π/ 2 on our interval of interest.
We calculate
[Equation]
Reordering terms, we can write
[Equation]
We can estimate numerically
[Equation]
which shows that C 1 , C 2 , C 3 , C 4 > 0 and both of the linear prefactors are decreasing functions of ν .
We have on all of (0 , π/ 2) by concavity of sin
[Equation]
using in particular sin ν ≤ ν and sin ν ≥ (2 /π )
ν .
Using similarly concavity of cos on this domain, in particular the inequalities cos ν ≤ π/ 2 − ν and cos ν ≥ 1 − (2 /π )
ν , we have
[Equation]
In total [Equation] we have a bound
[Equation]
We calculate the maximizer of the concave quadratic function of ν in the previous bound via differentiation; plugging in, we get
[Equation]
A numerical estimate gives
[Equation]
and using that − cos 3 is strictly decreasing for ν <
π , we can therefore guarantee q (cid:48) ≤ 0 as long as ν ≤ cos -1 3 (cid:112) 20 / 81 .
Writing c
= cos -1 3 (cid:112) 20 / 81 , we estimate numerically 0 .
90 ≥ c ≥ 0 .
89 , so that this bound is nonvacuous.
For ν ≥ c , we apply again concavity of cos to develop the lower bound cos ν ≥ (cid:18) π/
[Equation]
π/ 2] .
Using this to estimate the − cos 3 term in our upper bound for q (cid:48) , we obtain a bound q (cid:48)
[Equation]
ν ≤ π/
[Equation]
We deﬁne D = 20 / ( π/ 2 − c ) 3 , A = 2 C 2 /π + C 4 , B = 2 C 3 /π + πC 4 / 2 + C 1 , and C = C 3 , so that the RHS can be written as − D ( π/ 2 − ν ) 3 − Aν 2 + Bν − C .
Differentiating once and equating to zero results in the quadratic equation
[Equation]
which has roots M/ 2 ± 12 √ M 2 − 4 N .
Numerically estimating the constants, we get that the two roots lie in [0 . 99 , 1] and [3 . 3 , 3 . 4] , so that we need only consider the smaller root.
Differentiating once more to determine the class of the critical point, we ﬁnd for the second derivative at M/ 2 −
[Equation]
so that M/ 2 − 12 √ M 2 − 4 N is a maximizer for our cubic bound, and the bound is increasing for arguments less than this point and decreasing for arguments greater than it; we can conclude that the zero in [3 . 3 , 3 . 4] is a minimizer, so that our bound can be ascertained negative by checking its value at M/ 2 − 12 √ M 2 − 4 N .
We ﬁnd using a numerical estimate
[Equation]
which proves that q (cid:48) ≤ 0 on [ c, π/ 2] .
This shows that our lower bound on νh 1 ( ν ) + h 0 ( ν ) in (E.75) is nonincreasing on [0 , π/ 2] , so that we can assert
[Equation]
It remains to bound ν 2 h 2 ( ν ) =
ν 2 (3 π cos ν − 11 sin ν ) .
On [0 , π/ 2] , cos is decreasing and sin is increasing, so 3 π cos ν − 11 sin ν is decreasing here; it is positive at ν = 0 and negative at ν = π/ 2 , so that by continuity it has a unique zero in (0 , π/ 2) .
Denote this zero as ν 0 ; then using that ν 2 ≥with no zeros in the interior, we can write
[Equation]
and
[Equation]
which gives the bound ν 2 h 2 ( ν ) ≥ − 11 π 2 / 4 on [0 , π/ 2] .
Putting it all together, we have
[Equation]
where the last inequality follows from a numerical estimate of the constants.
Lemma E.48 (Uniformization) .
Let (Ω , F , P ) be a complete probability space.
For some t ∈ R , δ t ≥ 0 , S ⊂ R d , and event E ∈ F , suppose that f : S × Ω → R is second-argument measurable and satisﬁes
For all x ∈ S , P [ f ( x , · ) ≤ t ] ≥ 1 − δ t ; 2.
For all g ∈ E , f ( · , g ) is L -Lipschitz; 3.
There is M > 0 such that sup x ∈ S (cid:107) x (cid:107) 2 ≤ M .
Then g (cid:55)→ sup x ∈ S f
( x , g ) is measurable, and for every ε > 0 , one has
[Equation]
Proof.
Because S is a subset of the separable metric space ( R d , (cid:107) · (cid:107) 2 ) and all sample trajectories f ( · , g ) are assumed (Lipschitz) continuous, the supremum in the deﬁnition of g (cid:55)→ sup x ∈ S
f
( x , g ) can be taken on a countable subset of S , and the resulting function of g is measurable (e.g., (Ledoux & Talagrand, 1991, §2.2 p. 45)).
By (Vershynin, 2018, Proposition 4.2.12) and boundedness of S , for every ε > 0 there exists an ε -net of S having cardinality at most (1 + 2 M/ε ) d ; denote these nets as N ε .
Since each N ε is ﬁnite, we may also deﬁne for each x ∈ S a point x
ε such that (cid:107) x − x ε (cid:107) 2 ≤ ε ; then for every g ∈ E , we have | f ( x , g ) − f ( x ε , g ) | ≤ Lε .
We deﬁne a collection of events E ε by
[Equation]
The triangle inequality then implies that if g ∈ E ε ∩E , then for all x ∈ S , one has f ( x , g ) ≤ t + Lε .
Consequently, several union bounds yield
[Equation]
as claimed.
E. D EFERRED P ROOFS
Proof of Lemma E.5.
The function cos -1 is C ∞ on ( − 1 , 1) [Equation] and because f ( ν ) := cos ϕ ( ν ) is smooth and satisﬁes f (cid:48)
( ν ) =
( π − 1 ν − 1) sin ν < 0 if ν < π with f (0)
=
1 and f ( π ) = 0 , we see that ϕ is
C ∞ on (0 , π ) by the chain rule.
This also shows ϕ (0) =
cos -1 (1) = 0 and ϕ ( π ) = cos -1 (0) = π/ 2 .
Direct calculation gives
[Equation]
and
[Equation]
Calculating endpoint limits using these expressions will sufﬁce to show the derivatives are continuous on [0 , π ] and give the claimed values there.
We have
[Equation]
by L’Hôpital’s rule, whereas a direct evaluation gives
[Equation]
Continuity of the square root function gives the claimed results for ˙ ϕ .
Again by direct calculation, we ﬁnd
[Equation]
Since ˙ ϕ 2 is meromorphic in a neighborhood of 0 with, as we have shown, a removable singularity at 0 , it is actually analytic, and we can calculate further derivatives at 0 by expanding it locally at 0 .
We use the expansions sin ν = ν − ν 3 / 6 + O ( ν 5 ) and cos ν = 1 − ν 2 / 2 + ν 4 / 24 + O ( ν 6 ) nearto calculate
[Equation]
and 1 −
[Equation]
from which it follows
[Equation]
By the geometric series, we then obtain
[Equation]
Taking the square root of this expression and applying the binomial series, we thus have
[Equation]
from which we read off
[Equation]
It is clear from the analytical expression for ˙ ϕ
and the mean value theorem that ϕ is strictly increasing on [0 , π ] , since ( π − ν ) sin ν >
if 0 < ν < π .
To prove strict concavity for ν ∈ (0 , π ) , we start by simplifying notation.
Consider the function ϕ
r ( ν ) =
ϕ
( π − ν ) , which satisﬁes by the chain rule ¨ ϕ r ( ν ) =
¨ ϕ
( π − ν ) .
Because ϕ r is strictly concave if and only if ϕ is strictly concave, it sufﬁces to prove that ¨ ϕ ( π − ν ) < 0 .
We note
[Equation]
Multiplying both sides of the latter inequality by sin ν − ν cos ν , dividing through by ( ν cos ν − sin ν ) 2 (which is positive on (0 , π ) , since it equals cos 2 ϕ composed with a reversal about π ), and distributing and moving terms to the RHS gives the equivalent condition
[Equation]
and canceling once more gives equivalently
[Equation]
Using ν cos ν − sin ν < 0 , which follows from its derivative − ν sin ν being negative on (0 , π ) , and writing g ( ν )
=
π − 2 ( ν 2 − sin 2 ν ) , we have equivalently ν cos ν +sin
ν > g ( ν )( ν cos ν − sin ν ) , and rearranging gives the inequality
[Equation]
Strict concavity of sin on (0 , π ) gives sin ν < ν , and 0 < g ( ν ) < 1 follows after squaring; so
the
LHS is a convex combination of ν cos ν and sin ν , which in particular satisﬁes | (1 − g ( ν ))
ν cos ν + g ( ν ) sin ν | ≤ max {| sin ν | , | ν cos ν |} .
As argued before, we have sin ν − ν cos ν > 0 if ν ∈ (0 , π ) ; moreover, because ν > 0 we have ν cos ν > 0 if ν ∈ (0 , π/ 2) and ν cos ν < 0 if ν ∈ ( π/ 2 , π ) .
We can numerically determine sin(5 π/ 8)+(5 π/ 8) cos(5 π/ 8) > 0 , and given that 5 π/ 8 ≥ 1 .
95 > π/ 2 , it follows
[Equation]
which implies (E.83)
when 0 < ν ≤ 1 . 95 .
Recalling that we are arguing for ϕ r in this setting, we translate our results back to ϕ and conclude that ϕ
( ν ) <
if π − 1 .
95 ≤ ν < π .
To address the case where 0 < ν < π − 1 .
95 , we employ Lemma E.47; it allows us to conclude ¨ ϕ < 0 provided 0 < ν ≤ π/ 2 , and a numerical estimate gives that π − 1 .
95 < π/ 2 , so that we have ¨ ϕ < 0 for all 0 < ν < π .
Taking limits in ϕ gives concavity at the endpoints { 0 , π } as well.
To bound ¨ ϕ away from zero on [0 , π/ 2] , we apply Lemma E.47 to assert
[Equation]
The numerator in the last expression is nonpositive if 0 ≤ ν ≤ π/ 2 , and using the lower bound in
Lemma E.14 on [0 , π/ 2] , we have
[Equation] [Equation]
ν > 0 .
From nonpositivity of the numerator, it follows
[Equation]
We have 1 − 12 ν 2 ≥ 0 as long as 0 ≤ ν ≤ √ 2 ; so after removing the max, distributing, and cancelling, we have
[Equation]
The denominator of this last expression is nonnegative and has singularities at ± 2 , and is clearly even symmetric; so it is maximized on 0 < ν ≤ √ 2 at √ 2 , and we have
[Equation]
Taking limits ν (cid:38) 0 , we can assert this bound on [0 , √ 2] , and the bound is clearly an increasing function of ν , from which it follows
[Equation]
where the last inequality follows from a numerical estimate of the constants.
On the other hand, when √ 2 < ν ≤ π/ 2 , we have from (E.84) that
[Equation]
If we differentiate the degree four polynomial on the RHS of this bound and solve for critical points, we ﬁnd a double critical point at ν =
and a critical point at ν =
π 2 / 83 ; a numerical estimate conﬁrms that this critical point lies in the interior of [ √ 2 , π/ 2] .
The second derivative of the RHS is − (4 /π ) ν
+ 83 / (2 π 3 ) ν 2 , and plugging in ν = 12 π 2 / 83 gives a value of − 48 π/
+ 144 π/ 83 , which is positive
; hence the RHS is maximized on the boundary, i.e.,
[Equation]
A numerical estimate shows that the RHS of the last inequality is no larger than − 0 .
Since the intervals we have proved a bound over cover [0 , π/ 2] , this proves the claim with c = − 0 .
The bound ˙ ϕ < 1 on (0 , π ) follows from the fact that ϕ is strictly concave on (0 , π ) and the mean value theorem; we have already shown ˙ ϕ > 0 in proving strict increasingness of ϕ .
Similarly, the proof of strict concavity in the interior has already established ¨ ϕ < 0 .
To obtain the lower bound on ¨ ϕ , we use that ¨ ϕ is continuous on [0 , π ] and the Weierstrass theorem to assert that there is C ≥such that ¨ ϕ ≥ − C on [0 , π ] ; because ¨ ϕ (0) (cid:54) = 0 , we actually have C > 0 .
For the quadratic model, we use our previous results and Taylor expand ϕ about 0 ; we get immediately
[Equation]
For the upper bound, we can assert immediately on [0 , π/ 2] a bound
[Equation]
where c = 0 .
07 sufﬁces.
To extend the bound to ν ∈ [ π/ 2 , π ] , we employ a bootstrapping argument; because ϕ is concave, we have a bound
[Equation]
where the second line plugs into the formulas for ϕ and ˙ ϕ .
We will show that the graph of ν − cνlies entirely above the graph of the RHS of this inequality.
This condition is equivalent to
[Equation]
the LHS of this inequality is a concave quadratic with maximizer ν (cid:63) = 1 / (2 c ) (cid:0) 1 − π
[Equation]
1 (cid:1) , and numerical estimation of the constants gives ν (cid:63) ≥ π .
Since ν (cid:63) is outside [ π/ 2 , π ] and the quadratic is concave, we conclude that the bound is tightest at the boundary point π/ 2 , and one checks numerically
[Equation]
which establishes that the bound ϕ ( ν ) ≤ ν − cν 2 actually holds on all of [0 , π ]
This completes the proof of all of the claims.
F C ONTROLLING C HANGES D URING T RAINING F. P RELIMINARIES
We now consider the changes in the integral operator Θ k during gradient descent.
In this section we restore the iteration subscript (that is dropped in other sections to lighten notation) to various quantities.
Θ k changes during training as a result of both smooth changes in the features at all layers and non-smooth changes in the backward features { β (cid:96)t ( x ) } due to the non-smoothness of the derivative of the ReLU function.
Because of the difﬁculty of reasoning precisely about the changes in Θ t , we will bound these rather naively by controlling Θ t over all possible support patterns of the features given a bound on the norm change of the pre-activations.
We now deﬁne a trajectory in parameter space that interpolates between the iterates of gradient descent, given for any k (cid:48) ∈ { 0 , . . .
, k } and s ∈ [0 , 1] by
[Equation]
(with the formal derivative ˜ ∇ deﬁned in Appendix A.1).
We will henceforth use k (cid:48) to denote an integer indexing the iteration number and t to denote a continuous parameter taking values in [0 , k ] (such that k (cid:48)
= (cid:98) t (cid:99) , s = t − (cid:98) t (cid:99) ).
Quantities indexed by t are ones where the parameters take the value θ Nt .
To lighten notation, we will drop the N superscript when referring to time-indexed quantities (aside from ζ Nk and Θ Nk ), but all such quantities depend on the parameters as deﬁned by (F.1).
Instead of considering the change in the features (cid:8) α (cid:96)t ( x ) (cid:9) directly, it will be more convenient to work in terms of the pre-activations, which are given at layer (cid:96) by
[Equation]
We deﬁne a maximal allowable change in the pre-activation norm by
[Equation]
for q ≥ 0 and a constant C η to be speciﬁed later, where the scaling is chosen with foresight.
We can then deﬁne a maximal number of iterations such that the pre-activation norms at all layers along the trajectory (F.1) change by no more than η .
This number k η must satisfy
[Equation]
for η given by (F.2).
Our goal will be to show that we can in fact train for long enough so as to reduce the ﬁtting error without exceeding k η iterations.
F. C HANGES IN F EATURE S UPPORTS D URING T RAINING
Recalling the deﬁnition of the feature supports at layer (cid:96) , time t and x ∈ M by I (cid:96),t ( x ) =
supp( α (cid:96)t ( x ) > 0 ) ⊆
[ n ] , we denote by I t ( x ) =
( I 1 ,t ( x ) , . .
, I L,t ( x ))
the collection of these support patterns at all layers.
We would next like to relate the smooth changes in the pre-activation norms to the non-smooth changes in the supports of the features.
We denote by J =
( J 1 , . . .
, J L ) a collection of support patterns with J i ∈ [ n ] .
We now consider sets of support patterns that are not too different from those at initialization, as deﬁned by
[Equation]
Note that B ( ρ (cid:96) 0 ( x ) , η ) is simply the set of supports of the positive entries of ρ (cid:96)
( x ) + v for every possible perturbation v of norm at most η .
We consider all possible perturbations due to the complex nature of the training dynamics.
As a result of this worst-casing, the scaling we will require of the depth and width of the network in order to guarantee that the changes during training are sufﬁciently small is expected to be suboptimal.
For a given general support pattern J , we deﬁne generalized backward features and transfer matrices
[Equation]
where the weights are given by (F.1) (and thus β (cid:96)t ( x ) =
β N(cid:96)
I t ( x ) t ).
By controlling these objects for every possible set of supports J that can be encountered during training, we can control the smooth changes in the features themselves.
A ﬁrst step towards this end is understanding how many such support patterns we expect to see given the constraint in (F.2).
In order to bound the number supports that can be encountered during training, we need to control the diameter of B ( ρ (cid:96)
( x ) , η ) .
This can be done by deﬁning
[Equation]
where (cid:9) denotes the symmetric difference.
Since the pre-activation at a given layer are Gaussian variables conditioned on all the previous layer weights, bounding the size of δ η ( ρ (cid:96) 0 ( x )) can be reduced to showing concentration of a certain function of Gaussian order statistics.
This is achieved in the following lemma :
Lemma F.1.
For η given by (F.2) , if n, L, d satisfy the requirements of lemma F.6 and n > d 5 for some constant K , then for a vector g ∈ R n , g i ∼ iid N (0 , 1 n ) we have
[Equation]
for some constants c, C, C (cid:48) .
Proof.
Let
[Equation]
denote the order statistics of the magnitudes of the elements of g .
We will show that bounding δ η ( g ) can be reduced to understanding what is the smallest k such that
[Equation]
We denote this value of k by k η .
Deﬁne indices j i by | g J i | = | g | ( i ) (and breaking ties arbitrarily in case several order statistics are equal).
To see that
[Equation]
it sufﬁces to note that since k η − 1 (cid:80)
[Equation]
|
[Equation]
[Equation]
) <
η 2 one can choose ε > 0 small enough such that
[Equation]
which will give d s ( g , y )
=
k η − 1 ⇒ δ η ≥ k η − 1 .
To prove the second inequality, consider y =
g − δ η (cid:80) i
=1 g J i e J i .
Clearly for any y (cid:48)
such that d s ( g , y (cid:48) )
=
δ η we have (cid:107) g − y (cid:107) 2 ≤ (cid:107) g − y (cid:48) (cid:107) 2 .
Since there exists at least one such y (cid:48)
∈
B E ( g , η ) , it follows that (cid:107) g − y (cid:107) 2 ≤ (cid:107) g − y (cid:48)
(cid:107) 2 ≤ η , and hence the smallest k such that k (cid:80) i = (cid:12)(cid:12) g
( i ) (cid:12)(cid:12) 2 ≥ η 2 must obey k ≥ δ η .
For η deﬁned in (F.2), we can satisfy the requirement on η in lemma F.6 by requiring n > d 5 for some K .
Applying this lemma, we ﬁnd that there is a constant K (cid:48) such that for k = (cid:6) K (cid:48) nη
[Equation]
3 (cid:7) we have
[Equation]
from which it follows immediately that
[Equation]
Choosing some constant C such that (cid:6) K (cid:48)
nη 2 / 3 (cid:7) ≤
Cnη 2 / 3 and using (F.9) allows us to bound δ η ( g ) with the same probability.
With this result in hand, we can control the objects in (F.7) for all the supports in J η ( M ) .
Lemma F.2.
Assume d, L, n satisfy the assumptions of lemmas D.2, F.1, D.8, D.14 and additionally
[Equation]
for some constants K, K (cid:48) , C 0 , where q is the constant in (F.2) .
Then
i) for η , J η ( x ) deﬁned in (F.2) , (F.5) , on an event of probability at least 1 − e − cd , simultaneously sup x ∈M sup J∈J η ( x ) sup (cid:96) ∈
[ L ] 1 ≤ (cid:96) (cid:48) <(cid:96) (cid:13)(cid:13) ρ (cid:96)
( x ) (cid:13)(cid:13) 2 ≤ C 2 , sup x ∈M sup J∈J η ( x [Equation]
sup (cid:96) ∈ [ L ] 1 ≤ (cid:96) (cid:48) <(cid:96) (cid:13)(cid:13)(cid:13) β (cid:96) − 1 J [Equation] 0 (cid:13)(cid:13)(cid:13) 2 ≤ C 2 √ n
[Equation]
sup x ∈M sup J∈J η ( x ) sup (cid:96)
[Table]
ii) for T η deﬁned in (F.3) , on an event of probability at least 1 − e − cd , sup x ∈M ,t ∈ [0 ,T η ] ,(cid:96) ∈
[ L
[Table]
for some constants c, C .
Proof.
Deferred to F.5.
F. C HANGES IN F EATURES D URING T RAINING
We can now bound the smooth changes during training:
Lemma F.3 (Smooth changes during training) .
Set the step size τ and a bound on the maximal number of iterations k max such that
[Equation]
for some constant q .
Assume n, L, d satisfy the requirements of lemmas F.2, and in particular n ≥
KdL 9+2 q for some K .
Assume also that given some k ≤ k max − 1 , for all k (cid:48) ∈ { 0 , . . .
, k } ,
[Equation]
(F.10)
Then on an event of probability at least 1 − e − cd , one has simultaneously
[Equation]
[Table]
Proof.
We will bound the smooth changes in the network features during gradient descent with respect to either the population measure µ ∞ or the ﬁnite sample measure µ N .
We denote a measure that can be one of these two by µ N
[Equation]
For any collection of supports J ∈ J η ( M )
[Equation]
deﬁne generalized backward features and transfer matrices at t by β (cid:96) J t , ˜ Γ (cid:96) : (cid:96) (cid:48)
J t .
These are obtained by setting the network parameters to be θ Nt according to (F.1), but setting all the support patterns to be those in J .
We then deﬁne for any t ∈ [0 , k + 1] ,
[Equation]
[Table]
[Table]
[Table]
while
[Table]
[Equation]
It follows that we can control the difference norms of the pre-activations and backward features by controlling the magnitudes of ρ ηk (cid:48) , β ηk (cid:48) .
In order to control these, we also note that for any t ∈ [0 , k + 1] ,
[Equation]
and similarly
[Equation]
In particular, the above bounds hold when J
= I t ( x ) .
We would now like to understand how the quantities (cid:16) ρ η k (cid:48) , β η k (cid:48) , Γ η k (cid:48) (cid:17) evolve under gradient descent.
Towards this end, for any k (cid:48) ∈ { 0 , . . . , k } and s ∈ [0 , 1] we compute at any point of differentiability
[Equation]
+ s ( x ) = − τ ˜ ∂ ρ (cid:96) ( x ) ∂ θ (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ∗ θ k (cid:48) − sτ ˜ ∇ L N ( θ Nk (cid:48) )
˜ ∇ L N ( θ Nk (cid:48) )
= − τ (cid:32) ˜
∂
ρ (cid:96)k (cid:48)
+ s ( x ) ∂ θ (cid:33) ∗ ˜ ∇ L N ( θ Nk (cid:48) )
=
− τ (cid:96) (cid:88) i = n i (cid:88)
j = n i − 1 (cid:88) l = ˜ ∂
ρ (cid:96)k (cid:48)
+ s ( x ) ∂W ijl ˜ ∂
L N ( θ Nk (cid:48) )
∂W ijl = − τ (cid:96) (cid:88) i
=
(cid:90) x (cid:48) ∈M (cid:10) α
i − 1 k (cid:48)
+ s ( x ) , α i − 1 k (cid:48)
( x (cid:48) ) (cid:11) ˜
Γ (cid:96) : i +1 k (cid:48)
+ s ( x ) P
I i,k (cid:48)
+ s ( x ) β i − 1 k (cid:48)
( x (cid:48) )
ζ Nk (cid:48)
( x (cid:48) )
d µ N ( x (cid:48) ) .
[Equation]
where we used Jensen’s inequality in the second line and our assumption that the error up to iteration k has bounded L 2 µ N norm, and we additionally assumed ρ ηt , β ηt , Γ ηt ≥ 1 .
Arguing as in the proof of
Lemma B.7 for absolute continuity, it follows that
[Equation]
[Table]
[Equation]
Since the above holds for all choices of x , (cid:96) simultaneously, we conclude that
[Equation]
An analogous calculation for the other quantities in (F.11) gives the following set of coupled difference inequalities:
[Equation]
Instead of solving (F.18), we obtain sufﬁcient control by deﬁning k ∗ s.t.
[Equation]
For any t ∈ [0 , k ∗ ] , we obtain a sufﬁcient condition for satisfying the above constraint using (F.18), namely
[Equation]
) for some constant C (cid:48) .
Using the bounds for β ηt − β η 0 and Γ ηt − Γ η 0 in (F.18) to satisfy the second and third condition in (F.19) gives an identical constraint on τt .
In order to control these quantities at t =
we deﬁne an event
[Equation]
21) the probability of which can be controlled using lemma F.2.
On G , the upper bound on τt in (F.20) is at least
[Equation]
F.22) for
We would now like to pick τk max , and ensure that any t ∈ [0 , k + 1] satisﬁes the constraint above if k + 1 ≤ k max .
The analysis also assumes that τk max ≤ T η for which (F.3) holds.
We will then pick the scaling factor C η for the pre-activation norm bound in (F.2) in order to satisfy that constraint as well.
We choose
[Equation]
In order to ensure that k max ≤ k ∗ holds, we use (F.20) and (F.22), and require
[Equation]
which is satisﬁed by demanding n ≥ KdL 3+2 q for some constant K .
Using (F.17) and (F.22), on G we have
[Equation]
In order to ensure that k max ≤ k η we therefore require
[Equation]
which using (F.2) and (F.23) is equivalent to
[Equation]
and thus the constraint can be satisﬁed by choosing C η = C (cid:48)(cid:48) √ d .
Note that the constant C 2 in (F.21) (which enters C (cid:48)(cid:48) ) is set in lemma F.2 which takes C η as input (despite this, C 2 is independent of C η ).
This lemma holds as long as n ≥ C 30 C
η L 6+2 q , which we can guarantee by demanding n ≥ C 30 (cid:16) √ dC (cid:48)(cid:48) (cid:17) 2 L 6+2 q .
We have thus ensured that our choice of k max satisﬁes k max ≤ min { k ∗ , k η } .
We then obtain from the constraints in (F.19), the inequalities in (F.18) and the deﬁnition of G , that on this event
[Equation]
Then using (F.12) and (F.13), we obtain on an event of probability at least 1 − e − cd simultaneously
[Equation]
The combination of the last two lemmas allows us to control the changes in all the forward and backward features uniformly:
Lemma F.4.
Assume n, L, d, k satisfy the requirements of lemmas F.2 and F.3, and additionally n ≥ KL 36+8 q d 9 for some K .
Then one has simultaneously on an event of probability at least 1 − e − cd
[Equation]
for some constants c, C .
Proof.
Combine the results of lemmas F.2 and F.3 and take a union bound, using the triangle inequality to obtain the second two bounds.
The assumption n ≥ KL 36+8 q d 9 is required in showing (cid:13) (cid:13) β (cid:96) − 1 t ( x ) (cid:13) (cid:13) 2 ≤ C √ n .
F.
C HANGES IN Θ Nk D URING T RAINING
With these results in hand, control of the changes in Θ Nk during training is straightforward.
Lemma F.5 (Uniform control of changes in Θ during training) .
Denoting the gradient descent step size by τ , choose some k max such that
[Equation]
for some constant q .
Assume also that given some k ≤ k max − 1 , for all k (cid:48) ∈ { 0 , . . .
, k } ,
[Equation]
Deﬁne
[Equation]
Assume n ≥ KL 36+8 q d 9 , d ≥ K (cid:48)
d 0 log ( nn 0 C M ) for constants K, K (cid:48) .
Then on an event of probability at least 1 − e − cd
[Equation]
/for some constants c, C .
Proof.
Recall that
[Equation]
with the convention β Lt ( x ) = 1 for all t, x , and the parameters θ Nt given by (F.1).
We thus have
[Equation]
We consider a single summand in the above expression.
On the event deﬁned in lemma F.4, for all x , x (cid:48) ∈ M , (cid:96) ∈ { 0 , . . .
, L } ,
[Equation]
for some constants.
Summing this bound over (cid:96) gives the desired result
[Equation]
F. A UXILIARY L EMMAS AND
P ROOFS
Lemma F.6.
Consider a collection of n i.i.d.
variables X i
=
g 2 i , g i ∼ N (0 , 1 n ) and denote the order statistics by X ( i ) (so that X (1) ≤ X (2) . . . ).
For any d ≥ K (cid:48) log n , n ≥ K (cid:48)(cid:48)
d 3 , η > C d 9 / 8 n 3 /and integer Knη 2 / 3 ≤ k ≤ n , where K, K (cid:48) , K (cid:48)(cid:48) are appropriately chosen absolute constants, we have
[Equation]
where c, C, C (cid:48) are absolute constants.
Proof.
We will relate sums of order statistics of X i to functions of uniform order statistics and show that these concentrate.
We denote the CDF of the X i and its inverse by F and F − 1 respectively.
We use
[Equation]
where U ( i ) are order statistics with respect to Unif (0 , 1)
(David & Nagaraja, 2004).
Since X i ∼n Y i , Y i ∼ χ 2 1 we have
[Equation]
where in the inequality we used the series representation of erf − 1 .
This gives
[Equation]
The joint PDF of the ﬁrst k order statistics for any distribution admitting a density is given by
[Equation]
where x 1 ≤ x 2 ≤ ... ≤ x k (David & Nagaraja, 2004)
Applying this to the uniform order statistics, we can compute the mean of the summands
[Equation]
In order to show concentration, we appeal to the Rényi representation of order statistics (Boucheron et al., 2012).
This allows us to write k (cid:80) i =
1 U 2( i ) as a Lipschitz function of independent exponential random variables, and we can then apply standard concentration results for such functions (Talagrand, 1995).
This representation is due to a useful property unique to the exponential distribution whereby the differences between order statistics are independent exponentially distributed variables themselves when properly normalized.
If we deﬁne by E i , ..., E n a collection of independent standard exponential variables, the Rényi representation of the uniform order statistics gives
[Equation]
We now truncate the ( E 1 , ..., E k ) , so that w.p. P ≥ 1 − ke − K we have ∀ i : E i ∈ [0 , K ] , and denote this event by E K .
Using K < n 2 , it is evident that k (cid:80) i
=1 U 2( i ) is equal in distribution to a convex function of ( E 1 , ..., E k ) after truncation (which can be seen by calculating second derivatives).
The
Lipschitz constant of this function is bounded by 4 k n − k .
= λ .
If we deﬁne rescaled variables ˜ E i = λE i then with the same probability they take values in [0 , Kλ ] .
k (cid:80) i =
1 U 2( i ) written in terms of ˜ E
i is now 1-Lipschitz and convex, and we can apply Talagrand’s concentration inequality (Talagrand, 1995) to obtain
[Equation]
Setting t =
[Equation]
k
[Equation]
=
[Equation]
, if we now assume
[Equation]
for some c (cid:48) < 1 we obtain
[Equation]
We would also like to ensure that the truncation does not cause a large deviation in the mean.
We have
[Equation] [Equation]
Since we would like this to be small compared to E k (cid:80)
[Equation]
we can require K >
[Equation]
which gives
[Equation]
We can then choose the constant c 2 such that with probability
[Equation]
The upper bound on k can then be removed since the inequality then applies to all larger k automatically.
If we now set η according to equation (F.2), and choose K = d ≥ K (cid:48) log n and n satisfying n ≥ K (cid:48)(cid:48)
d 3 for appropriate constants K (cid:48) , K (cid:48)(cid:48) we obtain
P (cid:34) k (cid:88) i =
[Equation]
and due to our choice of η , this result holds for all k ≥ Cnη
[Equation]
Proof of lemma F.2.
i)
We begin by controlling the pre-activation norms.
Considering a point x ∈
N n − 3 n − 1 / 2, where N n − 3 n − 1 / 2is
the net deﬁned in Appendix D.3.1, rotational invariance of the
Gaussian distribution gives
[Equation]
where (cid:0) W (cid:96) 0 (cid:1) (: , 1) is the ﬁrst column of W (cid:96) 0 .
Bernstein’s inequality then gives
[Equation]
for appropriate constants.
As discussed in Lemma D.8, if we choose d to satisfy the requirements of this lemma then (cid:12)(cid:12)(cid:12) N n − 3 n − 1 /[-
-]
[Equation]
[Equation]
≤ e C (cid:48)(cid:48)
d for some constant.
We can then uniformize over the net using a union bound, obtaining
[Equation]
for some c (cid:48) , assuming n ≥
Kd .
We now need to control the feature norms and pre-activation norms off of the net.
From (D.62) and lemma G.10 we obtain that for d satisfying the requirements of lemma D.9,
[Equation]
By taking a union bound over the above two results
[Equation]
we obtain
[Equation]
F.
[Equation]
for some constants
We next turn to controlling the generalized backward features and transfer matrices.
Our ﬁrst task is to bound the number of support patterns that can be encountered, namely (cid:12)(cid:12) J η ( M ) (cid:12)(cid:12) .
In order to do this it will be convenient to introduce a set that contains J η ( M ) and is easier to reason about.
We deﬁne a metric between supports by
[Equation]
and denote by B s ( S, δ ) ⊂ P ([ n ]) a ball deﬁned with respect to this metric, where δ ∈ { 0 } ∪ [ n ] and
P ( A ) is the power set of a set A .
For δ η ( y ) and B ( y , η ) deﬁned in (F.8) and (F.4) respectively, it is clear that
[Equation]
and consequently
[Equation]
We will aim to control the volume of this set, which we will achieve by controlling it ﬁrst on a net.
This will require transferring control between different nearby points.
For any S, S (cid:48) ∈ [ n ] and δ ∈ { 0 } ∪ [ n ] , the triangle inequality implies
[Equation]
For some p ∈ B E ( g , r ) , we also have
[Equation]
p , g )
[Equation]
η +
[Equation]
g
[Equation]
(F
[Equation]
27) where we used B E ( p , η )
⊆
B E ( g , η + r ) .
It follows that
B s ( supp ( p > 0 ) , δ η ( p )) ⊆
B s ( supp ( g > 0 )
[Equation]
δ
[Equation]
p
[Equation]
p
[Equation]
g )
[Equation]
( supp ( g > 0 ) [Equation] δ η + [Equation]
( g
[Equation]
d
[Equation]
( p , g )
[Equation]
(F.28)
From (D.62) and lemma G.10 we obtain that for d satisfying the requirements of lemma D.8,
P 
[Equation] [Equation]
since under the assumptions of the lemma d s ( ρ (cid:96)
( x ) , ρ (cid:96)
( x )) ≤
L
[Equation]
= (cid:12)(cid:12) R (cid:96) (cid:0) x , Cn − 3 (cid:1)(cid:12)(cid:12) , with
R (cid:96) (cid:0) x , Cn − 3 (cid:1) denoting the number of risky features as deﬁned in section D.3.1.
We denote this event by E ρ .
On E ρ , we can transfer control of the ball of feature supports from a point on the net to any point on the manifold.
For some (cid:96), x we denote by x the point on the net that satisﬁes the above condition.
Considering (F.28), we choose g = ρ (cid:96) 0 ( x ) , p = ρ (cid:96)
( x ) , r = Cn − 5 / 2 and η =
C η L 3 / 2+ q n − 1 / 2 , obtaining
[Equation]
where we assumed C η L 3 / 2+ q n 2 > C .
We next turn to controlling δ 2 η ( ρ (cid:96) 0 ( x )) , which is now a random variable, for all (cid:96) ∈
[ L ] , x ∈
N n − 3 n − 1 / 2.
From lemma F.1 we have for a vector g with g
i ∼ iid N (0 , 1 n ) ,
[Equation]
Considering a vector ρ (cid:96)
( x ) for some (cid:96) ∈
[ L ] , x ∈
N n
[Equation]
3 n
[Equation]
[Equation]
, we have ρ (cid:96)
( x ) ∼
N (0 , 2 (cid:13)(cid:13) α (cid:96) − 1( x ) (cid:13)(cid:13) 2 2 n − 1 ) .
Lemma D.2 then gives
[Equation]
for some constants, assuming d > K log L for some K .
Since on the complement of this event we have √ 2 η
[Equation]
( x ) (cid:13)(cid:13)
[Equation]
1 2 ≤ 2 η , lemma F.1 and a rescaling gives
[Equation]
(F
[Equation]
31) for some constants.
Taking a union bound over N n − 3 n −
1 / 2and [ L ] we obtain
P
[Equation]
[Equation]
32) under the same assumptions on d as in lemma D.8, and additionally assuming d ≥ K log L for some
K .
Since n, d, η satisfy the assumptions of lemma F.1, we have nη 2 / 3 ≥ C (cid:48) n 1 / 2 d 3 / 4 ≥ C (cid:48)
d for some
C (cid:48) and hence
[Equation]
for some constants c, C, C 1 .
Denoting the complement of above event by E Nδ , we ﬁnd that on
E ρ ∩ E
Nδ , for every x
we can ﬁnd x ∈ N n − 3 n − 1 / 2∩
[Equation]
( x ) such that
[Equation]
where we used (F.30).
On E ρ ∩ E
Nδ , we can thus bound the size of the set that contains J η , denoting its size by
[Equation]
We ﬁrst note that for any p ,
[Equation]
for appropriate constants, assuming nη 2 / 3 > K log( nη 2 / 3 ) for some K .
It follows that on E ρ ∩ E
Nδ ,
[Equation]
for appropriate constants, since nη 2 / 3 ≥ C (cid:48)(cid:48)(cid:48)
d and d satisﬁes the assumptions of lemma D.8.
Since after worsening constants we have P (cid:2)
E
ρ ∩ E Nδ (cid:3) ≤ C (cid:48)
e − cd , we obtain
[Equation]
F.34)
We would next like to employ lemma D.14 in order to control the quantities of interest for a single
J ∈ J η ( M ) , and then take a union bound utilizing the upper bound above on (cid:12)(cid:12) J η (cid:12)(cid:12) .
This will require controlling the event E δK in the lemma statement with an appropriate choice of the constants δ s , K s .
As in other sections, we use the convention Γ (cid:96) : (cid:96) +1 J= I for any (cid:96) ∈
[ L ] .
At a given collection of supports J ∈ J η ( x ) for some x ∈ M , we choose x as the anchor point in lemma D.14.
From (F.27) we have, for any x ∈ N n
[Equation] [Equation]
Then using (F.29) we obtain to bound the two terms in the RHS gives
[Equation]
≥
[Equation]
where we used η = C η L 3 / 2+ q n − 1 / 2 and d satisﬁes the requirements of lemma D.8.
Using (F.33) to bound d + δ 2 η
( ρ (cid:96)
( x )) uniformly on N n − 3 n − 1 / 2and (cid:96) , and combining the failure probabilities of these events by a union bound, we obtain
[Equation]
for some constants.
Since δ η ( ρ (cid:96)
( x )) ≥ | J (cid:96) (cid:9) I (cid:96), 0 ( x ) | for any J (cid:96)
∈ J ∈
J η ( x ) , implies directly that
[Equation]
In the notation of lemma D.14 we denote this event by E δ , and choose δ s = C 1 nη 2 / 3 .
From the deﬁnition of J η , for every x ∈ M and J (cid:96) that is an element of J ∈ J η ( x ) ,
[Equation]
for some v such that (cid:107) v (cid:107) 2 ≤ η .
We now consider the vector
[Equation]
Note that for any element of w i that is non-zero, we must have | v i | ≥ (cid:12)(cid:12) ρ (cid:96)
( x ) i (cid:12)(cid:12) (since the perturbation must change the sign of this element), in which case we have | w i | = (cid:12)(cid:12) ρ (cid:96)
( x ) i (cid:12)(cid:12) .
Denoting the set of indices of these non-zero elements by Q , we have
[Equation]
This holds for all (cid:96) ∈
[ L ] .
Thus if we set K s = η for K s , the event E K in lemma D.14 holds with probability 1.
We therefore choose
E δK =
E δ with E δ deﬁned in (F.35).
In order to apply D.14 we must also ensure
[Equation]
Setting η
=
C
η
L 3 / 2+ q √ n as per (F.2), we can satisfy these requirements by demanding n ≥
C 3 0 C 2 η L 6+2 q .
We are now in a position to apply lemma D.14 to control the objects of interest.
We use rotational invariance of the Gaussian distribution repeatedly to obtain
[Equation]
[Table]
Recalling that W L +1 i ∼ N (0 , 1) , we use Bernstein’s inequality to obtain P (cid:2)(cid:13)(cid:13) W L +1 (cid:13)(cid:13) 2 > C √ n [Equation] ≤
C (cid:48)
e − cn , and another application of lemma D.14 gives P (cid:2) 1 E δK
[Equation]
Γ L :
[Equation]
+
[Equation]
Je
[Equation]
>
[Equation]
for some constants.
Hence after worsening constants
[Equation]
We also obtain
[Equation]
where we used an ε -net argument to bound (cid:13)(cid:13) W (cid:96)
(cid:13)(cid:13) and lemma D.14 to bound E δK (cid:13)(cid:13)(cid:13) Γ (cid:96) − 1: (cid:96) (cid:48) − 1 J ,(cid:13)(cid:13)(cid:13) .
We now combine this result with (F.36).
It remains to uniformize this result over the choice of (cid:96) and J .
Combining (F.26) and (F.34) gives
[Equation]
Denoting the complement of this event by E J , and setting η = C η L 3 / 2+ q √ n , on this event we have
P
[Equation]
∀J
[Equation]
M
[Equation]
[Equation] < (cid:96) ∈
[ L ] , :
[Equation]
assuming n ≥ KL 9+2 q d for some constant K .
Taking a union bound over the probabilities of E J or E Kδ not holding
[Equation]
we ﬁnally obtain
[Equation]
for appropriate constants, where we used ( F.35) to bound P [ E cδK ] , and in the last inequality we used n ≥ KLd for some K .
Combining this with (F.25) and taking a union bound gives the desired result.
ii)
We will control (cid:13)(cid:13)(cid:13) β (cid:96) I 0 ( x ) , 0 − β (cid:96) I t ( x ) , 0 (cid:13)(cid:13)(cid:13) 2 using lemma D.21.
For t ∈ [0 , T η ] we note that by deﬁnition of T η and J η ( x ) , [Equation] x [Equation]
η
[Equation]
As noted in the previous section, if we set d to satisfy lemma D.8 and n ≥ KdL 9+2 q for some K , then the requirements of lemma D.14 are satisﬁed with
[Equation]
and
[Equation]
where the last bound uses the deﬁnition of E δ in (F.35).
From the deﬁnition of d in lemma D.21, on the event E δK we have
[Equation]
Thus for some ﬁxed t ∈ [0 , T η ] , if we denote d i = | I i,t ( x ) (cid:9)
I i, 0 ( x ) | for i ∈
[ L ] , we can apply the second result of lemma D.21, choosing
[Equation]
for some appropriately chosen constants K, K (cid:48) , K (cid:48)(cid:48)(cid:48) .
Assuming n ≥ dL 2 and n 1 / 12 √ L 3+2 q/ 3 d 1 / 4 ≥ ˜ K for some constant ˜ K to simplify the result, we obtain
[Equation]
The constants K, K (cid:48) are chosen such that this result can be uniformized over the set of possible supports (cid:12)(cid:12) J η ( M ) (cid:12)(cid:12) and [ L ] .
Since on the event E J deﬁned in (F.37) the size of this set is bounded, we have
[Equation]
t ∈ [0 , T [Equation]
[Equation]
∈
[ L ] :
[Equation]
for some constant c, C, C (cid:48) , assuming n ≥ K (cid:48)(cid:48)(cid:48)(cid:48)
L 9+2 q d for some constant K (cid:48)(cid:48)(cid:48)(cid:48) .
Taking a union bound over the complements of E J and E δK using (F.37) and (F.35), we have [Equation] [Table]
[Equation]
for appropriate constants, assuming log( L ) L 2+2 q/ 3 n 2 / 3 > ˜ K for some constant ˜ K .
G A UXILIARY R ESULTS
Lemma G.1 (Hoeffding’s Inequality (Vershynin, 2018, Theorem 2.2.6)) .
Let X 1 , . . .
X N be independent random variables.
Assume that X i ∈ [ m i , M i ] for every i .
Then for any t > 0 , we have
[Equation]
Lemma G.2 (Bernstein’s inequality (Vershynin, 2018, Theorem 2.8.1)) .
Let X 1 , . . .
X N be independent mean-zero subexponential random variables.
Then, for every t ≥ 0 , one has
[Equation]
where c > 0 is an absolute constant, and (cid:107) · (cid:107) ψ 1 = inf { t > 0 | E [ e | · | /t ] ≤ 2 } is the subexponential norm.
Lemma G.3 (Bernstein’s inequality for bounded RVs - (Vershynin, 2018)
Thm. 2.8.4) .
For
X 1 , ..., X n independent, zero mean random variables such that ∀ i : | X i | < K , and every t ≥ 0 , we have
[Equation]
where σ 2 = n (cid:80) i =
E X 2 i .
Lemma G.4 (Hanson-Wright Inequality (Vershynin, 2018, Theorem 6.2.1)) .
Let g be a vector of n i.i.d., mean zero, sub-Gaussian variables and A be an n × n matrix.
Then for any t > 0 , we have
[Equation]
where max i (cid:107) g
i (cid:107) ψ 2 ≤ K (with (cid:107)·(cid:107) ψ 2 denoting the sub-Gaussian norm).
Lemma G.5 (Freedman’s Inequality (Freedman, 1975, Theorem 1.6)) .
Let (∆ i , F i ) be a sequence of martingale differences, with
[Equation]
and suppose that | ∆
[Equation]
≤ R a.s. .
Deﬁne the quadratic variation
[Equation]
Then
[Equation]
Lemma G.6 (Moment control Freedman’s (de la Peña, 1999)) .
Let (∆ i , F i ) be a sequence of martingale differences, with
[Equation]
and suppose that
[Equation]
Set
[Equation]
Then
P
[Equation]
[Equation]
Lemma G.7 (Martingales with subgaussian increments) .
Let (∆ i , F i ) be a sequence of martingale differences, and suppose that
[Equation]
Then
[Equation]
Proof.
By assumption, E [∆ i ] =
for each i ∈
[ L ] .
We calculate using standard properties of the conditional expectation
[Equation]
Moreover, one has E [ e λ ∆ 1 | F 0 ] =
E [ e λ ∆ 1 ] ≤ e λ 2 V 2 / 2 .
An induction therefore implies
[Equation]
and the result follows from standard equivalence properties of subgaussian random variables (Vershynin, 2018, Proposition 2.5.2).
Lemma G.8 (Azuma-Hoeffding Inequality (Azuma, 1967)) .
Let (∆ i , F i ) be a sequence of martingale differences, and suppose that
[Equation]
Then
[Equation]
Lemma G.9 (Chi and Inverse-Chi Expectations) .
Let X ∼ χ ( n ) be a chi random variable with n degrees of freedom, equal to the square root of the sum of n independent and identically distributed squared N (0 , 1) random variables.
Then
[Equation]
and, if n ≥ 2 ,
[Equation]
Proof.
We use the fact that the density of X is given by
[Equation]
which can be proved easily using the Gaussian law and a transformation to spherical polar coordinates (Muirhead, 1982, Theorem 2.1.3).
The expectation of X then results from a simple sequence of calculations using the change of variables formula:
[Equation]
Now we study X − 1 .
By the change of variables formula, its density is given by
[Equation]
A similar sequence of calculations then yields
[Equation]
provided n > 1 .
Lemma G.10 (Equivalence of (cid:96) p Norms) .
Let 1 ≤ p ≤ q ≤ + ∞ .
Then for every x ∈ R n one has
[Equation]
Lemma G.11 (Gaussian Moments) .
Let p ≥ 1 , and let g ∼ N (0 , 1) be a standard normal random variable.
Then
[Equation]
where [ x ] + = max { x, 0 } .
In particular E [ | g | p ] ≤ p p/ 2 , so that g is subgaussian and g 2 is subexponential.
Lemma G.12 ((Zhu, 1997, Lemma 3.6)) .
If M is a d -dimensional manifold with Ric ≥ ( d − 1) and r, ε > 0 are arbitrary constants, then for any p ∈ M there exists an ε -net (measured in the
Riemannian distance dist M ) of the metric ball { x ∈ M
| dist M ( p , x ) < r } with cardinality at most ψ (2 r ) /ψ
( ε/ 4) , where ψ : R + → R
+ gives the volume of a cap in S d having radius equal to its argument.