On E (cid:96) : (cid:96) (cid:48) B , the bound on (cid:12)(cid:12) ϕ (cid:96) − 1 ( ν ) − ν (cid:96) − 1 (cid:12)(cid:12) and lemma C.10 give ν (cid:96) − 1 ≤ C(cid:96)
a.s.. Additionally, on this event max i,j ∈ [ n ] (cid:12)(cid:12)(cid:12) ˜ B (cid:96) − 1: (cid:96) (cid:48) xx (cid:48) ij (cid:12)(cid:12)(cid:12) ≤
(cid:13)(cid:13)(cid:13) B (cid:96) − 1: (cid:96) (cid:48) xx (cid:48) (cid:13)(cid:13)(cid:13) ≤ C(cid:96) a.s.. It follows that (cid:12)(cid:12)(cid:12)(cid:12) 1 E (cid:96) : (cid:96) (cid:48) B
E W (cid:96) (cid:20) tr (cid:104) B (cid:96) : (cid:96) (cid:48) xx (cid:48) (cid:105) − (cid:18) 1 − ϕ (cid:96) − 1 ( ν ) π (cid:19) tr (cid:104) B (cid:96) − 1: (cid:96) (cid:48) xx (cid:48) (cid:105)(cid:21)(cid:12)(cid:12)(cid:12)(cid:12) (D.38) ≤ C(cid:115)
d 3 n log 3 n (cid:96) + 2 Cπ
+
Cπ(cid:96) ≤ C (cid:48) (cid:115)
[Equation]
D
[Equation] [Equation]
almost surely, and hence restoring a constant factor with magnitude bounded by 1, we have
[Equation]
Using lemma D.28 to bound P (cid:104)(cid:16) E
[Equation]
c (cid:105) from above then gives
[Equation]
An application of the triangle inequality and union bound then give
[Equation]
for some constants c, C
[Equation]
We proceed to control the remaining terms in (D.35), namely
L (cid:80) (cid:96) = (cid:96)
[Equation]
+2 ∆ (cid:96) .
Aiming to apply martingale concentration, we require an almost sure bound on the summands, which we achieve by truncation.
Towards this end [Equation] we deﬁne an event
[Equation]
Combining (D.43) and the result of lemma D.29 (after taking an expectation) we have
[Equation]
for appropriate constants.
We now decompose the sum that we would like to bound:
[Equation]
Since each summand in Σ 1 are equal to zero on the respective truncation event [Equation] a union bound and (D.45) give
[Equation]
for some constants.
The term Σ 2 is a sum of almost surely bounded martingale differences.
We can apply the Azuma-Hoeffding inequality (lemma G.8) directly to conclude
[Equation]
Considering a single summand in Σ
[Equation]
Jensen’s inequality and the Cauchy-Schwarz inequality give
[Equation]
This is an F (cid:96) − 1 -measurable function
[Equation]
and we can show that it is small on the event E (cid:96)
[Equation]
To control the ﬁrst factor
[Equation]
we note that
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
where to obtain the second to last line we used the deﬁnition of ∆ (cid:96) , then used Lemma D.29 and (D.40) to bound the ﬁrst and second term almost surely.
We proceed to control the second factor in (D.50), by bounding
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]

[Table]
[Equation]

[Equation]
Using (D.38) and (D.148) to bound the integrand above, we have
[Equation]
for appropriate constants.
Combining (D.51) and the above bound gives
[Equation]
for some c, c (cid:48) , C, C (cid:48) , and using lemma D.
[Equation]
for appropriate constants.
An application of the triangle inequality and a union bound (and introducing some slack to simplify the expression) then gives
[Equation]
for some constants.
Combining this bound with (D.47) and (D.48) gives
[Equation]
where in the last inequality we assumed K log L ≤ d .
Combining this with (D.44), we obtain
[Equation]
for appropriate constants.
This bound all the terms in the sum (D.32) aside from the ﬁrst one.
The ﬁrst term is bounded in (D.36), (D.37), and the ﬂuctuations due to the last layer are bounded in (D.31). Combining all of these gives
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
after worsening constants.
A ﬁnal union bound over (cid:96) (cid:48) and assuming d ≥ K log L gives
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
for appropriately chosen c, C (cid:48) , K .
If we additionally assume n > L we obtain the desired result.
D. U NIFORMIZATION E STIMATES D.3. N ETS AND C OVERING N UMBERS
We appeal to Lemma C.4 to obtain estimates for the covering number of M , which we will use throughout this section.
In the remainder of this section, we will use the notation N ε to denote the ε -nets for M constructed in Lemma C.4, and for any ¯ x ∈ N ε , we will also use the notation
N ε (¯ x ) =
B (¯ x , ε ) ∩M (cid:3) , where (cid:3) ∈ { + , −} is the component of ¯ x , to denote the relevant connected neighborhood of the speciﬁc point in the net we are considering.
Here we are implicitly assuming that M ± are themselves connected, but this construction evidently generalizes to cases where M ± themselves have a positive number of connected components, as treated in Lemma C.4.
Focusing on this simpler case in the sequel will allow us to keep our notation concise.
D.3. C ONTROLLING S UPPORT C HANGES U NIFORMLY
The quantities we have studied in Appendix D.2 are challenging to uniformize due to discontinuities in the support projections
P I (cid:96) ( · ) .
We will get around this difﬁculty by carefully tracking (with high probability) how much the supports can change by when we move away from the points in our net
N ε .
It seems intuitively obvious that when ε is exponentially small in all problem parameters, there should be almost no support changes when moving away from our net; the challenge is to show that this property also holds when ε is not so relatively small.
Introduce the following notation for the network preactivations at level (cid:96) , where (cid:96) ∈
[ L ] : [Equation]
so that α (cid:96) ( x ) =
[ ρ (cid:96) ( x )]
+ .
We also let F (cid:96) denote the σ -algebra generated by all weight matrices up to level (cid:96) in the network, and let F 0 denote the trivial σ -algebra.
Deﬁnition D.1.
Let ε, ∆ > 0 , and let ¯ x ∈ N ε .
For (cid:96) ∈
[ L ] , a feature ( α (cid:96) (¯ x ))
i is called ∆ -risky if | ( ρ (cid:96) (¯ x ))
i | ≤ ∆ ; otherwise, it is called ∆ -stable .
If for all x ∈ N ε (¯ x ) we have
[Equation]
we say that stable sign consistency holds up to layer (cid:96) .
We abbreviate this condition as SSC ( (cid:96), ε, ∆) at ¯ x , with the dependence on ¯ x , ε , and ∆ suppressed when it is clear from context.
If SSC ( (cid:96) ) holds at ¯ x
and if ( α (cid:96) (cid:48) (¯ x ))
i is stable, we can write for any x ∈ N ε (¯ x )
[Equation]
so that no stable feature supports change on N ε (¯ x ) , and we only need to consider changes due to the risky features.
Moreover, observe that
[Equation]
where g ∼ N ( 0 , (2 /n ) I ) is independent of everything else in the problem, since ∆ > 0 .
It follows that when considering the network features over any countable collection of points ¯ x ∈ M , we have almost surely that the risky features are witnessed in the interior of [ − ∆ , +∆] .
Below, we will show that with appropriate choices of ε and ∆ , with very high probability: (i) each point in the net ¯ x has very few risky features; and (ii) SSC ( L ) holds uniformly over the net under reasonable conditions involving n, L, d .
We write R (cid:96) (¯ x , ∆) ⊂
[ n ] for the random variable consisting of the set of indices of ∆ -risky features at level (cid:96) with input ¯ x ∈ N ε .
Lemma D.5.
There is an absolute constant K > 0 such that for any ¯ x ∈ M and any d > 0 , if n ≥ max { KdL, 4 } and ∆ ≤ d log n/ (6 n 3 / 2 L ) , then one has
[Equation]
Proof.
For any ¯ x ∈ N ε , Lemma D.2 (with a suitable choice of d in that context) gives
[Equation]
so that if additionally n ≥ (2 /c )
(cid:96) log( C ) , one has
[Equation]
Let G (cid:96) =
{ 1 / 2 ≤ (cid:107) α (cid:96) (¯ x ) (cid:107) 2 ≤ 2 } , so that G (cid:96) is F (cid:96) -measurable, and G = ∩ (cid:96) ∈
[ L − 1] G (cid:96) ; then by (D.53) and a union bound, we have P [ G ] ≥ 1 − L 2 e − cn/L .
We also let G 0 = ∅ c .
For i ∈ [ n ] and (cid:96) ∈
[ L ] , consider the random variables X i(cid:96)
=
| ( ρ (cid:96) (¯ x ))
i | , and moreover deﬁne
[Equation]
We have (cid:80) i,(cid:96) 1 X i(cid:96) ≤ ∆ =
(cid:80) (cid:96) | R (cid:96) (¯ x ) | , which is the total number of ∆ -risky features at ¯ x , and the corresponding sum with the random variables ˜ X i(cid:96) is thus an upper bound on the number of risky features at ¯ x .
Notice that X i(cid:96) and ˜ X i(cid:96) are F (cid:96) -measurable, and additionally notice that on G , we have X i(cid:96) / 2 ≤ ˜ X i(cid:96) ≤ 2 X i(cid:96) .
For any K ∈ { 0 , 1 , . . . , nL − 1 , nL } , we have by disjointness of the events in the union and a partition
[Equation]
so it is essentially equivalent to consider the ˜ X i(cid:96) .
By another partitioning we can write
P 
[Equation]
 [Equation] where { 0 , 1 } n × L is the set of n × L matrices with entries in { 0 , 1 } .
Using the tower rule and
F L − 1 -measurability of all factors with (cid:96) <
L
[Equation]
we can then write
[Equation]
We study the inner conditional expectation as follows: because ρ L (¯ x ) =
W L α L − 1 (¯ x ) , we can apply rotational invariance in the conditional expectation to obtain
[Equation]
where w L ∼ N ( 0 , (2 /n ) I ) is the ﬁrst column of W L , and the last equality takes advantage of the presence of the indicator for 1 G (cid:96) − 1 multiplying the conditional expectation.
We then write using independence
[Equation]
and putting p L = P [ | ( w L ) 1 | ≤ 2∆] , we have by identically-distributedness n
[Equation]
After removing the indicator for G L − 1 by nonnegativity of all factors in the expectation, this leaves us with
E
[Equation]
This process can evidently be iterated L − 1 additional times with analogous deﬁnitions—we observe that the fact that all weight matrices W (cid:96) have the same column distribution implies that p 1 = · · · = p L , so we write p = p 1 henceforth—and by this we obtain
[Equation]
and in particular
[Equation]
For i ∈ [ n ] and (cid:96) ∈
[ L ] , let Y i(cid:96)
denote nL i.i.d.
Bern( p ) random variables; we recognize this last sum as the probability that (cid:80) i,(cid:96) Y i(cid:96)
=
k .
In particular, using our previous work we can assert for any t >
[Equation]
so to conclude it sufﬁces to articulate some binomial tail probabilities and estimate p .
We have
[Equation]
D
[Equation]
54) and we can write with the triangle inequality and a union bound
P
[Equation]
i,
[Equation]
E [ Y i(cid:96) ] > t   .
By (D.54), we have (cid:80) i,(cid:96) E
[ Y i(cid:96) ] ≤ n 3 / 2 L ∆ .
We calculate using independence
E
[Equation]
so an application of Lemma G.3 yields
[Equation]
For any d > 0 , if we choose t = d log n and enforce ∆ ≤ d log n/ (6 n 3 / 2 L ) , we obtain
[Equation]
from which we conclude as sought
[Equation]
The next task is to study the stable sign condition at a point ¯ x as a function of ε and ∆ , assuming ∆ at least satisﬁes the hypotheses of Lemma D.5.
In particular, we will be interested in conditions under which we can guarantee that SSC ( (cid:96) − 1) holding implies that SSC ( (cid:96) ) holds.
Let S (cid:96) (¯ x , ∆) =
[ n ] \ R (cid:96) (¯ x , ∆) denote the ∆ -stable features at level (cid:96) with input ¯ x , and deﬁne for 0 ≤ (cid:96) (cid:48) ≤ (cid:96)
≤ L
[Equation]
so that Φ (cid:96),(cid:96) (cid:48) x x carries an input x ∈ N ε (¯ x ) applied at the features at level (cid:96) (cid:48) (in particular, (cid:96) (cid:48)
=corresponds to α 0 ( x ) = x , the network input) to the preactivations at level (cid:96) in a network restricted to only the stable features at ¯ x .
We can write
[Equation]
which gives us a useful representation if we disregard all levels with no risky features: let r = (cid:80) L(cid:96)
[Equation]
x [Equation] | > 0 be the number of levels in the network with risky features, and let (cid:96) 1 < (cid:96) 2 < · · · < (cid:96) r denote the levels at which risky features occur.
If no risky features occur at a level (cid:96) , we of course have P S (cid:96) (¯ x )
=
I .
Assume to begin that (cid:96) > (cid:96) r , and start by writing
[Equation]
Now we distribute from left to right, and recombine everything to right on the term corresponding to the projection onto the risky features at (cid:96) r ; this gives
[Equation]
We can write
[Equation]
where the restriction notation emphasizes that we are considering a column submatrix of the transfer operator induced by the risky features.
Iterating the previous argument, we obtain
[Equation]
It is clear that an analogous argument can be used in the case where (cid:96) ≤ (cid:96) r by adapting which risky features can be visited: we can thus assert
[Equation]
Furthermore, we note that under SSC ( (cid:96) − 1) , no stable feature supports change on N ε (¯ x ) , and so one has for every x ∈ N ε (¯ x )
[Equation]
so under SSC ( (cid:96) − 1) we have by (D.56)
[Equation]
The ReLU [ · ] + is 1 -Lipschitz with respect to (cid:107)·(cid:107) ∞ , and by monotonicity of the max under restriction and SSC ( (cid:96) − 1) we have
[Equation]
Thus, by the triangle inequality, we have under SSC (
[Equation]
− 1) a bound
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
This suggests an inductive approach to establishing SSC ( (cid:96) ) provided we have established it at previous layers—we just need to control the transfer coefﬁcients in (D.57).
Lemma D.6.
There are absolute constants c, c (cid:48) , C, C (cid:48) , C (cid:48)(cid:48) , C (cid:48)(cid:48)(cid:48) > 0 and absolute constants
K, K (cid:48) > 0 such that for any 1 ≤ (cid:96) (cid:48) <
(cid:96) ≤ L , any d ≥ K log n and any ¯ x ∈ S n 0 − 1 , if ∆ ≤ cn − 5 /and n ≥ K (cid:48) max { d 4 L, 1 } , then one has
[Equation]
[Table]
[Equation]
and for any ﬁxed S ⊂ [ n ] , one has
[Equation]
Proof.
We will use Lemmas D.14 and D.23 to bound the transfer coefﬁcients, so let us ﬁrst verify the hypotheses of these lemmas.
In our setting, the transfer matrices differ only from the ‘nominal’ transfer matrices by restriction to the stable features at ¯ x ; we have S (cid:96) (¯ x ) ∩ I [Equation]
(¯ x ) = [ n ] \ R (cid:96) (¯ x ) , which is an admissible support random variable for Lemmas D.14 and D.23, and in particular
[Equation]
by Lemma G.10 and the deﬁnition of R (cid:96) (¯ x ) .
Additionally, using Lemma D.5, we have if d ≥ 1 , n ≥
KdL , and ∆ ≤ cd/n 5 / 2 , there is an event E with measure at least 1 − 2 e − d − L 2 e − cn/L on which there are no more than d risky features at ¯ x .
Worsening constants in the scalings of n if necessary and requiring moreover d ≥ K (cid:48) log n and n ≥ K (cid:48)(cid:48)
d 4 , it follows that we can invoke Lemmas D. and D.23 to bound the probability of events involving transfer coefﬁcients multiplied by 1 E .
Let us also check the residuals we will obtain when applying Lemma {+D.23: in the notation there, the vector d has as its (cid:96) -th entry R (cid:96) (¯ x ) , and so we have bounds (cid:107) d (cid:107) 1 / 2 ≤ (cid:107) d (cid:107) 21 and (cid:107) d (cid:107) 
 = (cid:80) (cid:96) R (cid:96) (¯ x ) ,+} which means on the event E , the residual is dominated by the C √ d 4 nL term in the scalings we have assumed.
The (cid:96) 2 → (cid:96) ∞ operator norm of a matrix is the maximum (cid:96) 2 norm of a row of the matrix, and the (cid:96) ∞
→ (cid:96) ∞ operator norm is the maximum (cid:96) 1 norm of a row.
Thus
[Equation]
where ( W (cid:96) ) ∗ i is the i -th row of W (cid:96) , which is n 0 -dimensional when (cid:96)
= 1 and n -dimensional otherwise.
In particular, we have
[Equation]
and so taking a square root and applying Lemma G.2 and independence of the rows of W 1 , we have
[Equation]
for c > 0 an absolute constant.
When (cid:96) > 1 , we can write
[Equation]
[Equation]
where the second line applies Cauchy-Schwarz.
Using, say, rotational invariance, Gauss-Lipschitz concentration, and Lemma E.48 (or (Vershynin, 2018, Theorem 4.4.5)), we have
[Equation]
for absolute constants c, C > 0 .
On the other hand, note that (cid:107) [Equation] W [Equation]
[Equation] P I [Equation] x )
[Equation]
2 has the same distribution as the square root of one of the index- 0 diagonal terms studied in Lemma D.23 in a network truncated at level (cid:96) − 1 instead of L and scaled by 2 /n ; and so applying this result together with a union bound and the choice n ≥ max { K, K (cid:48)
d 4 L } for absolute constants K, K (cid:48) > 0 gives
[Equation]
where C (cid:48) , c (cid:48) , C (cid:48)(cid:48) > 0 are absolute constants.
We conclude by another union bound
[Equation]
We can reduce the study of the partial risky propagation coefﬁcients to a similar calculation.
We have
[Equation]
where by construction we have that (cid:96) > (cid:96) (cid:48) .
In the case (cid:96) =
(cid:96) (cid:48)
+ 1 , the form is slightly different; we have
[Equation]
where the inequality uses Lemma G.10.
The classical estimate for the gaussian tail gives
[Equation]
> (cid:114)
[Equation]
n (cid:35)
[Equation]
(D.58) for each k ∈ [ n ] , so a union bound gives P
[Equation]
(cid:114)
[Equation]
n (cid:35) ≤ 2 ne − d , and we conclude
[Equation]
S (cid:13)(cid:13)(cid:13)
[Equation]
∞ →
[Equation]
≤ | S | (cid:114) 2 d n (cid:35) ≥ 1 − 2 e − d/ 2 , where the ﬁnal bound holds if d ≥ 2 log n .
Next, we assume (cid:96) > (cid:96) (cid:48)
+ 1 .
In this case, Lemma G. gives
[Equation]
[Table] [Table]
[Table]
[Equation]
where g ∼ N ( 0 , (2 /n ) I ) is independent of everything else in the problem.
We have by
Lemma D.14 based on our previous choices of n and d
[Equation]
and (D.58) applied to g controls the remaining term.
Taking a union bound over [ n ] in these two estimates and partitioning with E [Equation] we conclude P
[Equation]
and thus
[Equation]
where the last bound holds if d ≥ 2 log n .
Choosing n ≥ KL log n for a suitable absolute constant
K > 0 allows us to simplify the residual terms in the probability bounds to the forms we have claimed.
Lemma D.7.
There is an absolute constant c > 0 and absolute constants k, k (cid:48) , K, K (cid:48) > 0 such that for any d ≥ K log n ,
if n ≥ K (cid:48) max { d 4 L, 1 } , ∆ ≤ kn − 5 / 2 , and ε ≤ k (cid:48) ∆ (cid:0) 1 + (cid:112) n 0 n (cid:1) − 1 , then one has for any ¯ x ∈ N ε
[Equation]
Proof.
We start by constructing a high-probability event on which we have control of every possible propagation coefﬁcient.
For any d ≥ K log n , choosing ∆ ≤ cn − 5 / 2 and n ≥ K (cid:48)
d 4 L and applying the ﬁrst conclusion in Lemma D.6 and a union bound, we have
[Equation]
and under the same hypotheses, for any 1 ≤ (cid:96) (cid:48) < (cid:96) ≤ L and any S ⊂ [ n ] , the second conclusion in
Lemma D.6 gives
[Equation]
Φ (cid:96),(cid:96) (cid:48)
¯ x (cid:12)(cid:12)(cid:12) S (cid:13)(cid:13)(cid:13) (cid:96) ∞
→ (cid:96) ∞ > C 2 | S | (cid:114) d n (cid:35) ≤ C (cid:48)
e − 16 d .
Using Lemma D.5, we have if n ≥ max { KdL, 4 } and ∆ ≤ K (cid:48) /n
P (cid:34) L (cid:88) (cid:96) = | R (cid:96) (¯ x , ∆) | > d (cid:35) ≤ 2 e − d + L 2 e − cn/L .
(D.60)
Denote the complement of the event in the previous bound as E .
On E , there are no more than d levels in the network with risky features.
There are at most (cid:80) (cid:100) d (cid:101) k =0 (cid:0) nk (cid:1) ways to choose a subset
S ⊂ [ n ] with cardinality at most (cid:100) d (cid:101) ; using n ≥ e and d ≥ 1 , we have (cid:100) d (cid:101) (cid:88) k
= (cid:18) n k (cid:19) ≤ 1 + (cid:100) d (cid:101) n (cid:100) d (cid:101) ≤ 4 dn 2 d .
In addition, there are at most L 2 ways to pick two indices 1 ≤ (cid:96) (cid:48) <
(cid:96) ≤ L .
Using n ≥ L , this yields at most 4 dn 2+2 d ≤ n 8 d items to union bound over, i.e.,
[Equation]
if d ≥ K log n and n ≥ max { K (cid:48)
d 4 L, n 0 } .
Denote the complement of the union of the events in the bounds (D.59) to (D.61) and E as G ; taking additional union bounds and worst-casing absolute constants, we have shown P
[Equation]
If we enumerate the levels of the network that have risky features as 1 ≤ (cid:96) 1 < · · · < (cid:96) r ≤ L , it follows from our previous counting argument that on G , we have transfer coefﬁcient bounds (for any (cid:96) ∈
[ L ] and any (cid:96) i < (cid:96) )
[Equation]
Now we begin the induction.
Let x ∈ N ε (¯ x ) .
For SSC (1) , we have from the deﬁnitions
[Equation]
where the last inequality holds on G .
So we have SSC (1) on G if ε ≤ ∆( C 1 (1 + (cid:112) n 0 n )) − 1 .
Continuing, we suppose that we have established SSC ( (cid:96) − 1) on G .
We can therefore apply (D.57) together with our transfer coefﬁcient bounds to get
[Equation]
Notice that the last bound does not depend on (cid:96) .
Thus, if we choose ε ≤ ∆(2 C 1 (1 + (cid:112) n 0 n )) − 1 and n ≥ 4 C 22 d 3 , we obtain (cid:13)(cid:13) ρ (cid:96) ( x ) − ρ (cid:96) (¯ x ) (cid:13)(cid:13) ∞ ≤ ∆ ; by induction, we can conclude that SSC ( L ) holds on G , which implies the claim; we obtain the ﬁnal simpliﬁed probability bound by worsening the constant in the exponent.
D.3. U NIFORMIZING F ORWARD F EATURES U NDER SSC
Under the SSC ( L ) condition, we can uniformize forward and backward features.
A prerequisite of our approach, which we also used to establish SSC ( L ) in the previous section, is control of certain residuals that appear when a small number of supports can change off the nominal forward and backward correlations.
These estimates are studied in the next section, Appendix D.3.4.
In previous sections, most of our results (e.g. Lemma D.1) feature a lower bound of the type n ≥ K in their hypotheses.
After uniformizing, we will discard this hypothesis using our extra assumption that n 0 ≥ 3 , which gives us a lower bound on the logarithmic terms of the form log( Cnn 0 ) that appear as lower bounds on d after uniformizing, and the fact that our lower bounds on n always involve a polynomial in d .
Thus, by adjusting absolute constants, we can achieve the same effect as previously.
Lemma D.8.
There are absolute constants c, C > 0 and an absolute constant K, K (cid:48) > 0 such that for any d ≥ Kd 0 log( nn 0 C M ) , if n ≥ K (cid:48)
d 4 L then one has
P   (cid:92) ¯ x
[Equation]
N n − 3 n − 1 / 2(cid:110) SSC
( L, n − 3 n − 1 / 2, Cn − 3 ) holds at ¯ x (cid:111) ∩ (cid:40) L (cid:88) (cid:96) =
| R (cid:96) (¯ x , Cn − 3 ) | ≤ d (cid:41)
[Equation]
Proof.
Following the discussion in Appendix D.3.1, if 0 < ε ≤ 1 then | N ε | ≤ e d 0 log( C M /ε ) ; to apply Lemma D.7 we at least need ∆ ≤ kn − 5 / 2 and ε ≤ k (cid:48) ∆ (cid:0) 1 + (cid:112) n 0 n (cid:1) − 1 , so it sufﬁces to put ∆ =
Cn − 3 when n is chosen larger than an absolute constant, and require ε ≤ min (cid:110) 1 , k (cid:48)
Cn − 5 / 2 (cid:0) 1 + (cid:112) n 0 n (cid:1) − 1 (cid:111) .
Fixing ε = n − 3 n − 1 / 2, which again is admissible when n is sufﬁciently large compared to an absolute constant, for any d ≥ Kd 0 log( nn 0 C M ) , we choose n ≥ K max { 1 , d 4 L } and take a union bound to obtain the claim (using here that C M ≥ 1 ).
Lemma D.9.
There is an absolute constant c > 0 and absolute constants K, K (cid:48) , K (cid:48)(cid:48) > 0 such that for any d ≥ Kd 0 log( nn 0 C M ) , if n ≥ Kd 4 L , then one has
[Equation]
Proof.
Let ¯ x ∈ N n − 3 n − 1 / 2.
Lemma D.2 and a union bound give
[Equation]
if n ≥ KdL and d ≥ K (cid:48) log n .
If additionally d ≥ K (cid:48)
d 0 log( nn 0 C M ) , we obtain by the discussion in Appendix D.3.1 and another union bound
[Equation]
Let E denote the event studied in Lemma D.8; choose d ≥ Kd 0 log( nn 0 C M ) and n sufﬁciently large to make the measure bound applicable.
A union bound gives
[Equation]
Let G denote the complement of the event in the previous bound.
For any x ∈ M , we can ﬁnd a point ¯ x ∈ N n − 3 n − 1 / 2∩N n −
[Equation]
n − 1 / 2( x ) .
On G , SSC ( L, n − 3 n − 1 / 2, Cn − 3 ) holds at every point in the net N n − 3 n − 1 / 2, which implies that on G ∀ (cid:96) ∈
[ L ] , (cid:13)(cid:13) ρ (cid:96)
[Equation]
and by the 1 -Lipschitz property of [ · ] + and Lemma G.10, this also implies that on G
[Equation]
Choosing n ≥ (4 C ) 2 / 5 , the RHS of this bound is no larger than 1 / 4 .
We write using the triangle inequality
[Equation]
Using the triangle inequality again, the ﬁrst term on the RHS is no larger than 1 / 4 for any (cid:96) on G .
The second term is also no larger than 1 / 4 on G by control over the net.
We conclude that on G
[Equation]
This implies that the event G is contained in the set (cid:92) x
[Equation]
which is closed, by continuity of (cid:107) · (cid:107) 2 and of the features as a function of the parameters, and is therefore also an event
The claim follows.
Lemma D.10.
There are absolute constants c, C > 0 and absolute constants K, K (cid:48) > 0 such that for any d ≥ Kd 0 log( nn 0 C M ) , if n ≥ K (cid:48)
d 4 L , then one has
P
[Equation]
Proof.
Let ¯ x , ¯ x (cid:48) ∈
N n − 3 n − 1 / 2.
Lemma D
[Equation]
1 and a union bound give
[Equation]
if d ≥ K log n and n ≥ max { K (cid:48)
d 3 L, K (cid:48)(cid:48)
d 4 , K (cid:48)(cid:48)(cid:48) } .
If additionally d ≥ Kd 0 log( nn 0 C M ) , we obtain by the discussion in Appendix D.3.1 and another union bound
P  
[Equation]
where with an abuse of notation we write S × 2 to denote S × S for a set S .
Let E 1 denote the event studied in Lemma D.8, and let E 2 denote the event studied in Lemma D.9; choose n sufﬁciently large to make the measure bounds applicable.
A union bound gives
P 
[Equation]
≤ e − cd after adjusting constants.
Let G denote the complement of the event in the previous bound.
For any ( x , x (cid:48) )
∈ M × M , we can ﬁnd a point ¯ x ∈ N n − 3 n − 1 / {+
 [Equation]+} ) and a point ¯ x (cid:48) ∈ N n − 3 n − 1 / 2∩ N n − 3 n − 1 / 2( x (cid:48) ) .
On G , SSC ( L, n − 3 n − 1 / 2, Cn − 3 ) holds at every point in the net N n − 3 n − 1 / 2, which implies that on G ∀ (cid:96) ∈
[ L ] , (cid:13)(cid:13) ρ (cid:96) ( x ) − ρ (cid:96) (¯ x ) (cid:13)(cid:13) ∞ ≤ C n 3 , and ∀ (cid:96) ∈
[ L ] , (cid:13)(cid:13) ρ (cid:96) ( x (cid:48) )
− ρ (cid:96) (¯ x (cid:48) )
(cid:13)(cid:13) ∞ ≤ C n 3 , and by the 1 -Lipschitz property of [ · ] + and Lemma G.10, this also implies that on G ∀ (cid:96) ∈
[ L ] , (cid:13)(cid:13)
α (cid:96) ( x ) − α (cid:96) (¯ x ) (cid:13)(cid:13) 2 ≤ C n 5 / 2 , and ∀ (cid:96) ∈
[ L ] , (cid:13)(cid:13)
α (cid:96) ( x (cid:48) ) − α (cid:96) (¯ x (cid:48) )
(cid:13)(cid:13) 2 ≤ C n 5 / 2 .
(D.63)
For any (cid:96) ∈
[ L ] , we write using the triangle inequality (cid:12)(cid:12)(cid:12)(cid:10) α (cid:96) ( x ) , α (cid:96) ( x (cid:48) ) (cid:11) − cos ϕ
( (cid:96) )
( ∠ ( x , x (cid:48) ))
(cid:12)(cid:12)(cid:12) ≤ (cid:12)(cid:12)(cid:10) α (cid:96)
[Equation]
[Equation] Using Cauchy-Schwarz
[Equation]
we have on G (cid:12)(cid:12)(cid:10) α (cid:96) ( x ) [Equation] α [Equation]
x
[Equation]
−
[Equation]
α [Equation] (¯ x ) [Equation]
α (cid:96)
[Equation]
x (cid:48) ) (cid:11)(cid:12)(cid:12) ≤ (cid:13) (cid:13) α (cid:96) ( x ) − α (cid:96) (¯ x ) (cid:13)(cid:13) 2 (cid:13) (cid:13) α (cid:96)
[Equation]
x
[Equation] n 5 / [Equation] , with the same bound holding for the second term in (D.64) by an analogous argument.
For the third term, we have on G
[Equation]
n . For the last term, we use 1 -Lipschitzness of cos and 1 -Lipschitzness of the ϕ ( (cid:96) ) , which follows from
Lemma E.5 and the chain rule, to obtain
[Equation]
Using Lemma C.5 and several applications of the triangle inequality
[Equation]
we get
[Equation]
and so returning to (D.64), we have shown
[Equation]
n for every (cid:96) ∈
[ L ] .
This implies that the event G is contained in the set (cid:92)
[Equation]
which is closed, by continuity of the inner product and of the features as a function of the parameters, and is therefore also an event
The claim follows.
Lemma D.11.
Assume n, L, d satisfy the requirements of lemma D.10 and additionally d ≥ 1 , n ≥
K √ L for some K .
Then
[Equation]
Deﬁne
[Equation]
Then under the same assumptions
[Equation]
for some numerical constant c .
Proof.
At some x ∈ M , we note that
[Equation]
where g is a standard normal independent of the other variables in the problem.
Similarly
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
where g (cid:48) is also standard normal.
With respect to the randomness of W L +1 , these two objects are Gaussian processes with variances (cid:13)(cid:13)
α L ( x ) (cid:13)(cid:13) 2
and (cid:13)(cid:13)(cid:13)(cid:13)
α L ( x ) − (cid:82)
M α L ( x (cid:48) )
d µ ∞ ( x (cid:48) )
(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
respectively.
We next note that
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
  where the ﬁrst inequality comes from an application of Jensen’s inequality.
Assuming n, d satisfy the requirements of lemma D.10 and denote the event deﬁned in it by G .
On G , angles between features concentrate uniformly around a simple function of the angle evolution function ϕ , in the sense that, for all x ∈ M simultaneously,
[Equation]
≤ C
[Equation]
n
[Equation]
for some constant C (cid:48) . From lemma C.
[Equation]
there exists a constant c 0 > 0 such that for all ν ∈ [0 , π ] ,
[Equation]
Using cos x ≥ 1 − x 2 / 2 and the above bound gives
[Equation]
and thus
[Equation]
Combining the above bound with D.67 and recalling the probability of G holding, we have P
[Equation]
On the same event we have
[Equation]
Thus on G the variances of the Gaussian processes (D.65) [Equation] (D.66) are uniformily bounded by 2 and 1 c 20 L 2 + C (cid:48) (cid:113) d 3 Ln respectively.
Writing for concision in the subsequent expression
[Equation]
[Table] gives [Table]
[Equation]
[Table]
for some constants, since d was chosen to satisfy the conditions of lemma D.10.
In addition, we see from (D.63) that on the same event, for every x ∈ M there exists x ∈ N n [Table] such that
[Equation]
Bernstein’s inequality also gives P (cid:2)(cid:13)(cid:13) W L +1 (cid:13)(cid:13)
2 > C √ n (cid:3) ≤ e − cn for some constants.
Denoting the complement of this event by E we have that on E ∩ G , for every x ∈ M there exists x ∈ N n − 3 n − 1 / 2such that
[Equation]
where we assumed d ≥ 1 , n ≥ K √ L for some K .
Combining the above bound with (D.69) and taking a union bound over the complements of E , G gives
[Equation]
≥ 1 − e − cd − P [ G c ] − P [ E c ] ≥ 1 − e − c (cid:48) d − e − c (cid:48)(cid:48) n ≥ 1 − e − c (cid:48)(cid:48)(cid:48)
[Equation]
From the ﬁrst result, we also obtain that with the the same probability (cid:107) ζ (cid:107) L ∞ ≤ 1 + √ d .
By worsening the constant in the tail we can simplify this to (cid:107) ζ (cid:107) L ∞ ≤ √ d .
Deﬁning
[Equation]
since ˆ ζ ( x ) − ζ ( x ) =
f θ 0 ( x ) − (cid:82)
M f θ 0 ( x (cid:48) )
d µ ∞ ( x (cid:48) ) , it follows that
[Equation]
Lemma D.12.
For some integer d 0 , assume n, L, d satisfy the requirements of lemmas D.8 and D.14, meaning that there exist absolute constants K, K (cid:48) , K (cid:48)(cid:48) , K (cid:48)(cid:48)(cid:48) , K (cid:48)(cid:48)(cid:48)(cid:48) > 0 such that for any d ≥ max { Kd 0 log( nn 0 C M ) , K (cid:48) log L } , if n ≥ max { K (cid:48)(cid:48)
d 4 L, K (cid:48)(cid:48)(cid:48) L log n, K (cid:48)(cid:48)(cid:48)(cid:48) } then If d 0 = 1 and n ≥ K (cid:48)(cid:48)(cid:48)(cid:48)(cid:48) max
[Equation]
d 2 L,
[Equation]
κc τ [Equation] [Equation] 3 , κ 2 / 5 (cid:27) where K (cid:48)(cid:48)(cid:48)(cid:48)(cid:48) is some absolute constant.
κ and c λ are the extrinsic curvature and injectivity coefﬁcient deﬁned in Section 2.1, then on an event of probability at least 1 − e − cd , one has
[Equation]
for a numerical constant c , and where the Lipschitz seminorm is taken with respect to the Riemannian distance on M ± .
If M = S n 0 − 1 so that d 0 = n 0 − 1 , then on an event of probability at least 1 − e − cd , one has
[Equation]
for a numerical constant c .
Proof.
We recall
[Equation]
Let M (cid:63) denote a connected component of M .
Let x 1 , x 2 ∈ M (cid:63) , and ﬁx a smooth unit-speed geodesic γ : [0 , dist M (cid:63) ( x 1 , x 2 )]
→ M (cid:63) such that γ (0) =
x 1 and γ (dist M (cid:63) ( x 1 , x 2 )) =
x 2 .
The absolute continuity of f θ 0 (cid:12)(cid:12) M ± ◦ γ follows from an argument almost identical to the one employed in the proof of Lemma B.8, and we obtain in particular the bound
[Equation]
Because γ is a unit-speed geodesic, we have for all t
[Equation]
and so in particular, by the triangle inequality and Cauchy-Schwarz
[Equation]
This implies
[Equation]
We next write
[Equation]
T x M (cid:63) can be identiﬁed with a linear subspace of R n 0 of dimension d 0 .
Since it is also a subspace of T x S n 0 − 1 , P T x M (cid:63) x = 0 and hence
[Equation]
where ˜ W 1 ∗ is a copy of W 1 ∗ that is independent of all the other variables in the problem (since β 0 ( x ) depends only on G 1 )
[Equation]
We ﬁrst consider the case d 0 = n 0 − 1 , and subsequently the case d 0 = 1 . We note that [Equation] Considering a single summand in the above expression
[Equation]
repeated application of the rotational invariance of the gaussian distribution gives [Equation] D
[Equation]
where g i, I ( x ) is a standard normal variable that depends on i and the support patterns I ( x )
= { I 1 ( x ) , . . . , I L ( x ) } , since β 0 ( x ) depends on x only through I ( x ) Similarly, the dependence on x in the ﬁrst factor in (D.72) is only through I ( x ) . We now show how to control such terms uniformly on M (cid:63) .
Deﬁne a net
[Equation]
as in Appendix D.3.1.
According to Lemma C.4, (cid:12)(cid:12)(cid:12) N
[Equation]
(cid:12)(cid:12)(cid:12) [Equation] e [Equation]
Assume n, L, d satisfy the requirements of Lemma D.8 and denote this event deﬁned in that lemma by E .
We also deﬁne sets of support patterns
[Equation]
On E ,
[Equation]
) , and additionally for any x ∈ M (cid:63) , then there exists some x ∈
[Equation]
such that x ∈ N n
[Equation]
( x ) and I ( x ) ∈ J ( x ) .
We now show that on E , the supports I ( x ) satisfy the requirements of Lemma D.14 with δ s =
d, K s =
Cn − 5 / 2 , with the anchor point in the statement of that lemma chosen to be x .
The value of δ s is satisﬁed by the supports at every point on the manifold on E from the deﬁnition of this event.
From the deﬁnition of the stable sign consistency property ( SSC ) in Appendices D.3.1 and D.3.2, the only features whose sign can differ between x to x are the risky features, and from the bound on their norm in the deﬁnition of SSC ( L, n − 3 n − 1 / 2, Cn − 3 ) we obtain for all (cid:96)
[Equation]
where in the last inequality we used Lemma G.10.
It follows that if n ≥ KLd for some K , the requirements of Lemma D.14 are satisﬁed if we make the choice E = E δK .
We would next like to apply Lemma D.14 for every possible support pattern in J ( N n − 3 n − 1 / 2) , which requires that we ﬁrst bound the cardinality of this set.
Note that J ( x ) is the .
Thus
[Equation]
≤
[Equation] Ln ) [Equation] for some C .
Using the bound on the cardinality of the net from Lemma C.4, the size of this set can be bounded, giving
[Equation]
[Equation]
We can now apply Lemma D.14 with E =
E δK to bound (cid:13)(cid:13) Γ (cid:96) :2 ( x ) e 1 (cid:13)(cid:13) 2 2 for all (cid:96) on E , taking a union bound over all possible supports.
Bernstein’s inequality and an exponential tail bound can be used to bound the second factor and third factors in (D.72) respectively.
Using (D.74) to bound the number of supports we need to uniformize over (since on E , (cid:83) x ∈M
[Equation]
{I [Equation] x
[Equation]
[Equation] J [Equation] N
[Equation]
) ) and the bound on the size of the net in Lemma C.4 and Appendix D.3.1, we obtain
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where we assume d ≥ K log( C M (cid:63) nn 0 ) d 0 , n ≥ K (cid:48)
Ld 2 for some K, K (cid:48) .
Since according to
Lemma D.8 the event E holds with probability greater than 1 − e cd for some c , we can remove the indicator in the bound above by assuming d ≥ K for some absolute constant K and worsening the constant in the bound. We can now complete the proof for d 0 = n 0 − 1 .
Since we are interested in bounding the sum (D.71) uniformly, we can bound n 0 (cid:80) i =1 g 2 i, I ( x ) as well using Bernstein’s inequality and uniformizing as above obtain [Equation] [Table]
[Equation]
[Table]
[Equation]
where we assume d ≥ K log( C M
[Table]
nn 0 ) d 0 , n ≥ K (cid:48)
Ld 2 for some K, K (cid:48) .
As above, we can remove the indicator in the bound above by assuming d ≥ K for some absolute constant K and worsening the constant in the bound.
Worsening constants in the failure probability, we can replace the residual in the above expression by d .
Using (D.70), we obtain that with the same probability the Lipschitz constant of f θ 0 on S n 0 − 1 is bounded by √ d .
We now consider d 0 = 1 .
Recall that for any x ∈ M (cid:63) , then there exists some x ∈ N n − 3 n − 1 / 2such that x ∈ N n − 3 n − 1 / 2( x ) where N n − 3 n − 1 / 2is
the net deﬁned earlier.
The gradient vector at x takes the form
[Equation]
We proceed to bound the ﬁrst term in the above equation uniformly over M (cid:63) .
Since in the d 0 = 1 case T x M (cid:63) can be identiﬁed with a linear subspace of R n 0 of dimension 1 , we can write the projection operator as
[Equation]
for some unit norm v x .
We then obtain from rotational invariance of the gaussian distribution that
[Equation]
where g x is a standard normal variable and the last bound comes from a calculation identical to (D.72).
Under the same assumptions on d
[Equation]
L
[Equation]
n as before
[Equation]
we can bound this expression uniformly using (D.75)
[Equation]
and additionally take a union bound over the net to account for all possible choices of x .
This gives
[Equation]
[Table]
where we assume d ≥ K log( C M (cid:63) nn 0 ) for some K .
This gives control of the ﬁrst term in (D.76) uniformly on M .
We now turn to controlling the second term in (D.76).
For some x , choose x ∈ N n − 3 n − 1 / 2such that x ∈ N n − 3 n − 1 / 2( x ) .
Deﬁne a unit-speed curve γ : [0 , s ] → M such that γ (0) =
x , γ ( s ) = x .
Since the curvature of M is bounded by κ , we have
[Equation]
Denote by r the geodesic distance between x and x (cid:48) .
Since the euclidean distance between them is bounded by n − 3 n − 1 / 2, assuming n > K for some K implies that r < C (cid:48) n − 3 n − 1 / 2for some C (cid:48) .
If we now demand n 3 ≥ C (cid:48) κc λ which implies
[Equation]
for some C > 0 .
For v x , v x deﬁned as in (D.77) we have γ (cid:48) (0) =
v x , γ (cid:48) ( s ) =
v x .
Combining the previous two results, it follows that
[Equation]
A straightforward calculation then gives
[Equation]
If we now use Lemma D.13 to control the norms of the backward features uniformly, a standard bound on the norm of a Gaussian matrix to give P (cid:104)(cid:13)(cid:13)(cid:13)
˜ W [Equation] > C [Equation]
1 + (cid:112) n 0 n (cid:1)(cid:105) ≤ e − cn , and assume n ≥ κ 2 / 5 we obtain that
[Equation]
Combining the above with (D.75) and using (D.76) and taking a union bound over the failure probability of E which results in a worsening of constants completes the proof.
We can additionally rescale d to obtain a ﬁnal bound on the Lipschitz constant of √ d instead of C √ d , which also results in a worsening of constants.
Lemma D.13.
There are absolute constants c, C > 0 and absolute constants K 1 , . . .
, K 4 > 0 such that for any d ≥ Kd 0 log( nn 0 C M ) , if n ≥ K (cid:48)
d 4 L , then there exists an event E such that
On E , we have
[Equation]
≤
[Equation]
√ d
[Equation]
simultaneously for every ( x , x (cid:48) )
∈ M × M ; 2.
P [ E ] ≥ 1 − e − cd .
Proof.
Let E 1 denote the event studied in Lemma D.8, with C 0 denoting the absolute constant appearing in the SSC ( L ) condition there; choose d ≥ Kd 0 log( nn 0 C M ) and n sufﬁciently large to make the measure bound applicable.
We will need to apply Lemma D.23 together with a derandomization argument to prove the claim; we appeal to the same residual checks at the beginning of the proof of Lemma D.6 to see that on E 1 , the dominating residual in Lemma D.23 under the scalings of d and n we enforce here is of size C √ d 4 nL .
For any subset S ⊂ [ L ] × [ n ] , we write S (cid:96) =
{ i ∈ [ n ] | ( (cid:96), i ) ∈ S } , and we deﬁne S ( S )
= {− 1 , +1 }
| S 1 | × · · · × {− 1 , +1 } | S L | for the set of “lists” of sign patterns with sizes adapted to these projections of S , with the convention {− 1 , +1 } 0 = { 0 } .
If Σ = { σ 1 , . . .
, σ L } ∈ S ( S ) is such a list of sign vectors and ∆ ≥ 0 , we deﬁne
[Equation]
, which is a sort of two-sided robust analogue of the support of α (cid:96) ( x ) : notice that when S = ∅
we have ˜
I (cid:96) ( x , S, Σ , ∆) =
I (cid:96) ( x ) .
We also deﬁne for (cid:96) = 0 , 1 , . . .
, L −
[Equation]
a generalized backward feature induced by these robust support patterns
. Writing for concision
[Equation]

[Equation]
where C 1 > 0 is an absolute constant we will specify below to make the event hold with high probability, we then deﬁne the event
[Equation]
There are no more than (cid:80) dk
=0 (cid:0) nLk (cid:1) ≤ n 4 d ways to choose the subset S in this union, and for a ﬁxed S there are no more than 2 d ways to choose the sign pattern Σ .
Thus, there no more than exp(10 d log n +12 d 0 log( nn 0 C M )) elements in the union, and under the condition on d this number is no larger than n 11 d .
For concision, write
[Equation]
For any instantiation of these parameters, Lemma D.23 and a union bound give
[Equation]
[Equation]
≤ e − cd for any d ≥ K log n and n ≥ K (cid:48)
d 4 L .
Thus, if we set C 1 = C and enforce d ≥
Kd 0 log( nn 0 C M ) / log n and n ≥ max K (cid:48)
d 4 L log 4 n , we have by a union bound
P [ E 1 ∪ E 2 ] ≤ n − cd .
Let G =
E c 1 ∩ E c 2 .
For any ( x , x (cid:48) )
∈ M × M , we can ﬁnd a point ¯ x ∈ N n − 3 n − 1 / 2∩ N n − 3 n − 1 / 2( x ) and a point ¯ x (cid:48) ∈ N n − 3 n − 1 / 2∩ N n − 3 n − 1 / 2( x (cid:48) ) .
On G , SSC ( L, n − 3 n − 1 / 2, Cn − 3 ) holds at every point in the net N [Table] , and there are no more than d Cn − 3 -risky features at any point in the net
N n − 3 n − 1 / 2, and in addition, following (D.52), we have almost surely on G that all risky features are realized for magnitudes in ( − ∆ , +∆) .
This implies that on G , the support sets (cid:70) (cid:96) ∈
[ L ] I (cid:96) ( x ) at any point x ∈ N n − 3 n − 1 / 2(¯ x ) differ by the support sets (cid:70) (cid:96)
∈
[ L ] I (cid:96) (¯ x ) at the base point in the net by no more than d entries, consisting only of a subset of the risky features at ¯ x ; the analogous statement is of course true for x (cid:48) and ¯ x (cid:48) .
At the same time, notice that on the event E c 2 we have constructed, we have control of every possible backward feature inner product obtained by modifying the supports at the base points ¯ x , ¯ x (cid:48) at no more than d risky features (each), since, for example, if ( ρ (cid:96) (¯ x ))
i is risky, then 1 ( ρ (cid:96) (¯ x ))
i > ∆ corresponds to “turning off” the feature, and 1 ( ρ (cid:96) (¯ x ))
i > − ∆ corresponds to “turning on” the feature.
Formally, we have established that on G ∀ (cid:96) ∈
[ L ] , (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:10) β (cid:96) [Table] [Table]
[Table]
(cid:11)
[Table]
n
[Table]
[Equation]
We can use differentiability properties for the remaining link: following the proof of Lemma D.10, we have
[Equation]
so we just need a Lipschitz property for the function q ( ν )
=
( n/ 2) (cid:81) L −
[Equation]
= [Equation] (1 − π − 1 ϕ (
[Equation]
) ( ν )) .
For this we appeal to Lemma E.5, which shows that the function ϕ is smooth, increasing and concave; therefore by the chain rule, the functions ϕ ( (cid:96) ) are increasing and concave, and by the Leibniz rule, q is decreasing and convex.
It therefore sufﬁces to calculate q (cid:48) (0) ; this is done in Lemma C.16, which gives q (cid:48) (0)
=
− n ( L − (cid:96) ) / (2 π ) , and in particular | q (cid:48) (0) | ≤ cnL .
It follows
[Equation]
π
[Equation]
so that by the triangle inequality
[Equation]
where the residual simpliﬁcation is valid when n ≥ KL .
We conclude that the set
[Equation]
contains the event G , which satisﬁes the claimed properties and completes the proof (after rescaling d by 1 / log n , which updates the lower bound on d ).
D.3. S MALL S UPPORT C HANGE R ESIDUALS
In this section, we prove generalized versions of our pointwise concentration lemmas for backward feature correlations and the matrices deﬁning the propagation coefﬁcients used in our study of
SSC ( L ) .
Lemma D.14.
Assume n ≥ max { KL log n, K (cid:48)
Ld, K (cid:48)(cid:48) } , d ≥ K (cid:48)(cid:48)(cid:48) log L for suitably chosen
K, K (cid:48) , K (cid:48)(cid:48) , K (cid:48)(cid:48)(cid:48) and integer L , and choose 1 ≤ (cid:96) (cid:48) ≤ (cid:96) ≤ L .
Deﬁne an anchor point x ∈ M and denote I i ( x ) = supp (cid:0) α
i 0 ( x ) > 0 (cid:1) for (cid:96) (cid:48) ≤ i ≤ (cid:96) .
Choose some δ s , K s > 0 and let J = { J (cid:96) (cid:48) , . .
, J (cid:96) } denote a collection of support sets such that each J i ⊂ [ n ] depends on the network parameters only through the pre-activation ρ i 0 ( x ) .
We deﬁne events implying that the supports at J are close to those at x :
[Equation]
Deﬁne
[Equation]
and ﬁx a unit norm vector v f .
If K s ≤ 12 L − 3 / 2 , δ s ≤ nL , then
[Equation]
and
[Equation]
For a vector g , g i ∼ iid N (0 , 1) , deﬁning H
[Equation]
for i ∈
[ L ] and
[Equation]
we have
[Equation]
for some numerical constants c, C .
Proof.
In the following, we will denote by v f ∈ S n − 1 a ﬁxed unit norm vector and by v u ∈ S n −a random vector uniformly distributed on S n − 1 .
When there is no need to distinguish between the two we will denote either by v p .
Our strategy in bounding
[Table]
[Equation] [Table] will be ﬁrst to bound 1 E δK [Table]
[Equation] v [Table] 2 with sufﬁciently high probability, and then apply an ε -net argument to uniformize the result (lemma D.20) and get control of the operator norm.
In achieving the ﬁrst goal, we will rely heavily on a decomposition of the weight matrices into terms that are conditionally independent given the pre-activations.
We will also utilize martingale concentration to control the terms that result from this decomposition.
Denoting S
i = span (cid:8)
α i ( x ) (cid:9) for i ∈
[ L ] , we decompose the weight matrices into
[Equation]
Note that (cid:8) H 1 , . . .
, H L (cid:9) are conditionally independent given σ ( G 1 , . .
, G L ) (by which we denote the sigma algebra generated by G 1 , . . .
, G L ).
Since the pre-activations obey
[Equation]
and the features are deterministic functions of the pre-activations, both (cid:8)
α 1 ( x ) , . . . , α L ( x ) (cid:9) and (cid:8) ρ 1 ( x ) , . .
, ρ L ( x ) (cid:9) are measurable with respect to σ ( G 1 , . .
, G L ) .
We deﬁne events
[Equation]
and aim to control 1 E δKρ (cid:13)(cid:13)(cid:13) Γ (cid:96) : (cid:96) (cid:48)
J ( x ) (cid:13)(cid:13)(cid:13) .
Since the supports J depends on the weights only through the pre-activations and are thus also σ ( G 1 , . .
, G L ) -measurable, this truncation does not affect the conditional independence of (cid:110) H (cid:96) (cid:48) , . . .
, H (cid:96) (cid:111) .
It will often be convenient to utilize the rotational invariance of the Gaussian distribution to replace all occurrences of H i in a given expression by ˜
W i P S
i − 1 ⊥ where ˜ W
i is a fresh copy of W i independent of all the other variables in the problem, which will not change the distribution of the original expression.
For (cid:96) (cid:48) ≤ i ≤ (cid:96), (cid:96) (cid:48) ≤ j ≤ i + 1 it will also be useful to denote
[Equation]
where we use the convention Γ i : i +1 G J =
Γ i : i +1 H J
= I .
Decomposing the weight matrices at every layer gives
[Equation]
[Table]
[Table]
[Equation]
[Table]
[Equation]
We next deﬁne
[Equation]
In accounting for all the terms in the decomposition (D.79) [Equation] there will be two simpliﬁcations that we use repeatedly.
One is
[Equation]
where we used P α i ( x
[Equation]
{+> 
+}
ρ i
[Equation]
[-α i ( x ) > 
 ρ i (-] x ) = (cid:2)
ρ i ( x ) (cid:3) + = α i ( x ) , from which it follows that [Equation] H i + [Equation]
D
[Equation]
We also have
[Equation]
i [Equation] and thus Γ i : j G J = i − 1 (cid:89)
[Equation]
j [Equation] x [Equation]
−
[Equation]
We refer to such a product as a G-chain.
We proceed to expand (D.79) into terms with different combinations of matrices
Γ i : j G J and Γ i : j H J .
There will be 2 (cid:96) − (cid:96) (cid:48) terms in total, and we denote the set of terms with r G-chains by G r,p (with the subscript p ∈ { u, f } denoting the choice of vector v p ).
We can clearly label each term by the start and end index of each G-chain, which may not be distinct.
We denote each such term by g rp ( i 1 ,i 2 ,...,i 2 r ) where
[Equation]
The constraints above ensure that every two G-chains are separated by at least one H i matrix.
To lighten notation, we denote a set of indices obeying the constraints by ( i 1 , . .
, i 2 r ) ∈ C r ( (cid:96), (cid:96) (cid:48) ) .
The maximal number of G-chains possible is bounded by r ≤ (cid:6) ( (cid:96) − (cid:96) (cid:48) ) / 2 (cid:7) .
Since the g rp ( i 1 ,i 2 ,...,i 2 r ) are non-negative, we have
[Equation]
Considering ﬁrst the form of the terms in G r
[Equation]
p , using (D.83) and (D.81) and recalling that Γ m − 1: m H J =
I
[Equation]
we have
[Equation]
The magnitudes the factors in this expression are bounded in the following lemma:
Lemma D.15.
For ˜ a k , ˜ b qij , ˜ c pts deﬁned in (D.86) , R u = (cid:113) dn , R f = {+
+}
and (cid:96) (cid:48) ≤ k < (cid:96) , (cid:96) (cid:48) + 2 ≤
[Equation]
[Equation]
for some constants c, c (cid:48) , C, C (cid:48) and d ≥ 0 .
Proof: Deferred to D.3.4.
We will use these results in order to bound 1 E
[Equation]
2 using (D.85).
While the sum over most of these terms can be controlled using the triangle inequality and the lemma above, there is a subset which will require special treatment since they are typically larger.
These are the terms where the leftmost or rightmost chain is a G-chain (meaning i 2 r = (cid:96) or i 1 = (cid:96) (cid:48) respectively) and they will be controlled using martingale concentration.
The or above is exclusive, since we can bound terms with i 2 r
= (cid:96) and i 1 = (cid:96) (cid:48) using a triangle inequality.
We denote these three sets of terms by ←−
G r,p , −→ G r,p , ←→ G r,p respectively, and elements in them by ←− g r,p , −→ g r,p , ←→ g r,p for clarity when needed.
Arranging the remaining terms into sets denoted G r,p , the sum in (D.85) decomposes into
[Equation]
We consider ﬁrst terms in ←− G r,p (and hence with i 2 r = (cid:96) ).
We denote such terms by
[Equation]
We show
Lemma D.16.
For p ∈ { f, u } and R u = (cid:113) dn , R f =i .
[Equation]
ii
[Equation]
for absolute constants c, C, C (cid:48) , and where d ≥ K log L for some constant K .
Proof: Deferred to D.3.4.
Turning next to bounding the terms in ←→ G r,p , we ﬁrst deﬁne an event ←→E
[Equation]
and from lemma D.15 and a union bound obtain
P
[Equation]
− c (cid:48)(cid:48)
[Equation]
assuming n ≥ KL log
L [Equation]
d ≥ K (cid:48) log L for some K, K (cid:48) .
It follows that
[Equation]
[Table]
[Equation]
[Table]
[Equation]
R p .
where we used L 3 / 2 K s ≤ 12 .
We also bound the number of summands in
[Equation]
by
L 2 r − 2 which is tight for small r .
It follows that
[Equation]
for appropriate constants.
It remains to bound the terms in G r,p by a similar argument.
Deﬁning
[Equation]
=
[Equation]
truncating on this event gives
[Equation]
n and bounding the probability of this even from below using lemma D.15 and a union bound gives
P
[Equation]
Combining the bound above with (D.90) and the results of lemma D.16 and worsening constants, the sum of all terms containing matrices G i is bounded by [Equation] n   + (cid:88) [Equation]
P
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
Bounding the ﬁrst term in (D.85) using lemma D.18, setting p = f above and choosing d = nL gives
[Equation]
We then apply lemma D.20 to obtain
[Equation]
Recalling (D.78), to obtain our ﬁnal bound on the operator norm it remains to control the probability of E ρ .
We consider some (cid:96) (cid:48) ≤ i ≤ (cid:96) and assume α i − 1 ( x ) (cid:54) =
(otherwise (cid:13)(cid:13)
ρ i ( x ) (cid:13)(cid:13) 2 ≤ C with probability 1).
Deﬁning an orthogonal matrix R such that
Rα i − 1 ( x
[Equation]
=
[Equation]
, rotational invariance of the Gaussian distribution gives
[Equation]
Since (cid:13)(cid:13)(cid:13) W i (: , 1) (cid:13)(cid:13)(cid:13) 2 2 is a sum of independent sub-exponential random variables with sub-exponential norm bounded by C (cid:48) n , Bernstein’s inequality (lemma G.2) and D.2 give
[Equation]
for appropriate constants.
Taking a union bound over i gives
[Equation]
for a suitable chosen constant c (cid:48)(cid:48) , where we used n ≥ KL log L for some K .
We then have
[Equation]
for appropriate constants, and similarly
[Equation]
This concludes the proof of the ﬁrst two statements.
For the ﬁnal result, we consider a vector g with g
i ∼ iid N (0 , 1) .
Bernstein’s inequality gives P (cid:104) (cid:107) g (cid:107) 22 > 2 n (cid:105) ≤
e − cn and since
[Equation]
where v u is uniformly distributed on S n − [Equation] we can use D.91 setting p =
u to obtain
P (cid:104)(cid:13)
[Equation]
[Table]
[Equation]
for appropriate constants, where we assumed n > KLd for some K .
Corollary D.17.
Deﬁning
[Equation]
under the same assumptions on n, L in D.14 we have
[Equation]
for some numerical constants c, C .
Lemma D.18.
Fix a collection of supports J =
{ J [Table]
J
[Table]
} for 1 ≤ (cid:96) (cid:48) ≤ (cid:96) ≤ L that satisfy the assumptions of lemma D.14 and denote by v p a unit norm vector.
Deﬁne an event
[Table]
[Equation]
[Table]
If n ≥ KL log n for some constant K then
[Table]
where c, C, C (cid:48) are absolute constants.
Proof.
In the following, we denote by ˜ W i an independent copy of W i , and by W i (: ,j ) the j -th column of this matrix.
Note that due to rotational invariance of the Gaussian distribution we can replace every occurrence of H i in an expression by ˜ W i P S i − 1 ⊥ without altering the distribution of the expression, which we will do presently.
We can repeatedly use this rotational invariance to give
[Equation]
[Table]
and rotational invariance of the Gaussian distribution gives
[Table]
and continuing in a similar fashion we obtain
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where in the last inequality we used the fact that multiplication by P S i ⊥ cannot increase the norm of a vector.
Denoting by { χ i } a collection of independent standard chi-squared distributed random variables where χ i has | J i | degrees of freedom, we have
[Equation]
Deﬁne
[Equation]
n
[Equation]
(cid:41) .
Denoting δ i =
| J i (cid:9) I i ( x ) | , on E
I we have min
[Equation]
n [Equation] n L
[Equation]
n
[Equation]
[Equation]
i
[Equation]
2 | J i | n ≤
[Equation]
i =
[Equation]
(cid:48) 2 (cid:0)
| I i | + δ i (cid:1) n ≤ (cid:96) (cid:89) i = (cid:96) (cid:48) 2 (cid:0)
| I i | + nL (cid:1) n = (cid:96) (cid:89)
i = (cid:96) (cid:48)
2 |
I i | n (cid:18) 1 + n L | I i | (cid:19) ≤ (cid:96) (cid:89) i
= (cid:96) (cid:48)
2 |
I
i
| n (cid:18) 1 + 4 L (cid:19) ≤ e(cid:96) (cid:89)
i = (cid:96) (cid:48)
2 |
I
i | n ≤ 2 e 4 .
where we used the assumption δ i ≤ nL and assumed n ≥ 8 L .
It follows that
[Equation]
An application of lemma D.26 and (D.92) then gives
[Equation]
for appropriate constants, assuming n ≥ KL log L for some K .
Using D.30 to bound P [ E cI ] we thus have
[Equation]
for some constants.
Having shown
[Equation]
for some ﬁxed v p = v f we can now apply lemma D.20 to obtain
[Equation]
where we used n ≥ KL log L for some K .
Choosing v p = (cid:98) e i for i ∈ [ n ] and taking a union bound, one obtains
[Equation]
for some constants, where we used n ≥ KL log n for an appropriate constant K .
A ﬁnal union bound over the last three events gives the desired result.
Proof of lemma D.15.
We ﬁrst consider the terms ˜ a k .
For k = (cid:96) [Equation] the deﬁnition of E δKρ in (D.78) gives
[Equation]
In order to handle the case (cid:96) (cid:48) ≤ k < (cid:96)
[Equation]
we use (D.81) and obtain that for any 2 ≤ j ≤ i ≤ L ,
[Equation]
where ˜ W j is an independent copy of W j , and we denote by ˜ W j (: , 1) the ﬁrst column of ˜ W j , and we used the rotational invariance of the Gaussian distribution.
Truncating on the event E i,j +1 H , which does not affect the distribution of ˜ W j (: , 1) , we have
[Equation]
almost surely for some constant C (cid:48) , and the Hanson-Wright inequality (lemma G.4) gives
[Equation]
1 E i,j +1 H Γ i : j +1 ∗ H J Γ
i : j +1 H J (cid:13)(cid:13)(cid:13)
     ≤ 2 exp (cid:18) − c (cid:48) n ( i − j + 1) (cid:19) for some constant c (cid:48) , where we used (cid:13)(cid:13)(cid:13) 1 E i,j +1 H Γ i :
[Equation]
J Γ i :
[Equation]
(cid:13)(cid:13)(cid:13) [Equation]
(cid:13)(cid:13)(cid:13) [Equation] H Γ i : j + [Equation]
H J (cid:13)(cid:13)(cid:13) F (cid:13)(cid:13)(cid:13) Γ i : j +1 H J (cid:13)(cid:13)(cid:13) 2 ≤
Cn ( i − j + 1) , and the fact that multiplying a matrix by P J j cannot increase its norm.
Writing for concision in the subsequent expression
[Equation]
it follows that
[Equation]
for appropriate constants
[Equation]
where we used D.18.
Since on E δKρ we have
[Equation]
we obtain
[Equation]
hence
P [˜
[Equation]
for appropriate constants.
We now turn to controlling the ˜ b qij .
Note that
[Equation]
where in the last inequality we used 2 K J L < 1 .
Additionally, we have
[Equation]
where ˜ W i − 1 is an copy of W i − 1 that is independent of all the other variables, we deﬁned
[Equation]
and g is a standard normal variable.
In the above expression,
[Equation]
Note also that Γ i − 2: j +1 H J is well-deﬁned since i ≥ j − 1 .
We therefore have
[Equation]
where we used (D.95) and the Gaussian tail probability to bound the ﬁrst and second terms in the second line respectively.
Combining the above with (D.97) we obtain
[Equation]
We now turn to controlling ˜ c pji .
If i ≥ (cid:96) (cid:48) we have ˜ c pj [Equation] + [Equation]
=
[Equation]
D
[Equation]
where g is a standard normal and
[Equation]
for some constant C where we used (D.97).
We also have (cid:13)(cid:13)(cid:13) Γ
[Equation]
[Equation]
J v p (cid:13)(cid:13)(cid:13)
[Equation]
2 ≤ ˜ C on E i − 1 ,(cid:96)
[Equation]
Lemma
D.18 and a Gaussian tail bound then give
[Equation]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
for appropriate constants.
Additionally, if i = (cid:96) (cid:48) − 1 we have from (D.97) for some ﬁxed v p = v f
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table]
[Equation]
If however v p = v u is drawn from Unif( S n − 1 ) , if we denote by g a vector with independent standard Gaussian entries we have
[Equation]
From Bernstein’s inequality it follows that P (cid:104) (cid:107) g
(cid:107) 22 < n 2 (cid:105) ≤
e − cn .
Combining this with a Gaussian tail bound gives
[Equation]
for some constants c, c (cid:48) .
From (D.97) it follows that
[Equation]
for appropriate constants.
Proof of lemma D.16.
Part (i).
We denote the set of all such terms with r G-chains by ←− G r,p .
Considering ﬁrst the contribution from the terms with a single G-chain, denoted ←− G 1 p , we have
[Equation]
where
[Equation]
Denoting by σ ( A 1 , . .
A k )
the sigma-algebra generated by a the random variables
A 1 , . .
A k , we deﬁne a ﬁltration
[Equation]
x ) , H (cid:96) (cid:48) , . . , H j (cid:17) , j = (cid:96) (cid:48) , . . . , (cid:96). (D.103)
The sequence { X i } = {+(cid:40) 
+}
+ i (cid:80) j = (cid:96) (cid:48) +1 ˜ c
(cid:96),j (cid:41) is adapted to the ﬁltration, and since the summands are linear in the zero mean (cid:8) H k (cid:9) the sequence is a martingale ( E X i +
| F i
=
X i ).
The martingale difference sequence is
[Equation]
giving
[Equation]
We cannot control this sum directly because we do not have almost sure control of the martingale differences.
To remedy this, we recall the event E i −
[Table]
deﬁned in lemma D.18, and decompose the sum of interest into [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
+ (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) (cid:96)
[Equation]
i
[Equation]
(cid:96) (cid:48)
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
(D.104)
Notice that the second sum is also a sum of zero-mean martingale differences.
Using (D.100), we have
[Equation]
where g ∼ N (0 , 1) and 1 E i − 1 ,(cid:96) [Equation]
+1 H σ
[Equation]
=
[Equation]
i − 1 ,
[Equation]
H
[Equation]
n
[Equation]
(cid:96) − 1 (cid:81) k
=
[Equation]
Γ
[Equation]
2 ≤ Cn almost surely for some constant C .
It follows that
[Equation]
and we can apply Freedman’s inequality for martingales with sub-Gaussian increments (lemma G.7) to conclude that for some d ≥
[Equation]
As for the ﬁrst term in (D.104), using lemma D.18 we have
[Equation]
for appropriate constants, where we assumed n ≥ KL log L for some K .
Combining the above with (D.94), and using (D.101) to give P (cid:104)(cid:12)(cid:12)(cid:12) ˜ a (cid:96) ˜ c p(cid:96),(cid:96)
[Equation]
+1 (cid:12)(cid:12)(cid:12) > ˜ C (cid:105) ≤ C (cid:48)
e − c nL for some constants and applying the triangle inequality, we have
[Equation]
for appropriate constants.
Having controlled the sum of terms in ←− G 1 p , we next consider a sum over the terms in ←− G r,p for r ≥ 2 .
The argument will be very similar to the ←− G 1 p case, with some additional technical details.
Note that since different G-chains must be separated by an H i matrix for some i
and we consider only terms with i 1 ≥ (cid:96) (cid:48)
+ 1 , the minimal starting index of the r -th chain (indexed by j below for clarity) is (cid:96) (cid:48)
+ 1 + 2( r − 1) .
The sum of all possible terms is thus
[Equation]
The constraints on the indices i 1 , . .
, i 2 r − 2 are similar to those in (D.84), with the starting index reﬂecting the constraint
i 1 > (cid:96) (cid:48) in the deﬁnition of ←− G r,p .
We once again deﬁne the ﬁltration F j as in (D.103) for (cid:96) − 1 ≤ j ≤ (cid:96) .
Noting that ˜ a (cid:96) , ˜ b
[Equation]
[Equation]
and
[Equation]
x
[Equation]
are all F l − 1 -measurable, the index constraints imply that
[Equation]
is F i -measurable and thus the sequence { X ri } is adapted to the ﬁltration.
˜ b (cid:96),i +1 ,i 2 r − 2 is a linear function of the zero-mean variables H i for any choice of i 2 r − 2 , and we can replace H i with ˜ W i P S
i − 1 ⊥ where ˜ W
i is an independent copy of W i without altering the distribution of X ri .
Since ˜ b klm for k ≤ i 2 r − 2 is independent of the ˜ W i for any choice of l, m , it follows that ˜ p r(cid:96),i +1 is also a linear function of the variables in ˜ W i which have zero mean.
Consequently
[Equation]
hence { X i } is a martingale sequence.
Deﬁning martingale differences
[Equation]
we have
[Equation]
We deﬁne an event E i ∆ ∈ F i by
E
[Equation]
) for i 1 ≥ (cid:96) (cid:48) + 1 and decompose the sum in (D.106) into
[Equation]
In order to control the second term
[Equation]
we note that
[Equation]
and using Γ i :[-i 
-]
[Equation]
where ˜ W i is an independent copy of W i gives
[Equation]
where g ∼ N (0 , 1) and
[Equation]
[Table]
[Equation]
[Table]
∗
[Table]
[Equation]
In the last inequality we used the deﬁnition of E i ∆ and the assumption L 3 / 2 K s ≤ 1 .
It follows that
[Equation]
and we can apply Freedman’s inequality for martingales with sub-Gaussian increments (lemma G.7) to conclude
[Equation]
It remains to bound the ﬁrst term in (D.108).
Using lemma D.15, (D.95) and taking a union bound over i 1 , i 2 , i 3 in (D.107) we have
[Equation]
where we assume n ≥ KL log L and d ≥ K (cid:48)
log L for some K [Equation] K (cid:48) .
An additional union bound over i gives
[Equation]
for appropriate constants.
Combining the above with (D.109) and recalling (D.106) gives
[Equation]
for some constants
[Equation] L 3 / 2 K s ≤ 12 implies
[Equation]
A union bound over r and a rescaling of d gives
P 
[Equation]
[Table]
[Equation]
110) for appropriate constants, again assuming assume n ≥ KL log L and d ≥ K (cid:48) log L for some K, K (cid:48) .
Combining the above with equation (D.105) and worsening constants gives
[Equation]
for appropriate constants.
Part (ii).
We consider terms in the sets −→ G r,p (with i 1 = (cid:96) (cid:48) and i 2 r ≤ (cid:96) − 1 ).
In contrast to the previous section, the bounds on these terms will differ based on the value of the p subscript (denoting whether we use a ﬁxed vector v f or a random vector v u ).
We ﬁrst consider −→ G 1 ,p , noting
[Equation]
Lemma D.15 and a union bound give
[Equation]
for appropriate constants, where we assume n ≥ KL log L and d 0 ≥ K (cid:48) log L for some constants
K, K (cid:48) .
With the same probability we have
[Equation]
[Equation]
where we deﬁned
[Equation]
and used LK J ≤ 1 .
We next consider sums of terms in −→ G r,p for r > 1 .
In controlling the sum of these terms, the proof will proceed along similar lines to the previous section.
The main tool we will be utilizing is martingale concentration.
Recall that since i 2 r ≤ (cid:96) − 1 and every two G-chains are separated by a matrix
H i , the starting index of the ﬁnal G-chain is no larger than (cid:96) − 2 r + 1 .
We thus have
[Equation]
r
[Equation]
Deﬁne a ﬁltration
[Equation]
(note the reversed indexing convention compared to the ﬁltration deﬁned in (D.103)).
Since ˜ p r,pj is
F (cid:96) − j -measurable (as can be seen from (D.86)), we can deﬁne
[Equation]
and it follows that X r,pi is F i -measurable.
Recalling from (D.86) that
[Equation]
and hence ˜ p r,pj is linear in the zero-mean variables H j +1 , we have
[Equation]
and thus the sequence { X r,pi } is a martingale with respect to this ﬁltration.
Deﬁning martingale differences ∆ r,pi
[Equation]
i the sum of interest can be expressed as
[Equation]
We now deﬁne an event which we will shortly show holds with high probability:
[Equation]
i [Equation] 
[Equation]
Γ i
[Equation] − 1: (cid:96) [Equation] i +
[Equation]
(cid:96) − i +1 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) [Equation] (cid:107) [Equation]
(cid:107)
[Equation]
  [Equation] (cid:107) [Equation]
(cid:96)
[Equation]
i
[Equation]
x
[Equation]
(cid:41) .
[-+
 H J-](D.113)
[-P J (cid:96) − i +1 H (cid:96) − i +1 P J (cid:96) − i ρ (cid:96) − i ( x ) (cid:107) α (cid:96) − i − 1 ( x ) (cid:107)= Γ i 3 − 1: (cid:96) − i +
 H J P J (cid:96) − i +1 H (cid:96) − i +1 Q J (cid:96) − i ρ (cid:96) − i ( x ) (cid:107) α (cid:96) − i − 1 ( x ) (cid:107)d =-]Truncating the martingale difference on such an event gives
[-Γ i 3 − 1: (cid:96) − i +
-][Equation]
×
[Equation]
[-i +
-][Table]
, where we used
[Table]
with ˜ W
[Table]
− {+i +1 an independent copy of W (cid:96) − i +
+}
and we have deﬁned σ
[Table]
Note that from (D.113), if we deﬁne
[Equation]
the standard deviation σ p can be bounded as
[Equation]
where in the ﬁrst inequality we used a triangle inequality, bounded the number of summands by
L 2 r − 2 .
In the second inequality we used L 3 / 2 K s ≤ 12 .
Writing the sum in D.112 as (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) (cid:96) − (cid:96) (cid:48) (cid:88) i =2 r −∆ r [Equation] pi (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
and recognizing that the second sum is over a zero-mean adapted sequence that obeys
[Equation]
an application of Freedman’s inequality for martingales with sub-Gaussian increments (lemma G.7) gives
[Equation]
Turning now to controlling the probability of E i − 1 ∆ holding, we use lemmas D.15, D.18, the deﬁnition of K J and a union bound to conclude
[Equation]
for appropriate constants, where we assumed n ≥ KL log L, d 1 ≥ K log L for some K, K (cid:48) .
Combining the previous two results gives
[Equation]
− [Equation]  
[Equation]
[Table]
[Equation]
[Table]
for some constants.
Noting as before that
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] ≤ 2 , using (D.111) to bound the terms with r = 1 , and setting d 0 = d 1 we obtain
P 
[Equation]
[Table]
[Equation] [Table] [Equation] for appropriate constants, where we used again n ≥ KL log L, d 1 ≥ K log L for some K, K (cid:48) .
Lemma D.19.
(Horn et al., 1994)
Given a semideﬁnite matrix A , for any partitioning
[Equation]
 we have (cid:107) A (cid:107) ≤ b (cid:80) i =1 (cid:107) A ii (cid:107) .
Lemma D.20.
Given a semideﬁnite matrix A and unit norm v , if
[Equation]
and n > 2log(9) (cid:96) c, then P
[Equation]
for some constants c, c (cid:48) , C, C (cid:48)(cid:48) , C (cid:48)(cid:48)(cid:48) .
Proof.
We partition A into blocks of size c 2 n(cid:96) for an appropriately chosen c 2 .
There are (cid:96)c 2 such blocks, and we similarly partition the coordinates { 1 , . . . , n } into (cid:96)c 2 sets K i = { 1 + ( i − 1) c
2 n(cid:96) :
i c 2 n(cid:96) } for i ∈
[ (cid:96)c 2 ] .
We proceed to bound the operator norm of the diagonal blocks using a standard ε -net argument (Vershynin, 2018).
The set of unit norm vectors supported on some K i forms a sphere S c 2 n(cid:96) .
We can thus construct a 14 -net N i on this sphere with at most e log(9) c 2 n(cid:96) points.
A standard argument gives
[Equation]
We control the RHS by a taking a union bound over the net, ﬁnding
[Equation]
We now choose c 2 to satisfy log(9) c
2 = c 1 2 , and the blocks will still have non-zero size because we assume n > 2log(9) (cid:96) c.
Taking a union bound over the (cid:96)c 2 blocks and using Lemma D.19 gives
[Equation]
[Table]
Lemma D.21.
Assume n ≥ max { KL log n, K (cid:48)
Ld b , K (cid:48)(cid:48) } , d b ≥ K (cid:48)(cid:48)(cid:48) log L for suitably chosen
K, K (cid:48) , K (cid:48)(cid:48) , K (cid:48)(cid:48)(cid:48) .
Deﬁne J as in Lemma D.14.
For x ∈ S n 0 − 1 and β (cid:96) J = (cid:0) W L [Equation] L W L . . . W (cid:96)
[Equation]
P
[Equation]
(cid:96)
[Equation]
1 (cid:1)
[Equation]
, denote
[Equation]
[Equation]
= (
[Equation]
, d
[Equation]
) , and d min = min i d i .
We then have
[Equation]
(cid:19)(cid:35) [Equation] for absolute constants c, c (cid:48) ,
[Equation] [Equation]
[Equation]
, where the event E δK is deﬁned in lemma D.14. Other useful forms of this result are
[Equation]
where s i ≥ 1 .
Proof.
Denoting by (cid:8) H i (cid:9) the weight matrices projected onto the subspace orthogonal to the features as in lemma D.14, we deﬁne
[Equation]
for (cid:96) = 0 , . . .
, L − 1 .
Note the additional matrix compared to the standard deﬁnition of the backward features.
Control of the norm of the difference between them can then be used to control the backward features and Lipschitz constant of the network.
Note also that H (cid:96) may not be a square matrix (and indeed in the case of the Lipschitz constant it will be rectangular).
We denote the number of columns of H (cid:96) +1 by n (cid:96) − 1 .
Writing
[Equation]
we begin by bounding the ﬁrst term.
For Γ i : j H ( x ) , Γ i : j H J deﬁned as in D.14 and
[Equation]
[Equation]
we have
[Equation]
We ﬁrst bound (cid:107) b
i (cid:107) 22 .
Repeated use of the rotational invariance of the Gaussian distribution in a similar manner to the proof of lemma D.18 gives
[Equation]
[Table]
where we deﬁned H L +1 (1 , :)
= (cid:113) 2 n W L +1 .
Denoting by ˜ W k an independent copy of W k , rotational invariance gives
[Equation]
where χ k is a standard chi-squared distributed random variable with | suppdiag 1 E δK P J k | degrees of freedom that is independent of all the other variables in the problem, and similarly for ξ I k ( x ) .
A product of such terms was bounded in lemma D.18, from which we obtain
[Equation]
We similarly have
[Equation]
Recalling (D.115) and since
[Equation]
we recognize that
[Equation]
where χ i is a standard chi-squared distributed random variable with d i degrees of freedom.
If d i =this variable is identically 0.
Otherwise, d i ≥ 1 and Bernstein’s inequality gives
[Equation]
for some constants and s ≥ 1 .
Clearly this result also holds if d i = 0 .
Similarly, (cid:13)(cid:13)(cid:13) H (cid:96) +1(1 , :) (cid:13)(cid:13)(cid:13) 2 2 is bounded almost surely by 2 n χ (cid:96) +1 where χ (cid:96) +1 has n (cid:96) − 1 degrees of freedom.
Bernstein’s inequality gives
[Equation]
≤ e − ct for t > Kn (cid:96) − 1 for some K .
Combining these results with (D.116) and taking a union bound, we obtain
[Equation]
118) for appropriate constants, assuming t ≥ K log L, n ≥ K (cid:48)
L log L for some K, K (cid:48) , which can be simpliﬁed to
[Equation]
) assuming s ≥ K (cid:48)(cid:48) log L for some K (cid:48)(cid:48) .
We next bound |(cid:104) b i , b j (cid:105)| for (cid:96) ≤ j < i ≤ L .
Once again using rotational invariance starting from the last layer weights, we obtain
[Equation]
(where we interpret an empty product as unity).
As before
[Equation]
we ﬁnd using lemma D
[Equation]
18 that
[Equation]
We proceed to bound the remaining factors in (cid:104) b i , b j (cid:105) , by ﬁrst writing
[Equation]
where u i +1:
j +1 = P
I
j +1 Γ
i : j +2 ∗ H
H i +1 ∗ (: , 1) and s k m ∈ {− 1 , 1 } are the signs of the elements in Q m ( x ) for m ∈ { i, j } .
In the above expression, k m index the entries on which diag Q m is supported, and we denote d m = | suppdiag Q m | and use the permutation symmetry of the Gaussian distribution to set these to be [ d m ] .
If i > j + 1 , deﬁning (cid:99) H
j +1 = (cid:99) W j +1 P S
j ⊥ where (cid:99)
W j +1 is an independent copy of W j +1 , with (cid:98) Γ i − 1: (cid:96) +1 H J denoting the matrix Γ i − 1: (cid:96)
+1 H J with (cid:99) H j +1(1 , :) in place of H j +1(1 , :) , and writing for concision
Ξ k
i ,k j i,j =
[Equation]
and Ψ
[Equation]
i ,
[Equation]
i
[Equation]
we have
[Equation]
where we used the invariance of the Gaussian distribution to reﬂections around the mean, { H m } d = (cid:110)
˜ W m P S m − 1 ⊥ (cid:111) , and the independence between the [Equation]
˜ W m [Equation] variables and the sign variables { s k m } to absorb the latter into the former.
Making a separate deﬁnition for concision
[Equation]
we ﬁrst consider the term
[Equation]
Lemma D.22 gives
[Equation]
D
[Equation]
We next consider B i,j 3 .
Writing
[Equation]
First, since the variables (cid:110) ˜ W j +1 ∗ ( k j , 1) (cid:111) are independent of
[Equation]
H
[Equation]
a Gaussian tail bound gives
[Equation]
for some constants and d ≥ K for some K .
Two applications lemma D.22 give
[Equation]
assuming t ≥ K log n , n ≥ K (cid:48)
L log n for some K .
Combining this bound with (D.123) we obtain P
[Equation]
for appropriate constants.
We now turn to bounding B i,j 2 .
Deﬁne by Q j a matrix such that Q jab =
(cid:12)(cid:12)(cid:12) Q jab ( x ) (cid:12)(cid:12)(cid:12) .
Then
[Equation]
In order to bound this term using the Hanson-Wright inequality, we ﬁrst note that since
[Equation]
Γ j − 1: (cid:96) +1 ∗ H J (cid:13)(cid:13)(cid:13) (cid:13)(cid:13) H j ∗ (cid:13)(cid:13) , (cid:13)(cid:13)(cid:13) P S [Equation] ⊥
[Equation]
⊥ (cid:13)(cid:13)(cid:13) [Equation] (cid:13)(cid:13) [Equation]
: j − [Equation] (cid:13)(cid:13) [Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
(cid:13)(cid:13)(cid:13) 2 F = (cid:13)(cid:13)
{+Φ j : j − 1 H j ∗ (cid:13)(cid:13) 
+}
d j , we can use lemma D.14, a standard ε -net argument to control the operator norm of a Gaussian matrix and a union bound to obtain P  
[Equation]
We also have
E
[Equation]
and using lemmas D.14 and D.22 gives
[Equation]
assuming n > KL log n for some K .
Denoting the union of these two events by G , an application of the Hanson-Wright inequality (lemma (G.4)) gives
[Equation]
for appropriate constants and s 2 ≥ 0 , and an additional union bound gives
[Equation]
We next turn to bounding (cid:12)(cid:12)(cid:12) B i,j 1 (cid:12)(cid:12)(cid:12) .
Rotational invariance of the Gaussian distribution gives
[Equation]
since P J i − 1 Γ i − 1: j +2 H J P J j +
[Equation] : , 1) and ˜ W i ( k i , 1) are independent.
Since H i +1(1 ,k i ) , ˜ W i (1 ,k i ) are both sub-Gaussian with sub-Gaussian norm bounded by C (cid:48) √ n , the product of two such variables is a sub-exponential variable with sub exponential norm satisfying (cid:13)(cid:13)(cid:13) H i +1(1 ,k i )
˜ W i ( k i , 1) (cid:13)(cid:13)(cid:13) ψ 1 ≤ Cn for some constants.
Thus the ﬁrst sum above is a sum of independent, zero-mean sub-exponential random variables, and Bernstein’s inequality gives [Equation]
(D.127) for s 1 ≥ 1 and some constant c .
Since (cid:13)(cid:13)(cid:13) 1 E δK P J
i − 1 Γ
i − 1: j +2 H J
P
[Equation]
2 ≤ (cid:13)(cid:13)(cid:13) 1 E δK Γ i [Equation]
[Equation]
+2 H J (cid:98) e 1 (cid:13)(cid:13)(cid:13) 2 we can apply lemma D.14 to obtain P [Equation] P
[Equation]
for appropriate constants.
Combining the last two results gives
[Equation]
for some constants.
Combining the above with (D.122), (D.124) and (D.126), we have
[Equation]
In the above proof we assumed i > j + 1 .
If instead i =
j
+ 1 we simply set (cid:98) Γ i − 1: (cid:96) +1 H J
= Γ i − 1: (cid:96) +1 H J in the expression for A i,j 2 in (D.121) and we have (cid:104) b
j +1 , b j (cid:105) =
A j +1 ,j.
We now turn to controlling the term A i,j 2 .
Since i > j , H i ( k i , :) d = ˜ W i [Equation] k
[Equation]
P [Equation] [Equation] and
P S i − 1 ⊥ (cid:98)
Γ i − 1: (cid:96) +1 H J Γ j − 1: (cid:96) +1 ∗
H J H
j ∗ (: ,k j ) is independent of ˜ W
i ( k i , :) , rotational invariance of the Gaussian distribution gives
[Equation]
We bound (cid:12)(cid:12) C i 1 (cid:12)(cid:12) using (D.127).
It remains to control (cid:12)(cid:12)(cid:12) C i,j 2 (cid:12)(cid:12)(cid:12) .
Since H j +1 ∗ ( k j , 1) d
= ( P S j ⊥ )
( k j ,k j ) ˜ W j +1 (1 ,k j ) and ˜ W j +1 (1 ,k j ) are independent of ( P S j ⊥ )
( k j ,k j ) (cid:13)(cid:13) (cid:13) P S
i −
[Equation]
⊥ (cid:98) Γ
[Equation]
− 1: (cid:96) +1 H
[Equation]
[Equation]
+1 ∗
[Equation]
H
[Equation]
[Equation]
k j ) (cid:13)(cid:13) (cid:13) 2 , the second factor in (D.130) is also zero-mean, and it follows that
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
for
[Table] and d [Table] k j ) (cid:13)(cid:13)(cid:13) 2 2 , applying lemmas D.22 and D.14 total of d j times and taking a union bound gives
[Equation]
where we assumed n ≥ KL log n, t ≥ K (cid:48) log n for some constants.
Combining the above three results with (D.122) and taking a union bound, we obtain
[Equation]
for appropriate constants.
Taking a union bound over this result as well as (D.129) and (D.120) allows us to bound the inner product by
[Equation]
) for some constants, again assuming n ≥ KLd .
At this point we obtain a bound on the sum of these inner products that will be useful in an application where the { d i } are expected to be small.
Subsequently, we will derive a different expression that will be useful when they are large.
We now choose s 1 =
d i sn , s
2 = d j Lsn , t = nn (cid:96) − 1 for some s ≥ 1 , which gives
[Equation]
for appropriately chosen constants.
Note that if d i = 0 or d j = 0 then |(cid:104) b i , b j (cid:105)| is identically zero, hence we can replace the term min { d i , d i } in the tail above by max { 1 , min { d i , d i }} and the result will still hold if d i = 0 or d j = 0 .
Lower bounding the second expression by d min = min
i d i gives
[Equation]
[Table]
Recalling the deﬁnition of d in the lemma statement, an additional union bound over the values of i, j in the expression above combined with (D.119) gives
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
for appropriate
[Table]
where we assumed d ≥ K
[Equation]
L log L for some K, K (cid:48) , K (cid:48)(cid:48) .
Taking a square root gives a bound on the ﬁrst term in (D.114).
We next consider a different bound for this term that will be useful when the { d i } are large.
Our starting point will be D.131.
If we set s 1 = s, s 2 =
Ls and use (D.118) we obtain
P
[Equation]
≤
[Equation]
for appropriate constants under similar assumptions on n, L, d .
To bound the remaining terms in (D.114) [Equation] since
[Equation]
and we can apply lemma D.14 and an ε -net argument to bound the ﬁrst and second factors respectively, to conclude
[Equation]
for some d such that d ≥ K log L and assuming n > K (cid:48) d .
An identical result holds for the last term in (D.114) where we simply choose J i
= I i ( x ) for all i .
In conclusion, using (D.132) we have
[Equation]
[Table]
[Equation]
[Table]
for appropriate
[Table]
, while if we use (
[Table]
) instead we obtain
[Equation] [Equation]
[Table]
[Equation]
[Table]
It remains to transfer control from
[Table]
to
[Table]
Note that
[Table]
where if (cid:96) =
we deﬁne P S 0 ⊥ =
I n 0 × n 0 .
Since
E
[Table]
[Equation]
Bernstein’s inequality gives (assuming n > K for some K )
[Equation]
and hence with the same probability
[Equation]
The bounds (D.134), (D.135) also apply to (cid:13)(cid:13) β (cid:96) H J − β (cid:96) H ( x ) (cid:13)(cid:13) 2 up to a constant factor, with the same probability up to a e − cn term which we can absorb into the existing tail by demanding n ≥ KL for some K .
Lemma D.22.
For any (cid:96) + 1 < m ≤ j + 1 , k ∈ [ n ] , if n ≥ KL for some K then
P (cid:104) 1 E δK (cid:13)(cid:13) (cid:13)
{+H j +
 ( k, :) Γ j : m H J (cid:13)(cid:13) (cid:13) 2 > C (cid:105) ≤ e − c nL+}
and if m = (cid:96) +
P (cid:34) 1 E δK (cid:13)(cid:13)(cid:13)
H j {++
+}
( k, :) Γ
j : (cid:96) +1 H J (cid:13)(cid:13)(cid:13) 2 > C (cid:114)nt (cid:35)
≤ e − c nL
+ e − c (cid:48) t assuming t ≥ K (cid:48) n (cid:96) − 1 for some K (cid:48) .
Proof.
If j + 1 > m ,
[Equation]
[Table]
If on the other hand j + 1 = m , we have
[Table]
inequality
[Table]
[-˜ W j +1 ∗ (1 , :) (cid:13) (cid:13) (cid:13) 2 > C (cid:105) ≤ C (cid:48) e − cn and P (cid:104)(cid:13) (cid:13) (cid:13) ˜ W (cid:96) +
 (1 , :) (cid:13) (cid:13) (cid:13) 2 > C (cid:113) 1 n t (cid:105) ≤ 2 e − ct-] for t ≥ 1 , while lemma D.14 gives
[Table]
Taking union bounds gives the desired results.
Lemma D.23 (Generalized backward features inner product concentration) .
Fix x , x (cid:48) ∈ S n 0 − 1 , ν = ∠
( x , x (cid:48) ) .
Deﬁne a collection of support sets J , generalized backward features β (cid:96) J , a constant δ s and event E δK as in lemma D.14.
Assuming n ≥ max { KL log n, K (cid:48) } , d ≥ K (cid:48)(cid:48) log n for suitably chosen K, K (cid:48) , K (cid:48)(cid:48) , we have
[Equation]
for absolute constants c, C, C (cid:48) .
If additionally we have P [ E δK ] ≥ 1 − e − c
[Equation]
d then the same result holds without the truncation on 1 E δK , with worse constants.
Proof.
Note that
[Equation]
[Table]
[Equation]
In order to bound the ﬁrst two terms, we use rotational invariance of the Gaussian distribution twice to obtain
[Equation]
Bernstein’s inequality gives P (cid:104)
[Equation]
W L +1 (cid:13)(cid:13) 2 2 > Cn (cid:105) ≤ 2 e − cn , while 1 E δK (cid:13) (cid:13) Γ L : (cid:96) +1 J [Equation] e 1 (cid:13)(cid:13) 2 2 can be bounded using D.14 to give
[Equation]
for appropriate constants.
Using lemma D.21 to bound 1 E δK (cid:13)(cid:13) β (cid:96) J − β (cid:96) ( x ) (cid:13)(cid:13) 2 we obtain
[Equation]
for some constants, assuming n ≥ KLd for some K .
Bounding the second term in (D.136) in an identical fashion and the last term in (D.136) using Lemma D.4 we obtain
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]

[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
for appropriate constants assuming d ≥ 1 .
Taking a union bound over all possible choices of (cid:96) ∈
[ L ] and using d ≥ K log L for some K gives the desired result.
If we additionally have
P [ E δK ] ≥ 1 − e − c [Equation]
d for some c (cid:48)(cid:48)(cid:48) , we can write
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Equation] [Table] [Table] c
[Equation]
D. A UXILIARY R ESULTS
Lemma D.24.
There are absolute constants c 1 , C, C (cid:48) > 0 and absolute constants K, K (cid:48) > 0 such that for any L ∈ N [Equation] if n ≥ max { K log 4 n, K (cid:48) L } , then for every (cid:96) ∈
[ L ] one has
[Equation]
The constant c 1 is the absolute constant appearing in Lemma E.1.
Proof.
The case of (cid:96) =
L follows immediately from Lemma E.1 with an appropriate choice of d ≥ K (cid:48)(cid:48) for K (cid:48)(cid:48) >
some absolute constant.
Henceforth we assume (cid:96) ∈
[ L − 1] .
We Taylor expand (with Lagrange remainder)
the smooth function ϕ
( L − (cid:96) ) about the point ϕ
(ˆ ν (cid:96) − 1 ) , obtaining for any t ∈ [0 , π ]
[Equation]
− [Equation] where ξ is some point of [0 , π ] lying in between t and ϕ (ˆ ν (cid:96) − 1 ) .
In particular, putting t =
ˆ ν (cid:96) , we obtain
[Equation]
where ξ is some point of [0 , π ] lying in between ˆ ν (cid:96) and ϕ (ˆ ν (cid:96) − 1 ) .
By (C.23) and (C.26), we have that ¨ ϕ
( L − (cid:96) ) ≤ 0 , whence
[Equation]
− ϕ
[Equation]
ˆ ν
[Equation]
≤
[Equation]
ν
[Equation] − [Equation] Using Lemma E.5 and an induction, we have that ˙ ϕ
( L − (cid:96) ) is decreasing, and moreover by the concavity property we have ϕ (ˆ ν (cid:96) − 1 ) ≥ ˆ ν (cid:96) − 1 / 2 .
An application of Lemmas E.1 and C.13 then yields
E (cid:104)
[Equation]
as long as d ≥ K and n ≥ K (cid:48)
d 4 log 4 n .
In particular, we can choose d = max { K, 2 /c 1 } to obtained the claimed error for the upper bound.
Next, for the lower bound, we make use of the estimate
[Equation]
, which follows from Lemma C.14 and
¨ ϕ ( L − (cid:96) ) ≤ 0 ; by that lemma, we have that f is increasing.
By Lemma E.3, as long as n ≥ K (cid:48) log 4 n , there is an event E on which
| ˆ ν (cid:96) − ϕ
(ˆ ν (cid:96) − 1 ) | ≤
C ˆ ν (cid:96) − 1 (cid:112) log n/n + C (cid:48)
n − 3 and which satisﬁes P [ E | F (cid:96) − 1 ] ≥ 1 − C (cid:48)(cid:48) n − 3 .
In particular, on the event E we have ˆ ν (cid:96) ≥ ˆ ν (cid:96) − 1 / 4 − C (cid:48) /n 3 provided n ≥ 16 C 2 log n , and so on the event E we have ξ ≥ min { ϕ (ˆ ν (cid:96) − 1 ) , ˆ ν (cid:96) − 1 / 4 − C (cid:48) /n 3 } ≥ ˆ ν (cid:96) − 1 / 4 − C (cid:48) /n
We can thus write ϕ (
[Equation]
(cid:96)
[Equation]
(ˆ ν (cid:96) ) − ϕ ( L − (cid:96) +1)
(ˆ ν (cid:96)
[Equation]
where the inequality in the third line follows from boundedness of the angles and the magnitude estimate on f in Lemma C.14, together with our estimate on ξ on E , and the inequality in the ﬁnal line is a consequence of f ≤ 0 , which allows us to drop the indicator for E and obtain a lower bound.
Taking conditional expectations using the previous lower bound and applying F (cid:96) − 1 -measurability of ˆ ν (cid:96) − 1 and boundedness of the angles together with our conditional measure bound on E c , we obtain
[Equation]
where we also apply the complementary bound obtained by our previous work following (D.137).
Since the CL estimate in Lemma C.14 applies also to f , and since f ≤ 0 , an application of
Lemma E.4 with an appropriate choice of d and the choice n ≥ K (cid:48) log 4 n then yields (with a larger absolute constant C (cid:48) )
[Equation]
If we choose n ≥ ( C 6 /C (cid:48) )
L
[Equation]
we can simplify this last estimate to
[Equation]
To conclude, we divide our analysis into two cases: when ˆ ν (cid:96) − 1 ≥ 8 C 4 /n 3 , we have ˆ ν (cid:96) − 1 / 4 −
C 4 n − 3 ≥ ˆ ν (cid:96) − 1 / 8 , and so
[Equation]
where the last inequality follows from Lemma C.14.
On the other hand, when ˆ ν (cid:96) − 1 ≤ 8 C 4 /n 3 , the
CL estimate in Lemma C.14 implies
[Equation]
where the last estimate holds when n ≥ (32 CC 24 /C (cid:48) )
L .
Adding these two estimates together, we obtain one that is valid regardless of the value of ˆ ν (cid:96) − 1 , and choosing n ≥ C 7 log n to combine the residuals, we obtain (after worst-case adjusting the constants)
[Equation]
Combining with our previous work, we obtain
[Equation]
n
[Equation]
after worst-casing constants.
Lemma D.25.
There are absolute constants c 1 , C, C (cid:48) , C (cid:48)(cid:48) , C (cid:48)(cid:48)(cid:48) > 0 and absolute constants K, K (cid:48) > 0 such that for any d ≥ K , if n ≥ K (cid:48)
d 4 log 4 n , then for every L ∈ N and every (cid:96) ∈
[ L ] one has
P (cid:34)(cid:12)(cid:12)(cid:12) ϕ ( L − (cid:96) [Equation] ˆ ν (cid:96) ) − ϕ
[Equation]
L − (cid:96) +
[Equation]
ˆ
[Equation]
(cid:96) − 1 )
[Equation]
and
[Equation]
The constant c 1 is the absolute constant appearing in Lemma E.1.
Proof.
We will ﬁx the meaning of the absolute constants C, C (cid:48) , C (cid:48)(cid:48) > 0 throughout the proof below.
By Lemma E.3, we have if d ≥ K and n ≥ K (cid:48)
d 4 log 4 n that for every (cid:96) ∈ [ L ]
[Equation]
By Lemma C.13, we have the estimate
[Equation]
valid for any (cid:96) ∈ N 0 .
Writing Ξ (cid:96)
=
ˆ ν (cid:96) − ϕ
(ˆ ν (cid:96) − 1 )
so that ˆ ν (cid:96) =
ϕ (ˆ ν (cid:96) − 1 )
+ Ξ (cid:96) , we have that (Ξ (cid:96) ) is adapted to ( F (cid:96) ) , and by the fundamental theorem of calculus
[Equation]
The integrand is nonnegative, so by Jensen’s inequality we have
[Equation]
and an integration then yields
[Equation]
Choosing d ≥ 1 /c
[Equation]
we can guarantee that whenever ν ≥ C (cid:48)
C n − c 1 d/ 2 , one has
[Equation]
and choosing n ≥ 64 C 2 d log n , we can guarantee that
[Equation]
In particular, the last condition guarantees 2 C (cid:112) d log n/n ≤ 1 / 4 .
By concavity of ϕ via Lemma E.5, we have
[Equation] and using (D.138) to obtain
[Equation]
we have by (D.140) and (D.141) as well as the concavity lower bound on ϕ
[Equation]
as long as ˆ ν (cid:96) − 1 ≥ ( C (cid:48) /C )
n − c 1 d/ 2 .
In particular, plugging these bounds into (D.139) and taking square roots [Equation] we obtain by a union bound
[Equation]
whenever ˆ ν (cid:96) − 1 ≥ ( C (cid:48) /C )
n − c 1 d/ 2 .
Meanwhile, when ˆ ν (cid:96) − 1 ≤ ( C (cid:48) /C )
n − c 1 d/ 2 , if we choose n ≥ d log n we have
[Equation]
and we can use the 1 -Lipschitz property of ϕ ( L − (cid:96) ) [Equation] which follows from Lemma E
[Equation] 5, to obtain using (D.138)
[Equation]
≤
[Equation]
−(cid:114)
[Equation]
log n n
[Equation]
F (cid:96) −(cid:35) ≥ 1 − C (cid:48)(cid:48) n − c 1 d . Because | ϕ ( L − (cid:96) ) (ˆ ν (cid:96) ) − ϕ
( L − (cid:96) +1)
(ˆ ν (cid:96) − 1 ) | ≥ 0 , we can then obtain using a union bound
P (cid:34)(cid:12)(cid:12)(cid:12) ϕ
( L − (cid:96) )
(ˆ ν (cid:96) ) − ϕ
( L − (cid:96) +1)
(ˆ ν (cid:96) − 1 ) (cid:12)(cid:12)(cid:12) ≤ (cid:114)[-d log n n (cid:12) (cid:12)(cid:12)(cid:12) ˆ ν (cid:96) −
-]
d log n n (cid:12)(cid:12)(cid:12)(cid:12) {+
+}
C ˆ ν (cid:96) −
+ ( c 0 / 8)
( L − (cid:96) )
ˆ ν (cid:96) − 1 (cid:12)(cid:12)(cid:12)(cid:12)
+ 2 C (cid:48)
n c 1 d/ 2 (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) F (cid:96) −(cid:35) ≥ 1 − 3 C (cid:48)(cid:48) n − c 1 d , which holds regardless of the value of ˆ ν (cid:96) − 1 . We can then obtain the second bound using this one, via a partition of the expectation: let
[Equation]
so that E ∈ F (cid:96)
[Equation]
and P [ E | F [Equation] 1 ] ≥ 1 − 3 C [Equation]
n −
[Equation]
1 d by our work above. Then we have [Equation] where the ﬁrst inequality uses the triangle inequality to obtain (cid:0) ϕ
[Equation]
−
[Equation]
− 1 ) (cid:1) 2 ≤ π 2 ; the second inequality applies the deﬁnition of E , uses nonnegativity to drop the indicator in the ﬁrst term, and applies the conditional measure bound on E c ; and the ﬁnal inequality integrates.
Using the fact that our previous choices of large n imply 2 C (cid:112) d log n/n ≤ 1 / 4 , and that
[Equation]
+
[Equation]
L
[Equation]
1 | ≤ π , we can distribute the square in this ﬁnal bound and worst-case constants to obtain
[Equation]
as claimed.
Lemma D.26.
Let X 1 , . . .
, X L be independent chi-squared random variables, having respectively d 1 , . . .
, d L degrees of freedom.
Write d min = min
i ∈
L d i and let ξ i =
d i X i .
Then there are absolute constants c, C > 0 and an absolute constant 0 < K ≤ 14 such that for any 0 < t ≤ K , one has
[Equation]
In particular, there are absolute constants C (cid:48) , C (cid:48)(cid:48) >
0 and an absolute constant K (cid:48) >
such that for any d > 0 , if d min ≥ K (cid:48) dL then one has
[Equation]
dL
[Equation]
Proof.
For any t ≥ 0 , we have by the AM-GM inequality
P (cid:34) L (cid:89) i = ξ i > 1 + t (cid:35) =
P   (cid:32) L (cid:89) i =
ξ i (cid:33) 1 /L > (1 + t ) 1 /L 
 ≤ P (cid:34)
L
L (cid:88) i = ξ i > (1 + t ) 1 /L (cid:35) .
By convexity of the exponential, we have (1 + t ) 1 /L ≥ 1 + 1 L log(1 + t ) , and by concavity of the logarithm we have log(1 + t ) ≥ t log 2 if t ≤ 1 .
This implies
P (cid:34) L (cid:89) i = ξ i > 1 + t (cid:35) ≤ P (cid:34) − L +
L (cid:88) i =
ξ i > Kt, (cid:35) , where K = log(2) .
Decomposing each X i into a sum of d i i.i.d.
squared gaussians and applying
Lemma G.2, we obtain
[Equation]
where the last inequality holds provided t ≤ CL , where C > 0 is an absolute constant.
Thus, as long as t ≤ CL/K , we have suitable control of the upper tail of the product (cid:81) i ξ i .
For the lower tail, writing log(0)
=
−∞ , we have for any 0 ≤ t <
[Equation]
where the inequality uses concavity of t (cid:55)→ log(1 − t ) .
By Lemma G.2, we have for each i ∈
[ L ] and every 0 ≤ t ≤ C (where C > 0 is an absolute constant)
[Equation]
so that by a union bound and for t ≤ C √ L , we have with probability at least 1 − 2 Le − cd min t 2 /L that 1 − t/ √ L ≤ ξ i ≤ 1 + t/ √ L for every i ∈
[ L ] .
Meanwhile, Taylor expansion of the smooth function x (cid:55)→ log x in a neighborhood of 1 gives
[Equation]
where k is a number lying between 1 and x .
In particular, if x ≥ 12 we have log x ≥ ( x − 1) − 2( x − 1) 2 , whence for t ≤ min { C √ L, 12 }
[Equation]
where the ﬁnal inequality requires in addition t ≤ 14 .
An application of (D.142) then yields the claimed lower tail provided t ≤ CL , which establishes the ﬁrst claim.
For the second claim, we consider the choice t = (cid:112) dL/cd min , for which we have t ≤ K whenever d min ≥ dL/cK 2 , and cd min t 2 /L =
d .
Lemma D.27.
Let X 1 , . . .
, X L be independent Binom( n, 12 ) random variables, and write ξ i =n X i .
Then for any 0 < t ≤ 14 , one has
[Equation]
In particular, for any d > 0 , if n ≥ 128 dL then one has
[Equation]
Proof.
The proof is very similar to that of Lemma D.26.
For any t ≥ 0 , we have by the AM-GM inequality
[Equation]
By convexity of the exponential, we have (1 + t ) 1 /L ≥ 1 + 1 L log(1 + t ) , and by concavity of the logarithm we have log(1 + t ) ≥ t log 2 if t ≤ 1 .
This implies
[Equation]
where K = log(2) .
Decomposing each X i into a sum of n i.i.d.
Bern( 12 ) random variables and applying Lemma G.1 twice, we obtain
[Equation]
This gives suitable control of the upper tail of the product (cid:81) i ξ i .
For the lower tail, writing log(0)
=
−∞ , we have for any 0 ≤ t <
P (cid:34) L (cid:89) i =
[Equation]
where the inequality uses concavity of t (cid:55)→ log(1 − t ) .
By Lemma G.1, we have for each i ∈
[ L ]
P
[Equation]
nt [Equation] [Equation] so that by a union bound, we have that 1 − t/ √ L ≤ ξ i ≤ 1+ t/ √ L for every i ∈
[ L ] with probability at least 1 − 2 Le − nt 2 / 2 L .
Meanwhile, Taylor expansion of the smooth function x (cid:55)→ log x in a neighborhood of 1 gives
[Equation]
where k is a number lying between 1 and x .
In particular, if x ≥ 12 we have log x ≥ ( x − 1) − 2( x − 1) 2 , whence for t ≤ 1 /
[Equation]
where the ﬁnal inequality requires in addition t ≤ 1 / 4 .
An application of (D.143) then yields the claimed lower tail, which establishes the ﬁrst claim.
For the second claim, we consider the choice t =
(cid:112) 8 dL/n , for which we have t ≤ 14 whenever n ≥ 128 dL , and nt 2 / 8 L = d .
Lemma D.28.
For 1 ≤ (cid:96) (cid:48) < (cid:96) ≤ L − 1 deﬁne events
[Equation]
xx
[Equation]
∩
[Equation]
(cid:96) (cid:48)
[Equation]
(cid:48) (cid:105) ≤ Cn (cid:111) and
E
[Equation]
∩  
[Equation]
C (cid:115)
[Equation]
n(cid:96) (cid:12)(cid:12)(cid:12)(cid:12) ∩
[Equation]
(cid:96) (cid:48)
[Equation]
If n, L satisfy the assumptions of corollary D.17 then
[Equation]
If n, L additionally satisfy the conditions of lemmas D.3, E.16 and n ≥ C (cid:48)(cid:48)
L log( n ) (log( L ) + d ) , then
[Equation]
where c, C, C (cid:48) , C (cid:48)(cid:48) are absolute constants.
Proof.
Since
[Equation]
applying corollary D.17 2 n times gives
[Equation]
With the same probability we also have
[Equation]
[Table]
and
[Table]
from which it follows that
[Table]
and
[Table]
It follows that ˜ E (cid:96) : (cid:96) (cid:48)
B holds with the same probability.
From lemma E.16,
[Equation]
[Table]
[Equation]
for some constants c, C (cid:48) .
Here (cid:98) ν
(cid:96) − 1 is the auxiliary angle process deﬁned in (D.2).
Using D.3, we obtain
[Equation]
for an appropriate choice of c, C (cid:48) .
We conclude that
[Equation]
for appropriately chosen constants [Equation] where we used n ≥ C (cid:48)(cid:48)(cid:48)(cid:48) (cid:96) log( n ) (log( (cid:96) ) + d ) .
Lemma D.29.
For ∆ (cid:96) deﬁned in (D.34) and E (cid:96) : (cid:96) (cid:48)
B deﬁned in lemma D.28 we have
[Equation]
Proof.
[Table]
and denoting
[Table]
we have
[Table]
) } we decompose W (cid:96) into a sum of two independent terms as
[Equation]
Note that each H (cid:96) is independent of every other random variable in the problem conditioned on the features. 1 E
[Equation]
tr
[Equation]
B
[Equation]
(cid:105) thus decomposes into a sum of four terms, which we proceed to consider individually and show that they concentrate.
The all G (cid:96) term is
[Equation]
where { u (cid:96) − 1 i } is an orthonormal basis of S (cid:96) − 1 .
If α (cid:96) − 1 ( x ) (cid:54) =
α (cid:96) − 1 ( x (cid:48) ) we choose ( u (cid:96)
[Equation]
, u (cid:96) − 1) = ( α (cid:96) − 1 (
[Equation]
(which are well-deﬁned on E (cid:96) : (cid:96) (cid:48) B ).Using
rotational invariance of the Gaussian distribution, we have
[Equation]
where g i ∼ iid N (0 , 2 n I ) .
If α (cid:96) − 1 ( x ) =
α (cid:96) − 1 ( x (cid:48) ) then dim S (cid:96) − 1 =
and we simply choose u (cid:96) − 1= α (cid:96) − 1 ( x ) (cid:107) α (cid:96) − 1 ( x ) (cid:107) 2 and end up with an identical expression, with ν (cid:96) − 1 = 0 .
Since P g 1 > 0 g j and P
[Equation]
1 > 0 g i are vectors of independent sub-Gaussian random variables with subGaussian norm bounded by (cid:113) Cn , their inner product is a sum of independent sub-exponential variables with sub-exponential norm bounded by Cn for some constant C .
Momentarily abbreviating ˜ v = g 1 cos ν (cid:96) − 1 + g 2 sin ν (cid:96) − 1 , Bernstein’s inequality then gives
[Equation]
for some constant c .
Since on E [Equation]
[Equation]
≤ C(cid:96) [Equation] we obtain
[Equation]
almost surely and thus
[Equation]
for some C (cid:48) , and hence P
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
xx
[Equation]
(cid:105) also contains the terms
[Equation]
[Table]
[Equation]
Considering the ﬁrst of these (since the second can be treated in an identical fashion), we recall that
H (cid:96) is independent of all the other random variables in the problem conditioned on the features, and we thus have
[Equation]
where ˜ W (cid:96) is an independent copy of W (cid:96) , v
[Equation]
= P [Equation] xx
[Equation]
u
[Equation]
w
[Equation]
= P S
[Equation]
⊥ B
[Equation]
− 1:
[Equation]
xx
[Equation]
u (cid:96)i .
Hence conditioned on all the other variables, v (cid:96) ∗ i ˜ W (cid:96) w (cid:96)i is a zero-mean Gaussian with variance 2 (cid:107) v (cid:96)i (cid:107) 2 2 (cid:107) w (cid:96)i (cid:107) 2n 1 E (cid:96) : [Equation] (cid:48) B . Again from the bound on (cid:13)(cid:13)(cid:13)
[Table]
implied by E (cid:96)
[Table]
B , we have [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
n almost surely
[Table]
Gaussian tail bound gives
[Table]
[Equation] The ﬁnal term in tr (cid:104) B (cid:96) :
[Equation]
(cid:48)
xx (cid:48) (cid:105) is
[Equation]
Due to the independence of ˜ W (cid:96) from the remaining random variables, this is simply a Gaussian chaos in n 2 variables. The Hanson-Wright inequality (lemma G.4) gives
[Equation]
[Table]
where in the last inequality we used the deﬁnition of E (cid:96) : (cid:96) (cid:48) B .
It follows that
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Table]
where in the last inequality we used (D.144) and the properties of E (cid:96)
[Table]
B .
Collecting terms and using (D.145), (D.146), (D.147) we obtain
[Table]
[Equation]
[Table] and hence
[Table] [Equation]
[Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
D.
[Equation]
Lemma D.30.
For x ∈ S n 0 − 1 and (cid:96) ∈
[ L ] , denote I (cid:96) ( x ) =
supp( α (cid:96) ( x ) > 0 ) .
If n ≥ K then
P
[Equation]
min
[Equation]
and for any 0 ≤ t ≤ P
[Equation]
[Equation]
|
[Equation]
x [Equation] |
[Equation]
where c, c (cid:48) , C, K are absolute constants.
Proof.
Consider the activations at layer (cid:96) .
From lemma E.16, if n ≥ K we have
[Equation]
Rotational invariance of the Gaussian distribution gives α (cid:96) ( x )
[Equation]
(cid:13)(cid:13)
α (cid:96) − 1 ( x ) (cid:13)(cid:13) 2 (cid:104) W (cid:96) (: , 1) (cid:105) + .
It follows that
E
[Equation]
I
[Equation]
[Table]
[Equation]
[Equation]
Since this variable is a sum of n independent variables taking values in { 0 , 1 }
[Equation]
an application of
Bernstein’s inequality for bounded random variables (lemma G.3) gives
P
[Equation]
for appropriate constants.
A union bound gives
[Equation]
from which
[Equation]
follows.
To prove the second inequality, we use the AM-GM inequality which gives
[Equation]
nL and hence
[Equation]
Convexity of the exponential gives (1+ t ) 1 /L ≥ 1+ 1 L log(1+ t ) and for t ≤ 1 we have log(1+ t ) ≥ t log 2 , giving
[Equation]
(cid:35)
We note that
[Equation]
where b (cid:96)i = 2 n θ (cid:96)i , θ (cid:96)i ∼ iid Bern ( 12 ) and E (cid:96) =
(cid:110) max i b (cid:96) − 1 i (cid:54) = 0 (cid:111) is the event that the features at layer (cid:96) − 1 are not identically 0.
Since n (cid:80) i = [Equation] i =
[Equation]
(cid:96)i a.s. we have P
[Equation]
Since this is a sum of independent bounded random variables, an application of Bernstein’s inequality for bounded random variables (lemma G.3) gives P
[Equation]
for some absolute constant c (cid:48) , where we used L ≥ 1 ≥ t .
Hence
P (cid:34) [Equation] (cid:89) (cid:96) [Equation]
2 | I (cid:96)
[Equation]
| n − 1 ≥ t (cid:35)
[Equation]
2 exp (cid:16)
[Equation]
c (cid:48)
n Lt 2 (cid:17) .
E S HARP B OUNDS ON THE O NE -S TEP A NGLE P ROCESS
In this section, we characterize the process by which angles between features for different pairs of points evolve as they are propagated across one layer of the zero-time network.
This section is self-contained, and as such it will occasionally overload notation used elsewhere in the document for different local purposes.
In particular, we will use the notation σ ( x ) =
[ x ] + for the ReLU in this section (and only in this section), and ˙ σ ( g )
=
1 g > 0 for its weak derivative.
E. D EFINITIONS AND P RELIMINARIES
Let n ∈ N , with n ≥ 2 .
Let g
and g 2 be i.i.d.
N ( 0 , (2 /n )
I ) random vectors; we use µ to denote the joint law of these random variables.
We write G ∈ R n × 2 for the matrix with ﬁrst column g 1 and second column g 2 , and g 1 , . . .
, g n for the n rows of G .
If S ⊂ [ n ] is nonempty and A ∈ R n × m , we write A S ∈ R | S |× m to denote the submatrix of A consisting of the rows indexed by S in increasing index order.
In such situations S c will always denote the complement relative to [ n ] .
For 0 ≤ ν ≤ 2 π , deﬁne random variables
[Equation]
and
[Equation]
Because ˙ v ν separates over coordinates of its arguments and has each of its coordinates the product of a nondecreasing function and a continuous function, it is Borel measurable.
A key property that we will use throughout this section is that the joint distribution of ( g 1 , g 2 ) is rotationally invariant; in particular, it is invariant to rotations of the type
[Equation]
where ν ∈ [0 , 2 π ] .
Since we can write
[Equation]
where all of the R 2 vectors appearing above are elements of S 1 , it follows by applying rotational invariance and the speciﬁc rotation given above that
[Equation]
This equivalence is useful for evaluating expectations and differentiating with respect to ν .
If 0 < c ≤ 0 .
5 and m ∈ N 0 with m < n , deﬁne an event
[Equation]
For each c, m , the set E c,m is closed, since (cid:107)
Av ν (cid:107) is a continuous function of ( g 1 , g 2 )
∈
E m for any linear map A .
We further deﬁne
[Equation]
2 k ) ,m
[Equation]
so that
[Equation]
and E 0 ,m is Borel measurable.
If c is omitted, we take the constant c in the deﬁnition to be 0 .
On
E c,m we guarantee that (cid:107) v ν (cid:107) 0 ≥ m uniformly on [0 , π ] .
Deﬁne a function X ν by
[Equation]
= [Equation] On E 1 , we guarantee that v ν (cid:54) = 0 for every ν , so X ν is well deﬁned; because E 1 is Borel measurable, we have that X ν is Borel measurable, and moreover | X ν | ≤ 1 , so X ν ∈ L pµ for every p ≥ 1 .
Finally, deﬁne for 0 ≤ ν ≤ π
[Equation]
E. M AIN R ESULTS
Lemma E.1.
There exist absolute constants c, C, C (cid:48) > 0 and absolute constants K, K (cid:48) > 0 such that if d ≥ K and n ≥ K (cid:48)
d 4 log 4 n , then one has
[Equation]
Proof.
Using the triangle inequality, we can write
[Equation]
Choose n sufﬁciently large to satisfy the hypotheses of Lemmas E.6 and E.7; applying these lemmas to bound the ﬁrst and second terms, we conclude the claimed result (after choosing n larger than an absolute constant multiple of d log n so that the n − cd error dominates the e − c (cid:48) n error).
Lemma E.2.
One has
[Equation]
Proof.
See (Cho & Saul, 2009).
Lemma E.3.
There exist absolute constants c, C, C (cid:48) , C (cid:48)(cid:48) > 0 and absolute constants K, K (cid:48) >such that if d ≥ K and n ≥ K (cid:48)
d 4 log 4 n , then one has with probability at least 1 − C (cid:48)(cid:48) n − cd
[Equation]
The constant c is the same as the constant appearing in Lemma E.1.
Proof.
Under our hypothess, the second result in Lemma E.6 together with Lemma E.1 and the triangle inequality imply the claimed result (after worst-casing multiplicative constants).
Lemma E.4.
There exist absolute constants c, C, C (cid:48) > 0 and absolute constants K, K (cid:48) > 0 such that if d ≥ K and n ≥ K (cid:48)
d 4 log 4 n , then one has
[Equation]
The constant c is the same as the constant appearing in Lemma E.1.
Proof.
Under our hypotheses, Lemma E.3 is applicable; we let E denote the event corresponding to the bound in this lemma.
By boundedness of cos -1 , nonnegativity of X ν , and ϕ ≤ π/ 2 from
Lemma E.2, we have (cid:107) cos -
X ν − ϕ
( ν ) (cid:107) L ∞ ≤ π .
Thus
[Equation]
as claimed.
Lemma E.5.
One has
1. ϕ ∈ C ∞ (0 , π ) , and ˙ ϕ
and ¨ ϕ extend to continuous functions on [0 , π ] ; 2. ϕ (0)
= 0 and ϕ ( π ) = π/ 2 ; ˙ ϕ (0)
= 1 , ¨ ϕ (0)
=
− 2 / (3 π ) , and ... ϕ (0)
= − 1 / (3 π 2 ) ; and ˙ ϕ ( π ) =
¨ ϕ
( π ) = 0 ; 3.
ϕ is concave and strictly increasing on [0 , π ] (strictly concave in the interior); 4.
¨ ϕ < − c < 0 for an absolute constant c > 0 on [0 , π/ 2] ; 5. 0 < ˙ ϕ < 1 and 0 > ¨ ϕ ≥ − C on (0 , π ) for some absolute constant C > 0 ; 6.
ν (1 − C 1 ν ) ≤ ϕ ( ν ) ≤ ν (1 − c 1 ν ) on [0 , π ] for some absolute constants C 1 , c 1 > 0 .
Proof.
Deferred to Appendix E.4.
E. S UPPORTING R ESULTS E.3. C ORE S UPPORTING R ESULTS
Lemma E.6.
There exist constants c, C, C (cid:48) , C (cid:48)(cid:48) , C (cid:48)(cid:48)(cid:48) , C (cid:48)(cid:48)(cid:48)(cid:48) > 0 and an absolute constant K >such that for any d ≥ 1 , if n and d satisfy the hypotheses of Lemmas E.9 and E.10 and moreover n ≥ Kd log n , then one has
[Equation]
, and with probability at least 1 − C (cid:48)(cid:48) n − cd , one has
[Equation]
Proof.
Fix ν ∈ [0 , π ] .
The function cos -1 is smooth on ( − δ, 1) if 0 < δ < 1 , and Taylor expansion with Lagrange remainder on this domain about the point E
[ X ν ] (by Lemma E.23, we have 0 ≤ E [ X ν ] < 1 if ν > 0 ; we will handle ν = 0 separately below) gives
[Equation]
where ξ lies between x and E [ X ν ] .
Using the fact that X ν (cid:54) = 1 almost surely if ν > 0 , which is established in Lemma E.23, we plug in x = X
ν to get {+cos -1 E [ X ν ] − cos -
 ( X ν [Equation] =+}
[Equation]
where we now express ξ as a function of X ν .
From Jensen’s inequality it is clear
[Equation]
so all that remains is to obtain a matching upper bound for the righthand side of (E.1).
We will make use of the following facts, proved in subsequent sections: there are absolute constants C i > 0 , i ∈
[6] , and c i > 0 , i ∈
[5] , such that
E [ X ν ] ≤ 1 − c 5 ν 2 + C 1 e − c 1 n .
(Lemma E.8) 2.
For each ν , Var[ X ν ] ≤ C 5 ν 4 log n/n + C 2 e − c 2 n .
( Lemma E.9) 3.
With probability at least 1 − C 3 n − c 3 d , one has | X ν − E
[ X ν ] | ≤ C 6 ν 2 (cid:112) d log n/n + C 4 e − c 4 n .
(Lemma E.10)
Let E denote the event on which property 3 holds.
Combining properties 1 and 3 , we obtain with probability at least 1 − C 3 n − [Equation] 3 d
[Equation]
provided n is chosen larger than an absolute constant multiple of d log n .
Thus, deﬁning
[Equation]
we obtain for ν ≥ ν
[Equation]
with the second bound holding with probability at least 1 − C 3 n − c 3 d .
Considering ﬁrst 0 ≤ ν ≤ ν 0 , we obtain using the triangle inequality, Lemma E.20 and property
[Equation]
with the ﬁnal inequality following from the triangle inequality for the (cid:96) 2 norm and the fact that ν ≤ ν 0 .
Meanwhile, if ν ≥ ν 0 , we have by (E.3)
[Equation]
νwith probability at least 1 − C 3 n − c 3 d .
Using 1 − x 2 = (1 + x )(1 − x ) and E [ X ν ] ≥ 0 , ξ ( X ν ) ≥ 0 , we have under this condition on ν
[Equation]
and similarly
[Equation]
Applying (E.6) and taking expectations in (E.1), we obtain by property
[Equation]
Together, (E.2), (E.4) and (E.7) establish the ﬁrst claim provided n is chosen larger than an absolute constant multiple of d log n .
For the second claim, we begin by using the triangle inequality to write
[Equation]
and then observe that our proof of the ﬁrst claim implies suitable control of the second term.
For the ﬁrst term, if ν ≤ ν 0 we use Lemma E.20 to immediately obtain with probability at least 1 − C 3 n − c 3 d that this term is at most Ce − cn .
For ν ≥ ν 0 , we apply property 3 and the bounds (E.5) and (E.6) in the expression (E.1) to obtain with probability at least 1 − C 3 n − c 3 d
[Equation]
which is of the claimed order when n is chosen larger than an absolute constant multiple of d log n .
Lemma E.7.
There exist absolute constants c, C, C (cid:48) , C (cid:48)(cid:48) > 0 such that if n ≥ C log n , one has
[Equation]
Proof.
Write f ( ν ) = cos ϕ ( ν ) , and h ( ν ) =
E [ X ν ] − f ( ν ) , so that h is the residual between the two terms whose images we are trying to tie together.
We will make use of the following results:
The function cos -1 is 12 -Hölder continuous on [0 , 1] , so that | cos -1 x − cos -1 y | ≤ (cid:112) | x − y | if x, y ≥ 0 .
(Lemma E.20) 2.
For ν ∈ [0 , π ] , we have 1 − 12 ν 2 ≤ f ( ν ) ≤ 1 − c 2 ν 2 .
(Lemma E.14) 3.
For all 0 ≤ ν ≤ π , | h ( ν ) | ≤ C 1 e − c 1 n + C 2 ν 2 /n .
(Lemma E.15)
We choose n large enough that the hypotheses of Lemma E.15 are satisﬁed.
Deﬁne ν 0 = 2 (cid:112) C 1 /c 2 e − c 1 n/ 2 .
We split the analysis into two sub-intervals: I 1 :=
[0 , ν 0 ] , and I 2 :
=
[ ν 0 , π ] .
Choosing n larger than an absolute constant multiple of log n , we guarantee that
I
and I 2 both have positive measure.
On I 1 , we proceed as follows:
[Equation]
The ﬁrst inequality uses Hölder continuity of cos -1 , the second uses our bound on the residual, the third uses the deﬁnition of I 1 , and the fourth worst-cases the constants.
On I 2 , we calculate
[Equation]
using the triangle inequality and our bounds on | h | and f .
Using the conditions ν ≥ ν 0 and choosing n ≥ 4 C 2 /c 2 , we can rearrange to get
[Equation]
which implies | f + h | ≤ 1 − c 2 ν 2 / 2 .
By the control f ( ν ) ≤ 1 − c 2 ν 2 , valid on I 2 , we get that both f and f + h are bounded above by 1 − c 2 ν 2 / 2 on I 2 ; moreover, because f ≥ 0 and f + h ≥ 0 , we can apply local Lipschitz properties of cos -1 on I 2 .
This yields
[Equation]
Above, the ﬁrst inequality is the instantiation of the local Lipschitz property; the second applies our upper and lower bounds on f and f + h derived above, and our bound on the residual | h | ; the fourth applies the bound 0 ≤ f ( ν ) ≤ 1 − 12 c 2 ν 2 to conclude 2 − 12 c 2 ν 2 ≥ 1 on I 2 , and cancels a factor of ν in the second term; and in the last line, we apply ν ∈ I 2 to get ν ≥ 2 (cid:112) C 1 /c 2 e − c 1 n/ 2 , which allows us to cancel the ν − 1 factor in the ﬁrst term of the previous line.
To wrap up, we can choose the largest of the constants appearing in the bounds derived for I 1 and
I 2 above and then conclude, since I 1 ∪ I 2 =
[0 , π ] under our condition on n .
[Equation]
Lemma E.8.
There exist absolute constants c, c (cid:48) , C, C (cid:48) , C (cid:48)(cid:48) > 0 such that if n ≥ C and if n is sufﬁciently large to satisfy the hypotheses of Lemma E.15, one has
[Equation]
Proof.
By the triangle inequality, we have
[Equation]
Applying Lemmas E.14 and E.15 with m = [Equation] we get
[Equation]
which proves the claim if we choose n ≥ 2 C (cid:48) /c .
Lemma E.9.
There exist absolute constants c, C, C (cid:48) > 0 such that if n satisﬁes the hypotheses of
Lemmas E.11 and E.12, then one has for each ν ∈ [0 , π ]
[Equation]
Proof.
We use the following elementary fact for a random variable with ﬁnite ﬁrst and second moments, easily proved using Var[ X ν ] =
E [ X 2 ν ] − E
[ X ν ] 2 and Fubini’s theorem: in this setting one has
[Equation]
By Lemma E.11, there is an event E of probability at least 1 − Ce − cn on which Var[ X ν ( g 1 , · )] ≤
C (cid:48)
ν 4 /n + C (cid:48)(cid:48)
e − c (cid:48) n .
Invoking as well Lemma E.12, we obtain
[Equation]
as claimed, where in the second line we applied nonnegativity of the variance and the Schwarz inequality, and in the third line we used the fact that (cid:107) X (cid:107) L 2 ≤ (cid:107) X (cid:107) L ∞ for any random variable X in L ∞ .
Lemma E.10.
There exist absolute constants c, c (cid:48) , C, C (cid:48) , C (cid:48)(cid:48) > 0 and absolute constants K, K (cid:48) >such that for any d ≥ 1 such that n and d satisfy the hypotheses of Lemmas E.11 and E.13 and n ≥ max { K log n, K (cid:48)
d } , for any ν ∈ [0 , π ] , one has
[Equation]
Proof.
By Lemma E. [Equation] we have
[Equation]
Let ψ = ψ 0 .
25 denote the cutoff function deﬁned in Lemma E.
[Equation]
and write
[Equation]
By Lemma E.13, we have
[Equation]
We have X ν
= Y ν on the event E 1 , by Lemma E.
[Equation]
and we thus calculate using the triangle inequality
[Equation]
where the last inequality uses Hölder’s inequality and the measure bound in Lemma E.16.
Again using the triangle inequality [Equation] we have
[Equation]
and so using our previous calculation and Markov’s inequality, we can assert
[Equation]
The claim then follows from the triangle inequality, a union bound [Equation] and a choice of n larger than an absolute constant multiple of log n and an absolute constant multiple of d .
Lemma E.11.
There exist absolute constants c, c (cid:48) , c (cid:48)(cid:48) , c (cid:48)(cid:48)(cid:48) , C, C (cid:48) , C (cid:48)(cid:48) , C (cid:48)(cid:48)(cid:48) , C 4 , C 5 > 0 , and absolute constants K, K (cid:48) > 0 such that for any d ≥ 1 , if n ≥ max { Kd, K (cid:48) log n } , then for every ν ∈ [0 , π ] one has with probability at least 1 − Ce − cn
[Equation]
and with ( g 1 , g 2 ) probability at least 1 − C (cid:48)(cid:48)
e − c (cid:48)(cid:48)
d one has
[Equation]
Proof.
Fix ν ∈ [0 , π ] .
Let E
1 = E 0 .
5 , 1 denote the event in Lemma E.16 which is in the deﬁnition of X ν .
We start by treating the case of ν = 0 or ν = π .
We have X π = 0 deterministically, so the variance is zero and it equals its partial expectation over g 2 with probability one.
For the other case, one has X 0 = 1 E 1 ; we have
[Equation]
and since E [ 1 E 1 ] = 1 − Cne − cn by Lemma E.16, we obtain by Markov’s inequality
[Equation]
This gives a suitable bound on the variance with suitable probability.
For deviations, we note that
[Equation]
and following our previous variance inequality but taking expectations over both g 1 and g 2 gives
Var[ X 0 ] ≤ Cne − cn , which implies by Chebyshev’s inequality
[Equation]
which is a suitable deviations bound that we can union bound with the event constructed below, which controls deviations uniformly for the remaining values of ν .
We therefore assume below that 0 < ν < π .
Let ψ ( x ) = max { x, 18 } , which is continuous and differentiable except at x = 18 , with derivative ψ (cid:48)
( x ) = 1 x> 1 / 8 .
We note in addition that x ≤ ψ ( x ) , and since ψ ≥ 18 we have for x ≥
the bound x/ψ ( x ) ≤ 1 .
Deﬁne
[Equation]
We ﬁrst show that it is enough to prove the claims for Y ν , which will be preferable for technical reasons.
On E 1 , we have Y ν =
X ν .
We have | Y ν | ≤ 1 , and we calculate
[Equation]
where the ﬁrst inequality uses the Schwarz inequality, and the last inequality uses that |
X ν | ≤ 1 and the triangle inequality, and where C > 0 is an absolute constant.
We have by Tonelli’s theorem and
Lemma E.
[Equation]
so Markov’s inequality implies
[Equation]
Thus, with probability at least 1 − e − cn/ 2 , we have
[Equation]
so that an application of Lemma E.32 yields that with probability at least 1 − e − cn/
[Equation]
where we have worst-cased constants and the exponent on n .
For deviations, we write using the triangle inequality
[Equation]
and then note that the ﬁrst term is identically zero on the event E 1 , which has probability at least 1 − Ce − cn , whereas for the third term, we have
[Equation]
where the ﬁrst inequality uses the triangle inequality and the Lyapunov inequality, and the second inequality holds with probability at least 1 − e − cn/ 2 , and leverages the argument we used to control the difference in variances.
Ultimately taking union bounds, we can conclude that it sufﬁcient to prove the claimed properties for Y ν .
With 0 < ν < π ﬁxed, we introduce the notation
[Equation]
so that
[Equation]
For ﬁxed g 1 , we will write Y ν
( g 2 )
=
Y ν ( g 1 , g 2 ) with an abuse of notation.
For ¯ g ∈ R n arbitrary and g 2 ﬁxed, we consider the function f ( t ) =
Y ν
( g 2 + t ¯ g ) for t ∈ [0 , 1] .
Writing f (cid:48) for the derivative of f where it exists, at any point of differentiability, we calculate by the chain rule
[Equation]
where
[Equation]
Using the fact that
[Equation]
where P { v g 1 , g 2 > 0 } is the orthogonal projection onto the coordinates where v g 1 , g 2 is positive, together with the fact that
[Equation]
we can also write
[Equation]
We next argue that f does not fail to be differentiable at too many points of [0 , 1] .
Because ψ > 0 , it will sufﬁce to show that (i) t
(cid:55)→ v g 1 , g 2 + t ¯ g and (ii) t (cid:55)→ ψ ( (cid:107) v g 1 , g 2 + t ¯ g (cid:107) 2 ) are differentiable at all but a set of isolated points in [0 , 1] .
For the latter function, we note that at any point where (cid:107) v g 1 , g 2 + t ¯ g (cid:107) 2 < 18 , by continuity we have that t (cid:55)→ ψ ( (cid:107) v g 1 , g 2 + t ¯ g (cid:107) 2 ) is locally constant, and therefore differentiable at such points.
At other points, by Lemma E.21 it sufﬁces to characterize t (cid:55)→ (cid:107) v g 1 , g 2 + t ¯ g (cid:107) 2 as differentiable at all but isolated points, and because (cid:107) v g 1 , g 2 + t ¯ g (cid:107) 2 ≥ 18 by assumption, the norm is differentiable and by the chain rule it sufﬁces to characterize differentiability of each coordinate of t (cid:55)→ v g 1 , g 2 + t ¯ g , which settles the question of all-but-isolated differentiability of (i) as well.
We have v g 1 , g 2 + t ¯ g = σ ( g 1 cos ν + g 2 sin ν + t
¯ g sin ν ) , so again by Lemma E.21, we conclude from differentiability of t (cid:55)→ g 1 cos ν + g 2 sin ν + t
¯ g sin ν that t (cid:55)→ v g 1 , g 2 + t ¯ g is differentiable at all but isolated points, and consequently so is f .
In particular, f is differentiable at all but countably many points of [0 , 1] .
Next, we show that f (cid:48) has suitable integrability properties.
Indeed, we calculate using (E.8)
[Equation]
where we used Cauchy-Schwarz and ψ ≥ 18 in the ﬁrst inequality and distributed and applied ( ψ (cid:48) )
2 = ψ (cid:48) and the estimate x/ψ ( x ) ≤ 1 in the second inequality.
In particular, this implies that | f (cid:48) ( t ) | ≤ C (cid:107) ¯ g (cid:107) 2 , which is a t -integrable upper bound for every ¯ g .
Because Y ν ( g 1 , · ) is continuous by continuity of σ , ψ , and the fact that ψ becomes constant whenever (cid:107) v g 1 , g 2 (cid:107) 2 < 18 , we can apply (Cohn, 2013, Theorem 6.3.11) to get
[Equation]
and since ¯ g was arbitrary, for any g (cid:48) 2 ∈ R n we can put ¯ g = g (cid:48) 2 − g 2 to get
[Equation]
Performing the expansion with g 2 and g (cid:48) 2 reversed and applying the triangle inequality and CauchySchwarz then implies the estimate
[Equation]
This relation is enough to conclude the result for angles satisfying ν ≥ c 0 , where 0 < c 0 ≤ π/is an absolute constant.
Indeed, (E.9) and (E.10) imply that Y ν is C -Lipschitz, where C > 0 is an absolute constant; so the Gaussian Poincaré inequality implies
[Equation]
and Gauss-Lipschitz concentration implies for any d ≥
[Equation]
Because ν ≥ c 0 , we can adjust these bounds to involve ν 4 and ν 2 (respectively) while only paying increases in the constant factors.
We proceed assuming 0 < ν ≤ c 0 .
Let 0 ≤ τ g 1 ≤ 1 denote a median of
Y ν
( g 1 , · ) , i.e., a number satisfying P g
[ Y ν ≥ τ g 1 ] ≥ 12 and
P g 2 [ Y ν ≤ τ g 1 ] ≥ 12 , and for each 0 ≤ s < τ g 1 deﬁne
[Equation]
For any 0 ≤ s < τ g 1 , notice that w s ≥ Y ν , which implies that P [ w s ≥ τ g 1 ] ≥ P [ Y ν ≥ τ g 1 ] ≥ 12 , because τ g 1 is a median of Y ν ; and similarly P [ w s ≤ τ g 1 ] ≥ P [ Y ν ≤ τ g 1 ] ≥ 12 , so that τ g 1 is also a median of w s .
The fact that w s ≥ Y ν implies for any t > 0 that P
[ Y ν − τ g 1 > t ] ≤ P [ w s − τ g 1 > t ] , and additionally if Y ν ≤ τ g 1 − s we have w s =
τ g 1 − s , so that P [ Y ν − τ g 1 ≤ − s ] ≤ P [ w s − τ g 1 ≤ − s ] .
In particular, the tails of Y ν can be controlled in terms of those of w s for appropriate choices of s .
Additionally, by Lemma E.21, we have that for each s , t (cid:55)→ w s ( g 2 + t ¯ g ) is differentiable at all but countably many points of [0 , 1] , and has derivative there equal to t (cid:55)→ (cid:104) ¯ g , ∇ g 2 w s ( g 2 ) (cid:105) , where
[Equation]
, which, following from (E.9), satisﬁes a strengthened gradient norm estimate
[Equation]
In particular, we obtain a nearly-Lipschitz estimate of the form (E.10):
[Equation]
For each g 1 , we deﬁne a set S g 1 = { g 2 | (cid:107) v g 1 , g 2 (cid:107) 2 ≥ 14 } .
Noting that the function g 2 (cid:55)→ (cid:107) σ ( g 1 cos ν
+ g 2 sin ν ) (cid:107) 2 is a convex 1 -Lipschitz function (given that | sin ν | ≤ 1 ), we have by
Gauss-Lipschitz concentration
[Equation]
and by Jensen’s inequality
[Equation]
where the last line holds because ν ≤ π/ 4 .
By Lemma E.16, there is a g 1 event E having probability at least 1 − Ce − cn on which (cid:107) u g 1 (cid:107) 2 ≥ 12 , so that for any g 1 ∈ E , we have by a suitable choice of t in our Gauss-Lipschitz bound P g
[ S g 1 ] ≥ 1 − e − cn .
Thus, using the ﬁrst line of (E.11), the Gaussian
Poincaré inequality and the Lipschitz property of w s (which follows from (E.12) after bounding by an absolute constant) and Rademacher’s theorem on a [Equation] e. differentiability of Lipschitz functions, we have whenever g 1 ∈ E
[Equation]
E [Equation] [Equation] where we also make use of the fact that 0 ≤ Y ν ≤ 1 .
Now
[Equation]
we calculate for g 1 ∈ E and g 2 ∈ S g
[Equation]
E.
[Equation]
where the second inequality uses g 1 ∈ E , the third uses nonexpansiveness of σ , and the ﬁrst requires a proof; we will show that for any nonzero vectors x , y ∈ R n , one has
[Equation]
To see this, write θ for the angle between x and y , and distribute to obtain equivalently
[Equation]
Divide through by (cid:107) x (cid:107) 22 , write K = (cid:107) y (cid:107) 2 (cid:107) x (cid:107) − 1 2 , and rearrange to obtain the equivalent expression
[Equation]
It sufﬁces to minimize the LHS of the previous inequality with respect to K subject to the constraint
K >
and then study the resulting function of θ to ascertain the validity of the bound.
Given that 1 + cos θ ≥ 0 , the LHS is a convex function of K , with minimizer K = 2 cos θ (1 + cos θ ) − 1 , and therefore for any θ ≥ π/ 2 , the LHS subject to the constraint K > 0 is minimized at K = 0 , where the inequality is easily seen to be true.
If θ < π/ 2 , we have that the minimizer is positive, and we verify that after substituting the bound becomes
[Equation]
which is also seen to be true for θ < π/ 2 , for example by showing that the polynomial x (cid:55)→ − 2 x 2 + x + 1 is nonnegative on [0 , 1] .
This proves the inequality, so returning to (E.14), we have
[Equation]
using Cauchy-Schwarz in the second inequality.
By Gauss-Lipschitz concentration (e.g. following the proof of the third assertion in Lemma E.17), there is a g 1 event E (cid:48) and a g 2 event E (cid:48)(cid:48) , each with probability at least 1 − Ce − cn , on which we have (respectively) (cid:107) g
i (cid:107) 2 ≤ 2 for i = 1 , 2 .
Then using (sin ν )(1 − cos ν ) ≥ 0 , we obtain that when g 1 ∈ E ∩ E (cid:48) and when g 2 ∈ S g 1 ∩ E
(cid:48)(cid:48)
Y
[Equation]
, where the ﬁnal inequality uses the standard estimate cos ν ≥ 1 − 0 .
5 ν 2 , which can be proved via
Taylor expansion.
By a union bound, we can assert that with g 1 -probability at least 1 − Ce − cn , with conditional (in g 2 ) probability at least 1 − C (cid:48)
e − c (cid:48) n we have Y ν ≥ 1 − 32 ν 2 , so that in particular, by nonnegativity of Y ν , and choosing n larger than an absolute constant, we guarantee with g 1probability at least 1 − Ce
[Equation]
Plugging the mean estimate into (E.13), we conclude with probability at least 1 − C (cid:48)(cid:48)
e − c
[Equation]
n
[Equation]
We could have just as well applied this exact argument to Y ν instead of w s , so we conclude the claimed variance bound from this expression.
We have stated the result in terms of the truncations w s so that it can be applied towards deviations control in the sequel.
As an immediate application, we use the fact that any median is a minimizer of the quantity c (cid:55)→ E [ | X − c | ] for any integrable X and c ∈ R to get with probability at least 1 − C (cid:48)(cid:48)
e − c [Equation] n
[Equation]
where we also applied Jensen’s inequality for the ﬁrst inequality and the Lyapunov inequality for the third.
In particular, the same argument yields
[Equation]
We turn to removing the t dependence in (E.12) without sacriﬁcing the dependence on τ g 1 .
To obtain a Lipschitz estimate on the subset S g 1 we need to control the norm of ∇ g 2 w s on the line segment between g 2 , g (cid:48) 2 ∈ S g 1 .
For this, write σ y ( x ) = max { x − y, 0 } for any y ∈ R , and make the following observations:
v g 1 , g 2 =
(sin ν ) σ −
[Equation]
1 cot ν ( g 2 ) , so that
[Equation]
for any x, y , σ y ( x ) = max { x, y } − y ; x (cid:55)→ max { x , y } is the projection onto the convex set { x | x i ≥
y i ∀ i } , so in particular x (cid:55)→ σ y ( x ) is nonexpansive, has convex range, and satisﬁes σ y ( σ y ( x ) + y )
=
σ y ( x ) ; and thus 3.
for any g 2 , Y ν ( g 2 ) =
Y ν ( σ − g 1 cot ν
( g 2 ) − g 1 cot ν ) .
We write ˜ g
2 = σ − g 1 cot ν ( g 2 ) − g 1 cot ν , ˜ g
[Equation]
2 = σ − g 1 cot ν ( g (cid:48) 2 ) − g 1 cot ν , so that (E.12) becomes | w s ( g (cid:48) 2 )
[Equation]
( g 2 ) | =
| w
[Equation]
(˜ g
[Equation] [Equation]
(˜ g 2 )
|
[Equation]
where the second line follows from nonexpansiveness and translation invariance of the distance.
Having reduced to the study of points along the segment between ˜ g 2 and ˜ g (cid:48)
[Equation]
we now observe
[Equation]
because σ − g 1 cot ν has image included in the nonnegative orthant
[Equation]
which is convex.
It then follows from (1) above that
[Equation]
and in particular
[Equation]
where the ﬁrst inequality uses that σ ≥ 0 and g 2 , g (cid:48) 2 ∈ S g 1 , and the second minimizes the convex function of t in the previous bound.
We conclude that g 2 , g (cid:48) 2 ∈ S g 1 implies that (cid:107) v g 1 ,t ˜ g (cid:48)
2 +(1 − t )˜
g 2 (cid:107) 2 > 18 for every t ∈ [0 , 1] , and consequently (E.12) becomes (after an additional simpliﬁcation of the quantity under the square root using τ g 1 ≤ 1 )
[Equation]
so that w s is 16 ν (cid:112) 1 − ( τ g 1 − s ) -Lipschitz on S g 1 .
Then by an application of the median bound in (E.16), if 0 ≤ s < 1 − 32 ν 2 , with g 1 probability at least 1 − Ce − cn we have that w s is 16 ν √ 32 ν 2 + s -Lipschitz on S g 1 .
For the previous assertion to be nonvacuous, we need to take ν small; in particular, we have 1 − 32 ν 2 > 12 if ν < 1 / 8 , which we can take to be the value of the absolute constant c 0 we left unspeciﬁed previously.
Then for each such s , deﬁne L s = 16 ν √ 32 ν 2 + s , and deﬁne
[Equation]
Then ˆ w s is L s -Lipschitz on R n , and satisﬁes ˆ w s = w s on S g 1 (Evans & Gariepy, 1991, §3.1.
Theorem 1).
By the Gaussian Poincaré inequality, we obtain immediately Var[ ˆ w s ] ≤ L s
[Equation]
and using ˆ w s = w s on S g 1 , we compute
[Equation]
[Equation]
where the second inequality follows from the Schwarz inequality, the third holds given that g 1 ∈ E and by the Minkowski inequality, and the ﬁnal uses that w s and ˆ w s are both Lipschitz with Lipschitz constants bounded above by absolute constants together with the Gaussian Poincaré inequality.
Meanwhile, by Gauss-Lipschitz concentration, we obtain a Bernstein-type lower tail
[Equation]
and for the upper tail, it will be sufﬁcient to consider ˆ w 0 , which satisﬁes a subgaussian tail (for any t ≥ 0 )
[Equation]
Using the results (E.18), (E.19), (E.21), and the fact that w s = ˆ w s on S g 1 , we get
[Equation]
Using d ≥ 1 , we put s = 2 Cν 2 (cid:112) d/n + C (cid:48)
e − cn in this bound; using that ν < 1 / 8 , and in particular 1 − 32 ν 2 > 12 , we can choose n larger than an absolute constant multiple of d to guarantee that for all 0 ≤ ν < 1 / 8 , this choice of s is less than 1 − 32 ν 2 , and that Cν 2 (cid:112) d/n ≤ 32 ν 2 .
Together with the lower tail bound (E.22), these facts imply
[Equation]
Meanwhile, for the upper tail, we have for any t ≥
[Equation]
and if we put t = 2 Cν 2 (cid:112) d/n + C (cid:48)
e cn
[Equation]
our previous requirements on n and the upper tail bound (E.23) yield
[Equation]
Combining these two bounds gives control of absolute deviations about the mean.
By independence, we conclude
[Equation]
To conclude, we have shown that for every ν ∈ [0 , π ] one has with probability at least 1 − Ce − cn
[Equation]
and with ( g 1 , g 2 ) probability at least 1 − 2 e − c (cid:48)(cid:48)
d + C (cid:48)(cid:48)(cid:48) ne − c (cid:48)(cid:48)(cid:48) n one has
[Equation]
To simplify these bounds, we may in addition choose n larger than an absolute constant multiple of log n , and n larger than an absolute constant multiple of d , to obtain that with probability at least 1 − Ce − cn
[Equation]
and with ( g 1 , g 2 ) probability at least 1 − C (cid:48)(cid:48)
e − c
[Equation]
d one has
[Equation]
which was to be shown.
Lemma E.12.
There exist absolute constants c, C, C (cid:48) > 0 and an absolute constant K > 0 such that if n ≥ K log 4 n , then for every ν ∈ [0 , π ] one has
[Equation]
Proof.
Deﬁne
[Equation]
where ψ = ψ 0 .
25 is as in Lemma E.31.
Then by Cauchy-Schwarz and property 2 in Lemma E.
(the case where either (cid:107) v 0 (cid:107) 2 = 0 or (cid:107) v ν (cid:107) 2 = 0 is treated separately, since in this case Y ν = 0 ), we obtain | Y ν | ≤ 4 , and
[Equation]
where we use the fact that if ( g 1 , g 2 ) ∈
E m then (cid:107) v ν (cid:107) 2 ≥ 12 for every 0 ≤ ν ≤ π and hence ψ ( (cid:107) v ν (cid:107) 2 ) =
(cid:107) v ν (cid:107) 2 in the ﬁrst line, apply Jensen’s inequality in the second line, and combine our bound on Y ν with Hölder’s inequality and the measure bound in Lemma E.16 in the third line.
An application of Lemma E.32 then yields
[Equation]
where the last inequality holds when n is chosen to be larger than an absolute constant multiple of log n .
It thus it sufﬁces to control the variance of Y ν .
Applying Lemma E.26, we get for almost all g 1 ∈ R n
[Equation]
where we follow the notation deﬁned in Lemma E.13.
We start by removing the term outside of the integral from consideration.
We have as above | Y ν | ≤ 4 , so that | E g 2 [ Y ν ] | ≤ 4 .
Moreover, following the proof of the measure bound in Lemma E.16, but using only the pointwise concentration result, we assert that if n ≥ C an absolute constant there is an event E on which 0 .
5 ≤ (cid:107) v 0 (cid:107) 2 ≤ 2 with probability at least 1 − 2 e − cn with c > 0 an absolute constant.
This implies that if g 1 ∈ E we have
[Equation]
and since
[Equation]
by the same argument used for Y ν , we can calculate
[Equation]
by the Minkowski inequality and the triangle inequality.
An application of Lemma E.32 implies that it is therefore sufﬁcient to control the variance of the quantity
[Equation]
By Lemma E.37, the Lyapunov inequality, and Fubini’s theorem, we have
[Equation]
) − E [ f ( ν, g 1 )]) 2 =
[Equation]
Using the elementary inequality
[Equation]
valid for any square integrable g :
[0 , π ] → R and proved with two applications of Jensen’s inequality, and Lemma E.37, we obtain
[Equation]
Thus, again by Lemma E.37, the Lyapunov inequality, Fubini’s theorem, and compactness of [0 , π ] , we have
[Equation]
We can control the variance under the integral using a combination of Lemmas E.35 and E.37, together with the deviations control given by Lemmas E.39, E.41 to E.44 and E.46, since we have chosen n according to the hypotheses of Lemma E.13.
In particular, these lemmas furnish deviation bounds of size at most
[Equation]
that hold with probabilities at least 1 − C
[Equation]
i n − c (cid:48)(cid:48) i d − C (cid:48)(cid:48)(cid:48)
i ne − c (cid:48)(cid:48)(cid:48) i n , for any d ≥ 1 larger than an absolute constant and suitable absolute constants speciﬁed above.
We can simplify these bounds as follows:
ﬁrst, choose n such that n ≥ (2 /c (cid:48)(cid:48)(cid:48) i ) log n for each i , which guarantees that the bounds hold with probability at least 1 − C (cid:48)(cid:48)
i n − c (cid:48)(cid:48) i
d − C (cid:48)(cid:48)(cid:48) i e − c (cid:48)(cid:48)(cid:48) i n/ 2 .
Next, choose n ≥ (2 c (cid:48)(cid:48) i /c
(cid:48)(cid:48)(cid:48) i )
d log n for all i , which implies that the bounds hold with probability at least 1 − 2 max { C (cid:48)(cid:48)
i , C (cid:48)(cid:48)(cid:48) i } n − c
(cid:48)(cid:48)
i d .
Similarly, we also choose n such that n ≥ (2 /c (cid:48) i ) log n for each i , which guarantees that the error terms that are exponential in n in the bounds are upper bounded by C (cid:48)
i e − c (cid:48)
i n/ 2 , and, choose n ≥ (2 c i /c (cid:48) i )
d log n for all i , which implies that for all
i
[Equation]
Finally, we make the particular choice d = 4 / min i { c i , c (cid:48)(cid:48) i } , or the minimum required value of d , whichever is larger, so that there are absolute constants C, C (cid:48) , C (cid:48)(cid:48) > 0 such that with probability at least 1 − C (cid:48)(cid:48) n − 4 we have for all
i
[Equation]
where the last inequality holds when n is larger than an absolute constant.
With these bounds, we can now invoke Lemma E.35 with Lemma E.37 to get
[Equation]
for different absolute constants C, C (cid:48) , C (cid:48)(cid:48) > 0 , and where the last inequality again holds n is larger than an absolute constant.
Plugging back into (E.26) and evaluating the integrals, we get
[Equation]
which is enough to conclude.
Lemma E.13.
Write
[Equation]
where ψ = ψ 0 .
25 is as in Lemma E.31.
There exist absolute constants c, c (cid:48) , C, C (cid:48) , C (cid:48)(cid:48) > 0 and absolute constants K, K (cid:48) > 0 such that for any d ≥ 1 , if n ≥ Kd 4 log 4 n and if d ≥ K (cid:48) , then there is an event E such that
One has
[Equation]
if g 1 ∈ E ; 2.
One has
[Equation]
Proof.
Fix d > 0 , and write
[Equation]
Applying Lemma E.26, we get for almost all g 1 ∈ R n
[Equation]
where Ξ 1 ( s, g 1 , g 2 ) = n (cid:88) i =
[Equation]
( g 1 i
[Equation]
[Equation]
v
[Equation]
ψ
[Equation]
(cid:107)
[Table]
[Equation]
Here we put Ξ 1 (0 , g 1 , g 2 )
=
Ξ 1 ( π, g 1 , g 2 ) = 0 , which does not affect the integral and which is equal to the limits
lim ν (cid:38)
Ξ 1 ( ν, g 1 , g 2 ) =
lim ν (cid:37) π Ξ 1 ( ν, g 1 , g 2 ) for every ( g 1 , g 2 ) .
Momentarily ignoring measurability issues, it is of interest to construct g 1 events E i of suitable probability on which we have
[Equation]
for each i = 1 , . . . , 6 , and a g 1 event E 7 on which we have
[Equation]
We can then consider the event E
= (cid:84) 7 i =
1 E i , possibly minus a negligible set on which (E.27) fails to hold, which has high probability via a union bound and on which we have simultaneously for all ν ∈ [0 , π ]
[Equation]
+
[Equation]
(cid:88)
i
[Equation]
(cid:90)
[Equation] [Equation] [Equation] ≤ Cν
[Equation]
(cid:32)(cid:114)
d
[Equation]
n n
[Equation]
n
[Equation]
(cid:33)
[Equation]
C (cid:48) ne [Equation] by Fubini’s theorem and Lemma E.37, the triangle inequality (for | · | and for the integral), (E.28), and using ν 2 ≤ π 2 and worst-casing the remaining constants.
To establish the bounds (E.28), we will employ lemma Lemma E.48, which shows that it is sufﬁcient to obtain pointwise control and show a suitable s -Lipschitz property for each i ∈ [6] ; following the lemma, these properties also imply Lebesgue measurability of the suprema immediately.
Reduction to product space events.
Fix ν .
By the triangle inequality, we have for each i = 1 , . . . ,
[Equation]
Suppose we can construct ( g 1 , g 2 )
events E (cid:48)
i such that
If ( g 1 , g 2 ) ∈ E (cid:48)
i , then
[Equation]
One has P
[Equation]
Then for each such i , we can write
[Equation]
using nonnegativity of the integrand and boundedness of the indicator for E (cid:48)
i in the second line.
The random variable remaining in the second line is nonnegative, and by Fubini’s theorem (with
Lemma E.37 for joint integrability) and the Schwarz inequality we have
[Equation]
where the second line applies Lemma E.37 and the Lyapunov inequality.
We can replace this last inequality with one equivalent to the measure bound on ( E (cid:48) i )
c using subadditivity of the square root and reducing the constants c (cid:48)
i and c (cid:48)(cid:48)
i by a factor of 2 .
Using this last inequality, Markov’s inequality implies for any t ≥
[Equation]
which, together with (E.29) and after worst-casing some exponents and constants, implies that there is a g 1 event E i that satisﬁes (the constants C and C (cid:48) are scoped across properties 1 and 2 )
If g 1 ∈ E i , then
[Equation]
One has
[Equation]
Thus, we can pass from ( g 1 , g 2 ) events to g 1 events with only a worsening of constants
[Equation]
and it sufﬁces to construct the events E (cid:48) i .
Additionally, we can leverage this same framework to pass ν -uniform control from the product space to g 1 -space.
Suppose we can construct ( g 1 , g 2 )
events E (cid:48)
i such that
If ( g 1 , g 2 ) ∈ E (cid:48)
i , then
[Equation]
One has P [ E (cid:48)
i ] ≥ 1 − C
[Equation]
Then following (E.30)
[Equation]
we can assert ∀
[Equation]
To get uniform control of this last random variable
[Equation]
we can use Lemma E.
[Equation]
which tells us that we have a bound
[Equation]
) where f i is in L 4 ( R n × R n ) , and has L 4 norm bounded by an absolute constant C i > 0 .
Then
Fubini’s theorem and the Schwarz inequality allow us to assert
[Equation]
which can be controlled exactly as in the pointwise control argument.
In particular, an application of Markov’s inequality gives
[Equation]
so that, returning to (E.31), we have uniform control of the quantity | E g
[Ξ i ( ν, g 1 , g 2 )] − E g 1 , g 2 [Ξ i ( ν, g 1 , g 2 )] | on an event of appropriately high probability.
In particular, we have incurred only losses in the constants compared to the pointwise case.
Approach to Lipschitz estimates.
We will use this framework for controlling the Ξ 1 and Ξterms only.
Accordingly, the sections for those terms below will produce results of the following type, for absolute constants
c i , c (cid:48)
i , c (cid:48)(cid:48)
i , c (cid:48)(cid:48)(cid:48) i , C i , C (cid:48) i , C (cid:48)(cid:48) i , C (cid:48)(cid:48)(cid:48)
i , C (cid:48)(cid:48)(cid:48)(cid:48) i > 0 for i = 1 , 2 , and parameters d ≥ 1 , δ > 0 such that d and δ are larger than (separate) absolute constants and n satisﬁes certain conditions involving d :
For each ν ∈ [0 , π ] ﬁxed, with probability at least 1 − C
[Equation]
n , we have that
[Equation]
With probability at least 1 − C (cid:48)(cid:48)
[Equation]
e −
[Equation]
C
[Equation]
2 n − δ , we have that | E g
[Ξ i ( ν, g 1 , g 2 ) − E [Ξ i ( ν, g 1 , g 2 )]] | is ( C 2 + C (cid:48)
2 n 1+ δ ) -Lipschitz.
We show here that we can use these properties to obtain uniform concentration of the relevant quantities.
Write M = C 1 (cid:112) d log n/n + C
[Equation]
1 n
[Equation]
+ C (cid:48)(cid:48) 1 ne − c (cid:48) 1 n ; we are interested in showing that uniform bounds of sizes close to M hold with probability not much smaller than that of the pointwise bounds.
By Lemma E.48, it follows from the assumed properties that for any 0 < ε < 1 one has
P (cid:34) sup
[Equation]
where K > 0 is an absolute constant.
To make the RHS of the bound on the supremum of size comparable to M , it sufﬁces to choose ε =
C 1 (cid:112) d log n/n/ ( C 2 + C (cid:48) 2 n 1+ δ ) .
We have
C 2 + C (cid:48)
2 n 1+ δ ≤ K (cid:48) n 1+ δ for K (cid:48) > 0 an absolute constant, and so we have ε − 1 ≤ K
[Equation]
n 3 /
+ δ for K (cid:48) > 0 another absolute constant.
This gives
[Equation]
where K (cid:48) > 0 is an absolute constant whose value changes from line to line; and where the ﬁrst inequality assumes that n ≥ (2 /c (cid:48)(cid:48)(cid:48) 1 ) log n , the second inequality assumes that n ≥ (2 c (cid:48)(cid:48)
1 /c (cid:48)(cid:48)(cid:48) 1 ) d log n , and the third assumes that δ ≤ c (cid:48)(cid:48) 1 d/ 2 − 3 / 2 .
Choosing d so that the value c (cid:48)(cid:48) 1 d/ 2 − 3 / 2 is larger than the minimum value for δ (i.e., larger than an absolute constant), then choosing δ
= c (cid:48)(cid:48) 1 d/ 2 − 3 / 2 , and ﬁnally choosing d ≥ 6 /c (cid:48)(cid:48) 1 , we obtain
P (cid:34) sup ν ∈ [
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where K > 0 is an absolute constant, which is an acceptable level of uniformization.
Completing the proof.
To obtain the desired control, we apply the uniform framework for the terms Ξ i , i = 2 , 3 , 4 , 6 ; and the pointwise with Lipschitz control framework for the terms Ξ i , i = 1 , 5 .
We also establish high probability control of the zero-order term in Lemma E.38.
The events we need for the pointwise framework terms are constructed in Lemmas E.39, E.40, E.44 and E.45.
The events we need for the uniform framework are constructed in Lemmas E.41 to E.43 and E.46.
Because n and d are chosen appropriately by our hypotheses here, we can invoke each of these lemmas to construct the necessary sub-events and obtain an event E which satisﬁes 1.
One has
[Equation]
if g 1 ∈ E ;
One has
[Equation]
We can adjust d and n slightly to obtain an event with the properties claimed in the statement of the lemma.
Indeed, choosing n to be larger than an absolute constant multiple of log n , we can obtain
C (cid:48) ne − c (cid:48) n ≤ C (cid:48)
e − c
[Equation]
n/ 2 and C (cid:48)(cid:48)(cid:48) ne −
[Equation]
n ≤ C
[Equation]
n/ 2 ; choosing n to be larger than an absolute constant multiple of d log n , we can obtain C (cid:48)(cid:48) n
[Equation]
[Table]
[Equation]
d + C
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
≤ 2 C
[Table]
n −
[Table]
d ; and choosing d to be larger than an absolute constant, we can assert (cid:112) d log n/n + n − cd ≤ 2 (cid:112) d log n/n .
This turns the guarantees of E into the guarantees claimed in the statement of the lemma, and completes the proof.
[Equation]
Lemma E.14.
One has bounds
[Equation]
Proof.
Write f ( ν ) = cos ϕ ( ν ) =
cos ν + π − 1 (sin ν − ν cos ν ) , where the last equality follows from
Lemma E.2.
We start by obtaining quadratic bounds on f ( ν ) for ν ∈ [0 , 0 . 1] .
In particular, we will show
[Equation]
We readily calculate
[Equation]
Taylor expanding at ν = 0 gives
[Equation]
We have f (cid:48)(cid:48) (0) = − 1 , and sin ν ≤ sin 0 .
1 on our interval of interest by monotonicity.
The derivative of ν cos ν is cos ν − ν sin ν ; ν sin ν is increasing as the product of two increasing functions (given ν ≤ 0 . 1 ), and one checks that cos(0 .
1) − 0 .
1 sin(0 . 1) > 0 ; therefore ν cos ν ≤ 0 .
1 cos(0 .
1) on our domain of interest.
One checks numerically
[Equation]
and this establishes f ( ν ) ≤ 1 − 14 ν 2 on [0 , 0 . 1] .
If ν ≤ π/ 2 , we have cos ≥ 0 and sin ≥ 0 , so that ν cos ν + sin ν ≥ 0 on this domain.
This implies f (cid:48)(cid:48) ( ν ) ≥ − cos ν ≥ − 1 for 0 ≤ ν ≤ π/ 2 , which proves inf t ∈ [0 ,π/ 2] f (cid:48)(cid:48) ( t ) = − 1 , and establishes the lower bound on [0 , π/ 2] .
To obtain (possibly) looser bounds on [0 , π ] , we use a bootstrapping approach.
The lower bound is more straightforward; to assert the lower bound on [0 , π ] , we evaluate constants numerically to ﬁnd that the lower bound’s value at π/ 2 is 1 − π 2 / 8 < 0 , and given that f ≥ 0 by Lemma E.5 and the concave quadratic bound is maximized at ν = 0 , it follows that the bound holds on the entire interval.
For bootstrapping the upper bound, we note that the equation
[Equation]
shows immediately that f is a strictly decreasing function of ν on (0 , π ) .
Therefore f ( ν ) ≤ f (0 . 1) on [0 . 1 , π ] , and so the quadratic function ν (cid:55)→ 1 − π − 2 (1 − f (0 . 1)) ν 2 , which is lower bounded by 1 − ν 2 / 4 on [0 , π ] by the fact that both concave quadratic functions are maximized at 0 and the veriﬁcation 1 − π 2 / 4 < 0 ≤ f (0 . 1) , is an upper bound for f on all of [0 , π ] ; so the claim holds with c = π − 2 (1 − f (0 . 1)) .
Lemma E.15.
There exist absolute constants c, C, C (cid:48) , C (cid:48)(cid:48) > 0 such that if n ≥ C log n , then one has
[Equation]
Proof.
Write h ( ν ) = cos ϕ ( ν ) − E
[ X ν ] .
By Lemmas E.24 and E.25, we have a second-order Taylor formula
[Equation]
We calculate h (cid:48) (0) = 0 , since E [ (cid:104) v 0 , ˙ v 0 (cid:105) ]
=
E [ (cid:104) σ ( g 1 ) , g 2 (cid:105) ] = 0 , and P ⊥ v 0 v 0 = 0 .
We also have h (0) =
E [ (cid:107) v 0 (cid:107) 22 ] − E
1 E m ]
= µ ( E c m ) (writing m = 1 ), so this formula yields
[Equation]
and we see that it sufﬁces to bound h (cid:48)(cid:48) .
We will use the (Lebesgue-a.e.) expression
[Equation]
Distributing over the inner product and applying rotational invariance to combine the two cross terms, then using the triangle inequality
[Equation]
we obtain the bound
[Equation]
[Table]
We proceed by giving magnitude bounds for Ξ i ( ν ) , i = 1 , 2 , 3 .
Because we are working with expectations, it sufﬁces to ﬁx one value ν ∈ [0 , π ] and prove pointwise ν -independent bounds; we will exploit this in the sequel to easily deﬁne extra good events without having to uniformize, and we will generally suppress the notational dependence of Ξ i on ν as a result.
We will also repeatedly use the fact that we have µ ( E c 1 ) ≤
Cne − cn for some absolute constants c, C > 0 by Lemma E.16.
We will accrue a large number of additive C/n and C (cid:48)
n pm e − cn errors as we bound the Ξ i terms; at the end of the proof we will worst-case the constants in each additive error and assert a bound of the form claimed.
Ξ 1 control.
Let E = {(cid:107) ˙ v ν (cid:107) 2 ≤ 2 } ∩ {(cid:107) ˙
v 0 (cid:107) 2 ≤ 2 } .
By Lemma E.17 and a union bound, we have µ ( E c ) ≤ Ce − cn .
Deﬁne an event E 1 = E m ∩ E .
The ﬁrst step is to pass to the control of
[Equation]
The triangle inequality gives
[Equation]
The ﬁrst term is readily controlled from two applications of the Schwarz inequality, a union bound, and rotational invariance together with Lemma E.29:
[Equation]
˙ v ν (cid:107) 42 (cid:3) 1 / 4 E (cid:2) (cid:107) ˙ v 0 (cid:107) 42 (cid:3) 1 /≤ (cid:0) µ ( E c m )
+ Ce − cn (cid:1) 1 / 2 E (cid:2) (cid:107) ˙
v 0 (cid:107) 42 (cid:3) 1 /≤ (cid:16)
Cne − cn + C (cid:48)
e − c (cid:48) n (cid:17) 1 / 2 (cid:18) 1 + C (cid:48)(cid:48) n (cid:19) 1 /≤
Cn 1 / 2 e − cn , where in the last line we require n to be at least the value of a large absolute constant.
The calculation is similar for the normalized term, except we also apply the deﬁnition of E m to get some extra cancellation:
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where in the last line we apply our bounds from the ﬁrst term and use n ≥ 1 to obtain the ﬁnal inequality.
Next, Taylor expansion of the smooth convex function x (cid:55)→ x − 1 / 2 on the domain x >about the point x = 1 gives
[Equation]
Given that E m guarantees (cid:107) v ν (cid:107) 2 ≥ 12 , we can apply this to get a bound
[Equation]
On E (cid:48)
[Equation]
we also have (cid:107) v 0 (cid:107) − 2 2 (cid:107) v ν (cid:107) − 2 2 ≤ 2 4 , so we can control the integral residual as
[Equation]
where we replace the tighter bound that we get in the case (cid:107) v
[Equation] [Equation]
v ν
[Equation]
22 ≥ 1 with the worst-case bound from the other case.
This gives bounds
[Equation]
0 (cid:107) 22 (cid:107) v ν (cid:107) 22 − 1 (cid:1) 2 (cid:19) ≤ 1 E 1 (cid:18) 1 − (cid:107) v 0 (cid:107) 2 (cid:107) v ν (cid:107)(cid:19) ≤ 1 E 1 1 2 (cid:0) (cid:107) v 0 (cid:107) 22 (cid:107) v ν (cid:107) 22 − 1 (cid:1) .
Given that (cid:107) ˙ v ν (cid:107) 2 ≤ 2 on E (cid:48)(cid:48) , it follows |(cid:104) ˙ v 0 , ˙ v ν (cid:105)| ≤ 4 on E 1 , so that (cid:104) ˙ v 0 , ˙ v ν (cid:105) + 4 ≥ 0 here.
Writing 1 E 1 (cid:104) ˙ v 0 , ˙ v ν (cid:105) (cid:18) 1 − (cid:107) v 0 (cid:107) 2 (cid:107) v ν (cid:107)(cid:19) = 1 E(cid:20)
( (cid:104) ˙ v 0 , ˙ v ν (cid:105)
+ 4) (cid:18) 1 − (cid:107) v 0 (cid:107) 2 (cid:107) v ν (cid:107)(cid:19) −(cid:18)
1 − (cid:107) v 0 (cid:107) 2 (cid:107) v ν (cid:107)(cid:19)(cid:21) , we can apply nonnegativity to obtain upper and lower bounds (cid:101) Ξ 1 ≤ E (cid:20) 1 E 1 (cid:104) ˙ v 0 , ˙ v ν (cid:105) (cid:18) 1 2 (cid:0) (cid:107) v 0 (cid:107) 22 (cid:107) v ν (cid:107) 22 − 1 (cid:1) + 4 C (cid:0) (cid:107)
[Table]
where =
[Table]
We continue with bounding the quadratic term arising in the previous equation.
We have
[Equation]
The ﬁrst inequality applies the triangle inequality for the integral, the deﬁnition of E 1 and CauchySchwarz, then drops the indicator for E 1 because the remaining terms are nonnegative; the second line is just distributing; the third line rearranges and applies the Schwarz inequality; and the fourth inequality applies Jensen’s inequality and Lemma E.18 to control the second term (to apply this lemma, we need to choose n larger than an absolute constant; we assume this is done), and
Lemma E.29 to control the third term.
Since n ≥ 1 , this gives a C/n + C (cid:48)
e − cn bound on the quadratic term.
Next is the linear term; our ﬁrst step will be to get rid of the indicator.
By the triangle inequality, it sufﬁces to get control of the corresponding term with the indicator for E c 1 instead; we control it as follows:
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
The ﬁrst line is the Schwarz inequality; the second line is the good event measure bound and CauchySchwarz; the third line distributes and drops the cross term, given that all factors are nonnegative; the fourth line applies subadditivity of the square root function, then the Schwarz inequality to the resulting separate terms; the ﬁfth line applies Lemma E.29; and the last line again uses square root subadditivity and treats the remaining terms as multiplicative constants, since n ≥ 1 .
Therefore passing to the linear term without the indicator incurs only an additional exponential factor.
Proceeding, we drop the indicator and distribute to get for the linear term
[Equation]
[Table]
[Equation]
it is of interest to apply Lemma E.30 to these two terms to get the proper cancellation, and for this we just need to check that the coordinates of each factor in the product have subexponential moment growth with the proper rate.
For even powers of (cid:96) 2 norms of v ν , this follows immediately from Lemma G.11 after scaling by (cid:112) 2 /n ; for the inner product term, the coordinate functions are ˙ σ
[Equation]
g 1 i ) g 2 i ˙ σ
[Equation]
g 1 i cos ν
+ g 2 i sin ν )
[Equation]
g 2 i cos ν − g 1 i sin ν ) , and we have from the Schwarz inequality and rotational invariance
[Equation]
which has subexponential moment growth with rate Cn − 1 by Lemma E.17 and Lemma G.11 after rescaling by (cid:112) 2 /n .
These formulas also show that when k = 1 , we have a bound of precisely n − 1 .
This makes Lemma E.30 applicable, so we can assert bounds (cid:12)(cid:12)(cid:12)
[Equation]
˙
[Equation]
[Table]
Because E (cid:2) σ ( g 11 ) 2 (cid:3) 2 = n − 2 , this is enough to conclude a C/n bound on the magnitude of the linear term.
Thus, in total, we have shown
[Equation]
where we combine the different constant that appear in the various exponential additive errors throughout our work by choosing the largest magnitude scaling factor and the smallest magnitude constant in the exponent to assert the previous expression.
Ξ 2 control.
The approach is similar to what we have used to control Ξ 1 .
We start with exactly the same E 1 event deﬁnition
[Equation]
and as previously deﬁne
[Equation]
and then calculating
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
using the same ideas as in the previous section
[Table]
plus several applications of the Schwarz inequality and a ﬁnal application of Lemma E.29.
We can therefore pass to (cid:101)
Ξ 2 with a small additive error.
Next
[Table]
we Taylor expand in the same way as previously
[Table]
except that larger powers in the denominator force the constant in our residual bound to be 3 ·
[Table]
and the event E 1 now gives us a bound |(cid:104) ˙ v 0 , v ν (cid:105)(cid:104) v ν , ˙ v ν (cid:105)(cid:107) v 0 (cid:107) 22 | ≤ 2 6 on the numerator, which we add and subtract as before to exploit nonnegativity.
We get
[Table]
with C = 3 · 2 27 .
Proceeding to control the quadratic term
[Table]
we have
[Table]
[Equation]
The justiﬁcations for the ﬁrst four lines are identical to those of the previous section.
In the last three lines, we use the binomial theorem twice to expand the sixth power term, and we assert the ﬁnal line by the fact that k > 0 , so that each term in the sum corresponding to a j = 0 has a positive inverse power of n attached, and when j = 2 k − 1 we pick up an exponential factor.
Moving on to the linear term, as in the previous section we start by dropping the indicator.
We control the residual as follows:
[Table]
[Equation]
The justiﬁcations are almost the same as the previous section, although we have compressed some steps into fewer lines here and we have omitted the ﬁnal simpliﬁcations which follow from applying the Schwarz inequality to each of the two expectations in the second-to-last line 3 times and then applying Lemma E.29.
Dropping the indicator and distributing now gives:
[Equation]
to apply Lemma E.
[Equation]
we check the two new coordinate functions that appear in this linear term: for (cid:104) ˙ v 0 , v ν (cid:105) , we have
E
[Equation]
34) and for (cid:104) ˙
v ν , v ν (cid:105) , we have likewise
E
[Equation]
both by the Schwarz inequality and rotational invariance.
As before, an appeal to Lemmas G. and E.17 implies that these two coordinate functions satisfy the hypotheses of Lemma E.30, so we have a bound
[Equation]
Noticing that
[Equation]
by rotational invariance and independence
[Equation]
we conclude by identically-distributedness of the coordinates of v ν and ˙ v ν
[Equation]
which establishes the desired control on Ξ 2 .
Thus
[Equation]
in total
[Equation]
we have shown
[Equation]
where we combine the different constant that appear in the various exponential additive errors throughout our work by choosing the largest magnitude scaling factor and the smallest magnitude constant in the exponent to assert the previous expression.
Ξ 3 control.
The argument for control of this term is very similar to the previous section, since the degrees of the denominators now match.
We start by deﬁning
[Equation]
[Table] [Table]
[Equation]
using the same ideas as in the previous section.
We can therefore pass to (cid:101)
Ξ 3 with an exponentially small error.
Next
[Equation]
we Taylor expand in the same way as previously, obtaining
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
with C = 3 · 2 27 .
Proceeding to control the quadratic term, we notice
[Equation]
[Table]
[Equation]
since the ﬁnal term was controlled in the previous section.
Moving on to the linear term, as in the previous section we start by dropping the indicator.
We control the residual as follows:
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
by the same argument as in the previous section.
Dropping the indicator and distributing now gives:
[Equation]
[Table]
[Equation]
; to apply Lemma E.30, we check the one new coordinate function that appears in this linear term: for (cid:104) v 0 , v ν (cid:105) , we have
[Equation]
36) by the Schwarz inequality and rotational invariance.
As before, an appeal to Lemmas G.11 and E. implies that this coordinate function satisﬁes the hypotheses of Lemma E.30, so we have a bound
[Equation]
As in the previous section, using that E [ (cid:104) v ν , ˙ v ν (cid:105) ] = 0 then allows us to conclude the desired control on Ξ 3 .
Thus, in total, we have shown
[Equation]
where we combine the different constant that appear in the various exponential additive errors throughout our work by choosing the largest magnitude scaling factor and the smallest magnitude constant in the exponent to assert the previous expression.
To wrap up, we take the largest of the scaling constants in the estimates we have derived, and the smallest of the constants-in-the-exponent that we have derived, in order to assert
[Equation]
Matching constants in the exponent and choosing n larger than an absolute constant multiple of log n , it follows
[Equation]
which was to be proved.
E.3. G ENERAL P ROPERTIES
Lemma E.16.
Consider the event
[Equation]
Suppose n ≥ max { 2 m, m + 20 } .
Then we have the following properties:
µ ( E c c,m ) ≤
Cn m e − c (cid:48) n ; 2.
We have E c,m = E c,m Q for every Q ∈ O(2) , so that in particular 1 E c,m ( GQ ) =
E c,m ( G ) .
Above, O( n ) denotes the set of n × n orthogonal matrices.
Proof.
We will show the second property ﬁrst.
For each c > 0 , if Q ∈ O(2) , notice that
[Equation]
since the vector [cos ν, sin ν ] ∗ ∈ S 1 , and O(2) acts transitively on S 1 .
This proves the second property when c > 0 ; the result for c = 0 is obtained by applying the preceding argument to each set in the inﬁnite union deﬁning the c = 0 , m event.
For the measure bound, we observe that E c,m ⊂ E c (cid:48) ,m if c ≥ c (cid:48) , so it sufﬁces to bound the measure of the complement for the particular choice c = 1 2 .
We start by controlling pointwise the measure of the complement of the event
[Equation]
for each u ∈ S 1 , then uniformize over the one-dimensional manifold S 1 ; we need to begin with c = 0 .
6 instead of c = 12 to survive some loosening of the bounds when we uniformize.
We have
[Equation]
so that a union bound implies
[Equation]
where the ﬁnal inequality follows from right-rotational invariance of µ and identicallydistributedness of the coordinates of g 1 .
Let ˜ g ∈ R n − m be distributed as N ( 0 , (2 /n ) I ) , so that σ (˜ g ) has the same distribution as I [ m ] c σ ( g 1 )
By Gauss-Lipschitz concentration (Boucheron et al., 2013, Theorem 5.6), we have
[Equation]
since σ is 1 -Lipschitz and nonnegative homogeneous.
After rescaling, we apply Lemma E.19 to get
[Equation]
Plugging these estimates into the Gauss-Lipschitz bounds gives
[Equation]
Putting t = 2 / 3 in the upper tail bound gives the control we need for one half of (E.37).
For the lower tail, we note that the assumption n ≥ max { 2 m, m + 20 } yields the estimates
[Equation]
so that
[Equation]
and one checks numerically that 2 − 1 / 2 − (1 / 10) > 0 .
Putting therefore t = 2 − 1 / 2 − (1 / 10) − 0 .in
the lower tail bound yields
[Equation]
Plugging these results into (E.37) gives the pointwise measure bound
[Equation]
for some constant c > 0 .
For uniformization, ﬁx S ⊂ [ n ] with | S | = m and consider the function f S : R 2 → R deﬁned by
[Equation]
By Gauss-Lipschitz concentration, we have
[Equation]
and by (Rudelson & Vershynin, 2011, Theorem 2.6), we have
[Equation]
Let E = {(cid:107) G (cid:107) ≤ 5 } ; then it follows that µ ( E ) ≥ 1 − e − cn .
On E , for every S , we have that f S is a 5 -Lipschitz function of u .
Let T ε ⊂ S 1 be a family of sets with the property that u ∈ S 1 implies that there is u (cid:48) ∈ T ε such that (cid:107) u (cid:48)
− u (cid:107) 2 ≤ ε for each ε > 0 ; by standard results (Vershynin, 2018,
Corollary 4.2.13), T ε exists and we have | T ε | ≤ (1 + 2 ε − 1 ) 2 .
Deﬁne
[Equation]
6 ,m, u .
Then a union bound together with our pointwise concentration result gives
[Equation]
On E ∩ E 0 .
6 ,m,ε , for any u ∈ S 1 and any S , there is u (cid:48) ∈ T ε such that | f S ( u ) − f S ( u (cid:48) ) | ≤ 5 ε .
But since on this event 0 .
6 ≤ f S ( u (cid:48) ) ≤ 5 / 3 , we conclude 0 .
6 − 5 ε ≤ f S ( u ) ≤ 5 / 3 + 5 ε , and therefore the choice ε = 1 / 50 gives 0 .
5 ≤ f S ( u ) ≤ 2 .
This implies
[Equation]
Thus, by a union bound and our previous results, we have
[Equation]
which is the desired measure bound.
Lemma E.17.
We have for each ﬁxed ν ∈ [0 , π ] that:
The coordinates of ˙ v ν have subgaussian moment growth
[Equation]
The event {(cid:107) ˙ v ν (cid:107) 2 ≤ 2 } has probability at least 1 − e − cn ; 3.
The event {∀ ν ∈ [0 , π ] (cid:107) ˙ v ν (cid:107) 2 ≤ 4 } has probability at least 1 − e − c (cid:48) n .
Proof.
We have that the coordinates of ˙ v
ν are i.i.d., and ( ˙ v ν ) i
d = ˙ σ ( g 1 i ) g 2 i , by rotational invariance.
By independence of g 1 and g 2 , we compute
[Equation]
for each p ≥ 1 ; the last inequality follows from Lemma G.11.
This shows that the coordinates of ˙ v ν are independent subgaussian random variables with scale parameters at most C (cid:112) 2 /n , so we have a tail bound (Vershynin, 2018, Theorem 3.1.1)
[Equation]
also taking into account that E (cid:2) ( ˙ v ν ) 2 i (cid:3) = 1 /n .
This shows that the event E (cid:48)(cid:48)
= {(cid:107) ˙
v ν (cid:107) 2 ≤ 2 } has probability at least 1 − e − cn .
For the third assertion, we use the triangle inequality to get (cid:107) ˙ v ν (cid:107) 2 ≤ (cid:107) g 2 (cid:107)
+ (cid:107) g 2 (cid:107) 2 , which has
RHS independent of ν ; then applying Gauss-Lipschitz concentration gives for t ≥
[Equation]
using that E [ (cid:107) g
i (cid:107) 2 ] ≤ (cid:112) E [ (cid:107) g i (cid:107) 22 ] .
Putting t = 0 .
5 in this bound and applying a union bound, we conclude that there is an event of probability at least 1 − e − cn on which (cid:107) ˙
v ν (cid:107) 2 ≤ 4 uniformly in ν .
Lemma E.18.
There exists an absolute constant C > 0 such that if n ≥ C , one has
[Equation]
where c, C (cid:48) , C (cid:48)(cid:48) > 0 are absolute constants.
Proof.
For the upper bound, we apply the Schwarz inequality to get
[Equation]
by rotational invariance and Lemma G.11.
For the lower bound, we will truncate and linearize the product using logarithms.
Let E =
E 0 .
5 , 0 ; by Lemma E.16, as long as n ≥ 20 we have µ ( E c ) ≤
Ce − cn .
Deﬁne X = (cid:107) v 0 (cid:107) 2 (cid:107) v ν (cid:107) 2 1 E + 1 E c , so that
X ( G ) =
(cid:26) (cid:107) v 0 ( G ) (cid:107) 2 (cid:107) v ν ( G ) (cid:107)
G ∈ E , otherwise .
We calculate
[Equation]
using the triangle inequality
[Equation]
the Schwarz inequality
[Equation]
rotational invariance
[Equation]
and Lemmas E. and E.29.
It follows
[Equation]
so it sufﬁces to prove the lower bound for X instead.
Factoring as X = (
[Equation] [Equation]
+
E c ) , we apply concavity of x (cid:55)→ log x , Jensen’s inequality
[Equation]
and convexity of x (cid:55)→ e x to get
[Equation]
where the last equality is due to rotational invariance.
Now write Y =
(cid:107) v 0 (cid:107) 2 1 E + 1 E c , so that by the deﬁnition of E we have Y ≥ 12 .
Taylor expansion with Lagrange remainder of the logarithm about E [ Y ] ≥ 1 2 gives
[Equation]
for some ξ ( Y ) between E [ Y ] and Y .
Using Y ≥ 12 and taking expectations on both sides, we get
[Equation]
[ Y ] .
Moreover, we have
[Equation]
by the Schwarz inequality, and this extra exponential error can be rolled into the exponential error accrued via our use of X .
In particular, we have
[Equation]
by Lemma E.19.
Since n ≥ 20 , if we also enforce n ≥ C 1 := c − 1 log(5 C/ 2) we have 2 /n +
Ce − cn ≤ 12 ; it follows by concavity of x (cid:55)→ log(1 − x ) that we have a bound log
[Equation]
which has the form claimed.
It remains to upper bound Var[ Y ] ; using that Y
= (cid:107) v 0 (cid:107) 22 1 E + 1 E c
, we have
[Equation]
which is sufﬁcient to conclude.
Lemma E.19.
One has
[Equation]
Proof.
By rotational invariance, it is equivalent to characterize the expectation of (cid:107) σ ( g 1 ) (cid:107) 2 .
By the
Schwarz inequality, we have
[Equation]
by Lemma G.11.
For the lower bound, we apply the Gaussian Poincaré inequality (Boucheron et al., 2013, Theorem 3.20) and the 1 -Lipschitz property of g (cid:55)→ (cid:107) σ ( g ) (cid:107) 2 to get
[Equation]
so that after distributing and applying E [ (cid:107) v 0 (cid:107) 22 ] = 1 , we see that
[Equation]
Because n ≥ 2 , it follows
[Equation]
where the last bound holds because 1 − 2 n − 1 ≤ 1 .
Lemma E.20.
If 0 ≤ x, y ≤ 1 , we have
[Equation]
Proof.
Let 0 ≤ x, y ≤ 1 , and assume to begin that x ≤ y .
We apply the fundamental theorem of calculus and knowledge of the derivative of cos -1 to get
[Equation]
The integrand is nonnegative, so cos -1 x − cos -1 y ≥ 0 .
Writing √ 1 − t 2 = √ 1 − t √ 1 + t and using x ≥ 0 , we get
[Equation]
This shows that | cos -1 x − cos -1 y | ≤ |√ 1 − x −√ 1 − y | when x ≤ y .
An almost-identical argument establishes the same when y ≤ x , via the inequalities 0 ≥ cos -1 x − cos -1 y ≥ − ( √
1 − x −√ 1 − y ) .
So we have shown | cos -1 x − cos -1 y
| ≤ |√ 1 − x − (cid:112) 1 − y | for arbitrary 0 ≤ x ≤ 1 and 0 ≤ y ≤ 1 .
Now notice
[Equation]
which establishes | cos -1 x − cos -1 y | ≤ (cid:112) | x − y | .
E.3. D IFFERENTIATION R ESULTS
Lemma E.21.
For a < b , let f : [ a, b ] → R be a continuous function that is differentiable on ( a, b ) except at a set of isolated points in ( a, b ) , and let c ∈ R .
Then max { f, c } is differentiable except at a set of isolated points in ( a, b ) .
Proof.
Let A ⊂ ( a, b ) denote the set of points of differentiability of f , and let B ⊂ ( a, b ) denote the set of points of nondifferentiability of max { f, c } .
Because ﬁnite unions of isolated sets of points in ( a, b ) are isolated in ( a, b ) , it sufﬁces to consider only points x ∈ A .
Fix x ∈ A , and consider the case f ( x ) (cid:54) =
c .
Then because f is continuous, there is a neighborhood of x on which f (cid:54)
=
c .
If f > c on this neighborhood, then we have max { f, c } = f on this neighborhood; if f < c , then we have max { f, c } = c .
In either case, this implies that max { f, c } is differentiable at x , and thus x is not in B .
Next, consider the case where f ( x ) =
c .
First, suppose f (cid:48)
( x ) > 0 ; then by Rolle’s theorem, we can ﬁnd a neighborhood of x on which f
( x (cid:48) )
> c if x (cid:48) >
x and f ( x (cid:48) ) < c if x (cid:48) <
x .
Possibly shrinking this neighborhood, we can assume every point of the neighborhood is a point of differentiability of f .
Thus, for x (cid:48) < x in this neighborhood, we have max { f ( x (cid:48) ) , c } = f
( x (cid:48) ) , and for x (cid:48) > x , we have max { f ( x (cid:48) ) , c } = c .
We conclude that max { f, c } is differentiable at all points of this neighborhood except x , and in particular x is an isolated point in B .
A symmetric argument treats the case where f (cid:48) ( x ) < 0 , with the same conclusion.
On the other hand, if f (cid:48) ( x ) = 0 , we can write f ( x (cid:48) )
=
c
+ o ( | x (cid:48) − x | ) for x (cid:48) in a neighborhood of x , which implies max { f ( x (cid:48) ) , c } = max { c,
c + o ( | x (cid:48) − x | ) }
=
c ± o ( | x (cid:48) − x | ) .
In particular, | max { f ( x (cid:48) ) , c } − max { f ( x ) , c }|
=
o ( | x (cid:48) − x | ) , which shows that max { f, c } is differentiable at x , and thus x is not in B .
This shows that every point of A ∩ B is isolated in A ∩ B , and we can therefore conclude that max { f, c } is differentiable except at isolated points of ( a, b ) .
Lemma E.22.
For 0 ≤ ν ≤ π , consider the function
[Equation]
where
[Equation]
Then ˜ ϕ is absolutely continuous on [0 , π ] , and satisﬁes the ﬁrst-order Taylor expansion
[Equation]
and moreover ˜ ϕ is 1 -Lipschitz.
Proof.
At points of (0 , π ) where each of the functions composed in φ is differentiable, the chain rule gives for the derivative of the integrand as a function of ν
[Equation]
where we have used the result
[Equation]
valid for any x (cid:54) = 0 .
Because E 1 guarantees that v ν (cid:54) = 0 for all ν ∈ [0 , π ] , we see that the integrand φ is continuous.
Similarly, given that (cid:107) v ν (cid:107) 2 ≥ 1 2 on E 1 , we note that there are just two obstructions to differentiability:
The inverse cosine is not differentiable at {± 1 } ; 2.
The activation σ is not differentiable at 0 .
First we characterize the issue of nondifferentiability with regards to the inverse cosine.
We note that cos φ ( ν, g 1 , g 2 ) =
if and only if the Cauchy-Schwarz inequality is tight, which is equivalent to v 0 and v ν being linearly dependent.
Suppose we have ( g 1 , g 2 )
∈
E 1 and ν 0 ∈ (0 , π ) such that v 0 ( g 1 , g 2 ) and v ν 0 ( g 1 , g 2 ) are linearly dependent.
Because two vectors u 1 , u 2 ∈ R n have σ ( u 1 ) and σ ( u 2 ) linearly dependent if and only if σ ( u 1 ) and σ ( u 2 ) have the same support and are linearly dependent on the support, and given that (cid:107) v ν (cid:107) 0 > 1 for each ν , we have that there is a 2 ×submatrix of GM ν 0 having positive entries and rank 1 (since the rank is zero
if and only if the submatrix is zero), where
[Equation]
Write the corresponding 2 × 2 submatrix of G as X .
Because rank M ν 0 = 2 by ν 0 ∈ (0 , π ) , we have rank X = 1 .
On the other hand, if G ∼ i .
i .
d . N (0 , 2 /n ) , we have
P [ G has a singular 2 × 2 minor ] ≤
[Equation]
where the ﬁrst line is a union bound, and the second line uses the fact that 2 × 2 submatrices of G are i.i.d.
N (0 , 2 /n ) , and that the complement of the set of full-rank 2 × 2 matrices is a positivecodimensional closed embedded submanifold of R 2 × 2 .
It follows that the subset of E 1 of matrices having no singular 2 × 2 minor has full measure in E 1 , and we conclude that for almost all ( g 1 , g 2 ) , we have cos φ ( ν, g 1 , g 2 ) <
1 for every ν ∈ (0 , π ) .
Next, we characterize nondifferentiability due to the activation σ ; by the chain rule, it sufﬁces to consider nondifferentiability of v ν as a function of ν , and then Lemma E.21 implies that for every ( g 1 , g 2 ) , v ν is differentiable at all but at most countably many points of [0 , π ] .
Next, we observe that whenever v ν is nonvanishing, one has
[Equation]
where the ﬁrst inequality is due squaring the orthogonal projection and Cauchy-Schwarz
[Equation]
and the second equality follows from distributing to evaluate the squared norm, cancelling
[Equation]
and taking square roots.
Using the fact that orthogonal projections have operator norm
[Equation]
we thus conclude
[Equation]
where the last inequality is valid whenever ( g 1 , g 2 )
∈
E 1 .
Since
[Equation]
and this upper bound is jointly integrable in ν
and ( g 1 , g 2 ) over [0 , π ] × R n × R n , we can apply (Cohn, 2013, Theorem 6.3.11) to obtain that
whenever ( g 1 , g 2 ) ∈ E 1 minus a negligible set, we have for every ν ∈ [0 , π ]
[Equation]
In particular, multiplying by the indicator for E 1 , taking expectations over ( g 1 , g 2 ) , and applying the previous joint integrability assertion for φ (cid:48) together with Fubini’s theorem yields
[Equation]
so to conclude the Lipschitz estimate, it sufﬁces to obtain a suitable estimate on E g 1 , g 2 [ φ (cid:48) ( ν, g 1 , g 2 )] .
In light of (E.39) we calculate more precisely
E
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
In the ﬁrst line, we apply rotational invariance and unpack notation; in the second line, we use nonnegativity of the integrand to pass to the containing event where v 0 is at least 2 -sparse; and in the third line, we condition on the size of the support of g 1 .
In the fourth line, we use several facts; ﬁrst, we note that P ⊥ v 0 ( ˙ σ ( g 1 ) (cid:12) g 2 ) =
P ⊥ v 0 P { σ ( g 1 ) > 0 } g 2 for any g 2 ∈ R n , and that the commutation relation P ⊥ v 0 P { σ ( g 1 ) > 0 }
= P { σ ( g 1 ) > 0 } P ⊥ v 0 implies that the operator P ⊥ v 0 P { σ ( g 1 ) > 0 } is itself an orthogonal projection, with range equal to the ( (cid:107) v 0 (cid:107) 0 − 1) -dimensional subspace consisting of vectors with support supp( v 0 ) orthogonal to v 0 .
In particular, σ ( g 1 ) and P ⊥ v 0 P { v 0 > 0 } g 2 are independent gaussian vectors, and conditioned on the size of the support of σ ( g 1 ) the quantities (cid:107) σ ( g 1 ) (cid:107) 2 and (cid:107) P ⊥ v 0 P { v 0 > 0 } g 2 (cid:107) 2 are distributed as independent chi random variables with (respectively) k and k − 1 degrees of freedom.
An application of Lemma G.9 then gives
[Equation]
which is sufﬁcient to conclude.
Lemma E.23.
The random variable X ν satisﬁes the following regularity properties:
If 0 < ν ≤ π , we have X ν < 1 almost surely.
If ( g 1 , g 2 ) ∈ E 1 , then X ν is absolutely continuous on [0 , π ] , with a.e. derivative
[Equation]
and moreover we have E g
g 2 [ | ˙
X ν |
] ≤
, so the analogous differentiation result applies to E
g 1 , g 2 [ X ν ] .
Proof.
The ﬁrst claim is a corollary of the proof of differentiability of the inverse cosine part of ˜ ϕ in Lemma E.22 and the observation that X π = 0 .
The second claim is also a direct consequence of the proof of Lemma E.22 and Fubini’s theorem.
Lemma E.24.
Consider the function
[Equation]
Then f is continuously differentiable, with derivative
[Equation]
Moreover, f (cid:48) is absolutely continuous, with Lebesgue-a
[Equation]
e. derivative
[Equation]
Proof.
The expression for f (cid:48) is a direct consequence of Lemma E.23.
To see that f (cid:48) is actually continuous, apply rotational invariance of the Gaussian measure and of 1 E 1 by Lemma E.16 to get
[Equation]
then notice that this expression is an integral of a continuous function of ν , which is therefore continuous.
Moreover, the ν dependence in this expression for f (cid:48) mirrors exactly that of f ; in particular, the integrand
[Equation]
is absolutely continuous whenever ( g 1 , g 2 )
∈ E 1 by Lemma E.23, with a.e. derivative
[Equation]
We can therefore conclude the claimed expression for f (cid:48)(cid:48) provided we can show absolute integrability over E 1 of this last expression, using Fubini’s theorem in a way analogous to the argument in
Lemma E.22.
But
[Equation]
using, in sequence, Cauchy-Schwarz and the lower bound in the deﬁnition of E 1 ; the operator norm of orthogonal projections being 1 , the Schwarz inequality, nonnegativity of the integrand, and rotational invariance; and Lemma E.17.
We can therefore conclude the claimed expression for f (cid:48)(cid:48) and complete the proof.
Lemma E.25.
For the heuristic cosine angle evolution function
[Equation]
we have the following integral representations for its continuous derivatives:
[Equation]
Proof.
The proof follows exactly the arguments of Lemma E.24, but with a simpler integrand and different integrability checks; the continuity assertion relies on Lemma E.5.
Indeed, this approach gives that (cid:104) v 0 , v ν (cid:105) is absolutely continuous, with Lebesgue-a.e. derivative (cid:104) v 0 , ˙ v ν (cid:105) ; we check
[Equation]
by Cauchy-Schwarz, the Schwarz inequality, rotational invariance, and Lemma E.17.
This veriﬁes the claimed expression for (cos ◦ ϕ ) (cid:48) .
For the second derivative, we apply rotational invariance to get
[Equation]
which has an absolutely continuous integrand, with Lebesgue-a.e. derivative
[Equation]
Checking absolute integrability, we have as before
[Equation]
by Cauchy-Schwarz, the Schwarz inequality, rotational invariance, and Lemma E.17.
This establishes the claimed expression for (cos ◦ ϕ ) (cid:48)(cid:48) .
Lemma E.26.
Let ψ : R → R be deﬁned by ψ ( x ) = ψ 0 .
25 ( x ) , where ψ 0 .
25 is the function constructed in Lemma E.31.
Then the function
[Equation]
satisﬁes for all ν ∈ [0 , π ] and Lebesgue-a
[Equation]
e. g 1 the second-order Taylor expansion
[Equation]
[Table]
(cid:21) (cid:33) d s. where previously-unspeciﬁed notation in this expression is introduced in (E.44) .
Proof.
Take g 1 ∈ R n such that f ( ν, · ) exists and is g 1 -integrable; by Fubini’s theorem such ghave full measure in R n .
Because ψ > 0 and ψ ( (cid:107) v ν (cid:107) ) is locally (as a function of ν ) constant whenever (cid:107) v ν (cid:107) < 14 , we need only consider nondifferentiability of σ when assessing differentiability of f ( · , g 1 ) .
By Lemma E.21, we conclude that f ( · , g 1 ) is differentiable at all but at most countably many points of (0 , π ) ; since ψ > 0 and ψ is smooth, f is continuous, and we can therefore apply Lebesgue differentiation theorems (Cohn, 2013, Theorem 6.3.11) to f provided we satisfy the standard derivative product integrability checks.
Writing
[Equation]
the chain rule gives (at points of differentiability)
[Equation]
In this expression, we follow the convention 0 /
= 0 to account for the possibility that (cid:107) v ν (cid:107) 2 =(in this case, the ψ (cid:48) term handles the denominator).
For product integrability, we Lemma E.31 to get | ψ (cid:48) | ≤ C for some absolute constant C > 0 together with Cauchy-Schwarz and the triangle inequality to get
[Equation]
and applying the Schwarz inequality, rotational invariance (to eliminate ν dependence in the resulting expectations) and Lemma E.17, we conclude that φ (cid:48) is jointly absolutely integrable over [0 , π ] × ( R n × 2 , µ ⊗ µ ) .
We have therefore a ﬁrst-order Taylor expansion f
[Equation]
, g
[Equation]
Ξ
[Equation]
t.
We have
[Equation]
since v 0 depends only on g 1 .
Next, we show t -differentiability of the inner expectation.
Our aim is to apply Lemma E.27 to differentiate Ξ 1 and Ξ 2 .
We ﬁrst focus on Ξ 1 ; distributing and applying linearity, we have
[Equation]
We have shown absolute integrability of the quantity inside the expectation above; we can therefore apply Fubini’s theorem and the previous deﬁnition to write
[Equation]
For each i ∈ [ n ]
[Equation]
write π i : R n → R n − 1 for the linear map that deletes the i -th coordinate from its input
[Equation]
and let ˆ
π
i : R × R n − 1 → R n be the linear map such that ˆ π i ( g i , π i ( g ))
=
g .
With g 2 ﬁxed (in the context of (E.41)), if we deﬁne
[Equation]
then we can write
[Equation]
Thus, to differentiate Ξ 1 , it sufﬁces to check the regularity of f 1 ( ν, g ) and apply Lemma E.27.
As before, ψ > 0 and ψ smooth implies that f 1 is continuous on [0 , π ] × R .
For integrability of f , we appeal to the Fubini’s theorem justiﬁcation that we applied previously.
For absolute continuity, we apply Lemma E.21 to get that the derivative of f with respect to ν is, by the chain rule,
[Equation]
at all but at most countably many values of ν ; and the triangle inequality, Cauchy-Schwarz
[Equation]
and
Lemma E.31 yield
[Equation]
(we apply square root subadditivity in the last line) which is jointly integrable over [0 , π ] × R , and moreover over [0 , π ] × R n .
We conclude absolute continuity of f 1 ( · , g ) and the integrability property of f (cid:48) 1 .
Finally, for the growth estimate, we obtain an estimate for f 1 similar to the one we just obtained for f (cid:48) 1 as follows:
[Equation]
the RHS of the ﬁnal inequality above is a linear function of | g | , and when | g | ≥ 1 we can therefore obtain | f 1 ( ν, g ) | ≤ 16( | g 1 i | + | g 1 i | 2 ) | g | , which is a suitable growth estimate with p = 1 .
Then as long as g 1 i (cid:54) = 0 for all i (such g 1 form a set of measure zero, which we can neglect), we can apply
Lemma E.27 to get
[Equation]
−
[Equation]
t
[Equation]
The estimates (E.42) and (E.43) show, respectively, that f (cid:48) 1 and f 1 are absolutely integrable functions of ( ν, g 2 ) .
We have
[Equation]
so that Lemma E.31 and nonnegativity give
[Equation]
As in the proof of Lemma E.37, in particular using the estimates (E.52) (E.53) to control the magnitude of the RHS for all values of t
[Equation]
we can conclude that the Dirac term is absolutely integrable over [0 , π ] × R n .
An application of Fubini’s theorem then allows us to re-combine the split integrals in the previous expression:
[Equation]
We notice that
[Equation]
and thus motivated introduce the notation
[Equation]
We can then write
[Equation]
Finally
[Equation]
we apply linearity of the integral to move the summation over i back inside the integrals, obtaining
[Equation]
Noting that, in the zero-order term
[Equation]
the only g 2 dependence is in ˙ v 0 = ˙
σ ( g 1 ) (cid:12) g 2 , we apply independence of g 1 and g 2 to obtain ﬁnally
[Equation]
d t
We run the same type of argument on Ξ 2 next.
Distributing and applying linearity, we have
[Equation]
where in the previous expression
[Equation]
By the preceding (product) absolute integrability check when taking ﬁrst derivatives
[Equation]
we can apply
Fubini’s theorem to split the integral as we did with Ξ 1 .
We deﬁne, with g 2 ﬁxed, the function
[Equation]
[Table]
where in the previous expression
[Equation]
so that
[Equation]
Now we check that the hypotheses of Lemma E.27 are satisﬁed for f 2 .
The continuity argument is identical to that employed for f 1 , as is the joint absolute integrability property of f 2 .
For absolute continuity, we again use ψ > 0 , ψ smooth
[Equation]
and Lemma E.21 to obtain the derivative at all but ﬁnitely many points of [0 , π ] (by the chain rule and the Leibniz rule) as
[Equation]
[Table] [Table]
ψ (cid:48) or ψ
[Table]
and our convention handle cancellation in the case where (cid:107) v ν (cid:107) 2 = 0 , we can proceed when necessary with the convenient estimate
[Equation]
which follows from the fact that (cid:107) u (cid:107) ∞ ≤ (cid:107) u (cid:107) 2 for any u ∈ R n .
As with Ξ
[Equation]
we then estimate the magnitude of f (cid:48)
2 using Lemma E.31, Cauchy-Schwarz, the triangle inequality, and square-root subadditivity (skipping some steps that we wrote out in the Ξ 1 estimate):
[Equation]
which is jointly integrable over [0 , π ] × R , and moreover over [0 , π ] × R n .
We conclude absolute continuity of f 2 ( · , g ) and the integrability property of f (cid:48) 2 .
For the growth estimate, we argue similarly to our bound on f (cid:48) 2 to get
[Equation]
[Equation] the RHS in the ﬁnal inequality is a quadratic function of | g | , and we therefore obtain a suitable growth estimate with p = 2 and C (cid:48)
= 64 C (cid:107) g 1 (cid:107) (1 + 2 | g 1 i | + | g 1 i | 2 ) as soon as | g | ≥ 1 .
We can therefore apply Lemma E.27 to get that for all but a negligible set of g
that Ξ
[Equation]
The estimates (E.45) and (E.46) show, respectively, that f (cid:48) 2 and f 2 are absolutely integrable functions of ( ν, g 2 ) .
Because σ ( g 1 i cos ν − g
i cot ν sin ν )
= 0 , we have (fortuitously)
{+f 2 ( t, − g 
+}
i cot t ) = 0 , so that there is no Dirac term in the derivative expression for Ξ 2 .
An application of Fubini’s theorem then allows us to re-combine the split integrals in the previous expression:
[Equation]
We have by linearity of the integral
[Equation]
where the last equality applied independence of g 1 and g
[Equation]
as in the zero-order term of Ξ 1 .
Finally
[Equation]
we apply linearity of the integral to move the summation over i back inside the remaining integrals, obtaining
[Equation]
Since f ( ν, g 1 ) =
f (0 , g 1 ) + (cid:82) ν
0 E g
[Ξ 1 ( t ) − Ξ 2 ( t )]
d t , the claim follows.
Lemma E.27.
Let µ denote the distribution of a N (0 , (2 /n )) random variable, and let ρ denote its density.
Let u ∈ R and u (cid:54) = 0 , and let f : [0 , π ] × R → R satisfy:
f is continuous in its second argument with its ﬁrst argument ﬁxed; 2.
f is absolutely continuous in its ﬁrst argument with its second argument ﬁxed, with a.e. derivative f (cid:48) ; 3.
f and f (cid:48) are absolutely integrable with respect to the product of Lebesgue measure and µ over [0 , π ] × R ; 4.
There exist p ≥ 1 and C > 0 constants independent of x such that | f
( ν, x ) | ≤ C | x | p whenever | x | ≥ 1 .
Consider the function
[Equation]
Then q is absolutely continuous, and the following ﬁrst-order Taylor expansion holds:
[Equation]
Proof.
For m ∈ N , deﬁne
[Equation]
Then 0 ≤ ˙ σ m ≤ 1 ; ˙ σ m is continuous, hence Borel measurable; ˙ σ m → ˙ σ pointwise as m → ∞ ; and ˙ σ m is differentiable on R except at x ∈ { 0 , m − 1 } , with derivative ¨ σ m = m 1 0 ≤ x ≤ m − 1 .
Moreover, we have (cid:90)
[Equation]
and the ﬁrst-order Taylor expansion
[Equation]
Deﬁne
[Equation]
Then at every ν ∈ [0 , π ] , we have by assumption
[Equation]
so that the dominated convergence theorem implies
[Equation]
By the chain rule, the expression ˙ σ m ( x ) = − max {− m max { x, 0 } , − 1 }
[Equation]
and Lemma E.21, ν (cid:55)→ ˙
σ m ( u cos ν + x sin ν ) is an absolutely continuous function of ν ∈ [0 , π ]
[Equation]
and we therefore have by the product rule for AC functions on an interval (Cohn, 2013, Corollary 6.3.9)
[Equation]
We have
[Equation]
by assumption, and
[Equation]
by the growth assumption on f and the Schwarz inequality.
Applying compactness of [
[Equation]
π ] and the lack of ν dependence in the ﬁnal inequality above
[Equation]
an application of Fubini’s theorem therefore yields
[Equation]
By dominated convergence and the ﬁrst of the preceding two product integrability checks
[Equation]
it is clear lim m →∞
[Equation]
For the second term
[Equation]
we need to proceed more carefully.
For k ∈ N sufﬁciently large for the integral to be over a nonempty interval, we consider
[Equation]
which is a truncated version of the integral constituting the second term in q m , with a change of variables applied to explicitly show the density corresponding to µ , and where we write c 2 = 2 /n .
In particular, by the calculation used to apply Fubini’s theorem in this context previously, we have by dominated convergence
[Equation]
By the product integrability assumption on f and Fubini’s theorem
[Equation]
we can consider the inner Rintegral for ﬁxed t
[Equation]
and due to our truncation we have 0 < t < π ; we therefore change variables x (cid:55)→ x sin − 1 t in the inner integral to get
[Equation]
If 0 < t < π and x ∈ R
[Equation]
deﬁne
[Equation]
so that, after an additional change of variables x (cid:55)→ x − u cos t
[Equation]
we obtain
[Equation]
Using the growth estimate for f , we have
[Equation]
where C > 0 depends only on c .
We are going to bound this quantity under the assumption that | x | ≤ | u | / 2 , where we use the assumption | u | > 0 .
First, note that when π/ 4 ≤ t ≤ 3 π/ 4 , we have sin t ≥ 1 / √ 2 , and we always have sin t ≤ 1 for 0 ≤ t ≤ π ; so in this regime
[Equation]
which is a continuous function of ( t, x ) , and is therefore bounded by a constant depending only on c, f, u over the compact set [ π/ 4 , 3 π/ 4] ×
[ − u/ 2 , u/ 2] .
Next, we consider the case 0 < t ≤ π/ 4 ; by the symmetry sin( π − t )
= sin t , controlling | g ( t, x − u cos t ) | in this regime implies control of it in the regime 3 π/ 4 ≤ t < π .
Here, we note that by our assumption on t and the triangle inequality
[Equation]
where we can take K = 2 − 1 / 2 − 2 − 1 > 0 .
Applying the triangle inequality and the condition on | x | gives
[Equation]
which only depends on t .
For any constants c (cid:48) , C (cid:48) > 0 , the continuous map y (cid:55)→ C | y | p +
e − c (cid:48)
y 2 is a bounded function of y ∈ R by L’Hôpital’s rule applied to determine lim y →±∞ | y | p
e − y
=
0 for any p > 0 .
It follows that there is a constant M ≥ 0 depending only on c, u, p such that | g ( t, x − u cos t ) | ≤ M whenever 0 ≤ tπ/ 4 ; we obtain the result for t = 0 by the previous limit calculation.
Applying symmetry and taking the sum of our two bounds then yields | g ( t, x − u cos t ) | ≤ M (cid:48) for
M (cid:48) ≥ 0 not depending on k, m whenever ( t, x ) ∈ [0 , π ] × [ − u/ 2 , u/ 2] .
Now, we have after one additional change of variables x
(cid:55)→ xm −
[Equation]
We can invoke our M (cid:48) bound when xm − 1 ≤ | u | / 2 , and the indicator enforces | x | ≤ 1 ; thus, taking m ≥ 2 / | u | (here we use | u | > 0 critically) implies
[Equation]
so that by dominated convergence, we have
[Equation]
By the same estimate together with second-argument continuity of f
[Equation]
hence of g
[Equation]
we have by the dominated convergence theorem
[Equation]
Combining with our results on q m and the ﬁrst term
[Equation]
we conclude
[Equation]
as claimed.
E.3.
M ISCELLANEOUS A NALYTICAL R ESULTS
Lemma E.28.
If m > 0 , then ¯ ϕ is 1 -Lipschitz.
Proof.
We recall
[Equation]
Considering instead the related function ˜ ϕ deﬁned by
[Equation]
] , where
[Equation]
we notice
[Equation]
It is therefore equivalent to show that ˜ ϕ is 1 -Lipschitz; but this follows from Lemma E.22.
Lemma E.29 (Even Moments) .
If k ∈ N and k ≤ n , one has
[Equation]
where C k ≤ ( k − 1) 2 4 k − 1 (2 k − 1)!! .
Proof.
First notice that the claim is immediate if k = 1 , since E (cid:2) (cid:107) v ν (cid:107) 22 (cid:3) = 1 .
We therefore proceed assuming k > 1 .
Also notice that Lemmas G.11 and E.17 show that ˙ v ν and v ν have matching even moments, so it sufﬁces to prove the claim for v ν .
By rotational invariance, we can write
[Equation]
where the last sum is taken over all elements of [ n ] k .
We split this sum into a sum over terms whose expectations contain no repeated indices, and a sum over all other terms.
There are exactly k !
(cid:0) nk (cid:1) ways to choose a k -multi-index from an alphabet of size n without repetitions—select the k distinct indices, then arrange them in every possible way—and multi-indices without repetitions correspond to terms in the sum where the expectation factors completely, by independence, so we can write
[Equation]
We will prove the elementary estimate
[Equation]
Assuming it for the time being, we use that E (cid:2) σ ( g
[Equation]
2 (cid:3) k
= 2 − k to conclude
[Equation]
Next we study the expectation-of-products arising in the sum.
The expectation factors over distinct indices; we can classify repeated indices in a multi-index by partitions j 1 + . . .
j m = k , where each j l is a positive integer.
Formally, for each multi-index ( i 1 , . .
, i k ) , there is a partition j 1 + . . .
j m = k such that
[Equation]
where p : [ m ] → [ k ] is injective.
We can evaluate these expectations using the result E (cid:2) σ ( g 1 ) 2 k (cid:3) =
12 (2 k − 1)!!
, because the coordinates of g are i.i.d.:
[Equation]
We claim that
[Equation]
) which is the expectation obtained from a term with all indices equal, whence
[Equation]
by (E.47), which gives a bound on the number of terms in the sum.
Noticing that this constant is larger than ( k − 1) 2 2 k − 2 , we can conclude the claimed estimate on C k provided we can justify (E.48).
For this, it sufﬁces to show
[Equation]
Observe that m ≥ 1 for any partition, so 2 m − 1 ≥ 1 and we need only study the second term on the righthand side.
We write this term as
[Equation]
The fact that j 1 + · · · + j m = k implies that there are k factors in the denominator, so we can put the factors in the numerator and denominator into one-to-one correspondence.
Consider the ordering of the factors in the denominator ( (cid:81) j 1 l =1 (2 l − 1)) . . .
( (cid:81)
j m l =1 (2 l − 1)) .
Then (cid:81) ki
=1 (2 i −
[Equation]
(cid:81)
j 1 i =1 (2 i −
[Equation]
k (cid:89) i =
j 1 +
[Equation]
(2 i − 1)
[Equation]
If j 1 =
k , then this product is empty and m = 1 , so the claim is established.
If not, then we proceed to the next group of factors in the denominator: we get
[Equation]
because j 1 > 0 implies that every term in the numerator (ordered in ascending order) is larger than the corresponding term in the denominator.
This gives the claim in the case m = 2 ; for m > 2 , we conclude the claim by induction.
To close the loop, we prove (E.47).
Using simple algebra, we observe
[Equation]
and we note bounds
[Equation]
Working on the upper bound ﬁrst, we obtain with the help of the binomial theorem
[Equation]
where the last expression removes cancellation by making each term in the sum nonnegative, then applies a change of index.
With the identity (cid:0) k − 1 j +1 (cid:1) =
( k − 1) /
( j + 1) (cid:0) k − 2 j (cid:1) , we proceed as k −n k − 2 (cid:88)
j =
[Equation]
n
[Equation]
n (cid:19) k −
[Equation]
, given that 1 / ( j + 1) ≤ 1 .
Since n ≥ k , this gives
[Equation]
The upper bound on the product gives immediately n k − k !
(cid:0) nk (cid:1) ≥ 0 , which completes the proof.
Lemma E.30 (Mixed Moments) .
Let g 1 , . .
. , g n denote the n (i.i.d.
according to N ( 0 , (2 /n ) I 2 ) ) rows of the matrix G .
Let k ∈ [ n ] , and for each 1 ≤ j ≤ k let f j : R 2 → R be a function such that
E [ | f j ( g 1 ) | p ] 1 /p ≤ Cn − 1 p , with C > 0 an absolute constant and p ≥ 1 ; 2.
E (cid:2) | f j ( g 1 ) | (cid:3) ≤ n
Consider the quantities
[Equation]
Then one has | A − B | ≤ Cn − 1 , with the constant depending only on k .
Proof.
Start by writing
[Equation]
[Table]
[Equation]
[Table]
[Equation]
as in Lemma E.29.
Applying the triangle inequality and the ﬁrst moment assumption on the functions f j , we get
[Equation]
with the last inequality following from the estimate (E.47).
For the remaining term, we have by the triangle inequality
[Equation]
[Table]
[Equation]
[Table]
[Equation]
, using again (E.47) to control the number of terms in the sum.
To control the supremum, we apply the Schwarz inequality k − 1 times to get (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
By the subexponential assumption on the functions f j
[Equation]
we have moment growth control
[Equation]
and we therefore have a bound
[Equation]
and consequently
[Table]
which proves the claim.
Lemma E.31.
For any 0 < c ≤ 12 , there exists a smooth function ψ c : R → R satisfying
ψ c ( x ) = x if x ≥ 2 c and ψ c ( x ) =
c if x ≤ c , and ψ c is between c and 2 c if c ≤ x ≤ 2 c ; 2.
ψ c ( x ) ≥ 12 x ; 3.
There are constants M 1 , M 2 > 0 depending only on c such that | ψ (cid:48) c | ≤ M 1 and | ψ (cid:48)(cid:48) c | ≤ M 2 .
Proof.
The function f ( x ) = 1 x> 0 e − 1 x is smooth on R , and satisﬁes 0 ≤ f ≤ 1 and f =
if x ≤ 0 .
The function
[Equation]
is therefore smooth, satisﬁes 0 ≤ φ c ≤ 1 , and satisﬁes φ c ( x )
=
if x ≤ 0 and φ c ( x ) =
if x ≥ c .
Simplifying using the deﬁnitions, we can write
[Equation]
[Equation]
It follows that x (cid:55)→ xφ c ( x ) is zero when x ≤ 0 , x when x ≥ c , and in between otherwise.
Thus, the function ψ c ( x ) =
c
+ ( x − c ) φ c ( x − c ) satisﬁes property 1.
For property 2, we note that ψ c ( x ) =
c
+ ( x − c ) φ c ( x − c ) implies that ψ c ≥ c , since φ c ( x − c )
=whenever x ≤ c and φ c ≥ 0 .
Since ψ c ( x ) = x when x ≥ 2 c , we can then conclude ψ c ( x ) ≥ 12 x , since 12 x ≤ c when x ≤ 2 c and 12 x ≤ x when x ≥ 2 c .
For property 3, we note that by property 1 , ψ (cid:48) c ( x ) =
if x ≥ 2 c and ψ (cid:48) c
( x ) =
if x ≤ 0 ; consequently ψ (cid:48)(cid:48) c ( x ) =
if x (cid:54)∈ [0 , 2 c ] , and it sufﬁces to control ψ (cid:48) c and ψ (cid:48)(cid:48) c in this region.
By translation equivariance of the derivative, it then sufﬁces to control the derivatives of h ( x ) =
xφ c ( x ) for 0 < x < c .
We calculate
[Equation]
E.49) and
[Equation]
f
[Equation]
−
[Equation]
−
[Equation]
x
[Equation]
f (cid:48)
[Equation]
(E.51)
Completely ignoring possible cancellation, we see that it sufﬁces to get a lower bound on f ( x ) + f ( c − x ) and upper bounds on f (cid:48) and f (cid:48)(cid:48) to bound | h (cid:48) | and | h (cid:48)(cid:48) | .
We calculate
[Equation]
and since f ( x ) > 0 if x > 0 and c > 0 , we see that any solution of f (cid:48) ( x ) − f (cid:48) ( c − x ) = 0 must occur for x ∈ (0 , c ) , which implies as well c − x ∈ (0 , c ) .
Writing g ( x ) = x 2 e − x and using c − 1 < x − 1 < ∞ for x ∈ (0 , c ) , we note from our previous work that f (cid:48) ( x ) − f (cid:48) ( c − x ) =
⇐⇒ g
( x − 1 )
=
g (( c − x ) − 1 ) .
We calculate g (cid:48)
( x ) =
xe − x (2 − x ) , so that if x > 2 then g (cid:48) ( x ) < 0 , which implies that g is injective on (2 , ∞ ) .
By assumption, we have c − 1 > 2 ; consequently there is at most one solution to f (cid:48)
( x ) − f (cid:48) ( c − x ) = 0 in 0 < x < c , and given that x = 12 c is a solution, there is exactly one solution.
We check
[Equation]
where the ﬁrst RHS is the value of f ( x ) + f ( c − x ) at both x = 0 and x = c , and since 1 /c ≥ 2 , we conclude that f ( x ) + f ( c − x ) ≥ 2 f
( c/ 2) > 0 .
Next, we use
[Equation]
together with the bound x p e − x ≤ p p e − p for p > 0 , which is proved by differentiating x (cid:55)→ x p e − x , equating to zero, and comparing the values of the function at x = 0 , x = p , and x → ∞ , to obtain with the triangle inequality
[Equation]
Combining these bounds with our lower bound on f ( x ) + f ( c − x ) and repeatedly applying the triangle inequality and modulus bounds in (E.50) and (E.51), then subsequently in (E.49) (using also | x | ≤ c ), we conclude the claimed bounds on | φ (cid:48)
c | and | φ (cid:48)(cid:48) c | .
Lemma E.32.
Let Z, ¯ Z ∈ L 2 be square-integrable random variables.
Suppose that ¯ Z ≤ C a.s. and (cid:107) Z − ¯ Z (cid:107) L 2 ≤ M .
Then
Var[ Z ] ≤ Var[ ¯ Z ] + CM + M 2 .
Proof.
This is a simple consequence of the triangle inequality and the centering inequality for the
L 2 norm.
We have
[Equation]
and additionally
[Equation]
so that, after squaring, we get
[Equation]
by centering and the a.s. boundedness assumption.
Lemma E.33.
Let X, Y be square-integrable random variables, and let d > 0 .
Suppose | X | ≤ Ma.s., and suppose P [ | Y − 1 | ≥ C (cid:112) d/n ] ≤ C (cid:48)
e − cd and (cid:107) Y − 1 (cid:107) L 2 ≤ M 2 .