D EEP N ETWORKS AND THE M ULTIPLE M ANIFOLD
P ROBLEM
A BSTRACT
We study the multiple manifold problem, a binary classiﬁcation task modeled on applications in machine vision, in which a deep fully-connected neural network is trained to separate two low-dimensional submanifolds of the unit sphere.
We provide an analysis of the one-dimensional case, proving for a simple manifold conﬁguration that when the network depth L is large relative to certain geometric and statistical properties of the data, the network width n grows as a sufﬁciently large polynomial in L , and the number of i.i.d.
samples from the manifolds is polynomial in L , randomly-initialized gradient descent rapidly learns to classify the two manifolds perfectly with high probability.
Our analysis demonstrates concrete beneﬁts of depth and width in the context of a practically-motivated model problem: the depth acts as a ﬁtting resource, with larger depths corresponding to smoother networks that can more readily separate the class manifolds, and the width acts as a statistical resource, enabling concentration of the randomlyinitialized network and its gradients.
The argument centers around the “neural tangent kernel” of Jacot et al.
and its role in the nonasymptotic analysis of training overparameterized neural networks; to this literature, we contribute essentially optimal rates of concentration for the neural tangent kernel of deep fullyconnected ReLU networks, requiring width n ≥ L poly( d 0 ) to achieve uniform concentration of the initial kernel over a d 0 -dimensional submanifold of the unit sphere S n 0 − 1 , and a nonasymptotic framework for establishing generalization of networks trained in the “NTK regime” with structured data.
The proof makes heavy use of martingale concentration to optimally treat statistical dependencies across layers of the initial random network.
This approach should be of use in establishing similar results for other network architectures.
I NTRODUCTION
Data in many applications in machine learning and computer vision exhibit low-dimensional structure (Fig. 1a).
Although deep neural networks achieve state-of-the-art performance on tasks in these areas, rigorous explanations for their performance remain elusive, in part due to the complex interaction between models, architectures, data, and algorithms in neural network training.
There is a need for model problems that capture essential features of applications (such as low dimensionality), but are simple enough to admit rigorous end-to-end performance guarantees.
In addition to helping to elucidate the mechanisms by which deep networks succeed, this approach has the potential to clarify the roles of various network properties and how these should reﬂect the properties of the data.
These considerations lead us to formulate the multiple manifold problem (Fig. 1b), a binary classiﬁcation problem in which the classes are two disjoint submanifolds of the unit sphere S n 0 − 1 , and the classiﬁer is a deep fully-connected ReLU network of depth L and width n trained on N i.i.d.
samples from a distribution supported on the manifolds.
The goal is to articulate conditions on the network architecture and number of samples under which the learned classiﬁer provably separates the two manifolds , guaranteeing perfect generalization to unseen data.
The difﬁculty of an instance of the multiple manifold problem is controlled by the dimension of the manifolds d 0 , their separation ∆ , and their curvature κ , allowing us to study the constraints imposed by these intrinsic properties of the data on the settings of the neural network’s architectural hyperparameters such that the two manifolds can be separated by training with a gradient-based method.
[Figure]
M +
M ! ; 1 = 5 "
[Figure]
Our main result is an analysis of the one-dimensional case of the multiple manifold problem, which reduces the analysis of the gradient descent dynamics to the construction of a certiﬁcate —showing that a certain deterministic integral equation involving the network architecture and the structure of the data admits a solution of small norm.
We construct such a certiﬁcate for the simple geometry in
Fig.
3, guaranteeing generalization in this setting.
Theorem 1 (informal) .
Let d 0 = 1 .
Suppose a certiﬁcate for M exists.
Then if the network depth satisﬁes L ≥ poly( κ, C ρ , log( n 0 )) , the width satisﬁes n ≥ poly( L, log( Ln 0 )) , and the number of training samples satisﬁes N ≥ poly( L ) , randomly-initialized gradient descent on N i.i.d.
samples rapidly learns a network that separates the two manifolds with overwhelming probability.
The constants C ρ , κ depend only on the data density and the regularity of the manifolds.
In addition, if
L (cid:38) ∆ − 1 , then a certiﬁcate exists for the conﬁguration of M shown in Fig.
Theorem 1 gives a provable generalization guarantee for a model classiﬁcation problem with deep networks on structured data that depends only on the architectural hyperparameters and properties of the data.
In addition, it provides an interpretable tradeoff between the architectural settings necessary to separate the two manifolds: the network depth needs to be set according to the intrinsic difﬁculty of the problem, and the network width needs to grow with the depth.
Our analysis gives further insight into the independent roles played by each of these parameters in solving the problem, with the depth acting as a ‘ﬁtting resource’, making the network’s output more regular and easier to change, and the width acting as a ‘statistical resource’, granting concentration of the network over the random initialization around a well-behaved object that we can analyze.
Moreover, the sample complexity of Theorem 1 is dictated by the intrinsic difﬁculty of the problem instance which is set by the geometry of the data.
As a consequence, we avoid any dependence of the width of the network on the number of samples, which is common in deep network convergence results in the literature (e.g. (Allen-Zhu et al., 2019b; Du et al., 2019), (Chen et al., 2021, Theorem 3.4)).
As is the case in practice, given a ﬁxed architecture, more data doesn’t have a detrimental effect on ﬁtting 1 .
Theorem 1 is modular, in the sense that a generalization guarantee is ensured for any geometry for which one can construct a certiﬁcate.
The key to our approach will be to approximate the gradient descent dynamics with a linear discrete dynamical system deﬁned in terms of the so-called neural tangent kernel Θ( x , x (cid:48) ) deﬁned on the manifolds.
Due to the structure in the data, diagonalizing the operator corresponding to this kernel is intractable in general, but we show that constructing a certiﬁcate—arguably an easier task, because it requires producing a bound on the norm of a solution to an equation rather than producing the solution itself—sufﬁces to guarantee that the error decreases rapidly during training given a suitably structured network.
We summarize the primary contributions of this work below.
• Generalization in deep networks: There are few generalization results for deep networks trained efﬁciently with gradient descent available in the literature.
2 Theorem 1 provides such a guarantee that does not depend on any property of the trained network (e.g., norms of ﬁnal weights) that is not readily available before training.
In this context, the certiﬁcate condition is equivalent to the initial network function having a controlled norm in a certain RKHS; this condition is natural in the training regime we consider, and appears ubiquitously in works on generalization in shallower networks (Ghorbani et al., 2020; Ji & Telgarsky, 2020; Nitanda & Suzuki, 2021).
• Uniform concentration of the neural tangent kernel for deep ReLU networks: As an intermediate step in the proof of Theorem 1, we establish essentially optimal rates of uniform concentration for the neural tangent kernel of an arbitrarily deep network (Theorem 2) using martingale concentration, where we require the width to grow only linearly with the depth.
We expect this martingale approach to be applicable to essentially any other compositionally-structured network architecture.
Our uniform result generalizes prior results on pointwise concentration (Arora et al., 2019b; Allen-Zhu et al., 2019b), analogous to our Theorem B.3, and proves useful in establishing generalization.
• Strong regularity estimates for random ReLU networks: As a further consequence of the uniform concentration framework we have developed, we obtain depth-logarithmic Lipschitz estimates for random ReLU networks of arbitrary depth and linear
width, as well as (for still wider networks) a uniform approximation for the network output by a constant which improves with depth, both with overwhelming probability (Section 3.3).
We also control the evolution of the Lipschitz constant during NTK regime training (Lemma B.7), showing that it scales polynomially in the depth.
These results may be of interest in applications where guaranteeing a Lipschitz property for networks is important, such as GAN training (Miyato et al., 2018) or denoising (Ryu et al., 2019; Sun et al., 2020).
1. R ELATED W ORK
Deep networks and low-dimensional structure.
The notion of modeling data as low-dimensional submanifolds has been widely considered in the context of clustering (Wang et al., 2015) and manifold learning (Donoho & Grimes, 2005; Fefferman et al., 2016).
Goldt et al.
(2020) independently proposed the “hidden manifold model”, a model problem for learning shallow neural networks for binary classiﬁcation of structured data with motivations very similar to ours and which admits a mean-ﬁeld analysis (Gerace et al., 2020).
The data model consists of gaussian samples from a low-dimensional subspace passed through a nonlinear function acting coordinatewise in the standard basis; although this models statistical variations around a base domain, a feature of real data that the model we study here lacks, we believe that the study of an arbitrary density supported on two Riemannian manifolds lends our data model increased structural generality.
In the context of kernel regression with the kernel given by the NTK of a two-layer neural network, Ghorbani et al.
(2020) study a data generating model that consists of uniform samples from a low-dimensional subsphere corrupted additively by independent uniform samples from a subsphere in the orthogonal complement, and a target mapping that depends only on the low-dimensional part.
The authors obtain asymptotic generalization guarantees for this data model that reveal conditions under which the corruption degrades the performance of neural tangent methods.
involves a-priori assumptions on the trained network weights, which are only resolved for two-layer networks, and entails an unnatural relationship between n and N and a possible exponential dependence of N on L , which Theorem 1 avoids.
Analyses of neural network training.
To reason analytically about the complicated training process, we adopt the neural tangent kernel approach (Jacot et al., 2018).
The ﬁrst works to instantiate these ideas in a nonasymptotic setting obtained convergence guarantees for training deep neural networks on ﬁnite datasets (Allen-Zhu et al., 2019b; Du et al., 2019).
By exploiting more structure in the data, generalization results have been obtained (Allen-Zhu et al., 2019a; Arora et al., 2019a;
Ji & Telgarsky, 2020; Oymak et al., 2019; Cao & Gu, 2019; Suzuki, 2020; Li et al., 2020; AllenZhu & Li, 2020) that apply to shallow networks, teacher-student learning scenarios, and/or hold conditional on the existence of certain small-norm interpolators.
Other works have obtained generalization guarantees using generalization bounds for kernel methods (Ghorbani et al., 2019; Liang et al., 2020; Ghorbani et al., 2020; Montanari & Zhong, 2020) using the fact that the linearized predictor in the NTK regime can be linked to a kernel method (Arora et al., 2019b).
A parallel line of works (Mei et al., 2018; Tzen & Raginsky, 2020; Mei et al., 2019; Chizat & Bach, 2020; Fang et al., 2020) approach the problem by studying an inﬁnite-width limit of neural network training that yields a different training dynamics.
Approaches of this type are of interest because there is no restriction to short-time dynamics, and the limit of the dynamics can often be characterized in terms of a well-structured object, such as a max-margin classiﬁer (Chizat & Bach, 2020).
On the other hand, it is often difﬁcult to prove ﬁnite-time convergence to the limit.
P ROBLEM F ORMULATION AND M AIN R ESULTS 2. D ATA M ODEL AND N ETWORK D EFINITIONS
We consider data supported on the union of two class manifolds M = M + ∪ M − , where M + and
M − are two disjoint smooth regular simple curves taking values in S n 0 − 1 , with n 0 ≥ 3 .
We denote the data measure supported on M that generates our samples as µ ∞ , with corresponding density ρ , and write ρ min = inf x ∈M ρ ( x ) and ρ max = sup x ∈M ρ ( x ) .
We denote by κ a uniform bound on the (extrinsic) curvature of the two curves, we write ∆ =
min x ∈M + , x (cid:48) ∈M − ∠ ( x , x (cid:48) ) for the separation between class manifolds, where ∠ ( x , x (cid:48) )
=
cos -1 (cid:104) x , x (cid:48) (cid:105) for unit vectors, and to have a quantitative characterization of ‘how simple’ the curves are, we assume there exist constants 0 < c λ ≤ 1 ,
K λ ≥ 1 such that for every 0 < s ≤ c λ /κ
and every x , x (cid:48) in a common connected component of
M , one has dist M ( x , x (cid:48) )
≤ K λ s if ∠ ( x , x (cid:48) )
≤ s , where dist M denotes the Riemannian distance.
Our target function is the signed indicator for each class manifold f (cid:63) ( x ) = 1 M + ( x ) − 1 M − ( x ) .
The model we consider is a fully-connected neural network with ReLU activations and access to i.i.d.
samples from µ ∞ and their corresponding labels.
We parameterize our neural network with weights W 1 ∈ R n × n 0 , W (cid:96)
∈ R n × n if (cid:96) ∈ { 2 , . . .
, L } , and W L +1 ∈ R 1 × n , which we collect as θ =
( W 1 , . . .
, W L +1 ) , and write the iterates of the forward pass as α (cid:96) θ ( x ) = (cid:2) W
(cid:96) α (cid:96) − 1 θ ( x ) (cid:3) + for (cid:96) = 1 , . . .
, L with α 0 θ ( x ) = x , which we also refer to as features or activations , with the network output written as f θ ( x ) =
W L +1 α L θ ( x ) , and the prediction error as ζ θ ( x ) =
f θ ( x ) − f (cid:63) ( x ) .
For an i.i.d.
sample ( x 1 , . . .
, x N ) from µ ∞ , we write µ N = 1 N (cid:80) Ni =
δ x i for the empirical measure associated to the sample, and we consider the training objective L µ N ( θ ) = 12 (cid:82) M ( ζ θ ( x ))
2 d µ N ( x ) , i.e. the empirical risk evaluated with the square loss.
We train with vanilla gradient descent with constant step size τ > 0 : after randomly initializing the parameters θ N 0 as W (cid:96) ∼ i .
i .
d .
N (0 , 2 /n )
if (cid:96) ∈ [ L ] and W L +1 ∼
i .
i .
d . N (0 , 1) we consider the sequence of iterates θ
Nk +
=
θ Nk − τ (cid:101) ∇L µ N ( θ Nk ) , where (cid:101) ∇L µ N represents a ‘formal gradient’ of the empirical loss, which we deﬁne in detail in Appendix A.1.
We say the parameters obtained at iteration k of gradient descent separate the manifolds
M if the classiﬁer implemented by the neural network with the parameters θ Nk labels the two manifolds correctly, i.e. if f (cid:63) ( x ) sign( f θ Nk ( x ))
= 1 for every x ∈ M .
As a shorthand, we will denote quantities evaluated at θ Nk with a subscript k ; an omitted subscript will denote k = 0 , and we will write explicitly θ
= θ N 0 .
Additional notation is provided in Appendix A.5.1.
2. E RROR D YNAMICS AND C ERTIFICATES
Because it is difﬁcult to endow the network parameters generated by the gradient iteration with a particular interpretation, we prefer to reason about how the network error ζ N k evolves under gradient descent.
We calculate (in Lemma B.8)
[Equation]
where we have deﬁned the integral kernel Θ
[Equation]
, where (cid:101) ∇ f θ 0 denotes a formal gradient of the initial network function with respect to the parameters, which is deﬁned in detail in Appendix A.1.
We then deﬁne a nominal error evolution by
[Equation]
with identical initial conditions ζ ∞ 0 = ζ and where Θ( x , x
[Equation]
) =
[Equation]
x
[Equation]
) (cid:105) is the socalled neural tangent kernel with associated integral operator Θ .
We prove that the error evolution (2.1) is well-approximated by the nominal error evolution under suitable conditions on the network width, step size, and number of samples, which together ensure that training proceeds in the “NTK regime” where Θ Nk stays close to Θ .
As for the nominal error evolution (2.2), we note that this system is linear, time-invariant, and stable when τ is set appropriately small, so the norm of the nominal error is guaranteed to decrease rapidly if the initial error ζ aligns well with eigenfunctions of Θ corresponding to large eigenvalues.
However, computation of these eigenfunctions is intractable for general data geometries and distributions because the operator Θ is not generally translationally invariant on M .
To overcome this issue, we prove this alignment implicitly by constructing an approximate solution to the linear integral equation Θ [ g ] = ζ such that (cid:107) g (cid:107) L 2 µ ∞ is sufﬁciently small.
To be precise, g ∈ L 2 µ ∞ will be called a δ 1 , δ 2 -certiﬁcate for the dynamics (2.2) if
[Equation]
2. M AIN R ESULTS AND P ROOF O UTLINE
Our main result is that conditional on the existence of a certiﬁcate of suitably small norm for M , gradient descent provably separates the two manifolds in time polynomial in the network depth.
Theorem 1.
Let M be a one-dimensional Riemannian manifold satisfying our regularity assumptions.
For any 0 < δ ≤ 1 /e , choose
[Equation]
N
[Equation]
and ﬁx τ such that C 3 nL 2 ≤ τ ≤ C 4 nL .
Then if there exists a certiﬁcate in the sense of (2.3) with δ 1 = C 5 C 1 / 2 ρ (cid:112) log(1 /δ ) log( nn 0 ) /L and δ 2 = C 6 (cid:112) log(1 /δ )
log( nn 0 ) /
( nρ 1 / 2 min ) , with probability at least 1 − δ over the random initialization of the network and the i.i.d.
sample from µ ∞ , the parameters obtained at iteration (cid:98) L 39 / 44 / ( nτ ) (cid:99) of gradient descent on the ﬁnite sample loss L µ N yield a classiﬁer that separates the two manifolds.
The constants C 1 , . .
, C 6 are suitably chosen absolute constants, the constants κ , K λ , c λ are respectively the extrinsic curvature constant and the global regularity constant deﬁned in Section 2.1, the constant C ρ is deﬁned as max { ρ min , ρ − 1min } , and the constant C µ ∞ is deﬁned as
C 15 ρ (1 + ρ max ) 6 (min { µ ∞ ( M + ) , µ ∞ ( M − ) } ) − 11 / 2 .
For one-dimensional instances of the two manifold problem with sufﬁciently deep and overparameterized networks trained in the small-step-size regime, Theorem 1 completely reduces the analysis of the gradient iteration to the certiﬁcate problem.
From a qualitative perspective, the network resource constraints imposed by Theorem 1 are natural:
(i)
The network depth L is set by geometric and statistical properties of the data with only a mild polylogarithmic dependence on the ambient dimension n 0 , which reﬂects the role of depth in controlling the capability of the network to ﬁt functions.
(ii)
The network width n is set by the depth L : the inductive structure of the network causes quantities that depend on the initial random weights θ 0 to concentrate worse as the depth is increased, which can be counteracted by setting the width appropriately large.
(iii)
The sample complexity of N ≥ L 10 reﬂects the capacity of the network via the depth, and is in particular independent of the width n , which can thus be interpreted as purely a statistical resource.
In addition, the conclusion of Theorem 1 implies not just that the expected generalization error with respect to µ ∞ of a binary classiﬁer is zero, but the stronger separation property, i.e. that the generalization error will be zero for any choice of test distribution supported on M simultaneously.
We give a brief sketch of the proof of Theorem 1 in Appendix A.4.
To obtain a generalization guarantee from Theorem 1, it only remains to construct a certiﬁcate for M .
We demonstrate this for the family of simple, highly-symmetric geometries shown in Figure 3, and leave the case of general one-dimensional manifolds for future work.
Proposition 1.
Let M be an r -instance of the two circles geometry studied in Appendix C.1.
and shown in Figure 3, with r ≥ 1 / 2 .
For any 0 < δ ≤ 1 /e , if L ≥ C 1 ∆ − 1 and n ≥
C 2 L 5 log 4 (1 /δ ) log 4 ( Ln 0 log(1 /δ )) , then there exists a certiﬁcate in the sense of (2.3) satisfying the requirements of Theorem 1 with probability at least 1 − 3 δ for some absolute constants C 1 , C 2 > 0 .
Taking a union bound, Proposition 1 shows that under the hypotheses of Theorem 1, with probability at least 1 − 4 δ a certiﬁcate exists for the geometry shown in Figure 3 as soon as L is larger than a constant multiple of the inverse separation ∆ − 1 , even as the separation approaches zero.
We conjecture that a similar phenomenon holds for more general geometries, possibly with additional dependencies on the curvature and global regularity parameters of M .
The dependence of L on the geometry is due to the “sharpening” effect the depth has on the kernel Θ governing the dynamics and thus on the ﬁtting capabilities of the network, as illustrated in Figure 2a.
[Figure]
To prove that the nominal error evolution (2.2) decreases rapidly and approximates the actual error evolution (2.1) throughout training, it is essential to have a precise characterization of the ‘initial’ neural tangent kernel Θ .
One of our main technical contributions is to show concentration of Θ in the regime where the width n scales linearly with the depth L
Theorem 2.
For any d 0 ∈ N , let M be a d 0 -dimensional complete Riemannian submanifold of S n
−.
Then if n ≥ C 1 Ld 40 log 4 ( C M n 0 L ) , one has with probability at least 1 − n − 10 that for every ( x , x (cid:48) )
∈ M × M
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
where we write ν = ∠ ( x , x (cid:48) ) with an abuse of notation, ϕ ( (cid:96) ) denotes the (cid:96) -fold composition of ϕ ( ν ) =
cos − 1 (cid:0) (1 − νπ ) cos ν + sin νπ (cid:1) , the constants C 1 , C 2 > 0 are absolute, and the constant
C M > 0 is absolute except if d 0 = 1 , where it depends only on the lengths of the class manifolds.
For networks of uniform width that are wider than they are deep by a certain constant factor, we believe that the scalings in Theorem 2 are essentially optimal: the variance calculations of Hanin & Nica (2020) give some heuristic evidence here, and we believe the idea of using diagonal concentration to prove deviation lower bounds could be generalized to rigorously establish optimality.
Figure 2b illustrates the phenomenon underlying Theorem 2.
We discuss the proof of Theorem 2 in more detail in Sections 3.1 and 3.3.
K EY P ROOF E LEMENTS 3. C ONCENTRATION AT I NITIALIZATION : M ARTINGALES AND A NGLE C ONTRACTION
The initial kernel Θ is a complicated random process deﬁned over the weights ( W 1 , . . .
, W L +1 ) .
To control it, we ﬁrst show for ﬁxed ( x , x (cid:48) )
that Θ( x , x (cid:48) ) concentrates with high probability, and then leverage approximate continuity properties to pass to uniform control of Θ .
Here we describe our approach to pointwise control; uniformization is discussed in Section 3.3.
The kernel can be written in the form
[Equation]
where β (cid:96) ( x ) =
( W L +1 P I L ( x ) · · · W (cid:96) +
P I (cid:96) +1 ( x ) ) ∗ will be referred to as backward features , and
P I (cid:96) ( x ) is a projection onto { α (cid:96) ( x ) > 0 } .
We consider (cid:104) β 0 ( x ) , β 0 ( x (cid:48) )
(cid:105) as a representative example: up to a small residual term, this random variable can be expressed as a sum of martingale differences.
Formally, for (cid:96) ∈
[ L ] , let F (cid:96) denote the σ -algebra generated by all weight matrices up to layer (cid:96) , with F 0 denoting the trivial σ -algebra.
We can then write
[Equation] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
(3.1) for some functions g (cid:96) and controllable residual R , where ν 0 = ∠ ( x , x (cid:48) ) .
If we ﬁx all the variables in
F (cid:96) − 1 , the ﬂuctuations in the (cid:96) -th summand will be due to W (cid:96) alone.
Intuitively, since each weight matrix appears at most once in β 0 ( x ) , 4 it will appear at most twice in g (cid:96) , and therefore g (cid:96) will have a subexponential distribution conditioned on F (cid:96) − 1 and concentrate well around its conditional expectation.
This property stems from the compositional structure of the network, with independent sources of randomness introduced at every layer, and is essentially agnostic to other details of the architecture.
The concentration of the summands in (3.1) implies concentration of the sum: even though the summands are not independent, they can be controlled using concentration inequalities analogous to those for sums of independent variables (Azuma, 1967; Freedman, 1975).
Showing that terms of the form (cid:104) α (cid:96) ( x ) , α (cid:96) ( x (cid:48) ) (cid:105) concentrate in the linear regime gives rise to additional challenges.
Here we exploit an essential difference between the concentration properties our NTK scales like nL rather than L .
Due to our scaling of the weights (Section 2.1) the contribution of the ﬁnal layer to the NTK is negligible and can be dropped.
This leads to discrepancies between the expression above and similar expressions found in the literature—we show essential equivalence between our NTK and others in Appendix A.3.
Technically, the features α (cid:96) ( x ) depend on all the weights up to layer (cid:96) and hence so does the projection matrix P I (cid:96) ( x ) , but our analysis shows that this dependence has only a minor effect on the statistical ﬂuctuations.
of the angles between features ν (cid:96) =
∠ ( α (cid:96) ( x ) , α (cid:96) ( x (cid:48) )) relative to those of the correlation process (cid:104)
α (cid:96) ( x ) , α (cid:96) ( x (cid:48) ) (cid:105) studied in prior works on concentration of Θ : when ν (cid:96) − 1 = 0 , we have that ν (cid:96) =deterministically , whereas the correlation process behaves like a subexponential random variable with small but nonzero deviations.
Together with smoothness, this clamping phenomenon allows us to show concentration of the angle at layer (cid:96) around the function ϕ
( (cid:96) )
( ν 0 ) , which is no larger than a constant multiple of (cid:96) − 1 .
This contraction of the angles with depth is the key to establishing
Theorem 2; in addition, it gives the invariant kernel ˆΘ (see Figure 2b) its sharpness at zero and localization properties, both of which increase as the depth is increased and which we exploit in the proof of Proposition 1.
We provide full details of our approach in Appendices D and E.
[Figure] [Figure]
By a simple argument that relies on positiveness of Θ , we show that if we can solve the certiﬁcate problem (2.3), then for a suitably chosen learning rate τ and number of iterations k (Lemma B.6)
[Equation]
If the network is sufﬁciently deep, the norm of the nominal error can thus be made arbitrarily small in a number of iterations that scales only polynomially with the problem parameters.
Because our formulation of the certiﬁcate problem (2.3) accommodates approximate solutions, under a minor condition on the network width n (see Proposition 1) it sufﬁces to solve an auxiliary system ˆ Θ [ g ]
=
ˆ ζ , where ˆ Θ and ˆ ζ are analytically-convenient approximations to Θ and ζ produced by our concentration analysis, including Theorem 2.
For the simple geometry in Fig.
3, we show in Appendix C.1.1 how to solve this auxiliary system using Fourier analysis, where we require
L (cid:38) ∆ − 1 .
The depth of the network is thus determined by the geometry of the data, and speciﬁcally by the inter-manifold distance which intuitively sets the “difﬁculty” of the ﬁtting problem.
In
Section 4 we discuss approaches to constructing certiﬁcates for general smooth curves.
3. U NIFORM C ONCENTRATION AND ITS C ONSEQUENCES
To uniformize the pointwise estimates of Section 3.1, we must overcome the issue that the backward features β (cid:96) ( x ) are not continuous functions of the input, due to the matrices P
I (cid:96) ( x ) .
Our approach is to discretize the input space, control the number of features that can change sign near each point in the discretization, then extend the pointwise estimates of Section 3.1 to the setting where a small number of features have changed sign—again, we ﬁnd martingale concentration a necessity to achieve linear width-depth scaling.
We give full details in Appendix D.3.
Although Theorem 2 is the main application of these estimates—with uniform control of Θ , we can prove operator norm bounds on its corresponding integral operator Θ , which is of great help in proving generalization results—they also imply useful regularity estimates for the initial random network f θ 0 .
For example, we prove that networks of uniform width n (cid:16) n 40 L are with high probability (cid:112) n 0 (log n 0 )(log L ) -Lipschitz as functions on R n 0 (Theorem B.5)—in particular, the Lipschitz constant depends only logarithmically on depth, in contrast to existing results in the literature (Nguyen et al., 2020).
For networks of larger width n (cid:38) d 30 L 5 , we prove that with high probability the network f θ 0 is approximately constant on the domain M ⊂ S n 0 − 1 (Lemma D.11):
[Equation]
We ﬁnd this result to be useful in simplifying the certiﬁcate construction problem of Section 3.2.
D ISCUSSION
Certiﬁcates for curves.
The most urgent task toward expanding the scope of Theorem 1 is the construction of certiﬁcates for geometries beyond the coaxial circles of Proposition 1.
The proof of Proposition 1 relies heavily on translation invariance of the intra- and inter-manifold distances in the coaxial circles geometry in order to avoid the need for certain sharp technical estimates for the decay of the kernel ˆΘ .
With sharper control of the decay of the kernel ˆΘ , it is possible to select the network depth in a way that grants appropriate worst-case control of the magnitude of the crossmanifold integrals in the action of ˆ Θ (as in Figure 2a), allowing us to reduce to what is essentially a one-manifold certiﬁcate construction problem that can be solved with harmonic analysis.
Beyond these considerations, it is important to extend Theorem 1 to manifolds of dimension d 0 > 1 , which should be relatively straightforward.
Our concentration results, notably including Theorem 2, are already applicable to manifolds of arbitrary dimension.
Convolutional networks and non-differentiable manifolds.
Although we have motivated our data model in the multiple manifolds problem using applications in computer vision, it is important to note that the spatially-structured image articulation manifolds that arise as data in these contexts do not carry a differentiable structure (Wakin et al., 2005), so the assumption of bounded curvature may not be realistic here.
On the other hand, in these applications it is standard to employ a convolutional network architecture.
We anticipate that our martingale concentration framework can be extended to these architectures, and beyond establishing analogues of Theorem 1 in this setting, we believe it should be possible to obtain similar guarantees for models of image articulation manifolds.
In particular, one might expect randomly-initialized convolutional networks to enjoy local invariance properties, like the scattering networks of Mallat (Mallat, 2012; Bruna & Mallat, 2013), which could achieve a degree of invariant classiﬁcation without expending additional network resources computing convolutions over general LCA groups (Cohen & Welling, 2016).
The importance of being low-dimensional.
Ghorbani et al.
(2019) show that kernel ridge regression with any rotationally invariant kernel on S d (including that of a deep network) is equivalent to polynomial regression with a degree p polynomial if the number of samples is bounded by d p +1 and d → ∞ .
For data lying on a low-dimensional manifold, as we consider here, one would expect less pessimistic rates; indeed, in a subsequent work (Ghorbani et al., 2020) the authors establish similar guarantees to Ghorbani et al.
for a linear data model with low-dimensional structure in terms of a smaller “effective dimension”.
In comparison, although our present certiﬁcate construction argument only implies dynamics for the restrictive coaxial circles geometry of Figure 3, for which one can obtain guarantees for kernel regression with a shallow NTK by the results of Ghorbani et al.
(2020), the general multiple manifold problem formulation allows one to model nonlinear structure in the data, and measures ﬁtting difﬁculty through intrinsic parameters like the curvature and separation.
The guarantees in Ghorbani et al.
(2019; 2020) depend on the degree of approximability of the target function by low-degree polynomials, and although this achieves additional generality over our model, it seems more challenging to relate this to geometric or other types of nonlinear low-dimensional structure.
The NTK regime and beyond.
In recent years there has been much work devoted to the analysis of networks trained in the regime where the changes in Θ Nk remain small and the dynamics in (2.1) are close to linear (Jacot et al., 2018; Lee et al., 2019; Arora et al., 2019b; Allen-Zhu & Li, 2019) (referred to as the NTK/“overparametrized”/kernel regime).
Concurrently, there have also been results highlighting the limitations of this regime.
In (Chizat & Bach, 2018) the authors coin the term “lazy training” in referring to dynamics where the relative change in the differential of the network function is small compared to the change in the objective during gradient descent.
While the dynamics we study indeed fall into this category, the analysis makes it evident that not all lazy training regimes are created equal.
Our performance guarantees depend on the structure of the kernel ˆΘ , and on controlling the ﬂuctuations of Θ Nk around it.
We are able to control these only if the width of the network is sufﬁciently large compared to the depth.
In contrast, lazy training can also be achieved in homogeneous models by simply scaling the output of the model (Chizat & Bach, 2018), in which case one cannot argue that the kernel has the decay properties that enable it to ﬁt data.
Our analysis hinges on staying in the NTK regime during training.
We obtain suboptimal scaling of n with L in Theorem 1 because we treat all changes that occur in Θ Nk during training as being adversarial to the algorithm’s ability to generalize.
It is likely that if an improved understanding of feature learning can be incorporated into an analysis of the dynamics, the resulting scaling requirements would be more realistic.
A CKNOWLEDGMENTS
This work was supported by the grants NSF 1733857, NSF 1838061, NSF 1740833, NSF 1740391,
NSF NeuroNex Award DBI-1707398 (DG), the Gatsby Charitable Foundation (DG) and a Swartz fellowship (DG), and by a fellowship award (SB) through the National Defense Science and Engineering Graduate (NDSEG) Fellowship Program, sponsored by the Air Force Research Laboratory (AFRL), the Ofﬁce of Naval Research (ONR) and the Army Research Ofﬁce (ARO).
The authors would like to thank Ethan Dyer, Guy Gur-Ari, Quynh Nguyen, Jeffrey Pennington, Sam Schoenholz,
Daniel Soudry and Tingran Wang for helpful discussions/feedback
C ONTENTS A Extended Problem Formulation
A.1 Regarding the Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.2 Regarding the Data Manifolds . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.3 Regarding the Initialization . .
A.4 Proof Outline for Theorem 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.5 Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.5.1 General Notation . .
A.5.2 Summary of Operator and Error Deﬁnitions . . . . . . . . . . . . . . . . .
B Proofs of the Main Results
B.1 Main Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.2 Supporting Results on Dynamics . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.3 Auxiliary Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C Skeleton Analysis and Certiﬁcate Construction
C.1 Certiﬁcate Construction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C.1.
Two Circles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C
Auxiliary Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C
Geometric
Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C
Analysis of the Skeleton . . . . . . . . . . . . . . . . . . . . . . . . . . .
D
Concentration at Initialization
D.1 Notation and Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D.2 Pointwise Concentration . .
D.2.1 Forward Concentration . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D.2.2 Backward Feature Control . .
D.3 Uniformization Estimates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D.3.1 Nets and Covering Numbers . .
D.3.2 Controlling Support Changes Uniformly . . . . . . . . . . . . . . . . . . .
D.3.
Uniformizing Forward Features Under SSC . . . . . . . . . . . . . . . . .
D.3.4 Small Support Change Residuals . .
D.4 Auxiliary Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E Sharp Bounds on the One-Step Angle Process
E.1 Deﬁnitions and Preliminaries . . .
E.2 Main Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E.3 Supporting Results . .
E.3.
Core Supporting Results . .
E.3.
Proving Lemma E.6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E.3.
Proving Lemma E.7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E.3.
General Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E.3.
Differentiation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E.3.
Miscellaneous Analytical Results . .
E.4 Deferred Proofs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
F Controlling Changes During Training
F.
Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
F.
Changes in Feature Supports During Training . . . . . . . . . . . . . . . . . . . .
F.
Changes in Features During Training . . . . . . . . . . . . . . . . . . . . . . . . .
F.
Changes in Θ Nk During Training . . . . . . . . . . . . . . . . . . . . . . . . . . .
F.
Auxiliary Lemmas and Proofs . . . . . . . . . . . . . . . . . . . . . . . . . . . .
G Auxiliary Results
A PPENDICES : S UMMARY OF C ONTENTS
We brieﬂy summarize the contents of each of the subsequent appendices.
A. We discuss the contents of the problem formulation section from the main
body, Section 2.1, in more technical detail, in particular giving technical deﬁnitions for formal gradients, regularity conditions, and so on.
We provide a proof sketch to offer some intuitions about the proof of the main result.
We also summarize notation and the key operator deﬁnitions that appear throughout the paper.
B. We give proofs for our main results.
We provide supporting results on the NTK regime dynamics of gradient descent and other relevant technical lemmas, as discussed in the proof sketch of Section 2.3.
C. We give technical deﬁnitions relevant to the cross-manifold perspective on certiﬁcate construction, construct a certiﬁcate for the two circles geometry of Figure 3, and provide technical estimates on the kernels ψ 1 and ψ that remain after applying our measure concentration arguments to the NTK Θ .
D. We collect results on measure concentration relevant to proving our main uniform concentration result for the NTK, Theorem 2.
Some of these results are also relevant for controlling changes during training.
E. We collect results relevant to proving a certain concentration result for the angles between features as they propagate across one layer of the initial neural network.
The main results of this section are fundamental to the study of the concentration of angles in Appendix D, and we provide them in a separate appendix due to their length.
F. We establish results on uniform control of the changes during training of the NTK Θ Nk from its “initial value” of Θ .
These are a key ingredient in our dynamics arguments in Appendix B. G.
We provide statements of general technical lemmas that are of a classical nature, which we rely on throughout the other appendices.
A E XTENDED P ROBLEM F ORMULATION A. R EGARDING THE A LGORITHM
We analyze a gradient-like method for the minimization of the empirical loss L µ N .
After randomly initializing the parameters θ N 0 as W (cid:96) ∼ i .
i .
d .
N (0 , 2 /n )
if (cid:96) ∈ [ L ] and W L +1 ∼
i . i .
d .
N (0 , 1) , independently of the samples x 1 , . . .
, x N , we consider the sequence of iterates
[Equation]
where τ > 0 is a step size, and (cid:101) ∇L µ N represents a ‘formal gradient’ of the loss L µ N , which we deﬁne as follows:
ﬁrst, we deﬁne formal gradients of the network output by
[Equation]
for (cid:96) ∈
[ L ] and x ∈ M , where we have introduced the deﬁnitions
[Equation]
for (cid:96) = 0 , 1 , . . .
, L − 1 , and where we additionally deﬁne
[Equation]
for the orthogonal projection onto the set of coordinates where the (cid:96) -th activation at input x is positive.
We call the vectors β (cid:96) θ ( x ) the backward features or backward activations —they correspond to the backward pass of our neural network.
We also deﬁne
[Equation]
We then deﬁne the formal gradient of the loss L µ N by
[Equation]
Let us emphasize again that the expressions above are deﬁnitions, not gradients in the analytical sense: we introduce these deﬁnitions to cope with nonsmoothness of the ReLU [ · ] + .
On the other hand, our formal gradient deﬁnitions coincide with the expressions one obtains by applying the chain rule to differentiate L µ N at points where the ReLU is differentiable, and we will make use of this fact to proceed with these formal gradients in a manner almost identical to the differentiable setting.
We reiterate here our notational conventions for quantities evaluated at these iterates: we denote evaluation of quantities such as the features and prediction error at parameters along the gradient descent trajectory using a subscript k , with an omitted subscript denoting evaluation at the initial k = 0 parameters, and we add a superscript N to parameters such as the prediction error to emphasize that they are evaluated at the parameters generated by (A.1).
For example, in this notation we express ζ θ Nk as ζ Nk .
In addition, we use θ 0 to denote the initial parameters θ N 0 .
We emphasize the dependence of certain quantities on these random initial parameters notationally, including the initial network function f θ 0 .
A. R EGARDING THE D ATA M ANIFOLDS
We now provide additional details regarding our assumptions on the data manifolds.
For background on curves and more broadly Riemannian manifolds, we refer the reader to (Lee, 2018; Absil et al., 2009).
We assume that M = M + ∪M − , where M + and M − are two disjoint complete connected
Riemannian submanifolds of the unit sphere S n 0 − 1 , with n 0 ≥ 3 .
In particular, M ± are compact.
We take as metric on these manifolds the metric induced by that of the sphere, which we take in turn as that induced by the euclidean metric on R n 0 .
We write µ ∞ + and µ ∞− for the measures on M + and M − (respectively) induced by the data measure µ ∞ , and we assume that µ ∞ admits a density ρ with respect to the Riemannian measure on M , writing ρ + and ρ − for the densities on M ± induced by the density ρ .
When d 0 = 1 , we add additional structural assumptions to the above: we assume that M ± are smooth, simple, regular curves.
Concretely, that M admits a density ρ with respect to the Riemannian measure means that
[Equation]
When d 0 = 1 , because M ± are smooth regular curves, they admit global unit-speed parameterizations with respect to arc length γ ± :
I ± → S n 0 − 1 , where I ± are intervals of the form [0 , len( M ± )] .
In this setting, the curvature constraint is expressed as
[Equation]
and we observe that the fact that M ± are sphere curves implies κ ≥ 1 .
Exploiting the coordinate representation of the Riemannian measure and the ﬁxed inherited metric from R n 0 , we thus have
[Equation]
We will exploit this formula in the sequel to compare between L pµ ( M ) and L p ( M ) norms of functions deﬁned on the manifold.
More generally, we will frequently make use of similar reasoning that leverages the existence of unit-speed parameterizations for the curves.
For clarity we rewrite the global regularity condition: we assume there exist constants 0 < c λ ≤ 1 ,
K λ ≥ 1 such that
[Equation]
where dist M denotes the Riemannian distance between points in a common connected component, and we deﬁne C λ
= K 2 λ /c 2 λ .
Because M ± are simple curves, they do not self-intersect; the assumption (A.2) gives a quantitative characterization of how far the curves are from self-intersecting.
We illustrate how the associated constants can be obtained from the assumption that the manifolds are simple curves: for either (cid:63) ∈ { + , −} , consider a connected component M (cid:63) ⊂ M , and for any
< s ≤ len( M (cid:63) ) , deﬁne
[Equation]
If r (cid:63) ( s ) = 0 , by compactness we can construct a sequence of pairs of points that converges to r (cid:63) ( s ) , but this would imply that M (cid:63) is self-intersecting, contradicting our assumption that it is simple.
It follows that r (cid:63) ( s ) > 0 for any value of s .
If we now deﬁne ˜ K s = r (cid:63) ( s ) /s , it follows that for any ( x , x (cid:48) )
∈ M (cid:63) × M (cid:63) ,
[Equation]
Our regularity assumption implies that a single such constant holds for a range of scales below the curvature scale, which is a mild assumption since ˜ K s approaches 1 as s approaches 0 .
A. R EGARDING THE I NITIALIZATION
The manner in which we have deﬁned our initial random neural network f θ 0 is sometimes referred to as “fan-out initialization” in the literature—it guarantees that feature norms are preserved from layer to layer in the network, and thereby avoids the vanishing and exploding gradient problems.
The difference between this initialization and the so called “standard" or “fan-in” initialization is only in the ﬁrst and last layer weights, yet in a sufﬁciently deep network trained in the NTK regime the effect of any single layer is negligible and the dynamics of our network will be essentially identical to one with standard initialization.
On the other hand, following the work of Jacot et al.
(2018), it has become common in the theoretical literature to consider a different construction of the neural network called “NTK parameterization”, which is in some ways more convenient for theoretical analysis.
In particular, Arora et al. (2019b) prove their results on NTK concentration using this parameterization; to facilitate a comparison between our concentration result (Theorem 2) and theirs, we discuss the connection between fan-out and NTK parameterization in this section.
This material is well-known and no doubt can be found already in the literature, but we believe it may be helpful to translate it into our notation.
Recall our deﬁnitions for the weights and features in our neural network: we have W (cid:96) ∼ i .
i .
d .
N (0 , 2 /n )
if (cid:96) ∈ { 0 , 1 . . .
, L } and W L
+1 ∼
i . i .
d .
N (0 , 1) , with features deﬁned for (cid:96) = 0 , 1 , . . .
, L by
[Equation]
otherwise , and output f θ 0 ( x ) =
W L +1 α L ( x ) .
Within this section—and only within this section—we shall deﬁne auxiliary weights by G (1) ∈ R n × n 0 , G ( (cid:96) )
∈ R n × n for integer 1 < (cid:96) <
L + 1 , and G ( L +1) ∈
R 1 × n , with distributions G ( (cid:96) ) ∼ i .
i .
d .
N (0 , 1) for (cid:96) ∈ { 0 , 1 . . .
, L + 1 } , independent of everything else in the problem.
As before, for (cid:96) ∈ { 0 , 1 , . . .
, L } we use α ( (cid:96) )
NTK ( x ) to denote the layer- (cid:96) features:
[Equation]
This network’s output will be written
[Equation]
By 1 -homogeneity (absolute) of σ , it follows that f θ 0 d = f NTK .
As the notation suggests, the network f NTK corresponds to a “NTK parameterization” network—although this network and f θare equivalent in terms of predictions, their “gradients” are not equivalent.
The NTK for the NTK parameterization network is obtained by differentiating (at points of differentiability): after calculating (as in Lemma B.8), we introduce notation as we did for the fan-out parameterization network in Appendix A.1, so that
[Equation]
with (for (cid:96) = 1 , . . .
, L + 1 )
[Equation]
∇
[Equation]
∗ where β
[Equation]
and
I
[Equation]
We shall relate the NTK parameterization NTK Θ NTK to our fan-out parameterization NTK Θ using homogeneity of the ReLU.
First, let us observe that
[Equation]
because [ · ] + is 1 -homogeneous
and we have G ( (cid:96) )
d = (cid:112) n/ 2 W (cid:96) when (cid:96) ≤ L .
For (cid:96) ∈ { 0 , 1 , . . .
, L } , we note that both α (cid:96) ( x ) and α ( (cid:96) )
NTK ( x ) depend only on the parameters ( W 1 , . .
, W (cid:96) ) and ( G (1) , . . .
, G ( (cid:96) ) ) , respectively.
If we write θ = ( W 1 , . .
, W L ) and θ NTK =
( G (1) , . . .
, G ( L ) ) , then it follows that α (cid:96) ( x ) is a (cid:96) -homogeneous function of θ (and likewise for α ( (cid:96) )
NTK ( x ) ).
In addition, the projection matrices P I (cid:96) ( x ) are 0 -homogeneous functions of θ , and so taking (cid:96) ∈ { 0 , 1 , . . .
, L − 1 } and counting parameters in the deﬁnitions of β (cid:96) ( x ) and β ( (cid:96) )
NTK ( x ) implies that these two functions are ( L − (cid:96) − 1) -homogeneous functions of θ and θ NTK , respectively.
Of course, for (cid:96) =
L , they are 0 -homogeneous.
Thus, using that G ( (cid:96) )
d = (cid:112) n/ 2 W (cid:96) for (cid:96) ≤
L again, we obtain
[Equation]
Although we have argued equidistributionality above for each index (cid:96) separately for simplicity, the elementary nature of our arguments (we are just moving scalars around) and the statistical dependencies across gradients allows us to apply the same argument ‘in parallel’ to the sum of inner products between gradients, yielding
[Equation]
This expression makes it immediately clear that our concentration framework proves sharp concentration of the NTK of a uniform-width NTK parameterization feedforward ReLU network that improves over the results of Arora et al.
(2019b) when the data are on the sphere 7 —a simple adaptation of the proof of Theorem B.3 will sufﬁce.
A. P ROOF O UTLINE FOR T HEOREM
In Appendix B, we prove a slightly more general version of Theorem 1 in Theorem B.1.
Here, we give a brief outline of the proof of this result.
Proving the separation property essentially requires us to obtain control of (cid:107)
ζ Nk (cid:107) L ∞ ( M ) , and by an interpolation inequality (Lemma B.14)
it sufﬁces to control the generalization error (cid:107) ζ Nk (cid:107) L 2 µ ∞ and the smoothness (measured through the Lipschitz constant) of ζ Nk .
We start with the generalization error, picking up from where we left off at the end of Section 2.2: the triangle inequality gives
[Equation]
which allows us to divide the analysis into two subproblems: characterizing the nominal dynamics (Lemmas B.6 and B.12), and the nominal-to-ﬁnite transition (Lemma B.7).
Beginning with the nominal dynamics, we use (2.2) to write
[Equation]
where Θ denotes the operator on L 2 µ ∞ corresponding to integration against the kernel Θ and Id denotes the identity operator.
The deﬁnition of Θ and compactness of M imply that Θ is a positive, compact operator (Lemma B.9), so these dynamics are stable when τ is chosen larger than the operator norm of Θ .
However, the rate of decrease of (cid:107) ζ ∞ k (cid:107) L 2 µ ∞ with k could still be extremely slow if the initial error ζ has signiﬁcant components in the direction of eigenfunctions of Θ corresponding to small eigenvalues, and because Θ acts roughly like a convolution operator, we expect there to exist eigenvalues arbitrarily close to zero.
By solving the certiﬁcate problem (2.3), we can assert that misalignment does not occur.
To solve the certiﬁcate problem, as we describe in Section 3.2, we work with analytically-convenient approximations for Θ and ζ : the exact deﬁnitions of these approximations ˆ Θ and ˆ ζ are given in Appendix A.5.2, and we prove their suitability as approximations in Theorem B.2 (a slightly more general version of Theorem 2) and Lemma D.11, respectively.
As we have discussed in Section 3.1, our rates of concentration for Θ about ˆ Θ are essentially optimal—the poor rates that end up appearing in Theorem B.1 are set by later parts of the argument.
With our approximation to Θ justiﬁed, we show that for any sufﬁciently small step size τ and number of iterations k , solving the certiﬁcate problem (2.3) guarantees appropriate decrease of the nominal generalization error; additional details are discussed in Section 3.2.
The key property that we use in constructing certiﬁcates in Proposition B.4 (the ‘appendix version’ of Proposition 1) is that as the depth L increases, the kernel ˆΘ sharpens and localizes (Fig.
2a): the conditions on L in
Theorem B.1 guarantee that the sharpness is sufﬁcient to ensure that the cross-manifold integrals in the certiﬁcate problem are small in magnitude, which leads to rapid decrease of the nominal error.
Our precise characterization of this phenomenon is presented in Appendix C.
To complete the proof, we will justify the nominal-to-ﬁnite transition in (A.3).
Starting from the update equations (2.1) and (2.2), subtracting and rearranging gives an update equation for the difference:
[Equation]
In particular, if τ is chosen less than the operator norm of Θ , we can take norms on both sides of the previous equation, apply the triangle inequality, then exploit a telescoping series cancellation to obtain the difference bound
[Equation]
There are two obstacles to controlling the norm terms on the RHS of (A.4): the kernels Θ Ns are distinct from the kernel Θ due to changes in the weights that occur during training, and the empirical measure µ N incurs a sampling error relative to the population measure µ ∞ .
To address the ﬁrst challenge, we measure the changes to the NTK during training in a worst-case fashion as
[Equation]
and train in the NTK regime , where the network width n is larger than a large polynomial in the depth L and the total training time kτ is no larger than L/n .
These conditions imply that with high probability ∆ Nk is no larger than a constant multiple of n 1 − δ poly( L, d 0 ) for a small constant δ > 0 , so that the amortized changes during training kτ ∆ Nk can be made small by sufﬁcient overparameterization.
We provide full details of this argument in Appendix F.
By the preceding argument, we can use the triangle inequality and Jensen’s inequality to pass from the norm term in (A.4) to a difference-of-measures term which integrates against Θ , and by Theorem B.2, we can replace the integration against Θ by an integration against a smooth, deterministic kernel, which leads to a bound
[Equation]
L 2 µ
[Equation]
where R k is a residual term that we argue is small in the NTK regime with high probability, and for concision we write ψ 1 to denote the function of ∠ ( x , x (cid:48) ) that appears in Theorem B.2.
To control the remaining term, we make use of a basic result from optimal transport theory, which states that for any probability measure µ on the Borel sets of a metric space X and corresponding empirical measure µ N , one has for every Lipschitz function f
[Equation]
where W ( · , · ) denotes the 1 -Wasserstein metric, and concentration inequalities for empirical measures in the 1 -Wasserstein metric Weed & Bach (2019).
To apply this result to our setting, it is necessary to control the change throughout training of the Lipschitz constant of ζ Nk , and one must also account for the fact that the metric space in our setting is M , which has two distinct connected components.
We treat the ﬁrst issue using an inductive argument, and our treatment of the second issue (Lemma B.13) leads to the dependence on the degree of class imbalance demonstrated in the constant C µ ∞ in Theorem B.1.
A. N OTATION A.5. G ENERAL N OTATION
If n ∈ N , we write [ n ] = { 1 , . . . , n } .
We generally use bold notation x , A for vectors, matrices, and operators and non-bold notation for scalars and scalar-valued functions.
For a vector x or a matrix
A , we will write entries as either x j or A ij , or ( x ) j or ( A ) ij ; we will occasionally index the rows or columns of A similarly as ( A ) i or ( A ) j , with the particular meaning made clear from context.
We write [ x ] + = max { x, 0 } for the ReLU activation function; if x is a vector, we write [ x ] + to denote the vector given by the application of [ · ] + to each coordinate of x , and we will generally adopt this convention for applying scalar functions to vectors.
If x , x (cid:48) ∈ R n are nonzero, we write ∠ ( x , x (cid:48) )
=
cos -1 ( (cid:104) x , x (cid:48) (cid:105) / (cid:107) x (cid:107) 2 (cid:107) x (cid:48) (cid:107) 2 ) for the angle between x and x (cid:48) .
The vectors ( e i ) denote the canonical basis for R n .
We write (cid:104) x , y (cid:105) =
(cid:80) i x i
y i for the euclidean inner product on R n , and if 0 < p <
+ ∞ we write (cid:107) x (cid:107) p
=
( (cid:80) i | x i | p ) 1 /p for the (cid:96) p norms (when p ≥ 1 ) on R n .
We also write (cid:107) x (cid:107)
= |{ i ∈ [ n ] | x i (cid:54) = 0 }| and (cid:107) x (cid:107) ∞ =
max i ∈ [ n ] | x i | .
The unit ball in R n is written B n = { x ∈ R n | (cid:107) x (cid:107) 2 ≤ 1 } , and we denote its (topological) boundary, the unit sphere, as S n − 1 .
We reserve the notation (cid:107) · (cid:107) for the operator norm of a m × n matrix A , deﬁned as (cid:107) A (cid:107) =
sup (cid:107) x (cid:107) 2 ≤ 1 (cid:107)
Ax (cid:107) 2 ; more generally, we write (cid:107) A (cid:107) (cid:96) p
→ (cid:96) q = sup (cid:107) x
(cid:107) p ≤ 1 (cid:107) Ax (cid:107) q for the corresponding induced matrix norm.
For m × n matrices A and B , we write (cid:104) A , B (cid:105)
=
tr( A ∗ B ) for the standard inner product, where A ∗ denotes the transpose of A , and (cid:107) A (cid:107) F = (cid:112) (cid:104) A , A (cid:105) for the Frobenius norm of A .
The Banach space of (equivalence classes of) real-valued measurable functions on a measure space ( X, µ ) satisfying ( (cid:82) X | f | p d µ ) 1 /p < + ∞ is written L p µ ( X ) or simply L p if the space and/or measure is clear from context; we write (cid:107) · (cid:107) L p for the associated norm, and (cid:104) · , · (cid:105) L 2 for the associated inner product when p = 2 , with the adjoint operation denoted by ∗ .
For an operator T : L pµ → L qν , we write T [ f ] to denote the image of f under T , T i to denote the operator that applies T i times, and (cid:107)T (cid:107) L pµ
→ L qν = sup (cid:107) f (cid:107) Lpµ ≤ 1 (cid:107)T [ f ] (cid:107) L qν .
We use Id to denote the identity operator, i.e. Id[ g ]
=
g for every g ∈ L pµ .
We say that T is positive if (cid:104) f, T [ f ] (cid:105)
L 2 ≥ 0 for all f ∈ L 2 ; for example, the identity operator is positive.
For an event E in a probability space, we write 1 E to denote the indicator random variable that takes the value 1 if ω ∈ E and 0 otherwise.
If σ > 0 , by g ∼ N ( 0 , σ 2 I )
we mean that g ∈ R n is distributed according to the standard i.i.d.
gaussian law with variance σ 2 , i.e., it admits the density (2 πσ 2 ) − n/ 2 exp( −(cid:107) x (cid:107) 22 / (2 σ 2 )) with respect to Lebesgue measure on R n ; we will occasionally write this equivalently as g ∼ i .
i .
d .
N (0 , σ 2 ) .
We use d = to denote the “identically-distributed” equivalence relation.
We use “numerical constant” and “absolute constant” interchangeably for numbers that are independent of all problem parameters.
Throughout the text, unless speciﬁed otherwise we use c, c (cid:48) , c (cid:48)(cid:48) ,
C, C (cid:48) , C (cid:48)(cid:48) , K, K (cid:48) , K (cid:48)(cid:48) , and so on to refer to numerical constants whose value may change from line to line within a proof.
Numerical constants with numbered subscripts C 1 , C 2 , . .
and so on will have values ﬁxed at the scope of the proof of a single result, unless otherwise speciﬁed.
We generally use lower-case letters to refer to numerical constants whose value should be small, and upper case for those that should be large; we will generally use K , K (cid:48) and so on to denote numerical constants involved in lower bounds on the size of parameters required for results to be valid.
If f and g are two functions, the notation f
(cid:46) g means that there exists a numerical constant C >such that f ≤ Cg ; the notation f (cid:38) g means that there exists a numerical constant C > 0 such that f ≥ Cg ; and when both are true simultaneously we write f (cid:16) g .
If f is a real-valued function with sufﬁcient differentiability properties, we will write both f (cid:48) and ˙ f for the derivative of f , and when higher derivatives are available we will occasionally denote them by f ( n ) , with this usage speciﬁcally made clear in context.
For a metric space X and a Lipschitz function f : X → R , we write (cid:107) f (cid:107) Lip to denote the minimal Lipschitz constant of f .
A.5. S UMMARY OF O PERATOR AND E RROR D EFINITIONS
We collect some of the important deﬁnitions that appear throughout the main text and the appendices in this section.
We begin with the NTK-type operators that appear in our analysis.
Recall from
Appendix A.1 our deﬁnition for the backward features: we have
[Equation]
for (cid:96) = 0 , 1 , . . .
, L − 1 , and where we additionally deﬁne
[Equation]
for the orthogonal projection onto the set of coordinates where the (cid:96) -th activation at input x is positive.
“The” neural tangent kernel is deﬁned as
[Equation]
with corresponding operator on L 2 µ ∞ ( M )
[Equation]
As shown in Lemma B.8, this is not exactly the kernel that governs the dynamics of gradient descent: the relevant kernels in this context are deﬁned as
[Equation]
We deﬁne operators Θ Nk on L 2 µ N ( M ) corresponding to integration against these kernel in a manner analogous to the deﬁnition of Θ :
[Equation]
We then move to the deterministic approximations for Θ that we develop: we deﬁne
[Equation]
cos
[Equation]
cos
[Equation]
, which governs the angle evolution process in the initial random network, as studied in Appendix E, and write ϕ ( (cid:96) ) to denote (cid:96) -fold composition of ϕ with itself.
We deﬁne
[Equation]
which is the “output” of our main result on concentration, Theorem B.2, and
[Equation]
which is at the core of the certiﬁcate construction problem.
We think of ψ as an analytically-simpler version of ψ 1 , with an approximation guarantee given in Lemma C.9.
Throughout these appendices, we will make use of basic properties of ψ 1 and ψ that follow from properties of ϕ without explicit reference; the source material for these types of claims is Lemma E.5, which gives elementary properties of ϕ (for example, that it takes values in [0 , π/ 2] , which implies that ψ and ψ 1 are no larger than nL/ 2 ).
For derived estimates, we call the reader’s attention to the contents of Appendix C.2.2; we will make explicit reference to these results when we need them, however.
Although we have mentioned approximations ˆΘ and ˆ Θ in the main text, we will prefer in these appendices to explicitly reference ψ and ψ 1 to avoid confusion; as an exception, we will use the ˆ Θ notation in Appendix C as discussed there.
Our approximation for the initial prediction error is
[Equation]
where we recall f θ 0 denotes the network function with the initial (random) weights.
In particular, this approximates the network function with a constant, and the error as a piecewise constant function on M ± .
This approximation is justiﬁed in Lemma D.11.
B P ROOFS OF THE M AIN R ESULTS B. M AIN R ESULTS
Theorem B.1.
Let M be a one-dimensional Riemannian manifold satisfying our regularity assumptions.
For any 0 < δ ≤ 1 /e , choose L so that
[Equation]
let N ≥ L 10 , set n = C 2 L 99 log 9 (1 /δ ) log 18 ( Ln 0 ) , and ﬁx τ > 0 such that
[Equation]
Then if there exists a function g ∈ L 2 µ ∞ ( M ) such that
[Equation]
with probability at least 1 − δ over the random initialization of the network and the i.i.d.
sample from µ ∞ , the parameters obtained at iteration (cid:98) L 39 / 44 / ( nτ ) (cid:99) of gradient descent on the ﬁnite sample loss
L µ N yield a classiﬁer that separates the two manifolds.
The constants C 1 , . .
, C 4 > 0 depend only on the constants q cert , C 5 , C 6 > 0 , the constants κ , C λ are respectively the extrinsic curvature constant and the global regularity constant deﬁned in Section 2.1, and the constant C µ ∞ is deﬁned as max { ρ q min , ρ − q min } (1 + ρ max ) 6 (min { µ ∞ ( M + ) , µ ∞ ( M − ) } ) − 11 / 2 , where q = 11 + 8 q cert .
Proof.
The proof is an application of Lemma B.7, with suitable instantiations of the parameters of that result; to avoid clashing with the probability parameter δ in this theorem
[Equation]
we use ε for the parameter δ appearing in Lemma B.7.
Deﬁne C ρ = max { ρ min , ρ − 1min } .
We will pick q = 39 / 44 and ε = 5 / 47 , so that the relevant hypotheses of Lemma B.7 become (after worst-casing in the bound on N somewhat for readability)
[Equation]
and the conclusion we will appeal to becomes
[Equation]
Under our choice of τ and enforcing
[Equation]
we have the equivalent result
[Equation]
where the last bound holds when d ≥ K log L , which is redundant with the hypotheses on n and d required to use Lemma B.7.
Thus, when in addition d ≥ (1 /c (cid:48) ) log(1 /δ ) , we obtain
[Equation]
Therefore to conclude, we need only argue that our choices of n , N , L , d , and δ in the theorem statement sufﬁce to satisfy the hypotheses of Lemma B.7.
We have already satisﬁed the conditions on ε and q .
We notice that (B.2) implies that it sufﬁces to enforce simply N ≥ L 10 , and following
Lemma C.4, we can bound C M as in (B.62) in the proof of Lemma B.7 by
[Equation]
Because n ≥ L 99 and L ≥ C ρ (1 + ρ max ) , we can eliminate C M from the lower bound on d while paying only an extra factor of 2 in the constant.
In addition, because κ ≥ 1 and C λ ≥ max { 1 , 1 /c λ } , we can remove the κ 2 / 5 and (cid:16) κc λ (cid:17) 1 / 3 lower bounds on n , since they are enforced through L already via the bound L ≥ K (cid:48)(cid:48) κ 2 C λ , worsening the absolute constant if needed.
These simpliﬁcations lead us to the sufﬁcient conditions (plus the certiﬁcate existence hypotheses)
[Equation]
We ignore the condition on N below, since it matches with the theorem statement.
When δ ≤ 1 /e , given that n 0 ≥ 3 we have nn 0 ≥ e and max { log(1 /δ ) , log( nn 0 ) } ≤ log(1 /δ )
log( nn 0 ) .
For the sake of simplicity, we can also round up the fractional constants in the lower bound on L .
We can eliminate d from these sufﬁcient conditions by substituting the lower bound into the conditions on n and L , and this also implies that our conditions on certiﬁcate existence in the theorem statement sufﬁce for the certiﬁcate existence hypothesis for Lemma B.7.
Thus, we have the remaining sufﬁcient conditions
[Equation]
Using Lemma B.15 and choosing L larger than a sufﬁciently large absolute constant and larger than log(1 /δ ) , we obtain that it sufﬁces to enforce for n
[Equation]
In the hypotheses of the theorem, we have chosen the equality n = KL 99 log 9 (1 /δ ) log 18 ( Ln 0 ) in the last bound.
This implies log( nn 0 ) ≤ C log( Ln 0 ) , so it sufﬁces to enforce the L lower bound
[Equation]
Deﬁning, as in the theorem
[Equation]
and using C µ ∞ ≥ 1 , we can worsen the absolute constant K (cid:48) in order to apply Lemma B.15 once again, obtaining the simpliﬁed condition
[Equation]
These conditions reﬂect what is stated in the lemma.
Theorem B.2.
Let M be a d 0 -dimensional Riemannian submanifold of S n 0 − 1 .
For any d ≥
Kd 0 log( nn 0 C M ) , if n ≥ K (cid:48)
d 4 L then one has on an event of probability at least 1 − e − cd
[Equation]
where we write ν = ∠ ( x , x (cid:48) ) in context with an abuse of notation, c, K, K (cid:48) > 0 are absolute constants, and C M > 0 depends only on the length of the manifold if d 0 = 1 , and otherwise only on the number of connected components of M .
Proof.
We have by the deﬁnition of Θ
[Equation]
Under the stated hypotheses
[Equation]
Lemmas D.10 and D.13 give uniform control of each of the terms appearing in this expression with suitable probability to tolerate 2 L + 1 union bounds, which gives simultaneous uniform control of the factors on an event E with probability at least 1 − e − cd .
Starting from (B.4), we can write with the triangle inequality
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
By the triangle inequality
[Table]
we have
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
Under the conditions on n , L , and d , we have on the event E that for each (cid:96)
[Table]
≤ 2 , so we can conclude that on E
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
The conditions on n , d , and L imply that this residual is larger than that incurred by the level- L features, which is no larger than 2 .
Returning to (B.5), we have shown that on E
[Equation]
After adjusting the other absolute constants to absorb C into d , this gives the claim.
Theorem B.3 (Pointwise Version of Theorem B.2) .
Let M be a d 0 -dimensional Riemannian submanifold of S n 0 − 1 .
For any d ≥ K log n , if n ≥ K (cid:48) max { 1 , d 4 L } then one has for any ( x , x (cid:48) )
∈ M × M
[Equation]
where we write ν = ∠ ( x , x (cid:48) ) in context with an abuse of notation, and c, K, K (cid:48) > 0 are absolute constants.
Proof.
Follow the proof of Theorem B.2, but invoke the pointwise versions of the uniform concentration results used there (i.e., Lemmas D.1 and D.4) after rescaling d to relocate the log n terms.
Proposition B.4.
Let M be an r -instance of the two circles geometry studied in Appendix C.1.1, with r ≥ 1 / 2 .
For any 0 < δ ≤ 1 /e , if n ≥ KL 5 log 4 (1 /δ ) log 4 ( Ln 0 log(1 /δ )) and L ≥ K (cid:48) (1 − r 2 ) − 1 / 2 , then there exist absolute constants C 5 , C 6 > 0 and a function g such that (B.1) is satisﬁed with the choice
q cert = 1 / 2 with probability at least 1 − 3 δ .
The constants K, K (cid:48) > 0 are absolute.
Proof.
Given r ≥ 12 and L ≥ max { K, ( π/ 2)(1 − r 2 ) − 1 / 2 } , we have by Lemma C.1 that there exists g such that (cid:82) M ψ ◦ ∠ ( · , x (cid:48) )
g ( x (cid:48) )
d µ ∞ ( x
[Equation]
=
ˆ ζ , with
[Equation]
By this bound, the triangle inequality, the Minkowski inequality, and the fact that µ ∞ is a probability measure, we have
[Equation]
An application of Theorem B.2 and Lemma C.9 gives that on an event of probability at least 1 − e − cd (cid:107) Θ − ψ ◦ ∠
[Equation]
≤ Cn/L if d ≥ Kd 0 log( nn 0 C M ) and n ≥ K (cid:48)
d 4 L 5 .
An application of Lemma D.11 gives
[Equation]
and
[Equation]
as long as n ≥ Kd 4 L 5 and d ≥ K (cid:48)
d 0 log( nn 0 C M ) , where we use these conditions to simplify the residual that appears in Lemma D.11.
In particular, combining the previous two bounds with the triangle inequality and a union bound and then rescaling d , which worsens the constant c and the absolute constants in the preceding conditions, gives
[Equation]
Combining these bounds using a union bound and substituting into (B.7), we get that under the preceding conditions, on an event of probability at least 1 − 3 e − cd we have
[Equation]
where we worst-case the density constant in the second line, and in addition, on the same event, we have by (B.6)
[Equation]
To conclude, we simplify the preceding conditions on n and turn the parameter d into a parameter δ > 0 in order to obtain the claimed form of the result.
We have in this setting d 0 = 1 , and also that
C M is bounded by an absolute constant; since n 0 ≥ 3 , we can thus eliminate the parameter C M from our hypotheses by adding an extra absolute constant factor.
Choosing d ≥ (1 /c ) log(1 /δ ) , we obtain that the previous two bounds hold on an event of probability at least 1 − 3 δ .
When δ ≤ 1 /e , given that n 0 ≥ 3 we have nn 0 ≥ e and max { log(1 /δ ) , log( nn 0 ) } ≤ log(1 /δ )
log( nn 0 ) , so that it sufﬁces to enforce the requirement d ≥ K log(1 /δ )
log( nn 0 ) for a certain absolute constant K > 0 .
We can then substitute this lower bound on d into the two certiﬁcate bounds above to obtain the form claimed in (B.1) with q cert = 1 / 2 .
For the hypothesis on n , we substitute this lower bound on d into the condition on n to obtain the sufﬁcient condition n ≥ K (cid:48)
L 5 log 4 (1 /δ ) log 4 ( nn 0 ) .
Using
Lemma B.15 and possibly worsening absolute constants, we then get that it sufﬁces to enforce n ≥ K (cid:48)
L 5 log 4 (1 /δ ) log 4 ( Ln 0 log(1 /δ )) , which is the hypothesis in the result.
Theorem B.5.
There exist absolute constants c, C, K, K (cid:48) > 0 such that for any d ≥ Kn 0 log n , if n ≥ K (cid:48)
d 4 L , then on an event of probability at least 1 − e − cd the natural extension of f θ 0 to R n 0 is 3 √ d -Lipschitz.
Proof.
The proof is a simple application of Lemma B.17, which (because f θ 0 is 1 -nonnegatively homogeneous and so are all its intermediate feature maps α (cid:96) θ 0 ( x ) ) implies that it sufﬁces to control the Lipschitz constants of the maps and bound them on the unit sphere, together with Lemmas D. and D.12.
In particular, for any d ≥ Kn 0 log( n ) and any n ≥ K (cid:48)
d 4 L , we have that there exists an event of probability at least 1 − e − cd on which
[Equation]
and
[Equation]
Applying Lemma B.17, it follows that f θ 0 : R n 0 → R is 3 √ d -Lipschitz on an event of probability at least 1 − e − cd .
B. S UPPORTING R ESULTS ON D YNAMICS
Lemma B.6 (Nominal) .
Suppose C err , C cert , q cert > 0 are absolute constants.
Then there exist absolute constants c, c (cid:48) , C (cid:48) , C (cid:48)(cid:48) , C (cid:48)(cid:48)(cid:48) > 0 and absolute constants K, K (cid:48) , K (cid:48)(cid:48) > 0 such that for any d ≥ Kd 0 log( nn 0 C M ) and any 1 / 2 ≤ q ≤ 1 , if n ≥ K (cid:48)
d 4 L 5 , if L ≥ K (cid:48)(cid:48) dC 2 q cert ρ , and if additionally there exists g ∈ L 2 µ ∞ ( M ) satisfying
[Equation]
and τ > 0 is chosen such that
[Equation]
then one has
[Equation]
and in addition
[Equation]
Moreover, one has
P
[Equation]
The constant C ρ = max { ρ min , ρ − 1min } .
Proof.
We will combine Lemma B.12 with various probabilistic results to obtain a simple ﬁnal form for the bound from this result.
Invoking Lemma B.12, we can assert that for any step size τ > 0 satisfying τ < (cid:107)
[Equation]
and for any k satisfying
[Equation]
the population dynamics satisfy
[Equation]
We state the bounds we will apply to simplify this expression.
An application of Lemma D.11 gives
[Equation]
and
[Equation]
as long as n ≥ Kd 4 L 5 and d ≥ K (cid:48)
d 0 log( nn 0 C M ) , where we use these conditions to simplify the residual that appears in the version of (B.11) quoted in Lemma D.11.
In particular, combining (B.11) and (B.12) with the triangle inequality and a union bound and then rescaling d , which worsens the constant c and the absolute constants in the preceding conditions, gives
[Equation]
In addition
[Equation]
we can write using the triangle inequality
[Equation]
and
[Equation]
so that, by (B
[Equation] [Equation]
we have if L ≥ 2 √ d
[Equation]
Because µ ∞ is a probability measure, Jensen’s inequality, the Schwarz inequality
[Equation]
and the triangle inequality give
[Equation]
and an application of Theorem B.2 and Lemma E.5 then gives that on an event of probability at least 1 − e − cd
[Equation]
15) provided d ≥ Kd 0 log( nn 0 C M ) and n ≥ K (cid:48)
d 4 L .
We will write E for the event consisting of the union of the events invoked for the bounds (B.11) to (B.15), which has probability at least 1 − e − cd by a union bound and a choice of d ≥ K .
We will conclude by simplifying (B.10) on E .
First, we note that by (B.15), the step size condition (B.8) is satisﬁed on E provided
[Equation]
which holds under our hypotheses.
Next, on E , we write using decreasingness of x (cid:55)→ − log x and (B.12)
[Equation]
By the hypothesis on g , we have on E
[Equation]
and so it follows that on E
[Equation]
The function x (cid:55)→ − x log x is a strictly increasing function on [0 , e − 1 ] , so when k is chosen such that
[Equation]
we have on E by (B.17)
[Equation]
Additionally, in the context of the condition (B.9), notice that by (B.14) and (B.18), on E we have
[Equation]
so that given d ≥ 1 , we have that the choice
[Equation]
implies both conditions (B.9) and (B.19).
We can simplify (B.20) using the hypothesis kτ ≤ L q /n with 1 / 2 ≤ q ≤ 1 : we get
[Equation]
where the last inequality requires L ≥ ρ q cert min /C , which implies
[Equation]
The conditions we need to satisfy on kτ can be stated together as
[Equation]
and it is possible to satisfy these conditions simultaneously as long as
[Equation]
We obtain an upper bound C 2 e 2 d ρ 2 q certmin for the quantity on the RHS of this inequality from q ≥ 1 / 2 ; it sufﬁces to choose L larger than this upper bound instead.
The other simpliﬁcations are easier: using the assumption on the norm of Θ [ g ] − ζ , we have
[Equation]
Worst-casing terms using our hypotheses on d and L to obtain a simpliﬁed bound, on E
, we have thus shown that when (B.21) is satisﬁed, we have
[Equation]
We have
[Equation]
which is implied by the hypothesis kτ ≤ L q /n as long as L ≥ e .
So we can simplify to
[Equation]
We also need a bound that works for k that do not satisfy (B.21).
From the update equation for the dynamics in the proof of Lemma B.12 and the choice of τ , we also have
[Equation]
where the last bound is valid on E .
Finally
[Equation]
we can obtain the claimed sum bound by calculating using our ‘small- k ’ and ‘large- k ’ bounds:
[Equation]
where the second inequality uses standard estimates for the harmonic numbers and
C (cid:48) √ d/ ( nτρ q cert min ) ≥ 1 , which follows from τ ≤ c (cid:48) / ( nL ) , d ≥ 1 and L ≥ Kρ q cert min for a suitable absolute constant K ; and the third inequality integrates and simpliﬁes, using kτ ≤ L/n and again d ≥ 1 and L ≥ Cρ q cert min .
Worst-casing constants and using nτ ≤ 1 , we simplify this last bound to
[Equation]
To see that the conditions on L in the statement of the result sufﬁce, note that we have to satisfy (say)
L ≥ Kρ q cert min and L ≥ K (cid:48) ρ − q cert min ; the ﬁrst of these lower bounds is tighter when ρ min ≥ 1 , and the second when ρ min < 1 , and so it sufﬁces to require L ≥ Kρ 2 q cert min and L ≥ K (cid:48) ρ − 2 q cert min instead.
Lemma B.7 (Nominal to Finite) .
Let d 0 = 1 , and suppose C err , C cert , q cert > 0 are absolute constants.
Then there exist absolute constants c, c (cid:48) , C (cid:48) , C (cid:48)(cid:48) , C (cid:48)(cid:48)(cid:48) > 0 and absolute constants
K, K (cid:48) , K (cid:48)(cid:48) , K (cid:48)(cid:48)(cid:48) > 0 such that for any d ≥ Kd 0 log( nn 0 C M ) , any 1 / 2 ≤ q < 1 and any 0 < δ ≤ 1 , if L ≥ K (cid:48) max { C 2 q cert ρ
[Equation]
and if
[Equation]
and if additionally there exists g ∈ L 2 µ ∞ ( M ) satisfying
[Equation]
d n and τ > 0 is chosen such that
[Equation]
then one has generalization in L 2 µ ∞ ( M ) :
[Equation]
and in addition, one has generalization in L ∞ ( M ) :
[Equation]
The constant C ρ = max { ρ min , ρ − 1min } .
Proof.
The proof controls the L ∞ norm of the error evaluated along the ﬁnite sample dynamics using an interpolation inequality for Lipschitz functions on an interval (Lemma B.14), which relates the L ∞ norm to a certain combination of the predictor’s Lipschitz constant and its L 2 µ ∞ norm.
We can control these two quantities at time zero using our measure concentration results; to control them for larger times 0 < k ≤ L q / ( nτ ) , we set up a system of coupled ‘discrete integral equations’ for the generalization error of the ﬁnite sample predictor and the Lipschitz constant of the ﬁnite sample predictor, and use the fact that kτ is not large to argue by induction that not much blow-up can occur.
Along the way, we control the generalization error of the ﬁnite sample predictor by linking it to the generalization error of the nominal predictor as controlled in Lemma B.6; the residual that arises is shown to be small by applying Corollary B.11 and applying basic results from optimal transport theory adapted to our setting, encapsulated in Lemmas B.13 and B.16.
To begin, we will lay out the probabilistic bounds we will rely on for simpliﬁcations, so that the rest of the proof can proceed without interruption.
We will want to satisfy
[Equation]
following the notation of Lemma B.10.
Using Jensen’s inequality, the Schwarz inequality, and the triangle inequality, we have for (cid:63) ∈ { N, ∞}
[Equation] [Equation]
where the notation ψ 1 follows the deﬁnition in Appendix C.2.2.
The ﬁrst term in (B.24) can be controlled using Theorem B.2: we obtain that on an event of probability at least 1 − e − cd (cid:107)
[Equation]
25) if d ≥ Kd 0 log( nn 0 C M ) and n ≥ K (cid:48)
d 4 L .
The second term in (B.24) can be controlled using the triangle inequality, Lemma E.5, and the deﬁnition of ψ 1 : we obtain that it is no larger than nL/ 2 .
Combining these two bounds, we have on an event of probability at least 1 − e − cd
[Equation]
26) provided d ≥ Kd 0 log( nn 0 C M ) and n ≥ K (cid:48)
d 4 L .
Thus, with probability at least 1 − e − cd , our choice of step size τ ≤ c/ ( nL ) satisﬁes (B.23).
Under our hypotheses on the function g in the statement of the result and taking a union bound with the event in (B.26), we can invoke Lemma B. to obtain
P   (cid:92)
[Equation]
and
[Equation]
) provided d ≥ Kd 0 log( nn 0 C M ) , 1 / 2 ≤ q < 1 , n ≥ K (cid:48)
d 4 L 5 , and L ≥ K (cid:48)(cid:48)
C 2 q cert ρ d .
We have by
Lemmas B.6 and B.10, a union bound with (B.26), and our condition on τ that
P
[Equation]
as long as d ≥ Kd 0 log( nn 0 C M ) and n ≥ K (cid:48)
L 48+20 q d 9 log 9 L , and where we used our conditions on τ and q to obtain that L q /nτ ≥ 1 and simplify the probability bound; and, following the notation of Corollary B.11, we have by this result (again under our condition on τ and a union bound) that there is an event of probability at least 1 − CLe − cd / ( nτ ) on which
[Equation]
under the previous conditions on n and d .
In addition, applying Lemma D.12 and a union bound gives that on an event of probability at least 1 − Ce − cd
[Equation]
provided d ≥ Kd 0 log( nn 0 C M ) and n ≥ K (cid:48) max { d 4 L, ( κ/c λ ) 1 / 3 , κ 2 / 5 } .
Finally, we have by
Lemma B.13 that for any 0 < δ ≤
P  
[Equation]
as long as d ≥ 1 and N ≥ 2 √ d/ min { µ ∞ ( M + ) , µ ∞ ( M − ) } .
We let E ( q, δ ) denote the event consisting of the union of the events appearing in the bounds (B.25) to (B.32) hold; by a union bound and the previous observation that L q /nτ ≥ 1 , we have
[Equation]
In the sequel
[Equation]
we will use the events deﬁning E to simplify our residuals without explicitly referencing that our bounds hold only on E to save time.
We start from the dynamics update equations given by Lemma B.
[Equation]
which we use to write
[Equation]
where Θ is deﬁned as in Lemma B.12.
Under the choice of τ and positivity of Θ (Lemma B.9)
[Equation]
we apply the triangle inequality and a telescoping series with the common initial conditions to obtain
[Equation]
We can write
[Equation]
and analogously
[Equation]
Using Jensen’s inequality and the Schwarz inequality, we have
[Equation]
since µ N is a probability measure.
Repeating an analogous calculation with µ ∞ for the other term and applying the triangle inequality
[Equation]
we have
[Equation]
We detour brieﬂy to simplify residuals appearing in (B.34) before using the result to update (B.33).
Using (B.25) and (B.30)
[Equation]
we get
[Equation]
where the ﬁnal bound holds when n ≥ d 3 .
Using (B.29)
[Equation]
we can further simplify the RHS of the last bound above to
[Equation]
With this last bound and (B.34)
[Equation]
we can use kτ ≤ L q /n to simplify (B.33)
to
[Equation]
To control the remaining term in (B.36)
[Equation]
we split the error ζ Ns into a Lipschitz component whose evolution is governed by the nominal kernel ψ 1 ◦ ∠ and a nonsmooth component which is small in
L ∞ .
Formally, we deﬁne Θ nom : L 2 µ N ( M )
→
L
[Equation]
N ( M ) by
[Equation]
and use the update equation from Lemma B.8 to write
[Equation]
so that ζ Ns = ζ N, Lip s + δ Ns , and ζ N, Lip= ζ , δ N 0 = 0 .
It is straightforward to control δ Ns in L ∞ : we have (as usual) by the triangle inequality, Jensen’s inequality, and the Schwarz inequality
[Equation]
, and then the triangle inequality together with (B.25), (B.29) and (B.30) yield
[Equation]
[Equation]
where the second line applies the same simpliﬁcations that led us to (B.35).
The triangle inequality gives
[Equation]
and simplifying as usual using Jensen’s inequality and the Hölder inequality, we obtain
[Equation]
where the last bound uses (B.37).
Then using the triangle inequality and kτ ≤ L q /n to simplify in (B.36), we obtain
[Equation]
To simplify the remaining term in (B.38), we aim to apply (B.32); to do this we will need to justify the notation and establish that ζ N, Lip s ∈ Lip( M ) regardless of the random sample from µ ∞ and the random instance of the weights.
Because ζ N, Lip s is a sum of functions, we can bound its minimal
Lipschitz constant by the sum of bounds on the Lipschitz constants of each summand.
We always have for either (cid:63) ∈ { + , −}
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
We note that because the ReLU [ · ] + is 1 -Lipschitz as a map on R n , we have
[Equation]
so we need only develop a Lipschitz property for the summands in the second term of (B.39).
To do this, we will start by showing that t
[Equation]
[Table]
[Equation]
[Table]
is absolutely continuous for each x (cid:48) .
Continuity is immediate.
The only obstruction to differentiability comes from the inverse cosine, which fails to be differentiable at ± 1 , and because M ⊂ S n 0 − 1 we have (cid:104) γ (cid:63) ( t ) , x (cid:48) (cid:105) = ± 1 only if γ (cid:63) ( t ) = ±
x (cid:48) ; because γ (cid:63) are simple curves, this shows that there are at most two points of nondifferentiability in [0 , len( M (cid:63) )] .
At points of differentiability, we calculate using the chain rule the derivative
[Equation]
and because γ (cid:63) is a sphere curve, it holds ( I − γ
[Equation]
t
[Equation]
γ ∗
[Equation]
t
[Equation]
t ) for all t , whence by
[Equation]
where we also used that γ (cid:63) are unit-speed curves.
In particular, the derivative is bounded, hence integrable on [0 , len( M (cid:63) )]
[Equation]
and so an application of (Cohn, 2013, Theorem 6.3.11) establishes that t
[Equation]
◦ cos
[Equation]
γ
[Equation]
( t ) , x (cid:48) (cid:105) is absolutely continuous, with the expansion
[Equation]
which gives an avenue to establish Lipschitz estimates for t (cid:55)→ ψ 1 ◦ cos -1 (cid:104) γ (cid:63) ( t ) , x (cid:48) (cid:105) .
Because x (cid:48) (cid:55)→ ζ Ni ( x (cid:48) ) is continuous and i ≤ s ≤ k ≤ L q / ( nτ ) < + ∞ , an application of Fubini’s theorem enables us to also use this result to obtain Lipschitz estimates for the summands examined in (B.39), to wit
[Equation]
41) after using the bound (B.40) in the ﬁrst inequality and the Schwarz inequality for the second.
Before proceeding with further simpliﬁcations, we note that the C 2 property of ψ 1 , continuity of ζ Ni , boundedness of i , and compactness of M let us assert using (B.41) and (B.39) that ζ N, Lip s ∈ Lip( M ) whether or not we are working on the event E .
Continuing, we develop a bound for the RHS of (B.41) that is valid on E .
Using the triangle inequality and the Minkowski inequality, we have for
[Table]
[Equation]
For the ﬁrst term in (B.42)
[Equation]
we use Lemmas C.5, C.20 and C.22 to obtain that x (cid:48)
(cid:55)→ ( ψ (cid:48) 1 ◦ ∠ ( x , x (cid:48) ))is bounded by Cn 2 L 4 and C (cid:48) n 2 L 5 -Lipschitz for every x
[Equation]
and then applying (B.32) gives
[Equation]
For the second term in (B.42)
[Equation]
we apply Lemmas C.6 and C.20 together with the choice L ≥ Kκ 2 C λ to get sup
[Equation]
Combining (B.43) and (B.44) to control the RHS of (B.42), we obtain from (B.41)
[Equation]
Lip
[Equation]
45) where in the second line we used N ≥ L 4+2 δ .
Plugging (B.45) into (B.39) and applying in addition (B.31), we get
[Equation]
(B.46)
Let us brieﬂy pause to reorient ourselves.
We do not have control of the empirical losses appearing in (B.46) by an outside result, so we need to make some further simpliﬁcations to this bound.
We will control the sum of empirical losses term in (B.46) by linking it to the difference population error, which we last saw in (B.38), and the population error using the triangle inequality and a change of measure inequality.
Meanwhile, with the Lipschitz property of ζ N, Lip s we have shown, we will be able to obtain a bound in terms of simpler quantities for the last term on the RHS of (B.38) using (B.32).
The two resulting bounds will give us a system of two coupled ‘discrete integral equations’ for the difference population error and the Lipschitz constants of ζ N, Lip s , which we will solve inductively.
First, we continue simplifying (B.46).
The triangle inequality and the fact that µ N is a probability measure give
[Equation]
[Table] [Table]
We have shown that ζ
[Table]
as well, with
[Table]
M (cid:63) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
Lip
[Equation]
2 (cid:13)(cid:13)(cid:13)
[Equation]
N
[Equation]
Lip i (cid:13)(cid:13)(cid:13)
[Equation]
( M ) (cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
N, Lip i (cid:12)(cid:12)(cid:12) M (cid:63) (cid:13)(cid:13)(cid:13)(cid:13) Lip .
Applying the previous equation with (B.32) to control (B.48), we get (cid:13)(cid:13)(cid:13) ζ N, Lip i (cid:13)(cid:13)(cid:13)
[Equation]
≤ (cid:13)(cid:13)(cid:13)
[Equation]
+ Cd
[Equation] [Equation]
Lip
[Equation]
(cid:12)(cid:12)(cid:12)
[Equation]
Lip
N
[Equation] [Equation]
) ≤ (cid:13)(cid:13)(cid:13) ζ N
[Equation]
Lip i
[Equation]
L
[Equation]
M )
+ Cd 1 /  (cid:107)
[Equation]
Lip i (cid:107)
[Equation]
N +
[Equation]
 
[Equation]
where the second line applies the Minkowski inequality
[Equation]
Using the triangle inequality and that µ ∞ is a probability measure
[Equation]
we have
[Equation]
Substituting (B.49) into (B.47) and using (B.37) to simplify gives
[Equation]

[Equation]
Following (B.46), we need to sum the previous bound over i .
To simplify residuals, we use (B.28) to get
[Equation]
where the second bound uses the control sτ ≤ kτ ≤ L q /n and holds under the condition n ≥ ( C/C (cid:48) )
12 L 48+32 q d 3 .
Summing in (B.50) and using the previous bound, it follows
[Equation]
Plugging (B.51) into (B.46), we obtain (cid:13)(cid:13)(cid:13)
[Equation]
N
[Equation]
M
[Equation]
) where for concision we have deﬁned
[Equation]
and simpliﬁed the √ d residual in (B.46) by worst-casing with the larger residual from the population error term in (B.51), and made other simpliﬁcations by worst-casing some constants.
We simplify (B.38) next: we have shown that ζ N, Lip s ∈ Lip( M ) and ζ N, Lip s ∈ L ∞ ( M ) above
[Equation]
and so for every x ∈ M , we have
[Equation]
as well
[Equation]
with
[Equation]
(cid:13)(cid:13)
L ∞ ( M ) (B.54) using the deﬁnition of ψ 1 , Lemmas E.5, C.5 and C.20, and
[Equation] [Equation]
, Lip
[Equation]
(B.55)
The bounds (B.54) and (B.55) retain no x dependence.
Applying (B.32) and integrating over x , we obtain from (B.54) and (B.55)
[Equation]
and we can combine the ﬁrst and third terms on the RHS of the previous bound by worst-casing, giving
[Equation]
Plugging the previous bound into (B.38)
[Equation]
we obtain
[Equation]
B.
[Equation]
To ﬁnish coupling (B.52) and (B.56), we need to remove the L ∞ ( M ) terms.
We accomplish this using Lemma B.
[Equation]
which gives
[Equation]
.57) where we have deﬁned
[Equation] [Equation]
For coupling purposes
[Equation]
it will sufﬁce to use a version of (B.57) obtained by simplifying with some coarse estimates.
Using (B.49)
[Equation]
(B.37) and (B.29)
[Equation]
we have
[Equation]
using iτ ≤ L q /n and n ≥ L 48+20 q d 9 log 9 L in the second line
[Equation]
and plugging this into (B.57) and using the Minkowski inequality gives
[Equation]
To make some of the subsequent bounds more concise
[Equation]
we introduce additional notation
[Equation]
Plugging (B.59) into (B.52) and using the Minkowski inequality, we obtain
[Equation]
To simplify (B.60), we use sτ ≤ L q /n , C 2 ≥ 1 , and q ≤ 1 , and so if additionally we choose
N ≥ C 2 max {√ d, L 2 } we obtain
[Equation]
Meanwhile, we recall
[Equation]
and an integration in coordinates gives
[Equation]
so that
[Equation]
Using (B.62) and plugging (B.59) into (B.56), we obtain
[Equation]
In (B.61) and (B.63), we now have a suitable system of coupled discrete integral equations for (cid:107) ζ ∞ k − ζ Nk (cid:107) L 2 µ ∞ ( M ) and Λ k .
We will solve these equations by positing bounds for each parameter that are valid for all indices 0 ≤ k ≤ (cid:98) L q / ( nτ ) (cid:99) based on inspection of (B.61) and (B.63), then proving the bounds hold by induction on k .
Positing the bounds is not too hard, because each term in (B.61) and (B.63) with a factor of N in its denominator can be forced to be small by requiring N to be large enough.
For all 0 ≤ k ≤ (cid:98) L q / ( nτ ) (cid:99) , we claim
[Equation]
where C diff and C lip are two absolute constants that we will specify in our arguments below.
We prove (B.64) and (B.65) by induction on k .
The case of k = 0 is immediate, since ζ ∞
= ζ N 0 for (B.64); and by construction ζ N, Lip= ζ , and (B.31) and d ≥ 1 then gives (B.65) if L ≥ e .
We therefore move to the induction step, assuming that (B.64) and (B.65) hold for k − 1 and showing that this implies the bounds for k .
We begin by verifying (B.64).
Applying the induction hypothesis for k − 1 via (B.65), we can write
[Equation]
where we worst-cased in the second line using C lip ≥ 1 and C 1 ≥ 1 , C 2 ≥ 1 , which follow from (B.53) and (B.58).
We use kτ ≤ L q /n with the last bound to note that
C
[Equation]
τ √ dnLe 14 /δ
[Equation]
where C (cid:48)(cid:48) ≥ 1 .
Using this bound and (B.65) once more, we can simplify (B.63) to
[Equation]
(B.66)
Noticing that the RHS of the bound (B.64) does not depend on k , let us momentarily denote it by
C diff M (i
[Equation]
e., the part of the RHS of this bound that does not involve C diff is denoted as M ).
Plugging into (B.66) and using kτ ≤ L q /n , we obtain
[Equation]
In particular
[Equation]
if C diff = 6 max { C, C (cid:48)(cid:48) } (for the constants in the ﬁrst line of the previous bound)
[Equation]
we can bound the RHS of the previous bound and obtain
[Equation]
We can conclude (B.64) from (B.67) provided we can show the second and third terms are no larger than C diff M/ 3 .
For the second term in (B.67), if we choose N such that
[Equation]
and n such that
[Equation]
then we have
[Equation]
For the third term in (B.67)
[Equation]
we proceed in cases: ﬁrst, when
[Equation]
68) we have by (B.64)
M =
[Equation]
, and if we require additionally C diff ≥
[Equation]
it follows that
[Equation]
using C 1 ≥ 1 , C 2 ≥
[Equation]
and C lip ≥ 1 and worst-casing exponents on d and log L in the ﬁrst line
[Equation]
and (B.68) in the second line.
In particular, by the value of M in this regime, if
[Equation]
then we obtain for the third term in (B.67)
[Equation]
as desired.
Next, we consider the remaining case
[Equation]
which by (B.64) implies
[Equation]
With this setting of M
[Equation]
the third term in (B.67) can be bounded as
[Equation]
and using the RHS of the ﬁnal bound in the previous expression
[Equation]
we see that if we choose
[Equation]
then we have for the case (B.69)
[Equation]
Combining the bounds on the third term in (B.67) over both cases (B.68) and (B.69), we have shown
[Equation]
which proves (B.64).
Next, to verify (B.65), we proceed with a similar idea: the bound claimed in (B.65) corresponds to a constant multiple of the ﬁrst term in parentheses in (B.61), so to establish (B.65) it sufﬁces to show that each of the other terms in (B.61) is no larger than a certain constant.
To work with the maximum operation in (B.64), we will again split the analysis into two cases.
First, we consider the case where (B.69) holds, so that the maximum in (B.64) is achieved by the second argument.
Plugging (B.64) and (B.65) into (B.61) and using kτ ≤ L q /n , we get Λ
[Equation]
Using C lip ≥ 1 , C 1 ≥ 1 , C ρ ≥ 1 , and C 2 ≥ 1 , we can worst-case constants in the previous expression to simplify.
We can then do some selective worst-casing of the exponents on d , N , and
L in all except the ﬁrst term: we have evidently (to combine the ﬁrst and last terms)
[Equation]
and (to combine the ﬁrst and second terms)
[Equation]
and because 0 < δ ≤ 1 , we have 1 / (2 + δ ) ≤ 1 / 2 and (5 + δ ) / (4 + 2 δ ) ≥ 1 , and if N ≥ d 1 / 12 this implies (to combine the ﬁrst and second-to-last terms)
[Equation]
We can worst-case the remaining three terms, and we thus obtain
[Equation]
We can then pick C lip = 3 C , and if
[Equation]
and
[Equation]
then it follows from the previous bound
[Equation]
which establishes (B.65) in the ﬁrst case, where (B.69) holds.
Next, we consider the remaining case where (B.68) holds, so that the maximum in (B.64) is saturated by the ﬁrst argument.
We start by grouping some terms in (B.61) so that it will be slightly easier to simplify later: we can write
[Equation]
By the case-deﬁning condition (B.68) and (B.64), enforcing
[Equation]
implies
[Equation]
and we can use this to simplify (B.70), obtaining Λ
[Equation]
Plugging (B.64) and (B.65) into (B.71) and using kτ ≤ L q /n and (B.68), we get the bound Λ k ≤
[Equation]
From (B.72), we see that if we choose n such that
[Equation]
and we choose N such that
[Equation]
then (B.72) implies the bound
[Equation]
which agrees with the previous choice C lip = 3 C and thus proves (B.65) in the remaining case of (B.71).
By induction, then, we have proved that (B.64) and (B.65) hold for each index 0 ≤ k ≤ (cid:98)
L q / ( nτ ) (cid:99) .
We can now wrap up the proof: we will obtain the desired conclusion by plugging the results we have developed into (B.57) and simplifying.
Plugging (B.37), (B.27) and (B.64) into (B.49) and bounding the maximum by the sum, we get
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Equation]
where in the third inequality we apply (cid:98) L q / ( nτ ) (cid:99) ≥ L q / (2 nτ ) , which follows from our choice of step size, and in the fourth inequality we simplify residuals using n ≥ ( C (cid:48) /C )
12 d 9 L 60+44 q and N 1 / (2+ δ ) ≥ C (cid:48)(cid:48)
C 1 C 1 / 2 2 C 4 / 3 ρ d 5 / 4 L 5 / 2+2 q log L .
Applying (B.73), the triangle inequality (with (B.37) and the fact that µ ∞ is a probability measure) and our previous choice of large n , we get
[Equation]
i.e. generalization in L 2 µ ∞ ( M ) .
We can bootstrap generalization in L ∞ ( M ) from (B.73) using the triangle inequality and (B.57): we get
[Equation]
where in the second line we apply (B.37) and our previous choice of large n to absorb the residual from δ N (cid:98)
L q / ( nτ ) (cid:99) , and apply (B.65) to bound the Λ 1 / 3 (cid:98) L q / ( nτ ) (cid:99) term.
Worst-casing the errors in the previous bound, we obtain
[Equation]
To conclude, we will tally dependencies and make some simpliﬁcations to show the conditions stated in the result sufﬁce.
Recalling (B.53) and (B.58) and using (B.62), we have
[Equation]
so we can simplify to
[Equation]
We can use this to obtain a simpliﬁed generalization in L ∞ ( M ) bound from our previous expression: it becomes
[Equation] [Equation]
which can be made nonvacuous when q > 3 / 4 .
Tallying dependencies, we ﬁnd after worst-casing (and using q ≥ 1 / 2 and some interdependences between parameters to simplify) that it sufﬁces to choose N such that
[Equation]
the depth L such that
[Equation]
the width n such that
[Equation]
and d such that d ≥ Cd 0 log
[Equation]
nn 0 C M ) .
Unpacking the constants in the condition on N , we see that it sufﬁces to choose N such that
[Equation]
B. A UXILIARY R ESULTS
Lemma B.8.
Deﬁning a kernel
[Equation]
and corresponding operator on L 2 µ N ( M
[Equation]
we have that Θ Nk is bounded, and
[Equation]
Proof.
By the deﬁnition of the gradient iteration, we have that
[Equation]
The total number of trainable parameters in the network is M = n ( n ( L − 1) + n 0 + 1) , and the euclidean space in which θ lies is isomorphic to R M .
For k ∈ N 0 , deﬁne paths γ Nk : [0 , 1] → R M by
[Equation]
so that
[Equation]
We will justify a ﬁrst-order Taylor representation in integral form based on the previous expression by arguing that for every x ∈ M , t (cid:55)→ f γ Nk ( t ) ( x ) is absolutely continuous on [0 , 1] , by checking the hypotheses of (Cohn, 2013, Theorem 6.3.11).
Because γ Nk is smooth and f ( · ) ( x ) is continuous, f γ Nk ( t ) is also continuous.
Continuity of the features as a function of the parameters and of γ Nk implies that for every (cid:96) ≥ 0 , the image of [0 , 1] under the map
[Equation]
is compact.
By repeated application of Lemma E.21, we conclude that t (cid:55)→ f γ Nk ( t ) ( x ) is differentiable at all but countably many points of [0 , 1] .
Following the proof of Lemma E.21, we see that the points of nondifferentiability of t (cid:55)→ f γ Nk ( t ) ( x ) are contained in the set of points of [0 , 1] where there exists a layer (cid:96) at which at least one of the coordinates of α (cid:96) γ Nk ( · ) ( x ) vanishes.
Applying the chain rule at points of differentiability of the ReLU [ · ] + and assigning 0 otherwise, it follows that the derivative of t (cid:55)→ f γ Nk ( t ) ( x ) at t ∈ [0 , 1] is equal to
[Equation]
at all but countably many points t ∈ [0 , 1] .
We ﬁnally need to check integrability of this derivative on [0 , 1] .
We have by linearity
[Equation]
and by deﬁnition
[Equation]
By construction of the network, the feature maps ( t, x ) (cid:55)→ α (cid:96) γ Nk ( t ) ( x ) are continuous.
For the backward feature maps, we can write for any θ 1 = ( W 11 , . . .
, W L +1) and any θ 2 =
( W 12 ,
[Equation]
W
[Equation]
+
[Equation]
using Cauchy-Schwarz
[Equation]
and the RHS of this bound is a continuous function of ( θ , x ) .
Because our domain of interest [0 , 1] × M is compact, we have from the triangle inequality, the previous bound on the backward feature inner products and the Weierstrass theorem
[Equation]
so that in particular, we can bound our expression for the derivative of t (cid:55)→ f γ Nk ( t ) ( x ) using the triangle inequality as
[Equation]
for some constant C > 0 .
The RHS of the previous bound does not depend on t , so by an application of (Cohn, 2013, Theorem 6.3.11), it follows that t (cid:55)→ f γ Nk ( t ) ( x ) is absolutely continuous, and we have the representation
[Equation]
Using (B.76), we can express this as
[Equation]
To conclude, it will be convenient to switch the order of integration appearing in the previous expression.
Applying (B.77), we have
[Equation]
and the RHS of this bound is integrable over [0 , 1] ×M because the network is a continuous function of the input.
By Fubini’s theorem, it follows
[Equation]
Deﬁning
[Equation]
and using (B.77), we can deﬁne bounded operators Θ Nk : L 2 µ N
[Equation]
L 2 µ N ( M ) by
[Equation]
and with this deﬁnition, (B.78) becomes
[Equation]
as claimed.
Lemma B.9.
For any network parameters θ , deﬁne kernels
[Equation]
, and for (cid:63) ∈ { N, ∞} , deﬁne corresponding bounded operators on L 2 µ (cid:63) ( M ) by
[Equation]
For any settings of the parameters θ , the operators Θ θ ,µ (cid:63) are self-adjoint, positive, and compact.
In particular, they diagonalize in a countable orthonormal basis of L 2 µ (cid:63) ( M ) functions with corresponding nonnegative eigenvalues.
Proof.
When (cid:63) =
N , an identiﬁcation reduces the operators Θ θ ,µ (cid:63) to operators on ﬁnitedimensional vector spaces, and the claims follow immediately from general principles and the ﬁnite-dimensional spectral theorem.
We therefore only work out the details for the case (cid:63) = ∞ .
Boundedness follows from an argument identical to the one developed in the proof of Lemma B.8, in particular to develop an estimate analogous to (B.77).
This estimate, together with separability and compactness of M , also establishes that Θ θ , ∞ is compact, by standard results for HilbertSchmidt operators (Heil, 2011, §B).
In addition, this estimate allows us to apply Fubini’s theorem to write for any g 1 , g
2 ∈ L 2 µ ∞ ( M )
[Equation]
since Θ θ ( x , x (cid:48) )
=
Θ θ ( x (cid:48) , x ) .
A similar calculation establishes positivity: we have for any g ∈
L 2 µ ∞ ( M )
[Equation]
where we applied Fubini’s theorem and linearity of the integral.
These facts and the spectral theorem for self-adjoint compact operators on a Hilbert space imply in particular that the operator Θ θ , ∞ can be diagonalized in a countable orthonormal basis of eigenfunctions ( v i ) i ∈ N ⊂ L 2 µ ∞ ( M ) with corresponding nonnegative eigenvalues ( λ i ) i ∈ N ⊂ [0 , + ∞ ) .
Lemma B.10.
Write Θ µ N for the operator deﬁned in Lemma B.9, with the parameters θ set to the initial random network weights and the measure set to µ N .
There exist absolute constants c, K, K (cid:48) > 0 such that for any q ≥ 0 and any d ≥ Kd 0 log( nn 0 C M ) , if
[Equation]
and if in addition n ≥ K (cid:48) L 48+20 q d 9 log 9 L , then one has
[Equation]
Proof.
Consider the nominal error evolution ζ N, nom k , deﬁned iteratively as
[Equation]
for a step size τ > 0 , which satisﬁes
[Equation]
We will prove the claim by showing that this auxiliary iteration is monotone decreasing in the loss, and close enough to the gradient-like iteration of interest that we can prove that the gradient-like iteration also retains a controlled loss.
These dynamics satisfy the ‘update equation’
[Equation]
Because M is compact and ζ is a continuous function of the input, we have ζ ∈ L ∞ ( M ) for all values of the random weights.
Because µ N is a probability measure, this means ζ has ﬁnite L pµ N ( M ) norm for every p > 0 .
Meanwhile, the choice of τ and positivity of the operator (by Lemma B.9) guarantees
[Equation]
from which it follows from the update equation
[Equation]
where the last inequality uses that µ N is a probability measure.
In particular, this nominal error evolution is nonincreasing in the relevant loss.
Now, we recall the update equation for the ﬁnitesample dynamics
[Equation]
which follows from Lemma B.8.
Subtracting and rearranging, this gives an update equation for the difference:
[Equation]
Under our hypothesis on τ , (B.80) and the triangle inequality imply the bound
[Equation]
Using Jensen’s inequality and the Schwarz inequality
[Equation]
we have
[Equation]
since µ N is a probability measure.
Deﬁning
[Equation]
by a telescoping series and the identical initial conditions, we thus obtain
[Equation]
and the triangle inequality and (B.79) then yield
[Equation]
Using a discrete version of (the standard) Gronwall’s inequality
[Equation]
the previous bound implies
[Equation]
To conclude, we will use Lemma F.5 and an inductive argument based on (B.81).
Let us ﬁrst observe that by Lemma D.11 (with a rescaling of d , which worsens the absolute constants), we have
[Equation]
≤ √ d
[Equation]
e
[Equation]
cd
[Equation]
B.82) as long as n ≥ Kd 4 L and d ≥ K (cid:48)
d 0 log( nn 0 C M ) .
Deﬁne events E Nk by
[Equation]
where d > 0 is sufﬁciently large to satisfy the conditions on d given above.
We are interested in controlling the probability of (cid:83) ki =
E Nk for k ∈ N 0 .
We can write
P (cid:34) k (cid:91) i =
E N i (cid:35) =
P (cid:34) k − 1 (cid:91) i =
E N i (cid:35) + P (cid:34)
E N k ∩ k − 1 (cid:92) i
= (cid:0) E N i (cid:1) c (cid:35) , and unraveling, we obtain
P (cid:34) k (cid:91) i =
E Nk (cid:35) =
k (cid:88) i =
P  
E Ni ∩ i − 1 (cid:92) j = (cid:0) E Nj (cid:1) c   .
In words, it is enough to control the sum of the measures of the parts of E Nk that are common with the part of the space where none of the past events occurs.
First, note that (B.82) implies
[Equation]
and so assume i > 0 below.
For any q > 0 , if kτ ≤ L q /n , n ≥ KL 36+8 q d 9 and d ≥
K (cid:48)
d 0 log( nn 0 C M ) , Lemma F.5 gives that there are events B Ni that respectively contain the sets { ∆ Ni
[Equation] [Equation]
2 q/
[Equation]
4 n
[Equation]
log
[Equation]
4 L } , and which satisfy in addition
[Equation]
We thus have by this last bound, a partition, and intersection monotonicity
[Equation]
, and by construction, one has ∆ Ni − 1 ≤ CL 4+2 q
[Equation]
3 d 3 / 4 n 11 / 12 log 3 / 4 L on (cid:0) B Ni (cid:1) c .
Another partition and (B.82) give
[Equation]
When the two events on the RHS of the last bound are active, we can obtain using (B.81)
[Equation]
ζ Nk (cid:13)(cid:13)
L 2 µN ( M ) ≤ √ d (cid:16) 1 + kτCL 4+2 q/ 3 d 3 / 4 n 11 / 12 log 3 / 4 L exp (cid:16) kτCL 4+2 q/ 3 d 3 / 4 n 11 / 12 log 3 / 4 L (cid:17)(cid:17) .
Given that kτ ≤ L q /n , we have kτCL 4+2 q/ 3 d 3 / 4 n 11 / 12 log 3 / 4 L ≤ (cid:18) C 12 L 48+20 q d 9 log 9 L n (cid:19) 1 /≤ 1 /e, where the last bound holds provided n ≥ KL 48+20 q d 9 log 9 L .
Thus, on the event (cid:0) B Ni (cid:1) c ∩ (cid:40) (cid:107) ζ (cid:107) L ∞ ( M ) ≤ √ d (cid:41) , we have (cid:13)(cid:13)
ζ Nk (cid:13)(cid:13)
L 2 µN ( M ) ≤ √ d, and thus
P (cid:34)
E Ni ∩ (cid:0) B Ni (cid:1) c ∩ (cid:40) (cid:107) ζ (cid:107) L ∞ ( M ) ≤ √ d (cid:41)(cid:35) = 0 .
By our previous reductions, we conclude
P  
E Ni ∩ i − 1 (cid:92) j = (cid:0) E Nj (cid:1) c   ≤ 2 e − cd , and in particular
P (cid:34) k (cid:91) i =
E Nk (cid:35) ≤ (2 k + 1) e − cd .
The claim is then established by taking k as large as L q / ( nτ ) .
Corollary B.11.
Write Θ µ N for the operator deﬁned in Lemma B.9, with the parameters θ set to the initial random network weights θ 0 and the measure set to µ N , and deﬁne for k ∈ N
[Equation]
There exist absolute constants c, C, C (cid:48) , K, K (cid:48) > 0 such that for any q ≥ 0 and any d ≥
Kd 0 log( nn 0 C M ) , if
[Equation]
and if in addition n ≥ K (cid:48) L 48+20 q d 9 log 9 L , then one has on an event of probability at least 1 −
C (cid:48) (1 + L q / ( nτ ))
e − cd
[Equation]
Proof.
Use Lemma B.10 to remove the hypothesis about boundedness of the errors from Lemma F.5, then apply this result together with a union bound.
Lemma B.12.
Write Θ for the operator deﬁned in Lemma B.9, with the parameters θ set to the initial random network weights and the measure set to µ ∞ .
Consider the (population) nominal error evolution ζ ∞ k , deﬁned iteratively as
[Equation]
for a step size τ > 0 , which satisﬁes τ <
[Equation]
Then for any g ∈ L 2 µ ∞ ( M ) and any k satisfying
[Equation]
we have
[Equation]
Proof.
The dynamics satisfy the ‘update equation’
[Equation]
Because M is compact and ζ is a continuous function of the input
[Equation]
we have ζ ∈ L ∞ ( M ) for all values of the random weights.
Because µ ∞ is a probability measure
[Equation]
this means ζ has ﬁnite
L pµ ∞ ( M ) norm for every p > 0 .
Using the eigendecomposition of Θ as developed in Lemma B.9, we can therefore write
[Equation]
in the sense of L 2 µ ∞ ( M )
[Equation]
Because Θ and Id − τ Θ diagonalize simultaneously
[Equation]
we obtain
[Equation]
where the inequality follows from the elementary estimate 1 − x ≤ e − x for x ≥ 0 and our choice of τ , which guarantees that 1 − τλ i > 0 for all i ∈ N so that the elementary estimate is valid after squaring.
We can split this last sum into two parts: for any λ ∈ R , we have
[Equation]
Because Θ is positive, we have further that λ i ≥ 0 for all i , so we can take λ ≥ 0 .
The ﬁrst sum consists of large eigenvalues: we use exp( − 2 kτλ i ) ≤ exp( − 2 kτλ ) to preserve their effect, and then upper bound the remainder of the sum by the squared L 2 µ ∞ norm of ζ .
The second sum consists of small eigenvalues: we replace exp( − 2 kτλ i ) ≤ 1 , and then plug in ζ =
Θ [ g ] − ( Θ [ g ] − ζ ) and use bilinearity, self-adjointness of Θ , and the triangle inequality to get
[Equation]
We then square both (nonnegative) sides of the inequality and use Cauchy-Schwarz to replace the squared sum with the sum of squares times a constant, obtaining
[Equation]
after re-adding indices i to the sum to obtain the third residual.
We will choose λ ≥ 0 to minimize the sum of the ﬁrst and second terms.
Differentiating and setting to zero gives the critical point equation
[Equation]
which can be inverted to give the unique critical point
[Equation]
where W is the Lambert W function, deﬁned as the principal branch of the inverse of z (cid:55)→ ze z ; we know that this critical point is a minimizer because the function of λ we differentiated diverges as λ → ∞ .
Plugging this point into the sum of the ﬁrst two terms gives
[Equation]
For x ≥ 0 , the function x (cid:55)→ W ( x ) is strictly increasing, as the inverse of y (cid:55)→ ye y ; by deﬁnition
W ( e ) = 1 ; and we have the representation W ( z )+log W
( z ) =
log z
(Corless et al., 1996), whence
W ( x ) ≤ log x if x ≥ e .
Because µ ∞ is a probability measure, we have
[Equation]
and therefore if
[Equation]
we can simplify the previous bound to
[Equation]
using also properties of the logarithm.
Taking square roots and using the Minkowski inequality then yields
[Equation]
where we used the previous lower bound on kτ to determine the sign that the absolute value of the logarithm takes.
This gives the claim.
Lemma B.13 (Kantorovich-Rubinstein Duality) .
Let Lip( M ) denote the class of functions f :
M → R such that both f (cid:12)(cid:12)
M ± are Lipschitz with respect to the Riemannian distances on M ± .
For any d ≥ 1 , any 0 < δ ≤ 1 and any N ≥ 2 √ d/ min { µ ∞ ( M + ) , µ ∞ ( M − ) } , one has that on an event of probability at least 1 − 8 e − d , simultaneously for all f ∈ Lip( M )
[Equation]
where
[Equation]
Proof.
The proof is an application of the Kantorovich-Rubinstein duality theorem for the 1Wasserstein distance (Weed & Bach, 2019, eq.
(1)), which states that for any two Borel probability measures µ, ν on M ± , one has
[Equation]
where M ± denotes either of M + or M − , and (cid:107)·(cid:107)
Lip is the minimal Lipschitz constant with respect to the Riemannian distance on M ± .
Therefore for any f : M ± → R Lipschitz, we have
[Equation]
where one checks separately the case where (cid:107) f (cid:107) Lip = 0 to see that this bound holds there as well.
To go from (B.83) to the desired conclusion, we need to pass from the measures µ ∞ and µ N , both supported on M , to measures µ (cid:63) ± (with (cid:63) ∈ { N, ∞} ), supported on the manifolds M ± (which we will deﬁne in detail below); the challenge here is that the number of ‘hits’ of each manifold M ± that show up in the ﬁnite sample measure µ N is a random variable, which requires a small detour to control.
Let us deﬁne random variables N + , N − by
[Equation]
so that N ± have support in { 0 , 1 , . .
. , N } , and N + + N − = N .
Deﬁne in addition
[Equation]
which represent the degree of imbalance between the positive and negative classes in the data.
By deﬁnition of the i.i.d.
sample, we have that N + ∼ Binom( N, p + ) .
Using N + and N − , we can deﬁne ‘conditional’ ﬁnite sample measures µ N + and µ N − by
[Equation]
so that ( N + /N )
µ N + + ( N − /N )
µ N − =
µ N , 8 and µ N + and µ N − are both probability measures except when N + ∈ { 0 , N } , in which case exactly one is a probability measure.
By the triangle inequality, we have for any continuous f : M → R
[Equation]
By Lemma G.1, we have
[Equation]
Using that N − N + = N − and 1 − p + = p − , the bound (B.85) implies if N ≥ 2 √ d/ min { p + , p − }
[Equation]
Now ﬁx an arbitrary f ∈ Lip( M ) .
For either (cid:63) ∈ { + , −} , we can write (cid:90)
M (cid:63)
[Equation]
x
[Equation]
=
[Equation]
and since M + and M − are separated by a positive distance ∆ > 0 , we have that x
i (cid:55)→ 1 x i ∈M (cid:63) are continuous functions on M .
Since f is continuous on M by the same reasoning and the fact that
M is compact, it follows that the functions ( x 1 , . . .
, x N ) (cid:55)→ (cid:82) M (cid:63) f
( x ) d µ N(cid:63) ( x ) are continuous on
M × · · · × M as well, and in particular for any t > 0 the sets
[Equation]
x
[Equation]
(cid:12)
[Equation]
(cid:12)
[Equation]
t (cid:27) are open in M
[Equation]
and so is their union over all f ∈ Lip( M )
[Equation]
By conditioning
[Equation]
we can then apply (B.86) to write
P  
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
B
[Equation]
Conditioned on { N (cid:63) = k } with 0 < k < N , the measure µ N(cid:63) is distributed as an empirical measure of sample size k from the probability measure µ ∞ (cid:63) /p
(cid:63) supported on M (cid:63) .
For (cid:98) Np (cid:63) / 2 (cid:99) ≤ k ≤ (cid:100) N (1 − p (cid:63) ) / 2 (cid:101) , any δ > 0 and any d ≥ 1 we have for both possible values of (cid:63) √
[Equation]
and so an application of Lemma B.16 thus gives for any 0 < δ ≤ 1 and any d ≥
P
[Equation]
Combining this last bound with (B.83) and (B.87) gives
P 
[Equation]
where we used max { p + , p − } ≤ 1 to remove the exponent of 1 / (2 + δ ) on these terms.
Taking a max over the Lipschitz constants and combining this bound with (B.84) and (B.85) and a union bound, we obtain
[Equation]
where the constant is deﬁned as in the statement of the lemma.
Lemma B.14.
Let d 0 = 1 .
There is an absolute constant C > 0 such that for any function f :
M → R with f (cid:12)(cid:12) M ± Lipschitz with respect to the Riemannian distances on M ± , one has
[Equation]
  .
Proof.
For any T > 0 and a nonconstant Lipschitz function f : [0 , T ] → R , we will establish the inequality
[Equation]
where the constant C > 0 is absolute.
We can use this result to establish the claim.
We start by writing
[Equation]
and for (cid:63) ∈ { + , −} , we have
[Equation]
where γ (cid:63) : [0 , len( M (cid:63) )]
→ M (cid:63) are the smooth unit-speed curves parameterized with respect to arc length parameterizing the manifolds.
Similarly, the curves’ parameterization with respect to arc length implies
[Equation]
Applying (B.88) with (B.89) and (B.90)
[Equation]
we obtain
[Equation]
For the ﬁrst term in the max
[Equation]
we have
[Equation]
using the triangle inequality.
For the second term in the last bound
[Equation]
we note that
[Equation]
and for the ﬁrst term, we have
[Equation]
where in the ﬁrst inequality we used the triangle inequality
[Equation]
and for the second we used that ρ (cid:63) ◦ γ (cid:63) integrates to µ ∞ ( M (cid:63) ) over [0 , len( M (cid:63) )] , which implies that there exists at least one t ∈ [0 , len( M (cid:63) )] at which ρ (cid:63)
[Equation]
[Equation]
so that the maximum of the difference in the second term on the RHS of the ﬁrst inequality is bounded by the maximum of the density term.
Thus, by
Hölder’s inequality and (B.91) and (B.92), we have
[Equation]
Similarly, for the second term in the max
[Equation]
we have
[Equation]
Thus, we have
[Equation]
  , and taking a maximum over (cid:63) ∈ { + , −} establishes the claim.
To prove (B.88), consider ﬁrst the trivial case where (cid:107)
f
(cid:107) L ∞ = 0 : here the LHS and RHS of (B.88) are identical, and the proof is immediate.
When (cid:107) f (cid:107) L ∞ > 0 , the Weierstrass theorem implies that there exists t ∈ [0 , T ] such that | f ( t ) | = (cid:107) f (cid:107) L ∞ ; we consider the case sign( f ( t )) > 0 .
For any t (cid:48) ∈ [0 , T ] , we can write by the Lipschitz property
[Equation]
and the RHS of the previous bound is nonnegative on the intersection of the interval [ t − (cid:107) f (cid:107) L ∞ (cid:107) f (cid:107) − 1 Lip , t + (cid:107) f (cid:107) L ∞ (cid:107) f (cid:107) − 1 Lip ] with the domain [0 , T ] (with standard extended-valued arithmetic conventions when (cid:107) f (cid:107) Lip = 0 ).
This gives the bound
[Equation]
where the second line follows from the changes of variables t (cid:48)
(cid:55)→ t (cid:48)
+ t .
The integrand on the
RHS of the second line in the previous bound is even-symmetric, and max (cid:110) − (cid:107)
f (cid:107) L
[Equation]
Lip , − t (cid:111) =
− min (cid:110) (cid:107) f (cid:107) L ∞
[Equation]
f (cid:107) Lip , t (cid:111) , so we can discard one side of the interval of integration to get
[Equation]
We proceed analyzing two distinct cases.
First, if (cid:107) f (cid:107) L ∞ ≤ max { t, T − t }(cid:107) f (cid:107) Lip , then we must have (cid:107) f (cid:107) Lip > 0 ; integrating the RHS of (B.94), we obtain
[Equation]
or equivalently
[Equation]
Next, we consider the case (cid:107) f (cid:107) L ∞ > max { t, T − t }(cid:107) f
(cid:107) Lip .
We split on two sub-cases: when (cid:107) f (cid:107) Lip = 0 , integrating (B.94) gives
[Equation] [Equation]
B.96) where we used max { t, T − t } ≥ T/ 2 .
When (cid:107) f (cid:107) Lip > 0 , integrating (B.94) gives
[Equation]
where the second line uses a standard algebraic identity, and the third line uses max { t, T − t } ≥ T/together with the deﬁnition of the case to get that (cid:107) f (cid:107) L ∞ − (cid:107) f (cid:107) Lip max { t, T − t } > 0 in order to discard all but the k = 0 summand.
Combining (B.97) and (B.96), we obtain for this case
[Equation]
) and combining (B.95) and (B.98) gives unconditionally
[Equation]
which establishes (B.88).
For the case sign( f ( t )) < 0 , apply the preceding argument to − f to conclude.
See (Brezis, 2011, Exercise 8.15) for a sketch of a proof that leads to more general versions of
B.88).
Lemma B.15.
For any p ∈ N , if C ≥ (4 p ) 4 p , then one has
[Equation]
Proof.
We ﬁrst give a proof for p = 1 , then build off this proof for the general case.
Consider the function f ( x ) =
cx − log x .
We have f (cid:48)
( x ) =
c − 1 /x , which is nonnegative for every x ≥ 1 /c , so in particular f is increasing under this condition.
By concavity of the logarithm, we have log x ≤ log(2 /c )
+ ( c/ 2)( x − 2 /c ) , whence
[Equation]
The RHS of this bound is equal to zero at x = (2 /c )
(log(2 /c )
− 1) , and
[Equation]
In particular, we have f ( x ) ≥ 0 for every x ≥ (2 /c )
log(2 /c ) .
Rearranging this bound, we can assert the desired conclusion that if C ≥ 3 , then n ≥ C log n for every n ≥ 2 C log 2 C .
Equivalently, we have for all such n that Cn − 1 log n ≤ 1 .
Next, we consider the case of p > 1 .
We will show
C log p n n ≤under suitable conditions.
Let us consider the choice n = KC log p KC , where K > 0 is a constant we will specify below.
Consider the function f ( x ) =
Cx − 1 log p x , which satisﬁes
[Equation]
In particular, f is decreasing as soon as p ≤ log p − 1 ( x ) .
Now, we can calculate
[Equation]
and by our result for the case p = 1 , we have for all p ≥
[Equation]
This condition is satisﬁed for KC ≥ (4 p ) 4 p , so if we set K = 2 p , we obtain the above conclusion when C ≥ (4 p ) 4 p .
Under these conditions, we then get
[Equation]
Similarly, we have log p − 1 ( KC log KC ) ≥ log p − 1 ((4 p ) 4 p ) =
(4 p ) p − 1 log p − 1 (4 p ) , which is larger than p because 4 p ≥ e .
It follows that f ( x ) ≤ 1 for every x ≥ KC log KC , which completes the proof.
Lemma B.16 (Concentration of Empirical Measure in Wasserstein Distance (Weed & Bach, 2019)) .
Let d 0 = 1 .
For either (cid:63) ∈ { + , −} , let µ be a Borel probability measure on M (cid:63) , and write µ N for the empirical measure corresponding to N i.i.d.
samples from µ .
Then for any d ≥ 1 and any 0 < δ ≤ 1 , one has
[Equation]
where the 1 -Wasserstein distance is taken with respect to the Riemannian distance.
Proof.
The proof is a direct application of the results of (Weed & Bach, 2019) on concentration of empirical measures in Wasserstein distance.
For the duration of the proof, we will work on the metric space ( M (cid:63) , len( M (cid:63) ) − 1 dist M (cid:63) ( · )) , i.e., the same metric space scaled to have unit diameter; we will then obtain the result in terms of the unscaled metric by the deﬁnition of the 1 -Wasserstein distance.
Because d 0 = 1 and M (cid:63) can be given as a unit-speed curve parameterized with respect to arc length, we have for any Borel S ⊂ [0 , 1] and any ε >
[Equation]
where N ε ( S ) denotes the ε -covering number of S by closed balls in the rescaled metric.
Following the notation of (Weed & Bach, 2019, §4.1), we then obtain for any s >
[Equation]
Invoking (Weed & Bach, 2019, Proposition 5), we obtain after some simpliﬁcations of the constants that for any 0 < δ ≤ 1 (putting s =
δ + 2 in the previous estimates)
[Equation]
where the ﬁnal inequality worst-cases constants for convenience.
Using (Weed & Bach, 2019,
Proposition 20), we have
[Equation]
and hence
[Equation]
if d ≥ 1 .
Lemma B.17.
Let n, m ∈ N .
Let F : R n → R m be 1 -nonnegatively homogeneous, and suppose there exist M, L ≥
such that
[Equation]
L -Lipschitz
[Equation]
Then for any x , x (cid:48) ∈ R n , one has
[Equation]
so that F is (2 L + M ) -Lipschitz.
Proof.
For any numbers a, b ≥ 0 and any u , v ∈ R m , one has by the triangle inequality
[Equation]
Using an elementary property of the min and the max, we thus have
[Equation]
Now we proceed to show the claim.
Noting that the case where both x , x (cid:48) are zero is trivial, ﬁrst consider the case where x is nonzero and x (cid:48) is zero.
By nonnegative homogeneity, it sufﬁces to proceed as
[Equation]
to conclude; for the inequality we used the boundedness assumption on F .
Now ﬁx x , x (cid:48) ∈ R n nonzero.
The inequality (B.99) can be applied to get
[Equation]
+ max (cid:26)(cid:13)(cid:13)(cid:13)(cid:13) F (cid:18) x (cid:107) x (cid:107) (cid:19)(cid:13)(cid:13)(cid:13)(cid:13) , (cid:13)(cid:13)(cid:13)(cid:13) F (cid:18) x (cid:48) (cid:107) x (cid:48)
(cid:107) 2 (cid:19)(cid:13)(cid:13)(cid:13)(cid:13)(cid:27) (cid:107) x − x (cid:48)
(cid:107) 2 , where in the inequality we also applied the (cid:96) 2 triangle inequality.
Using the assumed properties of
F , we thus have
[Equation]
By a classical inequality (e.g. proved in (E.15)), one has
[Equation]
whence
[Equation]
as was to be shown.
C S KELETON A NALYSIS AND C ERTIFICATE C ONSTRUCTION
In this section, we construct a certiﬁcate g for the certiﬁcate problem (B.1) in the context of a simple model geometry.
We also collect technical estimates relevant to the analysis of the skeleton ψ .
We point to Appendix A.5.2 for a summary of the operator and function deﬁnitions relevant to the certiﬁcate problem that we will use below.
We will use the notation
[Equation]
in this section; we call explicit attention to this notation to avoid confusion with the kernel ˆΘ = ψ 1 ◦ ∠ that we have deﬁned in the main text for convenience of exposition.
C. C ERTIFICATE C ONSTRUCTION
To construct a certiﬁcate, it sufﬁces to solve the integral equation
[Equation]
for a function g ∈ L 2 µ ∞ ( M ) and obtain estimates on the norm of g .
It is useful to consider separately the contributions of integration over the class manifolds M ± in the action of the operator ˆ Θ : we can write for any g
[Equation]
and it then makes sense to further subdivide based on whether the evaluation point x lies in M + or M − , and to introduce the density ρ explicitly by a change of variables.
With a slight abuse of notation, we will write d x (cid:48) to denote the Riemannian measure on M + and M − , for concision.
Because the kernel ψ ◦ ∠ is symmetric, if we deﬁne an operator ˆ Θ + : L 2 ( M + )
→ L 2 ( M + ) by
[Equation]
an operator ˆ Θ − : L 2 ( M − )
→ L 2 ( M − ) by
[Equation]
and an operator ˆ Θ ± : L 2 ( M + )
→ L 2 ( M − ) by
[Equation]
then we can write the certiﬁcate system (C.1) equivalently as the 2 × 2 block operator equation
[Equation]
where we write ρ + and ρ − for the restriction of the density ρ to M + and M − , respectively, and where the adjoint operation is viewed as occurring with operators between L 2 ( M + ) and L 2 ( M − ) (both Hilbert spaces).
We will make use of this notation in the sequel.
C.1. T WO C IRCLES
The two circles geometry is a highly-symmetric geometry where M + and M − are coaxial circles in the upper and lower hemispheres of S 2 , each of radius 0 < r < 1 .
Here we note that since the skeleton ψ depends only on the angle between points of S n 0 − 1 , the particular embedding of this geometry into S n 0 − 1 is irrelevant, and it is without loss of generality to consider the geometry in Sonce we have restricted ourselves to this conﬁguration.
We have unit-speed charts, for t ∈ [0 , 2 πr ]
[Equation]
which implies speciﬁc forms of the spherical distances
[Equation]
and
[Equation]
with the analogous results for the remaining possible combinations of domains, by symmetry.
Because ˆ ζ is piecewise constant on each connected component of M , there are constants C + , C − such that C
+ = ˆ ζ on M + and C − = ˆ ζ on M − .
The block-structured system we are interested in solving is then
[Equation]
where subscripts are used to denote the domain of each component of the certiﬁcate.
The coordinate representations (C.2) and (C.3) show that each of the operators appearing in the 2 × 2 matrix in (C.4) is invariant on the circle; we can obtain some useful simpliﬁcations by identifying these operators with their coordinate representations.
Deﬁning
[Equation]
and (self-adjoint) operators on 2 π -periodic functions g by
[Equation]
by a change of coordinates, it is equivalent to solve the system
[Equation]
where we have identiﬁed ρ
+ and ρ − with their coordinate representations
[Equation]
and with an abuse of notation used the same notation for the certiﬁcate as in (C.4).
We can use symmetry properties to determine
[Equation]
, so for purposes of analysis we need only consider f r .
Each of the invariant operators in (C.5) diagonalizes in the Fourier basis, and because the target ˆ ζ is a piecewise constant function, we only need to use the ﬁrst Fourier coefﬁcient.
In other words, we can solve this system by ﬁrst inverting the invariant operator, which responds to only the constant component of the target, and then inverting the density multiplication operators.
This approach is made precise in the following lemma.
Lemma C.1.
There is an absolute constant K > 0 such that if L ≥ max { K, ( π/ 2)(1 − r 2 ) − 1 / 2 } and r ≥ 12 , then the system (C.4) has a solution that satisﬁes
[Equation]
Proof.
Following the discussion by (C.5), it is equivalent to solve the system in the Fourier basis, with only the DC component.
We thus start by solving the system
[Equation]
where G + and G − are constants that we will show exist.
This is a 2 × 2 system, and the matrix is symmetric, with minimum eigenvalue 2 (cid:82) π 0 ( ψ ◦ f r − ψ ◦ g r )( t ) d t .
Using Lemma C.2, we have if
L ≥ max { K, ( π/ 2)(1 − r 2 ) − 1 / 2 } and r ≥
[Equation]
so the 2 × 2 matrix is invertible, and by an operator norm bound on its inverse we have the regularity estimate
[Equation]
It follows that the function
[Equation]
solves the system (C.4).
We conclude
[Equation]
(cid:18)
G
[Equation]
ρ +
[Equation]
+ ( t ) (cid:19)
[Equation]
+
[Equation]
γ + ( t ) d t + (cid:90)
[Equation]
(cid:18)
G − ρ − ◦ γ − ( t ) (cid:19)ρ − ◦ γ − ( t ) d t ≤ 2πn
[Equation]
(cid:0)
[Equation]
Taking square roots on both sides of the expression resulting from the last inequality will give the claim, after we simplify the expression (cid:113) C 2+ + C 2 − .
Since
[Equation]
we can conclude after adjusting constants.
Lemma C.2.
There exists an absolute constant K > 0 such that if L ≥ max { K, ( π/ 2)(1 − r 2 ) − 1 / 2 } and if r ≥ 12 , one has
[Equation]
Proof.
Write σ r = ψ ◦ f r − ψ ◦ g r for brevity, which is nonnegative, by Lemma C.3.
We consider the tangent line to the graph of σ r at 0 ; by Lemma C.3, this line has the form t (cid:55)→ σ r (0) − tnrL ( L + 1) / 4 π , and its graph hits the horizontal axis at t
=
πσ r (0) /nrL
( L + 1) .
Using that σ r (0) ≤ ψ (0)
=
nL/ 2 , we see that this point of intersection is no larger than 2 π/r ( L + 1) , which can be made less than K by choosing L ≥ K (cid:48) , where K > 0 is the absolute constant appearing in the convexity bound of Lemma C.3, and K (cid:48) > 0 is an absolute constant.
Under this condition, we obtain using Lemma C.
[Equation]
, and so (cid:90) [
[Equation]
π ]
[Equation]
t )
[Equation]
We have σ r (0) =
nL/ 2 − ψ (cos -1 (2 r 2 − 1)) , and using the estimate of Lemma C.18, we get
[Equation]
Together with the estimate cos -1 (2 r 2 − 1) ≥ 2 √ 1 − r 2 , we obtain
[Equation]
where the ﬁnal inequality requires the choice L ≥ π/ 2 √ 1 − r 2 .
Thus, we have shown
[Equation]
as claimed.
Lemma C.3.
There is an absolute constant 0 < K ≤ π/ 2 such that if L ≥ 3 , one has for all r ∈ (0 ,
[Equation]
(i) ψ ◦ f r − ψ ◦ g r ≥ 0 on [0 , π ] ; (ii) ( ψ ◦ f r − ψ ◦ g r ) (cid:48) (0) =
− nrL ( L + 1) / 4 π ; (iii) ψ ◦ f r − ψ ◦ g r is convex on [0 , K ] .
Proof.
In this proof, we will make use of basic results on the skeleton ψ , namely Lemmas E.5,
C.15 and C.16 without making explicit reference to them.
Property (i) follows from the fact that ψ is decreasing, cos -1 is decreasing, and the deﬁnitions of f r and g r .
We note that f r is smooth on (0 , π ) ; to prove property (ii), it will sufﬁce to show that f r admits a right derivative at 0 and π and apply the chain rule.
We have if t ∈ (0 , π
[Equation]
after some rearranging, and by periodicity and symmetry properties of f r , we have
[Equation]
We Taylor expand sin t (1 − cos t ) − 1 / 2 in a neighborhood of zero to evaluate the derivatives there.
We have sin t = t − t 3 / 6+ O ( t 5 ) and 1 − cos t = t 2 / 2(1 − t 2 /
+ O ( t 4 )) ; by the binomial series, we have (1 − cos t ) − 1 / 2 = √ 2 /t
(1+ t 2 / 4+ O ( t 4 )) ,
whence sin t
[Equation]
[Equation] [Equation]
, and
[Equation]
Thus
[Equation]
For property (iii), now consider t ∈ [0 , π/ 2] when necessary.
The chain rule gives
[Equation]
and we have if t ∈ (0 , π )
[Equation]
after some rearranging of the numerator.
We have 1 − cos t ≥ 0 , and so the estimate r 2 cos t + (1 − r 2 ) ≥ cos t (with equality only at t = 0 ) yields f (cid:48)(cid:48) r ( t ) ≤ 0 (with a strict inequality if 0 < t < π ).
By symmetry, this implies that g (cid:48)(cid:48) r ≥ 0 , and using that ψ (cid:48) ≤ 0 , we obtain
[Equation]
By symmetry, we have g r ( t ) =
f r ( π − t ) on [0 , π ] , and because f r is strictly concave we know as well that f (cid:48) r is strictly decreasing; it follows that f (cid:48) r − g (cid:48) r is also strictly decreasing, and its unique zero satisﬁes the equation
[Equation]
Noting that t = π/ 2 satisﬁes this equation, we conclude that f (cid:48)
r ≥ g (cid:48)
r on [0 , π/ 2] , so that on this interval we have
[Equation]
By Lemma C.17, if L ≥ 3 there is an absolute constant K > 0 such that ... ψ ≤ 0 on [0 , K ] .
The previous bound then yields
[Equation] [Equation]
as claimed.
C. A UXILIARY R ESULTS C.2. G EOMETRIC R ESULTS
Lemma C.4.
Let M be a complete Riemannian submanifold of the unit sphere S n 0 − 1 (with respect to the spherical metric induced by the euclidean metric on R n 0 ) with ﬁnitely many connected components K .
If d 0 = 1 , assume moreover that each connected component of M is a smooth regular curve.
Then for every 0 < ε ≤ 1 , there is a ε -net for M in the euclidean metric (cid:107) · (cid:107) 2 having cardinality no larger than ( C M /ε )
d 0 , where
[Equation]
Moreover, these nets have the property that if x ∈ M is given, there is a point in the net ¯ x within euclidean distance ε of x such that ¯ x lies in the same connected component of M as x .
Proof.
Consider a ﬁxed connected component M i with i ∈
[ K ] .
We write the Riemannian distance of M i as dist M i ; because M i is a Riemannian submanifold of R n 0 , we have dist M i ( x , y ) ≥ (cid:107) x − y (cid:107) 2 for every x , y in M i .
Because dist M i ( x , y ) ≥ (cid:107) x − y (cid:107) 2 , it sufﬁces to estimate the covering number in terms of the Riemannian distance.
We will consider distinctly the cases d 0 =and d 0 ≥ 2 , starting with d 0 = 1 .
In this case, we have assumed that M i are regular curves, so it is without loss of generality to assume they are moreover unit-speed curves parameterized by arc length, with lengths len( M i ) .
It follows that we can obtain an ε -net for M i in terms of dist M i having cardinality at most len( M i ) /ε
when 0 < ε ≤ 1 , and by the submanifold property these sets also constitute ε -nets for M i in terms of the (cid:96) 2 distance.
Covering each connected component M i in this way gives a ε -net for M by taking the union of each connected component’s net.
When d 0 ≥ 2 , we can make use of sectional curvatures and deep local-to-global theorems from geometry that constrain the topology of M in terms of its curvature.
Because M is a Riemannian submanifold of S n 0 − 1 , the space of constant sectional curvature 1 , the sectional curvatures of M i are also bounded below by 1 ; it follows that the Ricci curvature tensors Ric i of M i (respectively) satisfy the bound Ric
i ≥ d 0 − 1 .
By Myers’ Theorem (Zhu, 1997, Theorem 3.4(1)), we conclude that diam( M i ) ≤ π .
Using the diameter bound from Myers’ Theorem and Gromov’s Packing Lemma (Lemma G.12), we then obtain a ε -net for M i in terms of the euclidean distance with cardinality at most vol( S d 0 ) /ψ
( ε/ 4) .
To estimate ψ ( θ ) , we note that the cap is in particular a spherical geodesic metric ball; by a rigid motion we may assume the cap is centered at the north pole e d 0 +1 , and so we need to integrate the volume element of S d 0 over all x ∈ S d 0 with (cid:104) x , e d 0 +1 (cid:105) ≤ cos θ .
Using a spherical polar coordinate chart tailored to this setting (e.g. (Muirhead, 1982, Theorem 2.1.3)), we have
[Equation]
where the second line holds if θ ≤ π/ 2 .
Using vol( S d 0 ) / vol( S d 0 − 1 ) ≤ √ π and 2 d 0 ≤ 2 d 0 , we obtain the estimate
[Equation]
Thus, for any i ∈ [ K ] , any d 0 ≥ 1 and any 0 < ε ≤ 1 , we can conclude that there is a ε -net for M i in the euclidean metric having cardinality no larger than ( C M /ε )
d 0 , where
[Equation]
Taking the union of these nets, we conclude that for any d 0 ≥ 1 and any 0 < ε ≤ 1 , there is a ε -net for M in the euclidean metric having cardinality no larger than ( C M /ε )
d 0 , where
[Equation]
The additional property claimed is satisﬁed by our construction of the nets.
Lemma C.5.
For any x , x (cid:48) , ¯ x , ¯ x (cid:48) in S n 0 − 1 , one has
[Equation]
Proof.
Writing ∠ ( x , x
[Equation]
) = cos -
[Equation]
x , x
[Equation]
(cid:105) =
cos -1 (1 − (1 / 2) (cid:107) x − x
[Equation]
(cid:107) 22 ) , consider the function f
( x ) = cos -1 (1 − (1 / 2) x 2 ) for x ∈ [ −√ 2 , √ 2] , which is differentiable except possibly at 0 .
We calculate
[Equation]
and taking limits at 0 shows that f admits left and right derivatives on all of [ −√ 2 , √ 2] .
f (cid:48) is evensymmetric, so by checking values at 0 and √
we conclude that | f (cid:48) | ≤ √ 2 , which shows that f is √ 2 -Lipschitz.
The claim follows.
Lemma C.6.
Let d 0 = 1 .
Choose L so that L ≥ Kκ 2 C λ , where κ and C λ are respectively the curvature and global regularity constants deﬁned in Section 2.1, and K, K (cid:48) > 0 are absolute constants.
Then
[Equation]
where C is an absolute constant and M ± denotes either M + or M − .
Proof.
Recall that γ + , γ − denote unit-speed curves parameterized with respect to arc length whose images are M + , M − .
For convenience, deﬁne g ( ν ) = 1 / (1 + Lν/π ) .
We have
[Equation]
First, we note that | g | is strictly decreasing.
We claim that for any x ∈ M − , there is a x (cid:63) ∈ M + such that ∠ ( x (cid:63) , x (cid:48) )
≤ ∠ ( x , x (cid:48) )
for all x (cid:48) ∈ M + ; it is easy to see this is the case by choosing x (cid:63) to achieve the minimum in min x (cid:48) ∈M + ∠
( x , x (cid:48) ) and arguing by contradiction.
By monotonicity of the integral, this implies
[Equation]
and similarly for the term involving integration over M − .
Therefore
[Equation]
and it sufﬁces to analyze these two terms.
We bound the ﬁrst term, since the second can be bounded by an identical argument.
By compactness, the supremum in this term is attained at some x ∈ M + .
Taking t such that γ + ( t )
=
x , we can write
[Equation]
We split the interval [0 , S + ] into two disjoint sub-intervals [0 , S + ] ∩ [ t − K τ / √ L, t + K τ / √ L ] and [0 , S + ] \ [ t − K τ / √ L, t + K τ / √ L ] , corresponding to “large scale” and “small scale” behavior, where K λ is the global regularity constant deﬁned in (A.2).
If we now assume 1 √ L ≤ c λ κ , then from (A.2) we obtain
[Equation]
and hence
[Equation]
From the deﬁnition of g it follows that
[Equation]
Since | g | is a monotonically decreasing function we can bound the second integral in (C.9), obtaining
[Equation]
We next consider the remaining interval of integration in (C.9).
Deﬁning
[Equation] [Equation]
L
[Equation]
and ν ± ( s ) =
∠ ( γ + ( t ∗ ± s ) , γ + ( t ∗ )) , the integral of interest can be written as
[Equation]
max
[Equation]
It will be sufﬁcient to consider the ﬁrst integral here since the second one can be bounded in an identical fashion.
We aim to show that the integral above is not too large.
This will be the case if ν + ( s ) stays very small for a large range of values of s .
To show that this is does not occur, we will use our bounds on the curvature of M to bound ν
+ ( s ) uniformly from below, which will in turn provide an upper bound on the integral.
We will require an application of Lemma C.7, which will be applicable if S ++ ≤ πκ .
If L ≥ κ 2 K 2 τ π 2 we have
[Equation]
It follows immediately that Lemma C.7 applies to any restriction of γ + of length no larger than πκ .
Next deﬁne by ˜ γ : [0 , S ++ ] → S n 0 − 1 an unit-speed arc of curvature κ , and ˜ ν
( s ) =
∠ (˜ γ (0) , ˜ γ ( s )) .
We claim that
[Equation] [Equation]
The proof is by contradiction.
Assume there is some r such that
[Equation]
+ ( r ) <
˜ ν
( r ) .
(C.13)
Now deﬁne by γ r : [0 , r ] → S n 0 − 1 a restriction of γ + such that γ r (0) =
γ + ( t ∗ ) , γ r ( s ) =
γ + ( t ∗ + s ) , by ˘ γ r an arc with curvature κ and the same endpoints as γ r , and by ˜
γ r a restriction of ˜ γ with len(˜ γ r ) = len( γ r ) =
r.
Note that ∠ (˜ γ r (0) , ˜ γ r ( s )) =
˜
ν
( r ) .
However, an application of Lemma C.7 gives
[Equation]
where the second inequality is because ˘ γ r and ˜ γ r have identical curvature at every point, and by assumption (C.13) the endpoints of ˜ γ r are a greater geodesic (and hence euclidean) distance from each other than the endpoints of ˘ γ r (which are a distance ν
+ ( r ) apart).
This inequality contradicts the equality above it, and we conclude that no such r exists, and (C.12) holds.
We have that | g | is a monotonically decreasing function, hence we can write for the ﬁrst integral in (C.11)
[Equation]
We now bound this integral.
Since ˜ γ is an arc with curvature κ , from the proof of Lemma C.3 we have that ˜
ν is concave, and since ˜ ν (0)
=
we can write
[Equation]
, and since | g | is monotonically decreasing
[Equation]
ν
[Equation]
S
[Equation]
where we used the deﬁnition of g .
It remains to show that S ++ and ˜ ν
( S ++ ) are close.
Since ˜ γ is an arc with curvature κ and length S ++ , if we additionally assume L ≥ Kκ 2 C λ for some K chosen so that κ (cid:107) ˜ γ (0) − ˜ γ ( S ++ ) (cid:107) 2≤ κS ++ 2 ≤ κC λ 2 √ L ≤ 12 , we obtain
[Equation]
where in the ﬁrst line we used sin − 1 ( x ) ≤ x + x 3 for x .
Since
[Equation]
we obtain
[Equation]
and hence
[Equation]
We now choose L ≥ Kκ 2 K 2 τ for some K , so that the above term is smaller than 2.
We therefore have
[Equation]
for some C .
We can bound the second integral in (C.11) in an identical fashion.
Combining this result with (C.10) and recalling (C.9), we obtain
[Equation]
for some constant, which completes the proof.
Lemma C.7.
Given a smooth, simple open curve in R n of length S with unit-speed parametrization γ : [0 , S ]
→ R n such that for some κ >
[Equation]
πκ deﬁne by ˘ γ an arc of any circle of radius 1 κ such that ˘ γ (0)
= γ (0) , ˘ γ ( ˘ S ) = γ ( S ) , ˘ S ≤ πκ 9 .
We then have
[Equation]
Proof.
This result is a generalization of a well known comparison theorem of Schur’s to higher dimensions following the proof in (Sullivan, 2008), where we additionally specialize to the case where one of the curves is an arc.
Given a curve γ satisfying the conditions of the lemma, we ﬁrst consider an arc ˜ γ of a circle of radius 1 κ and length S , with a unit-speed parametrization.
At the midpoint of this arc, the tangent vector ˜ γ (cid:48) ( S 2 ) is parallel to ˜ γ ( S ) − ˜ γ (0) , hence
[Equation]
Similarly, for the curve γ we have
[Equation]
Denoting the angle between tangent vectors (cid:104) γ (cid:48) ( a ) , γ (cid:48) ( b ) (cid:105) =
cos θ ( a, b ) , we use the fact that for any smooth curve with unit-speed parametrization (cid:107) γ (cid:48)(cid:48) ( t ) (cid:107)
=
(cid:12) (cid:12) dθds ( t, s ) s
= t (cid:12) (cid:12) .
= | θ (cid:48) ( t ) | .
This gives for any t ∈ [0 , S/ 2]
[Equation]
  ≥
[Equation]
  κ
S
[Equation]
t (cid:90)
[Equation] [Equation]
where we have used monotonicity of cos over the relevant range which is ensured by assumption 2, and a similar argument follows for t ∈ (0 , − S/ 2] .
Combining these inequalities gives
[Equation]
We have shown that, unsurprisingly, if the curvature of γ is bounded and it is not too long, then the distance between its endpoints is greater than that of a curve of equal length but larger curvaturenamely the arc ˜ γ .
We now consider the arc ˘ γ deﬁned in the lemma statement.
If S > ˘ S , due to assumption 2 this would imply
[Equation]
contradicting the inequality proved above.
It follows that S ≤ ˘ S .
C.2. A NALYSIS OF THE S KELETON
Notation.
Deﬁne ϕ (0)
=
Id , and for (cid:96) ∈ N deﬁne ϕ
( (cid:96) )
as the (cid:96) -fold composition of ϕ with itself, where
[Equation]
is the heuristic angle evolution function.
We will make use of basic properties of this function such as smoothness (established in Lemma E.5) below.
In this section, we will study the skeleton
[Equation]
where we have not included the additive factor cos ϕ ( L ) ( ν ) , as it is easily removed along the lines of Theorem B.2.
We deﬁne
[Equation]
so that
[Equation]
We will also establish a convenient approximation to the skeleton.
Deﬁne
[Equation]
Lemma C.8 implies that ψ is convex; it is less trivial to obtain the same for ψ 1 .
We will prove several estimates below for the terms ξ ( (cid:96) ) and their derivatives that can be used to immediately obtain useful estimates for ψ and its derivatives.
Lemma C.8.
For each (cid:96) = 0 , 1 , . . .
, L , the functions ϕ ( (cid:96) ) are nonnegative, strictly increasing, and concave (positive and strictly concave on (0 , π ) ); if 0 ≤ (cid:96) <
L , the functions ξ ( (cid:96) ) , are nonnegative, strictly decreasing, and convex (positive and strictly convex on (0 , π ) ).
Proof.
These claims are a consequence of some general facts for smooth functions that we articulate here so that we can rely on them often in the sequel.
First, we have for any smooth function f :
[Equation]
These equations show that if f > 0 , f (cid:48) > 0 , and f (cid:48)(cid:48) < 0 , then f ◦ f also satisﬁes these three properties.
Lemma E.5 shows that ϕ satisﬁes these three properties on (0 , π ) ; we conclude from the mean value theorem and a simple induction the same for ϕ ( (cid:96) ) , as claimed.
Meanwhile, if f, g are smooth real-valued functions on (0 , π ) , we have
[Equation]
Thus, if f and g are both positive, strictly decreasing, strictly convex functions on (0 , π ) , then fg also satisﬁes these three properties.
Lemma E.5 implies that 0 < 1 − π − 1 ϕ ( (cid:96) ) <
1 on (0 , π ) , and the ﬁrst and second derivatives are scaled and negated versions of those of ϕ ( (cid:96) ) ; we conclude by another induction that the same three properties apply to the functions ξ ( (cid:96) ) .
Lemma C.9.
There is an absolute constant C > 0 such that if L ≥ 12 and n ≥ L , then one has
[Equation]
Proof.
We have from the triangle inequality
[Equation]
where we use Lemma C.8 to take ξ ( (cid:96) ) outside the absolute value.
Notice that (cos ϕ ( (cid:96) ) − 1) ξ ( (cid:96) ) ≤ 0 , so to control the L ∞ norm of this term it sufﬁces to bound it from below.
We will show the monotonicity property
[Equation]
from which it follows
[Equation]
using also ξ ( L − 1) ( ν ) ≤ 1 .
Since cos x ≥ 1 − (1 / 2) x 2 , and since Lemma C.10 gives that ϕ ( L − 1) ≤
C/
( L − 1) (and also estimates the constant), we have as soon as L ≥ 1 + C/ √
[Equation]
which gives the claim provided L ≥ 2 and n ≥ L .
So to conclude, we need only establish (C.15).
To this end, write the LHS of (C.15) as
[Equation]
to notice that it sufﬁces to prove nonnegativity of the bracketed quantity.
In addition, since (cid:96) ≥and
ϕ ( ν ) ≤ ν by Lemma E.5, we can instead prove the inequality
[Equation]
for all x ∈ [0 , π ] .
Using the closed-form expression for cos ϕ
( x ) in Lemma E.2, we can plug into the previous inequality and cancel to get the equivalent inequality
[Equation]
But this is immediate from the concavity estimate sin x ≤ x , and (C.15) is proved.
Lemma C.10.
If (cid:96) ∈ N 0 , one has the “ﬂuid” estimate for the angle evolution function
[Equation]
where c > 0 is an absolute constant.
In particular, if (cid:96) ∈ N one has ϕ ( (cid:96) ) ≤ 1 /c(cid:96) .
Proof.
The second claim follows from the ﬁrst claim and 1 + c(cid:96)ν ≥ c(cid:96)ν , so we will focus on establishing the ﬁrst estimate.
The proof is by induction on (cid:96) ∈ N , since the case of (cid:96) = 0 is immediate.
By Lemma E.5, there is a constant c 1 > 0 such that ϕ ( ν ) ≤ ν (1 − c 1 ν ) , and using the numerical inequality x (1 − x ) ≤ x (1 + x ) − 1 , valid for x ≥ 0 , we get
[Equation]
which establishes the claim in the case (cid:96)
= 1 .
Assuming the claim holds for (cid:96) − 1 , we calculate
[Equation]
where the ﬁrst inequality uses (C.16), and the second inequality uses the induction hypothesis and the relation x (1 + x ) − 1 = 1 − (1 + x ) − 1 to see that x (cid:55)→ x (1 + c 1 x ) − 1 is increasing.
Clearing denominators in the numerator and denominator of the RHS of this last bound, we see that it is equal to ν/ (1 + (cid:96)ν/π ) , and the claim follows by induction.
Lemma C.11.
If (cid:96) ∈ N 0 , the iterated angle evolution function satisﬁes the estimate
[Equation]
Proof.
The proof is by induction on (cid:96) ∈ N , since the case (cid:96) = 0 is immediate.
The case (cid:96) =follows from Lemma C.12.
Assuming the claim holds for (cid:96) − 1 , we calculate
[Equation]
where the ﬁrst inequality applies Lemma C.12, and the second uses the fact that the RHS of the bound in Lemma C.12 is strictly increasing and the induction hypothesis.
Clearing denominators in the numerator and denominator of the RHS of this last bound, we see that it is equal to ν/ (1+ (cid:96)ν/π ) , and the claim follows by induction.
Lemma C.12.
It holds
[Equation]
Proof.
After some rearranging using Lemma E.2, it sufﬁces to prove
[Equation]
Using Lemma E.5, we see that both the LHS and RHS of this bound are nonincreasing.
We will prove the estimate in three stages, using “small angle”, “large angle”, and “intermediate angle” estimates of the quantities on both sides of (C.17).
Since πν/ ( π + ν ) ∈ [0 , π/ 2] , we can use standard estimates for cos to get RHS estimates
[Equation]
and
[Equation]
As for the LHS, we can obtain an estimate near ν = π in a straightforward way.
Transforming the domain by ν (cid:55)→ π − ν , it sufﬁces to get estimates on sin ν − ν cos ν near ν = 0 , then divide by π .
Using cos ν ≥ 1 − (1 / 2) ν 2 and sin ν ≤ ν , it follows that sin ν − ν cos ν ≤ (
[Equation] [Equation]
ν 3 .
We conclude
[Equation]
C
[Equation]
We will develop a second-order approximation to the LHS near 0 for the small-angle estimates.
The ﬁrst, second, and third derivatives of the LHS are (1 − ν/π ) sin ν , (1 /π ) sin ν − (1 − ν/π ) cos ν , and (2 /π ) cos ν
+ (1 − ν/π ) sin ν , respectively.
To bound the third derivative, we will use the estimate cos ν ≤ 1 − ν 2 / 3 on [0 , π/ 2] .
To prove this, note that Taylor’s formula implies the bound cos ν ≤ 1 − ν 2 / 3 on [0 , cos -1 (2 / 3)] ; because cos is concave on [0 , π/ 2] , we also have the tangent line bound cos( ν ) ≤ − ν √ 5 / 3 + (2 / 3 + √ 5 cos -
(2 / 3) / 3) on [0 , π/ 2] .
We can then solve for the zeros of the quadratic polynomial 1 − ν 2 / 3 + ( √ 5 / 3)
ν − (2 / 3 + √ 5 cos -
(2 / 3) / 3) ; a numerical evaluation shows that both roots are real and outside the interval [cos -1 (2 / 3) , π/ 2] .
Since the tangent line touches the graph of cos at ν = cos -1 (2 / 3) , this proves that cos ν ≤ 1 − ν 2 / 3 on [0 , π/ 2] .
We can therefore write
[Equation]
The RHS of this inequality is a concave quadratic; we calculate its maximum analytically as 2 + 3 π 2 / 20 .
Meanwhile, if ν ∈ [ π/ 2 , π ] , we have 2 cos ν ≤ 0 , and ( π − ν ) sin ν ≤ π/ 2 .
We conclude that (2 /π ) cos ν +(1 − ν/π ) sin ν ≤ 2+3 π 2 / 20 on [0 , π ] .
Writing c
= 1 / (3 π )+ π/ 40 , this implies an estimate
[Equation]
Finally, we will need some estimates for interpolating the small and large angle regimes.
We note that the second derivative (1 /π ) sin ν − (1 − ν/π ) cos ν of the LHS of (C.17) is nonnegative if ν ≥ π/ 2 , because cos ≥ 0 here; meanwhile, the third derivative (2 /π ) cos ν + (1 − ν/π ) sin ν of the LHS of (C.17) is nonnegative if 0 ≤ ν ≤ π/ 2 , since cos ≥ 0 here, and it follows that the second derivative is increasing on [0 , π/ 2] .
Checking numerically that the value of the second derivative at 1 .
42 is positive, we conclude that the LHS of (C.17) is convex on [1 . 42 , π ] .
In addition, we use calculus to evaluate the ﬁrst and second derivative of the RHS of (C.18) as − νπ 3 / ( π + ν ) 3 and − π 3 ( π − 2 ν ) /
( π + ν ) 4 , respectively
; this shows that the RHS of (C.18) is convex for ν ≥ π/ 2 , and concave for ν ≤ π/ 2 .
Taking a tangent line to the graph of the RHS of (C.18) at π/ 2 , it follows that the function
[Equation]
is a concave lower bound for the RHS of (C.18) on [0 , π ] .
We proceed to using the estimates developed in the previous paragraph to prove (C.17).
We ﬁrst argue that for ν in a neighborhood of 0 , we have
[Equation]
[Equation] [Equation]
, which will in turn prove (C.17) in the same neighborhood.
Cancelling and rearranging, it is equivalent to show
[Equation]
The LHS is a concave quadratic, with value 2 /π − 2 c > 0 at 0 ; we calculate its two distinct roots numerically as lying in the intervals [ − 5 .
1 , − 5] and [1 . 42 , 1 . 43] , respectively.
It follows that (C.17) holds for ν ∈ [0 , 1 . 42] .
Next, we argue that for ν in a neighborhood of π , we have
[Equation]
which will in turn prove (C.17) in the same neighborhood.
Transforming with ν (cid:55)→ π − ν and rearranging, it is equivalent to show ν 2 (2 π − ν ) ≤ 4 π 2 in a neighborhood of 0 .
The LHS of this last inequality is 0 at 0 , and nonnegative on [0 , π ] ; its ﬁrst and second derivatives are ν (4 π − 3 ν ) and 4 π − 6 ν , respectively, which shows that it is a strictly increasing function of ν on [0 , π ] .
Verifying numerically the three distinct real roots of ν 3 − 2 πν 2 + 1 = 0 and transferring the result back via another transformation ν
(cid:55)→ π − ν , we conclude that (C.17) holds on [ π − 1 . 1 , π ] .
To obtain that (C.17) holds on [1 . 42 , π − 1 . 1] , we use that the function g deﬁned in (C.22) is a concave lower bound for the RHS of (C.18), so that it sufﬁces to show that the LHS of (C.17) is upper bounded by g on [1 . 42 , π − 1 . 1] .
The LHS of (C.17) is convex on [1 . 42 , π ] , so it follows that it is sufﬁcient to show that the values of the LHS of (C.17) at 1 .
42 and at π − 1 .
1 are upper bounded by those of g at the same points.
Conﬁrming this numerically, we can conclude the proof.
Lemma C.13.
If (cid:96) ∈ N 0 , one has
[Equation]
where c > 0 is the absolute constant also appearing in Lemma E.5 (property 4 ), and in particular c/ 2 is equal to the absolute constant appearing in Lemma C.10.
In particular, if (cid:96) ∈ N and ν ∈ [0 , π ] we have the estimate
[Equation]
Proof.
The case of (cid:96) = 0 follows directly (as an equality) from ϕ (0) ( ν )
=
ν .
Now we assume (cid:96) ∈ N .
Smoothness of ϕ ( (cid:96) ) follows from Lemma E.5.
Applying the chain rule and an induction, we have
[Equation]
and applying the chain rule also gives
[Equation]
By Lemma E.5, we have ˙ ϕ > 0 on [0 , π ]) , and the formula (C.23) then implies that ˙ ϕ
( (cid:96) ) > 0 on [0 , π ]) as well.
Considering only angles in this half-open interval and distributing, it follows
[Equation]
Applying an induction using the previous formula and distributing in the result
[Equation]
we obtain
[Equation]
By Lemma E.5, we have 0 < ˙ ϕ ≤ 1 on [0 , π ) and ¨ ϕ ≤ 0 .
Thus
[Equation]
¨ ϕ ◦ ϕ
( (cid:96) (cid:48) ) .
When (cid:96) (cid:48) > 0 , we have ϕ ( (cid:96) (cid:48) ) ≤ π/ 2 , and by Lemma E.5, we have ¨ ϕ ≤ − c < 0 on [0 , π/ 2] ; thus, − ¨ ϕ ◦ ϕ
( (cid:96) (cid:48) )
≥ c if (cid:96) (cid:48) > 0 .
When (cid:96) (cid:48)
= 0 , we can use the fact that ¨ ϕ ≤ 0 on [0 , π ] to get a bound ¨ ϕ ≤ − c 1 [0 ,π/ 2] .
We conclude
[Equation]
Next, we notice using the chain rule that
[Equation]
and using (C.23) and Lemma E.5, we have that ˙ ϕ
( (cid:96) )
= 1 .
For any ν ∈ [0 , π ) , we integrate both sides of (C.26) from 0 to ν to obtain using the fundamental theorem of calculus
[Equation]
where in the ﬁnal inequality we use the inequality min { ν, π/ 2 } ≥ ν/ 2 , valid for ν ∈ [0 , π ] .
Rearranging, we conclude for any 0 ≤ ν < π
[Equation]
and noting that the LHS of this bound is equal to 0 at ν = π and the RHS is positive, we conclude the claimed bound for every ν ∈ [0 , π ] .
The second estimate claimed follows by multiplying this bound by ν on both sides, and using 1 + ( c/ 2) (cid:96)ν ≥ ( c/ 2) (cid:96)ν .
Lemma C.14.
If (cid:96) ∈ N , one has
[Equation]
where C > 0 is an absolute constant, and c > 0 is the absolute constant also appearing in
Lemma E.5 (property 4 ), and in particular c/ 2 is equal to the absolute constant appearing in
Lemma C.10.
If ν ∈ [0 , π ] , the RHS of this upper bound is a decreasing function of ν , and moreover we have the estimates
[Equation]
Proof.
Smoothness follows from Lemma E.5; we make use of some results from the proof of
Lemma C.13, in particular (C.23) and (C.25).
We treat the case of (cid:96) = 1 ﬁrst.
By Lemma E.5, we have | ¨ ϕ | ≤ C for an absolute constant C > 0 , and since 1 / (1 + ( c/ 2) ν ) ≥ 1 / (3 / 2) = 2 / 3 by the numerical estimate of the absolute constant c > 0 in Lemma E.5, it follows
[Equation]
which establishes the claim when (cid:96)
=
1 (after worst-casing constants if necessary).
Next, we assume (cid:96) > 1 .
Multiplying both sides of (C.25) by ( ˙ ϕ
( (cid:96) ) ) 2 and cancelling using (C.23), we obtain
[Equation]
where the last equality holds at least on [0 , π ) , by Lemmas E.5 and C.13, and where empty products are deﬁned to be 1 .
If (cid:96) (cid:48) > 0 , we have ϕ ( (cid:96) (cid:48) ) ≤ π/ 2 , and by Lemma E.
we have that | ¨ ϕ | ≤ C and ˙ ϕ ≥ c (cid:48) > 0 on [0 , π/ 2] for absolute constants C, C (cid:48) > 0 .
Separating the (cid:96) (cid:48)
= 0 summand, this gives a bound (cid:12)(cid:12)(cid:12)
[Equation]
= ˙ ϕ
( (cid:96) (cid:48) ) .
(C.28)
By Lemma C.13, we have ˙ ϕ
( ν ) ≤ 1 / (1 + ( c/ 2) ν ) , and by Lemma E.5, we have ϕ ( ν ) ≤ ν , hence ϕ
( (cid:96) (cid:48) )
( ν ) ≤ ν .
Using concavity of ϕ , nonincreasingness of ˙ ϕ and nondecreasingness of ϕ ( (cid:96) (cid:48) )
(which follow from Lemma E.5) and a simple re-indexing, we can write
[Equation]
(cid:96) (cid:48)
[Equation]
[Equation] [Equation]
≤
[Equation]
c
[Equation] [Equation]
(cid:96)ν where the third-to-last line follows from (C.23), the second-to-last line follows from Lemma C.13, and the last line follows from the inequality (cid:96) − 1 ≥ (cid:96)/ 2 if (cid:96) ≥ 2 .
Following on from (C.28), we conclude by an application of Lemma C.
[Equation]
where the last line simply worst-cases the constants.
For any (cid:96) (cid:48) ∈ N
[Equation]
the function x (cid:55)→ 1 / (1 + ( c/ 8) (cid:96) (cid:48) x ) is nonincreasing, so we can estimate the sum in the previous statement using an integral, obtaining
[Equation]
after evaluating the integral—we deﬁne the quantity inside the parentheses on the RHS of the ﬁnal inequality to be (cid:96) − 1 when ν = 0 , which agrees with the integral representation in the previous line and with the unique continuous extension of the function on (0 , π ] to [0 , π ] —which establishes the ﬁrst claim.
We now move on to the study of the bound we have derived.
For decreasingness, we note that the functions
[Equation]
whose product is equal to our upper bound, are evidently both smooth nonnegative functions of ν at least on (0 , π ] , so that by the product rule for differentiable functions it sufﬁces to prove that these two functions are themselves decreasing functions of ν .
The ﬁrst function is evidently decreasing as an increasing afﬁne reparameterization of ν (cid:55)→ 1 /ν ; for the second function, after multiplying by the constant (cid:96) − 1 and rescaling by a positive number (when (cid:96) = 1 , the function is identically zero on (0 , π ] , and the function’s continuous extension as deﬁned above equals 0 at 0 as well), we observe that it sufﬁces to prove that x (cid:55)→ x − 1 log(1+ x ) is a decreasing function of ν on (0 , ∞ ) .
The derivative of this function is x (cid:55)→ ( x − (1 + x ) log(1 + x )) /
( x 2 (1 + x )) , so it sufﬁces to show that x − (1 + x ) log(1 + x ) ≤ 0 .
Noting that the function x (cid:55)→ x log x is convex (its second derivative is 1 /x ), it follows that x − (1+ x ) log(1+ x ) is concave as a sum of concave functions, and is therefore has its graph majorized by its supporting hyperplanes; its derivative is equal to − log(1 + x ) , which equals 0 at 0 , and we therefore conclude from our previous reduction that the second function in (C.29) is decreasing, and that our composite upper bound is as well.
For the remaining estimates, we use the concavity estimate log(1 + x ) ≤ x to obtain from our previous result
[Equation]
since the function x (cid:55)→ C/ (1 + cx ) is nonincreasing for any choice of the constants.
Next, we use the expression we have derived in the ﬁrst claim to obtain
[Equation]
For any K > 0 , the function x (cid:55)→ x/ (1 + Kx ) is nondecreasing, and using the numerical estimate π ( c/ 8) < 1 that follows from Lemma E.5, we obtain in addition
+ π ( c/ 8)( (cid:96) − 1) ≤ (cid:96) for (cid:96) ∈ N .
Thus
[Equation]
as claimed.
Lemma C.15.
One has for every (cid:96) ∈ { 0 , 1 , . . .
, L }
[Equation] [Equation]
[Equation] [Equation]
and for every (cid:96) ∈
[ L ]
[Equation]
Finally, we have ˙ ϕ (0)
( π ) = 1 and ¨ ϕ (0) ( π ) = 0 .
Proof.
The claims are consequences of Lemma E.5 when (cid:96) = 1 , and of ϕ (0)
=
Id for smaller (cid:96) ; assume (cid:96) >
1 below.
The claim for ϕ ( (cid:96) ) (0) follows from the fact that ϕ (0) = 0 and induction.
For the claim about ˙ ϕ
( (cid:96) )
(0) , we calculate using the chain rule
[Equation]
By induction and Lemma E.5, we obtain ˙ ϕ
( (cid:96) )
= 1 .
The claim about ˙ ϕ
( (cid:96) )
( π ) follows from the same argument.
For the remaining claims about ¨ ϕ ( (cid:96) ) , we calculate using the chain rule
[Equation]
whence
[Equation]
Using Lemma E.5 to get ¨ ϕ (0)
= − 2 / (3 π ) , this yields
[Equation]
Similarly, since we have shown ˙ ϕ
( (cid:96) − 1) ( π ) = 0 , we obtain ¨ ϕ
( (cid:96) )
( π ) = 0 .
Lemma C.16.
For ﬁrst and second derivatives of ξ ( (cid:96) ) , one has ˙ ξ ( (cid:96) )
= − π − 1 L − 1 (cid:88) (cid:96) (cid:48)
= (cid:96) ˙ ϕ
( (cid:96) (cid:48) )
L − 1 (cid:89) (cid:96) (cid:48)(cid:48) =
(cid:96) (cid:96) (cid:48)(cid:48) (cid:54) =
(cid:96) (cid:48) (cid:0) 1 − π − 1 ϕ ( (cid:96) (cid:48)(cid:48) ) (cid:1) , and ¨ ξ ( (cid:96) ) (C.30) =
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
  , (C.31) where empty sums are interpreted as zero, and empty products as 1 .
In particular, one calculates
[Equation]
and
[Equation]
Proof.
The two derivative formulas are direct applications of the Leibniz rule to ξ ( (cid:96) ) .
The claims about values at 0 follow from plugging the results of Lemma C.15 into our derivative formulas and the deﬁnition of ξ ( (cid:96) ) .
For values at π , we ﬁrst note that ϕ (0) ( π ) =
π , from which it follows ξ (0) ( π )
= 0 .
Next, we use Lemma C.15 to get that ¨ ϕ ( (cid:96) )
( π ) =
for all (cid:96) ∈ { 0 , 1 , . . .
, L } and ˙ ϕ
( (cid:96) )
( π ) = 1 (cid:96) =
to get ˙ ξ ( (cid:96) )
( π ) = − π − 1 ξ
[Equation]
1) ( π ) 1 (cid:96) =0 .
For ¨ ξ ( (cid:96) )
( π ) , we have
[Equation]
[Table]
[Equation]
[Table]
If L = 1 , the sum in the last expression is empty, and this quantity is 0 .
If L > 1 , the sum is nonempty, and every summand is equal to zero by Lemma C.15.
We conclude ¨ ξ ( (cid:96) )
( π ) = 0 .
Lemma C.17.
If L ≥ 3 , there exists an absolute constant 0 < C ≤ π/ 2 such that on the interval [0 , C ] , one has for every (cid:96) = 0 , 1 , . . .
, L −
[Equation]
Proof.
We consider functions only on [0 , π/ 2] in this proof.
Following the calculations in the proof of Lemma C.13, we have the expression
[Equation]
Using as well Lemma E.
[Equation]
we have ﬁrst and second derivative estimates
[Equation]
and
[Equation]
By Lemma E.5, ...
ϕ extends to a continuous function on [0 , π/ 2]
[Equation]
so in addition there exists a δ >such that on [0 , δ ] we have
[Equation]
We lower bound (C.33) on [0 , δ ] using these estimates.
For (cid:96) = 1 , we can do no better than (C.34).
For (cid:96) > 1 , we can write
[Equation]
We have the numerical estimate c 2 = 0 .
14 from Lemma E.5, and we check numerically that (0 . 14) 2 > 1 / 6 π 2 .
This implies that on [0 , δ ] and for every (cid:96) ≥ 2 , ... ϕ
( (cid:96) ) is lower bounded by a positive number plus a scaled version of ... ϕ
( (cid:96) − 1) .
We check precisely using the original formula (C.33) and Lemma E.5 for (cid:96) =
[Equation]
so that in particular
[Equation]
By continuity, it follows that there is a neighborhood [0 , δ (cid:48) ] on which we have ... ϕ (2)
+ ... ϕ (1) > 0 .
Thus, on [0 , min { δ, δ (cid:48) } ] , we guarantee that simultaneously
[Equation]
Now we consider the third derivative of the skeleton summands ξ ( (cid:96) ) .
Following the calculations of
Lemmas C.8 and C.16, in particular applying the Leibniz rule, we observe that every term in the sum deﬁning ...
ξ ( (cid:96) ) that does not involve a third derivative of one of the factors (1 − (1 /π )
ϕ
( (cid:96) (cid:48) ) ) will be nonpositive, because (1 − (1 /π ) ϕ
( (cid:96) (cid:48) ) ) ≥ 0 , ˙ ϕ
( (cid:96) (cid:48) ) ≥ 0 , and ¨ ϕ ( (cid:96) (cid:48) ) ≤ 0 .
Meanwhile, by our calculations above, on the interval [0 , min { δ, δ (cid:48) } ] , the only terms that can be positive are those with (cid:96) = 0 or (cid:96)
= 1 where we differentiate the (cid:96) (cid:48)
= 1 factor three times, i.e., the (cid:96) (cid:48)
= 1 term in the sum
[Equation]
with (cid:96) = 0 or (cid:96) = 1 .
We will compare the (cid:96) (cid:48)
= 1 summand with the (cid:96) (cid:48)
= 2 summand: we have that the sum of these two terms equals
[Equation]
At 0 , the quantity inside the right parentheses is equal to ... ϕ
+ ... ϕ (2) > 0 , by our calculations above.
Thus, by continuity, there is a possibly smaller δ (cid:48)(cid:48) > 0 such that on [0 , δ (cid:48)(cid:48) ] , the sum of terms (C.35) is negative.
We conclude that on [0 , min { δ, δ (cid:48) , δ (cid:48)(cid:48) } ] , we have for every (cid:96) ≥...
ξ ( (cid:96) ) ≤ 0 , and since we have chosen the neighborhood sizes δ, δ (cid:48) , δ (cid:48)(cid:48)
independently of the depth L , we can conclude.
Lemma C.18.
For all (cid:96) ∈ { 0 , . . .
, L − 1 } , one has
[Equation]
Proof.
We have
[Equation]
where the ﬁrst inequality applies the AM-GM inequality
[Equation]
and the second uses the standard exponential convexity estimate.
Using Lemma C.11, we have
[Equation]
where the last inequality uses the fact that (cid:96) (cid:48) (cid:55)→ ν/ (1 + (cid:96) (cid:48) ν/π ) is nonincreasing for every ν ∈ [0 , π ] together with a standard estimate from the integral test.
We calculate
[Equation]
which gives the claim after substituting into (C.36).
Lemma C.19.
For all (cid:96) ∈ { 0 , . . .
, L − 1 } , one has
[Equation]
Proof.
Using Lemma C.16, we have
[Equation]
where we directly treat the case (cid:96) = 0 to avoid dividing by zero at ν = π .
The triangle inequality and Lemmas E.5 and C.18 then give
[Equation]
Using Lemma C.
[Equation]
we have
[Equation]
where the last inequality uses the fact that (cid:96) (cid:48) (cid:55)→ 1 / (1 + c(cid:96) (cid:48) ν ) is nonincreasing for every ν ∈ [0 , π ] together with a standard estimate from the integral test.
Evaluating the integral, we obtain
[Equation]
where the second term on the RHS is deﬁned at ν = 0 by continuity.
Using the standard concavity estimate log(1 + x ) ≤ x , we have
[Equation]
whence
[Equation]
Combined with the result of Lemma C.18, we conclude
[Equation]
The numerical estimate c = 0 .
07 in Lemma E.5 then allows us to conclude
[Equation]
as claimed.
Lemma C.20.
One has
[Equation]
and
[Equation]
Proof.
We calculate using the chain rule
[Equation]
and the triangle inequality gives
[Equation]
Applying Lemmas C.13, C.18 and C.19 and Lemma E.5 to estimate the constant c in Lemma C.13, we then obtain
[Equation]
The proof of the second claim is nearly identical, since in this case we need only use the bounds on | ˙ ξ ( (cid:96) ) | .
Lemma C.21.
There are absolute constants c, C > 0 such that for all (cid:96) ∈ { 0 , . . .
, L − 1 } , one has
[Equation]
Proof.
By Lemmas E.5 and C.
[Equation]
we can write
[Equation]
 
Focusing ﬁrst on the second term, we have using Lemma E.5, (C.37) and Lemma C.
[Equation]
[Table]
We can then write using nonnegativity
L − 1 (cid:88) (cid:96) (cid:48)
= (cid:96)
L − 1 (cid:88) (cid:96)
[Table]
=
[Table]
=
[Table]
≤
[Table]
− 1 (cid:88) (cid:96) (cid:48) =
(cid:96)
L
[Equation]
1 (cid:88) (cid:96) (cid:48)(cid:48) =
(cid:96) ˙ ϕ
( (cid:96) (cid:48) ) ˙ ϕ
( (cid:96) (cid:48)(cid:48) )
= (cid:32)
L − 1 (cid:88) (cid:96) (cid:48)
= (cid:96) ˙ ϕ
( (cid:96) (cid:48) ) (cid:33), and using (C.37) and Lemma C.18, we obtain thus
[Equation]
Regarding the ﬁrst term, we have using Lemma C.
[Equation]
because the function (cid:96) (cid:48) (cid:55)→ (cid:96) (cid:48) / (1 + c(cid:96) (cid:48) ν ) is nondecreasing.
Applying also Lemma C.18, we obtain using the triangle inequality and worst-casing constants
[Equation]
Lemma C.22.
One has
[Equation]
and
[Equation]
where c, C > 0 are absolute constants.
Proof.
We calculate using the chain rule
[Equation]
and the triangle inequality gives
[Equation]
Using Lemmas C.13, C.14, C.18, C.19 and C.21 and worst-casing constants for convenience
[Equation]
we obtain from the last estimate
[Equation]
where in the second line we made some estimates along the lines of the proof of Lemma C.20 and worsened the constant C .
The proof for ψ follows from the same argument, since in this case we have the same sum of ¨ ξ ( (cid:96) )
terms but none of the extra residuals.
D C ONCENTRATION AT I NITIALIZATION D. N OTATION AND F RAMEWORK
We recall the expression for the neural tangent kernel, as summarized in Appendix A.5.2:
[Equation]
The objective of this section is to establish supporting results for the proof of Theorem B.2, which gives uniform concentration of Θ( x , x (cid:48) ) over M×M around the deterministic skeleton kernel.
We take a pointwise-uniformize approach to proving this result: Appendix D.2 establishes concentration results for the constituents of Θ( x , x (cid:48) ) when x , x (cid:48) are ﬁxed, and Appendix D.3 develops results that control the number of local support changes near points in a discretization of M × M in order to provide a suitable stand-in for the continuity properties necessary to uniformize these pointwise results.
We collect relevant technical results and their proofs in Appendix D.4.
D. P OINTWISE C ONCENTRATION
We ﬁx ( x , x (cid:48) ) in this section, and generally suppress notation involving the speciﬁc points for concision.
We separate our analysis into two distinct sub-problems: “forward concentration”, which consists of the study of the correlations (cid:104) α (cid:96) ( x ) , α (cid:96) ( x (cid:48) ) (cid:105) , and “backward concentration”, which consists of the study of the backward feature correlations (cid:104) β (cid:96) ( x ) , β (cid:96) ( x (cid:48) ) (cid:105) .
Forward concentration is a prerequisite of our approach to backward concentration, so we begin there.
D.2. F ORWARD C ONCENTRATION
Notation.
For (cid:96) = 0 , 1 , . . .
, L , deﬁne random variables
z (cid:96) 1 = (cid:107) α (cid:96) ( x ) (cid:107) 2 and z (cid:96)
= (cid:107) α (cid:96) ( x (cid:48) )
(cid:107) 2 .
With the convention 0 · + ∞ = 0 , we deﬁne for (cid:96) = 0 , . . .
, L , random variables ν (cid:96) by
[Equation] [Equation]
(cid:28)
[Equation]
α
[Equation]
(cid:19) .
These deﬁnitions guarantee that ν (cid:96) =
π whenever either feature norm z (cid:96)i vanishes.
These random variables are signiﬁcant toward controlling Θ( x , x (cid:48) )
because, for each
[Equation]
Let us deﬁne pairs of gaussian vectors g (cid:96) 1 , g (cid:96) 2 ∼ i .
i .
d . N ( 0 , (2 /n )
I ) that are independent of everything else in the problem.
For (cid:96) ≥ 1 , we have by rotational invariance of the Gaussian distribution and the probability chain rule
[Equation]
Since α 0 ( x ) = x and (cid:107) x (cid:107) 2 = 1 , we have by an induction with analogous deﬁnitions
[Equation]
Similarly, we have
[Equation]
As for the angles, we have by rotational invariance
[Equation]
so that an inductive argument gives
[Equation]
g (cid:96) (cid:48) 1 cos ν (cid:96) (cid:48) − 1 + g (cid:96) (cid:48) 2 sin ν (cid:96) (cid:48) − 1 (cid:105) + (cid:13)(cid:13)(cid:13)(cid:13)(cid:33) .
We will write ¯ z (cid:96)
= (cid:96) (cid:89) (cid:96) (cid:48)
= (cid:13)(cid:13)(cid:13)(cid:13)(cid:104) g
(cid:96) (cid:48) 1 (cid:105)
+ (cid:13)(cid:13)(cid:13)(cid:13) 2 ,
¯ z (cid:96)
= (cid:96) (cid:89) (cid:96) (cid:48) =
(cid:13)(cid:13)(cid:13)(cid:13)(cid:104) g (cid:96) (cid:48) 1 cos ν (cid:96) (cid:48) − 1 + g (cid:96) (cid:48) 2 sin ν (cid:96) (cid:48) − 1 (cid:105) + (cid:13)(cid:13)(cid:13)(cid:13) 2 , and similarly ¯ ν (cid:96) = cos -   1 ¯ z (cid:96) 1 ¯ z
(cid:96) 2 >(cid:42) (cid:2) g (cid:96) 1 (cid:3) + (cid:13)(cid:13)(cid:13)(cid:2) g (cid:96) 1 (cid:3) + (cid:13)(cid:13)(cid:13), (cid:2) g (cid:96) 1 cos ν (cid:96) − 1 + g (cid:96) 2 sin ν (cid:96)
[Equation]

 , so that we obtain for the angles by a similar inductive argument ν
[Equation]
d = ¯ ν (cid:96) .
(D.1)
For technical reasons, it will be convenient to consider an auxiliary angle process, deﬁned for (cid:96) ≥as
[Equation]
where we deﬁne with notation from Appendix E
[Equation] [Equation]
, and ˆ ν
= ν
= (cid:104) x , x (cid:48) (cid:105) .
We then observe
[Equation]
since the inductive structure of ¯ z (cid:96)i implies that all feature norms are nonvanishing if and only if the top-level feature norms ¯ z Li are nonvanishing, and since the statement (cid:81)
[Equation]
1 ¯
[Equation]
g
[Equation]
1 , g (cid:96)
[Equation]
2 (cid:1) =implies by deﬁnition that ¯ z L 1 ≥ 2 − L and ¯ z L 2 ≥ 2 − L .
By Lemma E.16, as long as n ≥ 21 the event ¯
E has overwhelming probability, and in particular a union bound implies
[Equation]
so that
[Equation]
We can therefore pass from ¯ ν (cid:96) to ˆ ν (cid:96) with negligible error.
From the expression for ˆ ν (cid:96) , we see that the angles ˆ ν 0 →
ˆ ν 1 → · · · → ˆ ν L form a Markov chain, and we will control them using martingale techniques.
For (cid:96) = 0 , 1 , . . .
, L , we write F (cid:96) to denote the σ -algebra generated by the gaussian vectors ( g 11 , g 12 , g 21 , g 22 , . . .
, g (cid:96) 1 , g (cid:96) 2 ) , so that ( F 0 , . .
. , F L ) is a ﬁltration, and the sequences of random variables (ˆ ν 1 , . .
, ˆ ν L ) and (¯ ν 1 , . .
, ¯ ν L ) are adapted to ( F 1 , . .
. , F L ) .
Moreover, with these deﬁnitions we have
[Equation]
where ¯ ϕ is the angle evolution function deﬁned in Appendix E.1, which is well-approximated by the function
[Equation]
(see Lemmas E.1 and E.2).
In the sequel, we will employ the notation ϕ
( (cid:96) ) to denote the (cid:96) -fold composition of ϕ with itself.
By Lemma E.5, the function ϕ is smooth, and the chain rule implies the same for ϕ ( (cid:96) ) ; we will employ the notation ˙ ϕ
( (cid:96) ) and ¨ ϕ ( (cid:96) ) for the ﬁrst and second derivatives of ϕ ( (cid:96) ) , respectively.
Main results.
Lemma D.1.
There are absolute constants c, C, C (cid:48) > 0 and absolute constants K, K (cid:48) > 0 such that for any d ≥ K , if n ≥ K (cid:48) max { 1 , d 4 log 4 n, d 3 L log 3 n } then one has for any (cid:96) = 1 , . . .
, L
[Equation] [Equation]
(cid:96) ( x (cid:48) ) (cid:11)
[Equation]
( (cid:96) )
[Equation]
( x
[Equation]
x (cid:48) ))
(cid:12)(cid:12)(cid:12) > C (cid:115)
d 3 (cid:96) log 3 n n   ≤ C (cid:48) n − cd
[Equation]
Proof.
We have
[Equation]
[Table]
where we also use | cos | ≤ 1 and that cos is 1 -Lipschitz.
Since z
[Table]
d = ¯ z (cid:96)i for i = 1 , 2 , we obtain using
Lemma D.2 and the choice n ≥ KdL
[Equation]
and as long as n ≥ C 2 dL , we obtain on one of the same events
[Equation]
By a union bound, we obtain
[Equation]
so that if we put d (cid:48) =
d log n and therefore choose n ≥ C 2 dL log n
[Equation]
we have
[Equation]
with the second bound holding if d ≥ 1 and n ≥ L .
For the remaining term
[Equation]
we have by the triangle inequality
[Equation]
By (D.4), the ﬁrst term on the RHS of the previous expression is equal to zero with probability at least 1 − CLe − cn as long as n ≥ 21 .
The second term can be controlled with Lemma D.3 provided we select n, L, d to satisfy the hypotheses of that lemma.
We thus obtain via an additional union bound
P
[Equation]
C
[Equation]
If n ≥ (2 /c (cid:48) ) log L and n ≥ (2 c/c (cid:48) )
d log n , we have C (cid:48)(cid:48) n − cd + C (cid:48)(cid:48)(cid:48) (cid:96)e − c (cid:48) n ≤ ( C (cid:48)(cid:48)
+ C (cid:48)(cid:48)(cid:48) )
n − cd .
The previous bound then becomes
P
[Equation]
and if we worst-case the dependence on (cid:96) and d in the residual in the previous bound, we obtain
[Equation]
as claimed.
Lemma D.2.
There are absolute constants c, C, C (cid:48) > 0 and an absolute constant K > 0 such that for i = 1 , 2 , every (cid:96) = 1 , . . .
, L , and any d > 0 , if n ≥ max { Kd(cid:96), 4 } , then one has
[Equation]
n
[Equation]
Proof.
Because z (cid:96)i d = ¯ z (cid:96) 1 , it sufﬁces to show
[Equation]
The proof will proceed by showing concentration of the squared quantity (cid:81) (cid:96)(cid:96)
[Equation]
=
[Equation]
g
[Equation]
+ (cid:13)(cid:13)(cid:13)(cid:13)  around 1 , so that we can appeal to results like Lemma D.26, and then conclude by applying an inequality for the square root to pass to the actual quantity of interest.
To enter the setting of Lemma D.26, it makes sense to normalize the factors in the product by their degree, but we must avoid dividing by zero.
We have (cid:81) (cid:96)(cid:96)
[Equation]
=
if and only if (cid:81) (cid:96)(cid:96) (cid:48)
=1 (cid:13)(cid:13)(cid:13)(cid:13)(cid:104) g
(cid:96) (cid:48) 1 (cid:105)
+ (cid:13)(cid:13)(cid:13)(cid:13) = 0 , and whenever (cid:81) (cid:96)(cid:96) (cid:48)
=1 (cid:13)(cid:13)(cid:13)(cid:13)(cid:104) g
(cid:96) (cid:48) 1 (cid:105)
+ (cid:13)(cid:13)(cid:13)(cid:13) (cid:54) = 0 , we can write (cid:96) (cid:89) (cid:96) (cid:48) =
(cid:13) (cid:13)(cid:13)(cid:13) (cid:104) g
(cid:96) (cid:48) 1 (cid:105)
+ (cid:13) (cid:13)(cid:13)(cid:13) 2 = (cid:32) (cid:96) (cid:89) (cid:96) (cid:48) =
(cid:13) (cid:13)(cid:13)(cid:13) (cid:104) g
(cid:96) (cid:48) 1 (cid:105)
+ (cid:13) (cid:13)(cid:13)(cid:13)  (cid:33) 1 /(D.6)
= (cid:32) (cid:96) (cid:89) (cid:96) (cid:48) =
n (cid:13)(cid:13)(cid:13)(cid:13)(cid:104) g (cid:96) (cid:48) 1 (cid:105) + (cid:13)(cid:13)(cid:13)(cid:13) (cid:33) 1 / 2   (cid:96) (cid:89) (cid:96) (cid:48)
=  (cid:13)(cid:13)(cid:13)(cid:112) n 2 (cid:2) g (cid:96) (cid:48) 1 (cid:3) + (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:114) n (cid:104) g (cid:96) (cid:48) 1 (cid:105) + (cid:13)(cid:13)(cid:13)(cid:13)  
 1 /, (D.7) using 0 -homogeneity of the (cid:96) 0 “norm”.
This leads to an extra product-of-degrees term; we will make use of Lemma D.27 to show that the product of degrees itself concentrates.
We will also show that the event where a degree is zero is extremely unlikely and proceed with the degree-normalized main term by conditioning.
By symmetry, the random variables (cid:107) (cid:2) g (cid:96) 1 (cid:3) + (cid:107) 0 are i.i.d.
sums of n Bernoulli random variables with rate 12 .
By Lemma G.1, we then have
[Equation]
and so
P
[Equation]
min
[Equation]
(cid:48)
[Equation]
0 < n
[Equation]
where the ﬁrst inequality applies a union bound.
Putting t = n/ 4 , we conclude
[Equation]
so that whenever n ≥ 16 log (cid:96) , we have (cid:107) (cid:104) g (cid:96) (cid:48) 1 (cid:105) + (cid:107) 0 ≥ n/ 4 for every (cid:96) (cid:48) ≤ (cid:96) with probability at least 1 − e − n/ 16 .
This gives us enough to begin working on showing concentration of the squared version of (D.5): partitioning, we can use the previous simpliﬁed bound to write
[Equation]
[Table]
Using (D.7) and the triangle inequality, we can write whenever no terms in the product vanish
[Table]
(cid:33) − 1 (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) .
(D.10)
Moreover, we have by Lemma D.
P (cid:34)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
(cid:96) (cid:89) (cid:96) (cid:48) =
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:104)
[Equation]
(cid:48)
[Equation]
(cid:105)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) >(cid:114) d(cid:96) n (cid:35) ≤ 4 (cid:96)e − cd as long as n ≥ 128 d(cid:96) .
Choosing in addition n ≥ 4 d(cid:96) and using nonnegativity, this implies
P
[Equation]
(cid:96)
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
(cid:35)
[Equation]
(cid:96)e
[Equation]
cd , occurring on the same event.
Combining the previous two bounds with (D.10) and (D.9) via another partition, we get
P
[Equation]
[Table]
where we use here that on the event { min (cid:96) (cid:48)
=1 ,...,(cid:96) (cid:107) (cid:104) g
(cid:96) (cid:48) 1 (cid:105)
+ (cid:107) 0 ≥ n/ 4 } , the quantity (cid:81) (cid:96)(cid:96) (cid:48)
=1 (cid:13)(cid:13)(cid:13)(cid:13)(cid:104) g
(cid:96) (cid:48) 1 (cid:105)
+ (cid:13)(cid:13)(cid:13)(cid:13) 2 is nonzero almost surely, which allowed us to invoke the identities (D.7).
For ( k 1 , . . .
, k (cid:96) ) ∈
[ n ] (cid:96) , we deﬁne events E k 1 1 , . . .
, E k (cid:96) (cid:96) by
[Equation]
=
[Equation]
(cid:27) .
Conditioning and then relaxing the bounds, we can write
P
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
+ (cid:96) (cid:89)
[Table]
[Equation] [Equation]
(cid:35) ∗ P (cid:104) E
[Equation]
, E k
[Equation]
(cid:96) (cid:105) .
Conditioned on
E k 1 1 , . . .
, E k (cid:96) (cid:96) with k (cid:96) (cid:48) > 0 , the random variable (cid:81) (cid:96) (cid:96) (cid:48) =
[Equation]
(cid:107) (cid:112)
[Equation]
(cid:104) g (cid:96) (cid:48)
(cid:105)
+ (cid:107)
[Equation]
/ (cid:107) (cid:112) n 2 (cid:104) g (cid:96) (cid:48) 1 (cid:105) + (cid:107) 0 is distributed as a product of independent degree-normalized standard χ 2 random variables with minimum degree min { k 1 , . .
k (cid:96) } .
An application of
Lemma D.26 then yields immediately
[Equation]
(cid:104)
[Equation]
(cid:96) (cid:48)
[Equation]
(cid:105) + (cid:13)(cid:13)(cid:13)(cid:13)  (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Table] [Table]
as n ≥ K
[Table]
this
[Table]
yields
[Table]
(cid:96) (cid:48)
[Equation]
(cid:105)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)  (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
(cid:114)
[Equation]
n (cid:35)
[Equation] [Equation]
(cid:48)
[Equation]
cd , where we worst-cased constants in the probability bound.
If we choose n ≥ 4 C 2 d(cid:96) , we have
C (cid:112) d(cid:96)/n ≤ 1 / 2 , and we obtain on the event in the previous bound
P
[Equation]
(cid:12)
[Equation]
(cid:96)
[Equation]
(cid:96) (cid:48)
[Equation]
(cid:13)(cid:13)(cid:13) (cid:13) (cid:104)
[Equation]
(cid:96) (cid:48) 1 (cid:105) + (cid:13)(cid:13)(cid:13) (cid:13)  (cid:12)(cid:12)(cid:12)(cid:12) (cid:12)
[Equation]
(cid:35)
[Equation]
e
[Equation]
n
[Equation] [Equation]
C (cid:48) (cid:96)e
[Equation]
cd .
In particular, on the complement of the event in the previous bound, the product lies in [1 / 2 , 3 / 2] .
To conclude, we can linearize the square root near 1 to obtain an analogous bound for the product of the norms.
Taylor expansion of the smooth function x (cid:55)→ x 1 / 2 about the point 1 gives
[Equation]
where k lies between x and 1 .
In particular, if x ≥ 12 , we have
[Equation]
so that
[Equation]
Thus, when x ≥ 12 we have by the triangle inequality
[Equation]
from which we conclude based on a partition and our previous choices of large n
[Equation]
which yields the claimed probability bound when n ≥ 16 d .
Lemma D.3.
There are absolute constants c, C, C 0 > 0 and absolute constants K, K (cid:48) > 0 such that for any L max ∈ N and any d ≥ K , if n ≥ K (cid:48) max { 1 , d 4 log 4 n, d 3 L max log 3 n } , then one has
P 
[Equation]
∈
[ L
[Equation] [Equation]
Proof.
The proof uses a recursive construction involving L ∈
[ L max ] .
Before beginning the main argument, we will deﬁne the key quantities that appear and enforce bounds on the parameters to obtain certain estimates.
For each L ∈ [ L max ] , we deﬁne the event
[Equation]
where C 0 > 0 is an absolute constant whose value we will specify below, so that E L ∈ F L , and our task is to produce an appropriate measure bound on (cid:83)
L ∈
[ L max ] E L .
For notational convenience, we also deﬁne E 0 = ∅ .
For each L ∈ [ L max ] and each (cid:96) ∈
[ L ] , we deﬁne
[Equation]
so that for every L , ∆ 1 L , . . . , ∆ LL is adapted to the sequence F 1 , . .
. , F L , and we have the decomposition
[Equation]
In particular, we have
[Equation]
The sequences (∆ (cid:96)L ) (cid:96) ∈
L are not quite martingale difference sequences, but we will show they are very nearly so: writing
[Equation]
, we have that ( ¯∆ (cid:96)L ) (cid:96) ∈
L is a martingale difference sequence, which can be controlled using truncation and martingale techniques, and the extra conditional expectation term can be controlled analytically.
In particular, we have the following estimates: by Lemma D.24, we have if n ≥ max { K 1 log 4 n, K 2 L max } that for every L ∈ [ L max ] and every (cid:96) ∈
[ L ]
[Equation]
by the ﬁrst result in Lemma D.25 we have for every d ≥ max { K 3 , 6 /c 1 } that if n ≥ K 4 d 4 log 4 n , then for every L ∈ [ L max ] and every (cid:96) ∈
[ L ] (and after worsening constants)
[Equation]
and by the second result in Lemma D.25, we have by our previous choices of n , d , and L max that for every L ∈ [ L max ] and every (cid:96) ∈
[ L ] (after worsening constants
[Equation]
The main line of the argument will consist of showing that a measure bound of the form (D.12) on (cid:83) (cid:96) ∈
[ L − 1] E (cid:96) implies one of the same form on (cid:83) (cid:96) ∈
[ L ] E (cid:96) .
For any L ∈ [ L max ] , on the event E c L we have
[Equation]
where the second inequality follows from Lemma C.10, and the third follows from the choice n ≥ ( C 0 c 0 ) 2 d 3 L log 3 n .
In particular, if we make the choice n ≥ ( C 0 c 0 ) 2 d 3 L max log 3 n , we have (D.16) on E c L for every L ∈
[ L max ] .
Accordingly, for every L ∈ [ L max ] and every (cid:96) ∈
[ L ] we deﬁne truncation events G (cid:96)L by
[Equation]
D.17)
We have G (cid:96)L ∈ F (cid:96) , and a union bound and (D.14) imply
[Equation]
where the second line uses the fact that E (cid:96) (cid:48) ∈ F (cid:96)
[Equation]
In particular, taking expectations recovers
[Equation]
In addition, by (D.16) we have on E c (cid:96)
[Equation] [Equation]
where the ﬁnal inequality requires L ≥ 2 .
Thus, when L ≥ 2 , we have on G (cid:96)L that
[Equation]
where the ﬁnal inequality holds when d ≥ 1 and n ≥ ( C 2 c 0 / 128 C 3 ) 2 / 3 L 2 / 3 .
Similarly, when
L ≥ 2 , on E c (cid:96) − 1 we have by (D.15)
[Equation]
where the second inequality holds when d ≥ 1 and n ≥ ( C 4 c 20 / 2 17 C 23 ) 1 / 3 L 2 / 3 ; and in the same setting we have by (D.13)
[Equation]
where the second inequality holds when n ≥ ( C 2 c 0 / 128 C 1 ) L .
In particular, if we enforce these conditions with L max in place of L , we have that (D.19) to (D.21) hold for all 2 ≤ L ≤ L max (with (D.20) and (D.21) holding on E c (cid:96) − 1 ).
We begin the recursive construction.
We will enforce C 0 = max { 4 πC 3 , 6 K 0 } for the absolute constant in the deﬁnition of E (cid:96) .
The main tool is the elementary identity
[Equation]
which allows us to leverage an inductive argument provided we can control P
[ E L ∩ (cid:84)
[Equation]
] , the probability that the L -th angle deviates above its nominal value subject to all prior angles being controlled.
The case L = 1 can be addressed directly: (D.14) gives
[Equation]
and as long as d ≥ 1 and n ≥ ( C 2 /πC 3 ) 2 / 3 , this implies
[Equation]
This gives a suitable measure bound on E 1 , after choosing d ≥ 1 and n ≥ e so that d 3 log 3 n ≥ d log n .
We now assume L ≥ 2 .
By the triangle inequality, we have
[Equation]
, (D.24) and we therefore have for any t >
P
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
=
[Table]
[Equation]
By (D.21)
[Equation]
we have
[Equation]
For the remaining term, we have by the triangle inequality
[Equation]
F (cid:96) − 1 (cid:105) − E (cid:2) ∆ (cid:96)L (cid:12)(cid:12) F (cid:96) − 1 (cid:3)(cid:12)(cid:12)(cid:12) .
(D.27)
By (D.17), an integration of (D.14), and a union bound, we have
P (cid:34) 1 (cid:84) (cid:96) ∈
[ L − 1] E c (cid:96) (cid:12)(cid:12) (cid:12) (cid:12) (cid:12)
L (cid:88) (cid:96) =
∆ (cid:96) L − ∆ (cid:96) L 1 G (cid:96)L (cid:12)(cid:12) (cid:12) (cid:12) (cid:12) >(cid:35) ≤ P   (cid:91) (cid:96)
∈
[ L ] (cid:40)(cid:12)(cid:12) ∆ (cid:96)L (cid:12)(cid:12) > 2 C(cid:114)
d log n n ˆ ν (cid:96) −
+ ( c 0 / 64)( L − (cid:96) )
ˆ ν (cid:96) − 1 + 2 Cn(cid:41) 
≤ C 5 Ln − c 1 d , (D.28)
[Table]
[Equation]
where the ﬁrst line uses linearity of the conditional expectation and the triangle inequality for sums and for the integral; the second line uses the worst-case bound of π on the magnitude of the increments ∆ (cid:96)L ; the third line uses (D.17); and the fourth line uses a union bound, E (cid:96) − 1 ∈ F (cid:96) − 1 , and (D.14).
Multiplying both sides of the ﬁnal bound by 1 (cid:84) (cid:96)
∈
[ L − 1] E c (cid:96) , we conclude
[Equation]
For the remaining term in (D.27), we ﬁrst observe
[Equation]
where the ﬁrst line uses the centering property of the L 2 norm
[Equation]
and the second line uses (cid:0) ∆ (cid:96)L (cid:1) 2 ≥to drop the indicator for G (cid:96)L .
For notational simplicity, we deﬁne
[Equation]
so that our previous bound and (D.20) imply
[Equation]
This implies that for any t >
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Setting t = 3 K 0 (cid:113) d 3 log 3 n/nL , we obtain
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
D.30) where the last line uses the bounds L ≥ 1 and d log n/ (1 + d log n ) ≥ 12 if d ≥ 1 and n ≥ e .
Combining (D.28) to (D.30) in (D.27) via a union bound, we obtain
P
[Equation]
Applying this result and (D.26) to (D.25) via a union bound, we obtain
P
[Equation]
If d ≥ 2 /c 1 and n ≥ L max , we have C 5 Ln − c 1 d ≤ C 5 n − c 1 d/ 2 ; under these condition on d and n , we have πC 5 Ln − c 1 d ≤ πc 5 n − 1 , and so πC 5 n − c 1 d/ 2 + (256 C 1 /c 0 )(1 + log L )(log n ) /n ≤
C (1 + log L )(log n ) /n ; and if d ≥ 1 and n ≥ L max , we have
[Equation]
n provided n ≥ C (cid:48)
( C/ 3 K 0 )
2 L max log L max .
Under these conditions, our previous bound simpliﬁes to
[Equation]
In particular, applying this bound to (D.22), we have shown that for any L ≥
[Equation]
Unraveling the recursion with (D.23) (and worst-casing the constants there), we conclude
[Equation]
which proves the claim, after possibly choosing n to be larger than another absolute constant multiple of L max to remove the leading L factor.
D.2. B ACKWARD F EATURE C ONTROL
Having established concentration of the feature norms and the angles between them, it remains to control the inner products of backward features that appear in Θ .
The core of the technical approach will once again be martingale concentration.
We establish the following control on the backward feature inner products:
Lemma D.4.
Fix x , x (cid:48) ∈ S n 0 − 1 and denote ν = ∠
( x , x (cid:48) ) .
If n ≥ max { KL log n, K (cid:48)
Ld b , K (cid:48)(cid:48) } , d b ≥ K (cid:48)(cid:48)(cid:48) log L for suitably chosen K, K (cid:48) , K (cid:48)(cid:48) , K (cid:48)(cid:48)(cid:48) then
[Equation]
If additionally n, L, d satisfy the requirements of lemmas D.3 and E.16, we have
P
[Equation]
where ϕ ( i ) denotes i applications of the angle evolution function deﬁned in lemma E.2, and c > 0 , C are absolute constants.
Proof.
For (cid:96) ∈
[ L ] , write F (cid:96) for the σ -algebra generated by all the weights up to layer (cid:96) in the network, i.e., W 1 , . . .
, W (cid:96) , with F
given by the trivial σ -algebra.
Consider some (cid:68) β (cid:96) (cid:48)
( x ) , β (cid:96) (cid:48) ( x (cid:48) ) (cid:69) for 0 ≤ (cid:96) (cid:48) ≤ L − 1 .
Deﬁning
[Equation]
for (cid:96) ∈ { (cid:96) (cid:48) + 1 , . . .
, L } , and setting Γ (cid:96) (cid:48)
+1: (cid:96) (cid:48)
+
( x ) =
I , B (cid:96) (cid:48) : (cid:96) (cid:48) xx (cid:48)
=
12 I , we deﬁne the event ˜
E
[Equation]
B
[Equation]
∩
[Equation]
Since (cid:68) β (cid:96)
[Equation]
( x ) , β (cid:96) (cid:48) ( x (cid:48) ) (cid:69) =
W L +1 B L : (cid:96) (cid:48) xx (cid:48)
W L +1 ∗ is a Gaussian chaos in terms of the W L +1 variables (and recalling W L +1 ∼ iid N (0 , I ) ) and ˜ E LB is F L -measurable, the Hanson-Wright inequality (lemma G.4) gives
[Equation]
Using lemma D.28 to bound P (cid:104)(cid:16)
˜ E L +
[Table]
(cid:17) c (cid:105) from above gives
P
[Table]
(D.31) for appropriately chosen constant, if t (cid:46) n log n/L .
Choosing t = n/L in the bound above and using the bound on tr (cid:104) B L :
[Equation]
xx (cid:105) from lemma D.28 we obtain
P
[Equation]
for appropriate constants.
Taking a union bound over the possible values of (cid:96) (cid:48) proves the ﬁrst part of the lemma.
We next control (cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
using martingale concentration (in a similar manner to the control of the angles established in previous sections).
We write tr
[Equation]
(note the change in the indexing).
Consider the ﬁltration F 0 ⊂ · · · ⊂ F L and adapted sequence
[Equation]
so that
[Equation]
We begin by considering the ﬁrst term in the sum since it takes a distinct form.
Denoting by W (cid:96)
[Equation]
+1 (: ,i ) the i -th column of W (cid:96) (cid:48)
+1 , rotational invariance of the Gaussian distribution gives
[Equation]
and hence
[Equation]
where ( g 1 , g 2 ) ∼ N (0 , I ) .
Moving to spherical coordinates, we obtain
[Equation]
We now note that conditioned on F (cid:96) (cid:48) , tr (cid:104) P W 0(: , 1) > 0 P W 0(: , 1) cos ν + W 0(: , 2) sin ν> 0 (cid:105) is a sum of n independent variables taking values in { 0 , 1 } .
An application of Bernstein’s inequality for bounded random variables (lemma G.3) then gives
[Equation]
√
[Equation]
for some c (cid:48) , where we used the fact that the angle evolution function ϕ is bounded by π/ 2 .
Note also from Lemma D.3 that
[Equation]
for some constant c , where we assumed d > K log n for some K .
Having controlled the ﬁrst term in (D.35)
[Equation]
we now proceed to bound the remaining terms.
We deﬁne events
[Equation]
[Table]
(which from lemma D.28 hold with high probability).
Note that as a consequence of the ﬁrst event in E (cid:96) : (cid:96) (cid:48)
B the angle ν (cid:96) is well-deﬁned.
Note that E (cid:96) : (cid:96) (cid:48)
B is F (cid:96) − 1 -measurable.
We will ﬁrst control (D.35) by considering each summand truncated on the respective event E (cid:96) : (cid:96) (cid:48) B .
Our task is therefore to control
[Equation]
Since
[Equation]
the ﬁrst sum is over a zero-mean adapted sequence and hence a martingale, and can thus be controlled using the Azuma-Hoeffding inequality.
We will ﬁrst show that the remaining term is small.
We begin by computing
[Equation]
where we used the fact that E (cid:96) : (cid:96)
[Equation]
− 1 and is thus independent of W (cid:96) .
There exists a matrix R such that
[Equation]
Rotational invariance of the Gaussian distribution gives
[Equation]
where we denote by W (cid:96) (: ,i ) the i -th column of W (cid:96) .
Deﬁning ˜ B (cid:96) − 1: (cid:96) (cid:48) xx (cid:48)
= RB
[Equation]
− 1: (cid:96) (cid:48) xx (cid:48)
R ∗ we have
E tr (cid:104) B
[Equation] [Equation]
xx
[Equation]
|
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
If i / ∈ { 1 , 2 } we get
[Equation]
If i ∈ { 1 , 2 } then the Q (cid:96) − 1 ij (cid:54) = 0 only if j ∈ { 1 , 2 } .
In these cases we have
[Equation]
where ( g 1 , g 2 ) ∼ N (0 , I ) .
Moving to spherical coordinates, we obtain
[Equation]
and similarly
[Equation]
Combining terms and using tr (cid:104) B
[Equation]
(cid:105) we obtain
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]

[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
12 + ˜ B (cid:96) − 1: (cid:96) (cid:48) xx (cid:48) 21 (cid:17)  