Convergence beyond the over-parameterized regime using Rayleigh quotients
Abstract
In this paper, we present a new strategy to prove the convergence of deep learning architectures to a zero training (or even testing) loss by gradient  ow.
Our analysis is centered on the notion of Rayleigh quotients in order to prove Kurdyka- Ł ojasiewicz inequalities for a broader set of neural network architectures and loss functions.
We show that Rayleigh quotients provide a uni  ed view for several convergence analysis techniques in the literature.
Our strategy produces a proof of convergence for various examples of parametric learning.
In particular, our analysis does not require the number of parameters to tend to in  nity, nor the number of samples to be  nite, thus extending to test loss minimization and beyond the over-parameterized regime.
Introduction
In order to understand the performance of vastly over-parameterized networks, various works have investigated the properties of neural tangent kernels (NTK, see Jacot et al., 2018) and their eigenspaces.
While the study of these spectra has led to proofs of convergence to global minima despite the nonconvexity of the problem, these analyses typically rely on an over-parameterization assumption, or even in  nite-width limits, casting a shadow on their applicability.
Positive-de  niteness of the
NTK in particular, granted by the in  nite-width limit, does not hold with  nite width and a growing number of samples, despite observed successes of neural networks in this regime.
We provide a (toy) counter-example in dimension two to better outline this issue and  x this  aw by re-centering the discussion on Rayleigh quotients, corresponding to  xed directions, rather than positive de  niteness, i.e. uniformly bounding in all directions.
We give several ideas to obtain bounds on Rayleigh quotients, and provide non-trivial examples for each of the presented ideas, including a recovery of known results, but also a new convergence speed guarantee for the multi-class logistic regression.
Overview.
In a typical supervised learning task, one is given a training dataset of n ∈ N labeled samples D
=
(( x i , y i ) ∈ R d × R ) i ∈ [ n ] , and a parametric model with m ∈ N parameters, f : R m × R d → R .
The task is to  nd parameters  tting the training data, i.e.  nd θ ∗ ∈ R m such that ∀ i ∈ [ n ] , f ( θ ∗ ; x i )
≈ y i .
Aggregating these into a single vector F : θ → f θ =
( f ( θ ; x i ))
i ∈
[ n ] , this becomes a satisfaction of a system of equations F ( θ ) ≈ y ∈ R n .
After choosing a functional loss ℓ : R n → R + , one can learn the associated parameters by gradient  ow ∂ t
θ
= − DF ( θ ) T · ∇ ℓ ( F ( θ )) , where the jacobian of the parameterization F is a matrix DF ( θ ) ∈ R n × m .
This corresponds exactly to the usual practice of de  ning a parametric function F , a functional loss ℓ , and training by gradient  ow on the parameters to minimize the parametric loss L = ℓ ◦ F .
The question is then when does this algorithm converge, and how fast ?
Our focus is on the regime of  nitely many parameters ( m ∈ N ) and large data ( n → + ∞ ), where the over-parameterization arguments ( m ≫ n ) are insuf  cient.
Context.
Early arguments for the proof of convergence of this system to a loss of zero revolved around strong convexity hypotheses on the loss [see Boyd and Vandenberghe, 2004, Section 9.3.1].
However the parameterization F , typically as a neural network, leads to non-convex parametric losses L even when the functional loss ℓ is convex, sometimes even parametric losses that are not locally quasi-convex [for details, see Liu et al., 2022].
Recently, a common solution has been the leverage of Polyak- Ł ojasiewicz inequalities ∥∇ L ( θ )
∥ 22 ≥ µ L ( θ ) , which grant linear convergence by integrating with Grönwall’s lemma since for gradient
 ows it holds − ∂ t L ( θ )
=
∥∇ L ( θ ) ∥( thus for µ ∈ R ∗ + , − ∂ t L ( θ ) ≥ µ L ( θ )
⇒ L ( θ t ) ≤ L ( θ 0 ) exp( − µt )) .
For examples in continuous time, see Chizat [2020, Theorem 3.3 and 3.4].
Other results with discrete time include Arora et al.
[2019, Theorem 4.1], Oymak and Soltanolkotabi [2019, Theorem 2.1], Liu et al.
[2020, Theorem 5.1] and Liu et al.
[2022, Eq (3)] .
Generally speaking, discretized versions with suf  ciently small learning rate have very similar dynamics, at the cost of some local smoothness assumption, and similarly, stochastic versions can leverage the same Ł ojasiewicz inequalities to prove convergence rates, so the continous-time dynamics proof can be viewed as a  rst step in the analysis of these more complex cases.
These inequalities ensure that there are no critical points that are not global minima, and can hold even for non-convex losses L , although they can be hard to prove.
The behavior of the dynamical system ∂ t θ = −∇ L ( θ ) has been shown to be closely tied with the eigenspaces of the Neural Tangent Kernel (NTK
) matrix K ( θ ) =
DF ( θ ) · DF ( θ ) T ∈ R n × n , introduced precisely, decrease loss is − ∂ t L ( θ )
= ∇ ℓ ( f θ ) T · K ( θ ) · ∇ ℓ ( f θ ) .
As an example, for the quadratic loss, the gradient satis  es ∥∇ ℓ ( f θ ) ∥ 22 = 4 ℓ ( f θ ) = 4 L ( θ ) , such that a positive de  niteness condition K ( θ ) ⪰ µ > 0 guarantees the Polyak- Ł ojasiewicz condition − ∂ t L ( θ ) ≥ 4 µ L ( θ ) , and thus by integration, convergence to zero with a linear convergence speed.
Several works, starting with Jacot et al.
[2018, Proposition 2] but also Du et al.
[2018], have shown that the smallest eigenvalue of this K ( θ ) operator is indeed strictly positive if the network is suf  ciently overparameterized ( m ≫ n ).
Subsequent papers have also anayzed how overparameterized the network needs to be for this argument to hold, with interesting asymptotic bounds on the number of parameters required [Ji and Telgarsky, 2020, Chen et al., 2021].
Challenges.
However, this argument for convergence is bound to fail when there are fewer parameters than datapoints ( m < n ).
In particular, for a  xed number of parameters m ∈ N , it is impossible to have both n → + ∞ and λ min ( K ( θ )) > 0 , since K ( θ ) ∈ R n × n has rank m < n by de  nition.
As argued by Liu et al.
[2022, Proposition 3] for the quadratic loss ( ℓ : f → ∥ f − y ∥ 22 , satisfying ∇ ℓ ( f θ ) =
( f θ − y ) ∈ R n ), this implies that for underparameterized systems, the Ł ojasiewicz condition cannot be satis  ed for all y , since inf u ∈ R n u T K ( θ ) u/u T u = λ min ( K ( θ ))
= 0 .
Nonetheless, if some knowledge y i = f ∗ ( x i ) for some f ∗ ∈ F 0 is available, then it is suf  cient to show that inf u ∈ Y 0 u T K ( θ ) u/u T u > 0 , where Y
= { ( f ∗ ( x i ) − f θ ( x i ))
i , f ∗ ∈ F 0 } ⊆
R n is only a subset of the responses R n on which the smallest eigenvalue of the NTK might be positive.
Bounding the eigenvalues of the NTK away from zero is suf  cient, but not necessary, and for cases where the smallest eigenvalue is zero, one can bound the Rayleigh quotient of the gradient and enjoy similar guarantees despite the null eigenvalue(s).
Although stated differently in their respective context, previous uses of this restricted eigenvalue argument can be found for instance in Nitanda and Suzuki [2019, Assumption A4: response is NTK-separable], or Arora et al.
[2019, Section 6, bounded inverse-NTK response] .
We show how the argument used in these particular cases can be extended to a broader setting, and introduce tools to make calculations easier and obtain such guarantees.
Rayleigh quotient bounds enable convergence guarantees in the underparameterized regime ( m < n ) and in particular, for  xed number of parameters m , the guarantees hold even when the number of datapoints grows ( n → + ∞ ) and the domain becomes continuous.
Letting n → + ∞ requires a slightly different formalism than the vectors and matrices used in this introduction, we will therefore use functional spaces in the following, and the usual notations of differential geometry, with parameters in indices for instance.
Contrary to results such as Arora et al.
[2019], Du et al.
[2019], the formulation using functional spaces, from Jacot et al.
[2018], extends to the case where datapoints are arbitrarily close and even identical, allowing guarantees on the expected loss with respect to a continuous distribution and not just the empirical loss measured on  nitely many well-separated samples.
In particular, these conditions need not rely on properties satis  ed only with high-probability by random initialization when m → + ∞ , they can be proven even for  xed initialization and m ∈ N .
Lastly, our analysis ties together in a more general framework the convergence arguments formulated in the functional space [Du et al., 2018, 2019] studying dynamics of the network response, and similar arguments formulated in the parameter space [Li and Liang, 2018, Zou et al., 2020], by centering the work on the singular values of the network differential DF ( θ ) ∈ R n × m rather than the functional-space DF ( θ ) · DF ( θ ) T ∈ R n × n or parameter-space DF ( θ ) T · DF ( θ ) ∈
R m × m kernels.
Contributions.
We provide de  nitions in Sec. 2, then present Kurdyka- Ł ojasiewicz inequalities,
Rayleigh quotients, and their link in Sec.
We show in Sec.
4.1 that this recovers previously known linear bounds for the quadratic case.
We illustrate a two-dimensional counterexample to the NTK positive-de  niteness in Sec. 4.2, and how to overcome it with Rayleigh quotients.
In Sec.
4.3 we prove a new bound on logistic regression obtained by the same technique.
In Sec.
4.4 and Sec. 4.5, we outline arguments of convergence in more realistic settings and highlight future challenges.
De  nitions for gradient  ows and neural tangent kernels
Let X be a set with no particular structure.
We consider the problem of learning a target function f ∗ : X → R , by having access only to samples ( x, f ∗ ( x ))
∈
X × R , where x ∼ D are random samples from a probability distribution D on X .
Let F
= R X be the vector space of functions from X to R .
The setting presented in the introduction corresponds to X being  nite containing the examples x
i so that functions are represented as vectors f = ( f ( x i ))
i ∈ [ n ] and D is the empirical measure on X .
De  nition 2.1.
A network map is a function F : Θ → F , from Θ a vector space of  nite dimension equipped with an inner product ⟨ · , · ⟩ Θ , to F equipped with the topology of pointwise convergence.
To avoid confusions as much as possible, we will reserve lowercase letters ( f, g, h ) for functions in F , and the uppercase F for network maps.
We will usually put the parameters in index, and inputs between parenthesis, so that for θ ∈ Θ , the function f θ : X → R sends inputs x ∈ X to outputs f θ ( x ) ∈ R .
Readers familiar with differential geometry will note that the assumption that Θ is a vector space is a simpli  cation, and could be relaxed for instance to a differentiable manifold.
However, we are interested in easily readable results closest to applications, and this assumption will avoid cumbersome discussions on the parameter manifold’s tangent space, and keep results readable with only some background in linear algebra.
In all the examples, it is suf  cient for our needs to set Θ = R m with canonical inner product and ∥ · ∥ Θ = ∥ · ∥ 2 , for some number of parameters m ∈ N .
De  nition 2.2 ( D -seminorm) .
Any probability distribution D on X induces on F a bilinear symmetric positive semi-de  nite form ⟨ · , · ⟩ D : F × F → R , de  ned for ( g, h ) ∈ F × F as
[Equation]
The associated seminorm ∥ · ∥ D : F → R + is de  ned as ∥ g ∥ 2 D = ⟨ g, g ⟩ D =
E x ∼ D
 g ( x ) .
This seminorm does not in general separate points, it is therefore not a norm on F .
In particular, if D does not have full support, then there are non-null functions g ∈ F with null seminorm ∥ g ∥ D = 0 .
De  nition 2.3 (Gradient  ow) .
A gradient  ow with respect to the differentiable loss L : Θ → R + is an absolutely continuous curve θ : R + →
Θ satisfying the differential equation ∂ t θ =
−∇
L ( θ ) .
Additionally, we say that a gradient  ow is trivial if L ( θ 0 ) = 0 , since it implies that for all t, θ t
= θ 0 .
For U ⊆ Θ , if θ : R + → Θ is a gradient  ow such that θ ( R + ) ⊆
U then we write just θ : R + → U .
A common choice for regression with target f ∗ ∈ F is the quadratic loss L : θ → ∥ F ( θ ) − f ∗ ∥ 2 D .
If a network map F : Θ → F is differentiable for the pointwise convergence, we will write d F θ : Θ → F for the differential of F at θ ∈ Θ , with parameters in index for shortness.
Evaluation at x ∈ X and derivation with respect to θ ∈ Θ commute, easing computations (see Appendix A.2.2).
We write the corresponding gradient ∇ F θ : X → Θ , de  ned by ⟨∇ F θ ( x ) , ν ⟩ Θ = ( d F θ · ν )( x ) for all x ∈ X and ν ∈ Θ .
De  nition 2.4 (Neural Tangent Kernel, NTK form) .
A differentiable network map F : Θ → F de  nes at every point θ ∈ Θ a kernel function K θ : X × X
→ R as
[Equation]
This function induces a bilinear symmetric positive semi-de  nite form K ⋆θ : F × F → R as
[Equation]
In exponent notation, this bilinear form has signature K ⋆θ : R X × R X
→ R , while the kernel
K θ ∈ R X ×
X is an n × n matrix when X is  nite with n ∈ N elements.
Importantly, the (primal) kernel K θ is independent of the distribution D , while the (dual) kernel form K ⋆ θ changes with D .
De  nition 2.5 ( D -compatibility, functional gradient) .
A function ℓ : F → R + is said D -compatible if ∀ ( f, g ) ∈ F × F , it holds that ( f = g ) D -almost everywhere implies ℓ ( f ) =
ℓ ( g ) .
Moreover, if ℓ is D -compatible and differentiable, we say ∇ ℓ : F → F is a gradient of ℓ if it satis  es
[Equation]
This formalizes the idea that the loss depends only on the training samples, and the use of a gradient simpli  es the following statements.
When it exists, the functional gradient is usually not unique, for it is de  ned only D -almost everywhere.
See Appendix A.2.1 for some examples of conditions under which it is well de  ned (for instance D has  nite support, or ℓ is the expectation of a pointwise loss).
Rayleigh quotients to obtain Kurdyka- Ł ojasiewicz inequalities 3.1 Context: Kurdyka- Ł ojasiewicz inequalities for convergence
All convergence proofs presented in this paper rely on inequalities introduced by Kurdyka [1998] of the form of Proposition 3.1.
These are used for instance to prove  nite length of trajectories in dynamical systems (see e.g. Bolte et al.
[2007, Corollary 4.1]), and suf  cient to prove convergence to a loss of zero even for non-convex losses.
We will therefore direct all later efforts to the construction of such inequalities.
This was introduced as an extension to the Polyak- Ł ojasiewicz inequalities for linear convergence [see e.g. Nguyen, 2017, Section 1.3 for examples], to more general dynamics, and the proof of the following proposition is a simple application of the chain rule to φ ◦ L (see A.2.3).
Proposition 3.1 (Convergence by Kurdyka- Ł ojasiewicz inequality) .
Let U ⊆
Θ .
If L : U → R + is such that there exists µ ∈ R ∗ + and a strictly increasing differentiable function φ : R ∗ + → R satisfying
[Equation]
Then all non-trivial gradient  ows θ : R + → U of L satisfy ∀ t ∈ R + , L ( θ t ) ≤ φ − 1 ( φ ( L ( θ 0 ))
− µt )
Moreover, if such a  ow exists, then inf θ L ( θ ) = 0 and φ ( u ) →
−∞ if u → 0 (see Appendix A.2.4).
The central idea, similar to the one used in the following sections, is that a desingularizing function φ : R ∗ + → R transports the loss evolution L ( θ ) :
I → R ∗ + in dom( φ ) =
R ∗ + to the space Im( φ ) =
R where the evolution is easy to understand, since ( φ ◦ L )( θ ) is bounded by an af  ne function of time.
The desingularizing function provides a way to transfer the understanding of the convergence in the image of φ back to the domain of φ , where the loss evolution is a little more complicated.
The condition is also sometimes written ∇ L · ∇ L ≥ ψ ( L ) , where ψ : R + → R + is ( ψ ( u )) − 1 =
d φ u .
For the case of a linear convergence speed guarantee, the Polyak- Ł ojasiewicz condition from the introduction (i.e. − ∂ t L ( θ ) =
∥∇ L ( θ ) ∥
22 ≥ µ L ( θ ) ) corresponds to the choice φ : u → log( u ) .
To accurately describe systems with more intricate dynamics, more complicated choices of φ may be necessary, see the case of logistic regression in Sec.
4.3 for one such example.
3.2 Contribution: Kurdyka- Ł ojasiewicz inequalities
by composition
De  nition 3.2 (Rayleigh quotients of bilinear maps) .
Let ( V, ∥ · ∥ V ) and ( W, ∥ · ∥ W ) be two vector spaces equipped with seminorms, and let A : V × W → R be a bilinear map.
Then for ( x, y ) ∈ V × W such that ( ∥ x ∥ V ∈ R + \ { 0 } ) , and ( ∥ y ∥ W ∈ R + \ { 0 } ) , de  ne the Rayleigh quotient
[Equation]
With a symmetric map A : V × V → R , the Rayleigh quotient R( A ; x, x ) is a convex combination of the eigenvalues of A (which are real-valued), whose weighting depends on x .
Moreover, the minimal value is attained when x is an eigenvector corresponding to the minimal eigenvalue, and λ min ( A ) = inf x ∈ V \{ 0 } R
( A ; x, x ) .
Lastly, when the map is an inner product, then the Rayleigh quotient R ( ⟨ · , · ⟩ Θ ; a, b ) = ⟨ a, b ⟩ Θ / ∥ a ∥ Θ ∥ b ∥
Θ is a form of cosine similarity.
The most common usage is with x = y , but the asymmetric de  nition will be necessary later for the variational bound.
Proposition 3.3 (Kurdyka- Ł ojasiewicz inequality by composition) .
Let F : Θ → F be a differentiable network map, and K θ the associated neural tangent kernel (by Def 2.4).
Let U ⊆
Θ be a subset of parameters and F U = F ( U ) ⊆
F its image by F .
Let ℓ : F U → R + be a D -compatible differentiable loss with gradient ∇ ℓ : F U → F whose seminorm is  nite ∀ f ∈ F U , ∥∇ ℓ f ∥ D < + ∞ .
Assume that there exists a strictly increasing differentiable φ : R ∗ + → R satisfying
[Equation]
If the K ⋆θ -Rayleigh quotient of the gradient of ℓ is bounded below, i.e. if there exists µ ∈ R ∗ + such that
[Equation]
Then, for L = ( ℓ ◦ F ) : U → R + , it holds
[Equation]
The proof of this statement is deferred to Appendix A.3.1, and similar to the usual NTK arguments.
If K ⋆ θ is µ -uniformly conditioned, then in particular K ⋆ θ ( ∇ ℓ f , ∇ ℓ f ) ≥ µ ⟨∇ ℓ f , ∇ ℓ f ⟩ D , which is exactly the Rayleigh quotient condition.
The main difference is that it is not necessary to require uniform conditioning, it is suf  cient for this property to hold on any subspace containing the gradient (and in particular the one-dimensional subspace de  ned by the gradient, i.e. the Rayleigh quotient).
Kurdyka- Ł ojasiewicz (K Ł ) inequalities provide a reasonable path to convergence bounds, outside the usual convex framework.
However, they can still be very dif  cult to obtain.
This proposition splits the parametric-space K Ł inequality into a functional-space K Ł inequality which is easier to obtain (trivial for quadratic losses, see Sec.
4.1; available for cross-entropy for instance, see Sec. 4.3) and a
Rayleigh quotient bound, which is the focus of the following propositions.
Similarly, we provide hereafter several variational forms that can help break the Rayleigh quotient bounding problem down into smaller blocks that can be easier to compute independently before reassembling.
Proposition 3.4 (Variational bound) .
Let F : Θ → F be a differentiable network map, K θ the associated neural tangent kernel (by Def 2.4), and θ ∈ Θ .
If h ∈ F satis  es ∥ h ∥ D ̸ = 0 , then it holds
[Equation]
Where d F ⋆θ is the bilinear form ( ν , h ) → ⟨ d F θ · ν , h ⟩ D associated with the linear operator d F θ .
This property is particularly useful to avoid dealing with the square of the differential, and instead obtain lower-bounds on the Rayleigh quotient by carefully selecting (suboptimal) inputs ν ∈ Θ \ { 0 } .
Proposition 3.5 (Split cosine - singular value) .
Let F : Θ → F be a differentiable network map, K the associated neural tangent kernel, θ ∈ Θ , and h ∈ F such that ∥ h ∥ D ̸ = 0 .
If there exists a subspace Θ 0 ⊆
Θ and some µ ∈ R ∗ + such that there exists ν ∈ Θ 0 satisfying R( ⟨ · , · ⟩ D ; d F θ · ν , h ) ≥ µ , then for λ = inf ν ∈ Θ 0 ∥
d F θ · ν ∥ 2 D / ∥ ν ∥ 2 Θ ∈ R + , it holds R( K ⋆θ ; h, h ) ≥ µ 2 λ .
This proposition is a trivial consequence of the following one, but is easier to parse while still making apparent the distinction between a geometric quantity µ and the singular value λ .
See Sec.
4.2 for an example in dimension two, where µ is de  ned only by the angle between the gradient and the lemniscate’s tangent, independently of the parameterization.
Observe on the other hand that as λ , the speed at which the lemniscate is traveled, changes, so does the gradient  ow’s convergence speed.
Proposition 3.6.
Let F : Θ → F be a differentiable network map, θ ∈ Θ , and h ∈ F s.t.
∥
h ∥ D ̸ = 0 .
Let k ∈ N ∗ .
Let ( a i ) i ∈
[ k ] ∈ ( Θ \ { 0 } )
k and ( g i ) i ∈ [ k ] ∈ ( F \ ( ∥ · ∥ D ) − 1 (0))
k .
If h ∈ Span( g ) , then
[Equation]
where the smallest singular value of A ∈ R k × k is λ min ( A ) =
min u ̸ =
u T Au/u T u (resp. max ).
If the vectors ( a, g ) are taken orthogonal and such that d F θ · a i = σ i
g i for some σ i ∈ R , then the three matrices are the identity, and only the minimal Rayleigh quotient remains.
If they are chosen only approximately orthogonal, then a corresponding multiplicative penalty is incurred.
The proofs of the preceding three propositions are deferred to Appendix A.3.2, A.3.3 and A.3.
respectively.
Case studies 4.1 Linear models with quadratic loss, recovering known bounds
As a sanity check and simple  rst contact with the variational bound, we consider a model linear in its parameters, with quadratic loss, and recover the (known optimal) linear convergence rate.
This proposition is the continuous time form of Karimi et al.
[2016, Theorem 1].
Proposition 4.1 (Convergence of quadratic-loss linear models) .
Let X
=
Θ = R d , and F : Θ → F , be the linear network map F : θ → f θ de  ned by f θ ( x ) = ⟨ x, θ ⟩ .
Let f ∗ : X → R be a linear function.
Let L : Θ → R + be the quadratic loss L : θ → ∥ F ( θ ) − f ∗ ∥ 2 D where D a distribution over X such that L is well-de  ned and  nite.
If θ : R + → Θ is a gradient  ow of L , then for all t ∈ R + , it holds L ( θ t ) ≤ L ( θ 0 )
e − 4 λ +min ( A ) t , where A = E x ∼ D
 xx T  ∈
R d × d is the (uncentered) covariance matrix of the samples, and λ +min ( A ) its smallest non-null eigenvalue.
Moreover, there exists D such that this bound is an equality.
The idea is to apply Proposition 3.3.
The functional Kurdyka- Ł ojasiewicz inequality is immediate, and we bound the Rayleigh quotient with Proposition 3.5 applied to the subspace Θ
=
Ker( A ) ⊥ .
Proof.
Let ℓ : F → R + , f → ∥
f − f ∗ ∥ 2 D be the functional-space quadratic loss, whose gradient ∇ ℓ f = 2( f − f ∗ ) satis  es the Polyak- Ł ojasiewicz inequality ∥∇ ℓ f ∥ 2 D ≥ 4 ℓ ( f ) .
Hence, let us show
L ( θ ) ̸
= 0 ⇒ R
( K ⋆θ ; ∇ ℓ F ( θ ) , ∇ ℓ F ( θ ) ) ≥ λ +min
( A ) , which is suf  cient by applying Proposition 3.3.
Let θ ∗ ∈
Θ be any parameter such that f ∗ =
f θ ∗ , where existence is guaranteed by linearity of f ∗ .
Observe that the loss can be written L ( θ )
= ( θ − θ ∗ ) T A ( θ − θ ∗ ) .
Let θ ∈ Θ such that L ( θ ) ̸ = 0 .
In particular, θ − θ ∗ / ∈
Ker( A ) .
Then, let Θ 0 =
Ker( A ) ⊥ .
On one hand, it follows that
[Equation]
with the maximum attained for u ∈ Ker( A ) ⊥ \{ 0 } the orthogonal projection of ( θ − θ ∗ ) to Ker( A ) ⊥ , satisfying A ( θ − θ ∗ ) =
Au and ⟨ θ − θ ∗ , u ⟩ = 0 , thus u T Au = u T A ( θ − θ ∗ ) =
( θ − θ ∗ ) T A ( θ − θ ∗ ) .
Then by de  nition inf ν ∈ Θ 0 \{ [Equation]
[Equation]
+min ( A ) .
Conclude by Proposition 3.5, with µ = 1 and λ = λ +min ( A ) .
Equality is recovered for A
= I d .
This is to be contrasted with a direct proof of the Kurdyka- Ł ojasiewicz inequality, i.e. showing that
[Equation]
Although the proof seems a bit convoluted, the interesting part here is that the original bound can be split into two (hopefully simpler) subproblems, while still allowing the use of knowledge on ( f θ − f ∗ ) , leveraged here by the assumption ( θ − θ ∗ )
∈
Ker( A ) ⊥ .
Note that knowledge of a property such as ( θ − θ ∗ ) ∈
Θ 0 ⊆
R d for any subspace Θ 0 could have been used to eliminate any eigenvalues of A on Θ ⊥ 0 , including strictly positive eigenvalues, there is nothing speci  c to Ker( A ) ⊥ other than the existence of the prior knowledge ( θ − θ ∗ )
/ ∈
Ker( A ) granted by L ( θ ) ̸ = 0 .
4.2 Lemniscate-constrained optimization, singular values
We now present a toy example simple enough to allow for explicit computations and constructed to illustrate the importance of parametrization.
We consider linear functions in two dimensions where the function f ( a,b ) : R 2 → R , f ( a,b ) : ( x, y ) → ax + by is simply identi  ed with ( a, b ) ∈ R 2 .
We will still consider a quadratic loss but we now assume that the target function f ∗ =
f ( a ∗ ,b ∗ ) is linear and with ( a ∗ , b ∗ ) ∈ F
= { ( a, b ) ∈ R 2 | ( a 2 + b 2 ) 2 = a 2 − b 2 } .
Although we are looking for a two dimensional linear functions f ∗ , knowing that f ∗ ∈ F 0 reduces the "degrees of freedom".
In such a scenario in machine learning, we typically incorporate this information in the parametrization.
As a result, we now have only one parameter to estimate, i.e. Θ = R and our network maps F : R → Rwill satisfy Im( F ) = F 0 .
Note that Bernoulli’s lemniscate F 0 (pictured in Fig .2a) is neither a convex set, nor a manifold (due to the crossing at zero).
There is no "natural" parametrization of F 0 and as shown below, the chosen parametrization will matter.
For more clarity on the consequences of this parameterization, we use two parameterizations of the lemniscate F 0 :
[Equation]
The graph of these parameterizations { ( θ , F ( θ )) | θ ∈ R } ⊆ R 3 is depicted in Fig.
The  rst,
F S is differentiable 2 π -periodic and surjective, satisfying F S ([0 , 2 π ])
=
F 0 .
The second, F L is differentiable, but it is neither injective (since F L ( − 1) =
(0 , 0) =
F L (+1) ) nor surjective.
It is a punctured lemniscate
Im( F L ) =
F 0 \ { ( − 1 , 0) } , it is only dense in the lemniscate Im( F L ) =
F 0 .
[Figure]
Note that in both cases, the neural tangent kernel K θ has rank one (because there is only one parameter), thus λ min ( K ⋆θ ) = 0 by rank de  ciency but we can still prove convergence to zero loss.
To make things even more clear, we assume that all samples are lying on a line: D is a distribution supported on the one-dimensional subspace R t with t =
( u, v ) ∈ R 2 \ { 0 } .
In words, all the labeled samples are of the form z ( t, a ∗ u + b ∗ v )
∈ R 2 × R for some z ∈ R and any function f ( a,b ) with ( a − a ∗ ) u
+( b − b ∗ ) v = 0 will achieve a loss of zero.
Indeed as shown in previous section, a standard linear regression in this case converges to a loss of zero but the parameters inferred will not be on the lemniscate F 0 .
With the parametrization F S or F L , we will  nd a solution living on F 0 , namely one of the two points in ℓ − 1 (0) ∩ F 0 , as seen in Figure 2a.
[Figure]
Proposition 4.2 (Lemniscate convergence with varying speed) .
Let ( u, v ) ∈
R 2 such that u > 0 and | v | < | u | .
Let y ∈ R − such that the equation ( au + bv = y ) has exactly two solutions ( a, b ) ∈ F 0 .
Let ℓ : R 2 → R + be the quadratic loss ℓ ( a, b ) → ( au + bv − y ) 2 .
Let θ S : R + → R (resp.
θ L : R + → R ) be a gradient

ow with respect to the loss ℓ ◦ F S (resp.
ℓ ◦ F L ) such that θ (0) = 0 .
Then there exists a constant µ 0 ∈ R ∗ + such that it holds ℓ ( F S ( θ S ( t )))
≤ ℓ (0) exp( − 4 µ 20 λ ∗ S t ) and ℓ ( F L ( θ L ( t ))) ≤ ℓ (0) exp( − 4 µ 20 λ ∗ L t ) , where λ ∗ S = 12 and λ ∗ L = ∥∇ F L ( θ ∗ L ) ∥ 22 , for θ ∗ L =
lim t θ L ( t ) .
The sketch of this proof is given in Appendix A.4.1.
For the numerical values taken in Fig.
2a, we have λ ∗ L ≈ 4 .
× 10 − 3 showing that our bounds capture the speed of convergence.
The idea is as previously, to use the quadratic loss Polyak- Ł ojasiewicz property ( ∥∇ ℓ ∥ 22 ≥ 4 ℓ ) that will grant linear convergence provided we can show R( K ⋆θ ; ∇ ℓ F ( θ ) , ∇ ℓ F ( θ ) ) ≥ µ 20 λ ∗ S for all θ ∈ θ S ( R + ) (resp.
λ ∗ L for θ ∈
θ L ( R + ) ), achieved by a variational bound (Proposition 3.4) split according to Proposition 3.5.
4.3 Cross-entropy minimization with linear models
We now consider a classi  cation task with c ≥ 1 classes.
Let ∆ c = { u ∈ ( R + ) c |  i ∈
[ c ] u i = 1 } be the set of distributions over those classes.
The samples x live in X = R d and the target function is f ∗ : X → ∆ c .
Let E : R c → ∆ c , u → (exp( u i ) /  j exp( u j ))
i be the softargmax map.
Let Θ
= R c ×
d be the parameter space, and X : Θ → ( X → R c ) be the operator mapping parameters to linear functions, such that X ( θ ) : x → θ · x .
We use the parameterization F : θ → E ( X ( θ )) where E is applied pointwise.
For any  xed sample x ∈ X , we de  ne the loss for this sample as
H x : ∆ c → R + , p → −  i ∈
[ c ] f ∗ i ( x ) log( p i ) .
The complete loss used to train this model is then the logistic regression L : θ → E x ∼ D
[ H x ( F ( θ )( x ))] , for which we give a new convergence bound.
[Equation]
De  nition 4.3 (Isolation) .
A real-valued random variable Y ∈
L 1 is
κ -isolated if P ( Y ≥ E [ Y ]) ≥ κ .
All L 1 variables are κ -isolated for some κ > 0 , but we will need a notion of uniform isolation.
A random variable Y with  nite support, i.e. P ( Y = y i ) =
p i for some y ∈ R n and p ∈ ∆ n is (min i ∈
[ n ] p i ) -isolated, regardless of the values y .
This bounds the isolation of the maximal value in a sense.
Moreover, if ψ : R → R is increasing and Y is
κ -isolated, then it holds E [ ψ ( Y )] ≥ κψ ( E [ Y ]) .
We use κ = 1 /n in our experiments (see A.5.6), where n ∈ N ∗ is the number of training points.
De  nition 4.4 (Multi-class separating rays) .
We say that a parameter ζ ∈ R c × d is an ε -separating ray for the distribution D if it holds for D -almost all x ∈ X that
[Equation]
where ζ i ∈ R
d is the i -th row of ζ , i.e. if ( ζ · x ) ∈ R c has a unique maximum (with a  xed margin).
This property is invariant by rescaling of ζ and generalizes the notion of "separation margin" usual in two-class logistic regression.
If ζ is ε -separating for some ε > 0 , then for D -almost all inputs x , the softargmax classi  er f = F ( ζ ) : X → ∆ c induces a unique label i ∈ [ c ] as i = argmax j f ( x ) j .
Proposition 4.5 (Convergence speed of logistic regression) .
Let D be a distribution such that the point-loss random variable L x =
H x
( f ( x )) , where x ∼ D , is κ -isolated for all f ∈ F ( Θ ) .
Let L : θ ∈ Θ → E x ∼ D
[ H x ( F ( θ )( x ))]
∈ R
+ be the multi-class cross-entropy loss.
If there exists an ε -separating ray ζ such that inf λ ∈ R L ( λζ ) = 0 , then for all non-trivial gradient  ows θ : R + → Θ ,
[Equation]
where W 0 : R + → R + is the Lambert function, and C = log( e L ( θ 0 ) − 1) − ( e L ( θ 0 ) − 1) − 1 ∈ R .
The Lambert function W 0 is de  ned by W 0 ( x ) e W 0 ( x ) =
x , see Corless et al.
The proof is deferred to Appendix A.5.3.
The idea is to prove a functional Kurdyka- Ł ojasiewicz inequality by leveraging the isolation property, then bound the Rayleigh quotient by leveraging the separation and inf L = 0 hypotheses to obtain a parametric Kurdyka- Ł ojasiewicz inequality by Proposition 3.3.
Being a convex problem, the classical argument of Boyd and Vandenberghe [2004] gives a bound L ( θ t ) ≤ C 0 /t as long as there is a  nite optimum θ ∗ ∈ Θ .
This bound becomes vacuous ( C 0 → + ∞ ) in this setting with dirac labels, common in machine learning, because the in  mum is located “at in  nity”.
This assumption has been previously lifted (under separability in Soudry et al.
Nacson et al.
[2019] , without separability in Ji and Telgarsky [2019]) to recover the O (1 /t ) asymptotic behavior, but without explicit bounds for  nite times.
This result is consistent (see Appendix A.5.4) with the asymptotic O (1 /t ) bounds from Soudry et al.
[2018, Theorem 5] with similar hypotheses, this proposition only makes quantitative the nonasymptotic behavior of this system, and the characteristic quantities driving the convergence speed.
To do so, the separation assumption had to be made quantitative, hence the use of ε -separating rays for a  xed positive ε , where previous work used only non-quanti  ed data separation (i.e. ∃ ε , ∃ ζ s.t.
ζ is an ε -separating ray for the data), see Appendix A.5.5 for more details.
Similarly to the previous section, and contrary to the parameter-direction convergence theorems Soudry et al.
[2018, Theorem 5], Nacson et al.
[2019, Theorem 3], and Ji and Telgarsky [2019, Theorem 1.1], this proposition does not, on its own, yield any insights on implicit bias (which in  mum is reached) towards max-margin rays, additional arguments are required for this purpose.
The focus here is on the precise quanti  cation of convergence speed under separability assumptions, with continuous time.
4.4 Overparameterized two-layer networks with quadratic loss
Let X = R d , and σ : R → R be a non-polynomial Lipschitz map.
For m ∈ N \ { 0 } a number of neurons.
Let Θ ( m ) =
R m × d × R m be a parameter set and F ( m ) :
Θ ( m ) → F be the associated network map F ( m ) ( w, a ) : x →  i ∈ [ m ] a i σ ( w i · x ) , i.e. a two-layer network 1 with non-linearity σ .
Let K ⊆
X be compact, and D a distribution supported on K .
Let f ∗ ∈ F be a continuous function.
Over Θ ( m ) =
R m × d × R m , let I m be the (usual in practice) iid normal rescaled initialization with density p ( w, a ) =  i ∈
[ m ] ,j ∈ [ d ] N ( w i,j ; 0 , 1)  k ∈ [ m ] N ( a k ; 0 , 1 / √ m ) .
We write ( x ) + = max(0 , x )
Proposition 4.6.
Let ε ∈ R ∗ + , and δ ∈ ]0 , 1[ .
There exists c ∈ R ∗ + such that, for all radii R ∈ R ∗ + , there exists a neuron count m ∈ N such that with probability (1 − δ ) over initializations θ 0 ∼ I m , the quadratic loss L : θ ∈ Θ ( m ) →
∥ F ( m ) ( θ ) − f ∗ ∥ 2 D satis  es the inequality
[Equation]
Therefore, for any desired precision ε 0 ∈ R ∗ + , there exists ( m, κ ) ∈ N ∗ × R ∗ + such that with probability at least (1 − δ ) over initialization θ 0 ∼ I m , a gradient  ow θ : R + → Θ of L with θ (0)
= θ 0 satis  es ∀ t ∈ R + , L ( θ [Equation] ≤
[Equation]
+
[Equation]
[Equation]
Proof in Appendix A.7.
The idea for the proof is to use universal approximation property on compacts [Cybenko, 1989, Barron, 1993, Leshno et al., 1993], to get ∥ F ( θ ) + d F θ · ν − f ∗ ∥ 2 D ≤ ε for some ν ∈ Θ , then derive a Kurdyka- Ł ojasiewicz inequality from that with a variation of Proposition 3.5.
Knowledge of a Kurdyka- Ł ojasiewicz inequality in a ball around initialization alone is not suf  cient to show loss convergence to arbitrary precision in general, but the separable form of this inequality makes it possible, following Scaman et al.
[2022, Proposition 4.6].
This proposition shows convergence outside the vastly overparameterized regime ( m is  nite even with in  nite data), but still relies heavily on a (very) large number of neurons.
In the next section, we give a partial convergence argument using similar techniques in a much more constrained regime.
4.5 Periodic signal recovery
Let X = R .
Among functions F =
( R → R ) , we are interested in continuous periodic antisymmetric functions, which we parameterize with Θ
= R m × R m , as F : Θ → F , de  ned for ( a, ω ) ∈ Θ as
F ( a, ω ) : x →  i ∈ [ m ] a i sin( ω i x ) , and K ( a, ω ) the associated NTK at the point ( a, ω ) ∈ Θ .
The central property of this application, separating it from the most common machine learning applications, is the inability to obtain good samples.
Let R ∈ R ∗ + be a  nite window size, and de  ne the training data distribution D =
U
− R, + R ) , the uniform distribution on the interval [ − R, + R ] .
Let F 0 ⊆
F be the set of continuous periodic antisymmetric functions with period less than R .
Crucially, we are interested not just in learning the function on the interval, akin to just data retrieval, but rather in learning the function in ( R → R ) as a whole.
This problem is well de  ned, i.e. if f ∗ ∈ F 0 , then argmin g ∈ F 0 ∥
g − f ∗ ∥ 2 D = { f ∗ } .
The periodicity assumptions makes the data suf  cient to recover the target function among the hypotheses, however neither the assumption that the training and testing data distributions are identical, nor the assumption that the model has more parameters than there are data points are satis  ed.
There is in  nite data, but there is bias in the sampling.
We will rely on two properties of frequency parameters to show bounds.
First, we say that ω ∈ R m is δ -separated if inf i ̸ =
j | ω i − ω
j | ≥ δ and inf i | ω i | ≥ δ .
Then, we say that the pair ( ω , ω ∗ ) ∈
R m × R m is ε -paired
if sup i ∈ [ m ] | ω i − ω ∗
i | ≤ ε .
Moreover, let x 0 ∈ R + be the  rst zero of sinc ′′ .
( x 0 ≈ 2 . 0815 ).
Proposition 4.7 (Polyak- Ł ojasiewicz region) .
Let ( η , µ ) ∈
R ∗ + × R ∗ + such that η ≤ x 0 and η < 12 µ .
Let f ∗ ∈ F be a target, and ℓ : f ∈ F → 12 ∥ f − f ∗ ∥ 2 D the quadratic loss, with gradient ∇ ℓ f = f − f ∗ .
Assume that there exists ( a ∗ , ω ∗ )
∈
Θ such that f ∗ =
F ( a ∗ , ω ∗ ) , and ω ∗ is µ R -separated.
Then for all ( a, ω ) ∈ Θ such that ℓ ( F ( a, ω ))
̸ = 0 , ( ω , ω ∗ ) is η R -paired, and ∃ α ∈ [0 , 1] , ∀ k, a 2 k ≥ α ,
[Equation]
where with ψ = − sinc ′ , ϕ = − sinc ′′ , and H =  k ≤ m 1 k ≤ 1 + log( m ) ∈ R + , the constants are
[Equation]
Moreover, ∃ µ 0 ∈ R + , ∀ µ > µ 0 , ∃ η > 0 , s.t. κ 0 > ρ 0 .
(non degeneracy if enough periods observed)
Proof in Appendix A.6, leveraging Prop. 3.4 (variational bound) and Prop. 3.6.
This shows that when each frequency present in the signal is correctly estimated, then a gradient  ow is well-suited for  ne-tuning both frequencies and amplitudes.
There are suf  ciently few interactions to allow each neuron ( a i , ω i ) to descend towards its target ( a ∗ i , ω ∗ i ) .
If the modelling hypothesis is veri  ed (the target is a sum of sine waves), there is a  nite and small number of neurons giving a suf  cientlyparameterized system, and no need to go for vast overparamterization.
Letting the number of neurons tend to in  nity is one way to ensure there is at least one neuron in each bassin, but not the only way.
Conclusion
We have shown that Kurdyka- Ł ojasiewicz inequalities can be leveraged to prove convergence of gradient  ows to a loss of zero, even when the convergence speed is not linear.
In contrast, Polyak- Ł ojasiewicz inequalities granted by positive-de  niteness of the neural tangent kernel only covered least-squares losses enjoying linear convergence speed.
Furthermore, we have shown that by focusing on lowering-bounding Rayleigh quotients rather than all eigenvalues at once, one can prove convergence even when the neural tangent kernel is not positive-de  nite, the most striking example being the  nite-width in  nite-data regime, where the neural tangent kernel must have null eigenvalues by rank de  ciency.
We have provided several simple examples of such convergence proofs outside the vastly over-parameterized regime where there are more parameters than samples, along with tools and preliminary results that lead us to believe that obtaining the crucial Kurdyka- Ł ojasiewicz inequalities is feasible in more reasonable machine learning settings.
Acknowledgements
The authors would like to thank Thomas Le Corre, Lucas Weber, and Luca Ganassali for their help with various details of the proofs presented here, along with the anonymous reviewers, for their corrections and help in improving the readability of this work.
The authors acknowledge support from the French government under the management of the Agence Nationale de la Recherche as part of the “Investissements d’avenir” program, reference ANR-19-P3IA-0001 (PRAIRIE 3IA Institute).