Original: Global convergence of ResNets: From finite to infinite width using linear parameterization
Revised: On global convergence of ResNets: From finite to infinite width using linear parameterization

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Overparameterization is a key factor in the absence of convexity to explain globalconvergence of gradient descent (GD) for neural networks.
Revised: Overparameterization is a key factor in the absence of convexity to explain global convergence of gradient descent (GD) for neural networks.

Original: Beside the well studiedlazy regime, infinite width (mean field) analysis has been developed for shallownetworks, using convex optimization techniques.
Revised: Beside the well studied lazy regime, infinite width (mean field) analysis has been developed for shallow networks, using convex optimization techniques.

Original: To bridge the gap between thelazy and mean field regimes, we study Residual Networks (ResNets) in whichthe residual block has linear parameterization while still being nonlinear.
Revised: To bridge the gap between the lazy and mean field regimes, we study Residual Networks (ResNets) in which the residual block has linear parameterization while still being nonlinear.

Original: Such
Revised: Such

Original: ResNets admit both infinite depth and width limits, encoding residual blocks in a
Revised: ResNets admit both infinite depth and width limits, encoding residual blocks in a

Original: Reproducing Kernel Hilbert Space (RKHS).
Revised: Reproducing Kernel Hilbert Space (RKHS).

Original: In this limit, we prove a local PolyakLojasiewicz inequality.
Revised: In this limit, we prove a local PolyakLojasiewicz inequality.

Original: Thus, every critical point is a global minimizer and alocal convergence result of GD holds, retrieving the lazy regime.
Revised: Thus, every critical point is a global minimizer and a local convergence result of GD holds, retrieving the lazy regime.

Original: In contrast withother mean-field studies, it applies to both parametric and non-parametric casesunder an expressivity condition on the residuals.
Revised: In contrast with other mean-field studies, it applies to both parametric and non-parametric cases under an expressivity condition on the residuals.

Original: Our analysis leads to a practicaland quantified recipe: starting from a universal RKHS, Random Fourier Featuresare applied to obtain a finite dimensional parameterization satisfying with highprobability our expressivity condition.
Revised: Our analysis leads to a practical and quantified recipe: starting from a universal RKHS, Random Fourier Features are applied to obtain a finite dimensional parameterization satisfying with highprobability our expressivity condition.

Original:  Introduction
Revised:  Introduction

Original: State of the art supervised learning methods are based on deep neural networks, sometimes heavilyoverparameterized, which perfectly fit training data or even noisy data while exhibiting good generalization properties.
Revised: State of the art supervised learning methods are based on deep neural networks, sometimes heavily overparameterized, which perfectly fit training data or even noisy data while exhibiting good generalization properties.

Original: Such a behaviour appears as a paradox and questions the established theoryof “bias-variance trade-off” [9].
Revised: Such a behaviour appears as a paradox and questions the established theory of “bias-variance trade-off” [9].

Original: That an overparameterized model can fit data perfectly comes asno surprise but this capability does not explain the observed generalization properties.
Revised: That an overparameterized model can fit data perfectly comes as no surprise but this capability does not explain the observed generalization properties.

Original: Towards abetter understanding of it, one first needs to understand the optimization procedure in the parameterspace that selects the interpolation map.
Revised: Towards a better understanding of it, one first needs to understand the optimization procedure in the parameter space that selects the interpolation map.

Original: This question is tightly linked with the parameterization ofthe space of maps that are explored and state of the art parameterizations have emerged in the pastyears.
Revised: This question is tightly linked with the parameterization of the space of maps that are explored and state of the art parameterizations have emerged in the past years.

Original: One key architecture that is ubiquitous in deep learning are skip connections, heavily usedin Residual Neural Networks (ResNets) [25] and it has led to state of the art results in supervisedlearning.
Revised: One key architecture that is ubiquitous in deep learning are skip connections, heavily used in Residual Neural Networks (ResNets) [25] and it has led to state of the art results in supervised learning.

Original: ResNets actually allow one to consider a very large number of layers [59].
Revised: ResNets actually allow one to consider a very large number of layers [59].

Original: Continuous models.
Revised: Continuous models.

Original: Passing to the limit of infinite depth allows the connection with continuousmodels (Neural ODE) for which theoretical methods and new algorithms can be designed [11, 56].
Revised: Passing to the limit of infinite depth allows the connection with continuous models (Neural ODE) for which theoretical methods and new algorithms can be designed [11, 56].

Original: Indeed, the similarities between ResNet architectures and discrete numerical schemes motivated theintroduction of a continuous neural ODE
Revised: Indeed, the similarities between ResNet architectures and discrete numerical schemes motivated the introduction of a continuous neural ODE

Original: [Equation]
Revised: [Equation]

Original: where W ∈ L 2 ([0 , 1] , R m ) is the parameter of the model and v : R m × R q → R q is a residualtransformation whose output is the residual term .
Revised: where W ∈ L 2 ([0 , 1] , R m ) is the parameter of the model and v : R m × R q → R q is a residual transformation whose output is the residual term .

Original: These models correspond to limiting modelsof a discrete ResNet whose depth D tends to infinity.
Revised: These models correspond to limiting models of a discrete ResNet whose depth D tends to infinity.

Original: Therefore, their study brings a theoretical framework for understanding deep ResNet architectures, and more generally very deep NNs [19, 20].
Revised: Therefore, their study brings a theoretical framework for understanding deep ResNet architectures, and more generally very deep NNs [19, 20].

Original: Moreover, their mathematical analysis is facilitated since it allows one to leverage a large body ofworks and tools from analysis and in particular the theory of optimal control [47].
Revised: Moreover, their mathematical analysis is facilitated since it allows one to leverage a large body of works and tools from analysis and in particular the theory of optimal control [47].

Original: Conversely,methods from numerical analysis can bring inspiration for designing new architectures and newoptimization algorithms [39].
Revised: Conversely, methods from numerical analysis can bring inspiration for designing new architectures and new optimization algorithms [39].

Original: RKHS parameterization.
Revised: RKHS parameterization.

Original: Most often in the literature studying the training properties of ResNets,the considered residual transformations are Multi-Layer Perceptrons (MLP)
Revised: Most often in the literature studying the training properties of ResNets, the considered residual transformations are Multi-Layer Perceptrons (MLP)

Original: Those consist in the composition of several trained linear layers alternatively composed with a non-linearactivation function.
Revised: Those consist in the composition of several trained linear layers alternatively composed with a non-linear activation function.

Original: A 2-layer MLP with width r reads:
Revised: A 2-layer MLP with width r reads:

Original: [Equation]
Revised: [Equation]

Original: where U ∈ R r × q and W ∈ R q × r are the parameters for the “hidden” and the “visible” layer respectively and σ : R → R is a non-linear activation function applied component-wise.
Revised: where U ∈ R r × q and W ∈ R q × r are the parameters for the “hidden” and the “visible” layer respectively and σ : R → R is a non-linear activation function applied component-wise.

Original: Popular activationfunctions are for example the ReLU or the Swish function.
Revised: Popular activation functions are for example the ReLU or the Swish function.

Original: Provided with these activations, MLPsenjoy a nice universal approximation property as shown in the seminal work of Barron [6].
Revised: Provided with these activations, MLPs enjoy a nice universal approximation property as shown in the seminal work of Barron [6].

Original: In contrast, we consider here a setting where the residual term is linear w.r.t.
Revised: In contrast, we consider here a setting where the residual term is linear w.r.t.

Original: the parameters whilestill being nonlinear w.r.t the inputs.
Revised: the parameters while still being nonlinear w.r.t the inputs.

Original: Given a feature map φ : R q → R r , we consider as space ofresiduals the space:
Revised: Given a feature map φ : R q → R r , we consider as space of residuals the space:

Original: [Equation]
Revised: [Equation]

Original: ) where the matrices W ∈ R q × r are the trained parameters.
Revised: where the matrices W ∈ R q × r are the trained parameters.

Original: Compared to Eq.
Revised: Compared to Eq.

Original: (2), this can be seen asan MLP where the hidden layer is fixed by introducing the feature map φ : z (cid:55)→ σ ( Uz ) for somefeature matrix U .
Revised: (2), this can be seen as an MLP where the hidden layer is fixed by introducing the feature map φ : z (cid:55)→ σ [Equation] Uz [Equation]

Revised: for some feature matrix U .

Original: As is standard, the gradient of some loss L w.r.t.
Revised: As is standard, the gradient of some loss L w.r.t.

Original: W is computed in the sense ofthe Frobenius metric on the set of matrices:
Revised: W is computed in the sense of the Frobenius metric on the set of matrices:

Original: [Equation]
Revised: [Equation]

Original: Such an L 2 penalization induces a metric structure on V through the identification v ↔ W in Eq.
Revised: Such an L 2 penalization induces a metric structure on V through the identification v ↔ W in Eq.

Original: [Equation]
Revised: [Equation]

Original: As a finite dimensional space of continuous maps, V has the structure of Reproducing Kernel Hilbert
Revised: As a finite dimensional space of continuous maps, V has the structure of Reproducing Kernel Hilbert

Original: Space (RKHS).
Revised: Space (RKHS).

Original: Moreover, as pointed out in [5], the space V has a natural infinite width limit or meanfield limit which is an infinite dimensional RKHS.
Revised: Moreover, as pointed out in [5], the space V has a natural infinite width limit or mean field limit which is an infinite dimensional RKHS.

Original: In this paper, we are interested in understanding the convergence properties of Gradient Descent(GD) on a ResNet model for which the residual layers are encoded in a – possibly infinitedimensional – vector-valued RKHS V .
Revised: In this paper, we are interested in understanding the convergence properties of Gradient Descent (GD) on a ResNet model for which the residual layers are encoded in a – possibly infinitedimensional – vector-valued RKHS V .

Original: For V as in Eq.
Revised: For V as in Eq.

Original: (3), we stress out that, as the metric on
Revised: (3), we stress out that, as the metric on

Original: V is induced by the one on R q × r , GD on V for this metric is strictly equivalent to GD on R q × r withthe Frobenius metric.
Revised: V is induced by the one on R q × r , GD on V for this metric is strictly equivalent to GD on R q × r with the Frobenius metric.

Original: Our model is defined as follows:
Revised: Our model is defined as follows:

Original: Definition 1 (RKHS Neural ODE (RKHS-NODE)) .
Revised: Definition 1 (RKHS Neural ODE (RKHS-NODE)) .

Original: Let V be a RKHS of vector-fields over R q and
Revised: Let V be a RKHS of vector-fields over R q and

Original: A ∈ R q × d , B ∈ R d ′ × q .
Revised: A ∈ R q × d , B ∈ R d ′ × q .

Original: Then for v ∈ L 2 ([0 , 1] , V ) and a data input x ∈ R d , the RKHS-NODE’soutput is F ( v, x ) :
Revised: Then for v ∈ L 2 ([0 , 1] , V ) and a data input x ∈ R d , the RKHS-NODE’s output is F ( v, x ) :

Original: =
Revised: =

Original: Bz 1 , where z is the solution to the forward problem˙
Revised: Bz 1 , where z is the solution to the forward problem

Original: [Equation]
Revised: [Equation]

Original: z

Original: [Equation]

Original: Ax.

Original: The variable v will thereafter be called control parameter .
Revised: The variable v will thereafter be called control parameter .

Original: Remark 1.
Revised: Remark 1.

Original: Note that the matrices A and B are fixed and only the control parameter v is trained.
Revised: Note that the matrices A and B are fixed and only the control parameter v is trained.

Original: However, we argue that our approach can be simply adapted to the case where B is trained, following for example the proof of [43].
Revised: However, we argue that our approach can be simply adapted to the case where B is trained, following for example the proof of [43].

Original: Training A seems more challenging as the model is highlynon-linear w.r.t.
Revised: Training A seems more challenging as the model is highly non-linear w.r.t.

Original: this parameter.
Revised: this parameter.

Original: Relevance of the RKHS model.
Revised: Relevance of the RKHS model.

Original: The main difference between the model of Definition 1 andstandard ResNets is linearity in the parameters of the residual blocks.
Revised: The main difference between the model of Definition 1 and standard ResNets is linearity in the parameters of the residual blocks.

Original: As a comparison, a 2-layer
Revised: As a comparison, a 2-layer

Original: MLP is nonlinear w.r.t.
Revised: MLP is nonlinear w.r.t.

Original: the parameters of the hidden layers.
Revised: the parameters of the hidden layers.

Original: However, this linearity assumptiondoes not impact the expressiveness of the model, it only impacts the training dynamic.
Revised: However, this linearity assumption does not impact the expressivity of the model, but only its training dynamic.

Original: (i)
Revised: (i)

Original: Indeed,considering V to be a Random Features approximation (c.f. Eq.
Revised: Indeed, considering V to be a Random Features approximation (c.f. Eq.

Original: (20)) of some universal RKHS, theresidual blocks are as expressive as a 2-layer MLP since both are dense in the space of continuousfunctions.
Revised: (20)) of some universal RKHS, the residual blocks are as expressive as a 2-layer MLP since both are dense in the space of continuous functions.

Original: (ii) Up to the cost of adding a supplementary variable, the dynamical system parameterized by a 2-layer MLP can be expressed as a model which is linear w.r.t.
Revised: (ii) Up to the cost of adding a supplementary variable, the dynamical system parameterized by a 2-layer MLP can be expressed as a model which is linear w.r.t.

Original: its parameters [56, Section 
Revised: its parameters [56, Section 3.2].

Original: Only the training dynamic between these two architectures differs.
Revised: Only the training dynamic between these two architectures differs.

Original: Also, this assumption oflinearity in the parameters also prevents the use of normalization layers.
Revised: Also, this assumption of linearity in the parameters also prevents the use of normalization layers.

Original: In this direction, [61] hasshown that ResNets without normalization but proper initialization of the weights can lead to robusttraining and similar performance on the train set than standard ResNets.
Revised: In this direction, [61] has shown that ResNets without normalization but proper initialization of the weights can lead to robust training and similar performance on the train set than standard ResNets.

Original: Finally, the model of Definition 1 still retains the effect of depth and the nonlinearity w.r.t.
Revised: Finally, the model of Definitionstill retains the effect of depth and the nonlinearity w.r.t.

Original: the input.
Revised: the input.

Original: Due to composition ofthese residual blocks the model’s output is still highly non-linear w.r.t.
Revised: Due to composition of these residual blocks the model’s output is still highly non-linear w.r.t.

Original: parameters.
Revised: parameters.

Original: Therefore, weconsider this model as an important stepof study towards the  general case.
Revised: For these reasons, we consider this model as an important step towards the study of the general case.

Original: In turn, this linearity in parameters naturally leads to an RKHS parameterization which has twoimportant benefits on the theoretical side: (i) Flows of vector-fields as implemented by our modelin Eq.
Revised: In turn, this linearity in parameters naturally leads to an RKHS parameterization which has two important benefits on the theoretical side: (i) Flows of vector-fields as implemented by our model in Eq.

Original: (6) have already been studied theoretically and for applications in image registration problems [58, 8, 44].
Revised: (6) have already been studied theoretically and for applications in image registration problems [58, 8, 44].

Original: Under some regularity assumptions on the considered RKHS V , one can show thatthe model’s output corresponds to the invertible action of a diffeomorphism by composition on theinput [55].
Revised: Under some regularity assumptions on the considered RKHS V , one can show that the model’s output corresponds to the invertible action of a diffeomorphism by composition on the input [55].

Original: This property was already used in [51] to implement models of Normalizing Flows [29]with applications in generative modeling.
Revised: This property was already used in [51] to implement models of Normalizing Flows [29] with applications in generative modeling.

Original: (ii)
Revised: (ii)

Original: There is an important literature in Machine Learningabout Kernel methods [52].
Revised: There is an important literature in Machine Learning about Kernel methods [52].

Original: In practice, various sub-sampling methods exist in order to approximateinfinite-dimensional RKHSs with finite-dimensional spaces generated by Random Fourier Features(RFF) [48, 49].
Revised: In practice, various sub-sampling methods exist in order to approximate infinite-dimensional RKHSs with finite-dimensional spaces generated by Random Fourier Features (RFF) [48, 49].

Original: Thereby, leveraging results on the approximation bound for RFF [54, 53], we showthat the expressiveness properties of universal kernels, such as the Gaussian kernel, can be efficientlyrecovered using residuals of the form Eq.
Revised: Thereby, leveraging results on the approximation bound for RFF [54, 53], we show that the expressiveness properties of universal kernels, such as the Gaussian kernel, can be efficiently recovered using residuals of the form Eq.

Original: (3) with a finite number of neurons.
Revised: (3) with a finite number of neurons.

Original: To further support the practical applicability and the relevance of our model in comparison with standard architectures, we report in the supplementary material (Appendix A) numerical experiments on
Revised: To further support the practical applicability and the relevance of our model in comparison with stan dard architectures, we report in the supplementary material (Appendix A) numerical experiments on

Original: MNIST and CIFAR10 datasets.
Revised: MNIST and CIFAR datasets.

Original: They show that – as predicted by our theory – our model can betrained in these cases to almost zero loss.
Revised: They show that – as predicted by our theory – our model can be trained in these cases to almost zero loss.

Original: But more importantly, they show that our model is able togeneralize well on the test dataset with performances that are similar to those of classical ResNets.
Revised: But more importantly, they show that our model is able to generalize well on the test dataset with performances that are similar to those of classical ResNets.

Original: Supervised learning.
Revised: Supervised learning.

Original: We consider a map F from H× R d to R d ′ for some Hilbert space of parameter
Revised: We consider a map F from H× R d to R d ′ for some Hilbert space of parameter

Original: H (e.g. the model of Definition 1 with H =
Revised: H (e.g. the model of Definition 1 with H =

Original: L 2 ([0 , 1] , V ) ) and a training dataset consisting on afamily of inputs ( x i ) 1 ≤ i ≤ N ∈
Revised: L 2 ([0 , 1] , V ) ) and a training dataset consisting on a family of inputs ( x i ) 1 ≤ i ≤ N ∈

Original: ( R d ) N and target outputs ( y i )
Revised: ( R d ) N and target outputs ( y i )

Original: 1 ≤ i ≤ N ∈
Revised: 1 ≤ i ≤ N ∈

Original: ( R d ′ ) N .
Revised: ( R d ′ ) N .

Original: Then for everyparameter v ∈ H , we define the associated Empirical Risk as:
Revised: Then for every parameter v ∈ H , we define the associated Empirical Risk as:

Original: [Equation]
Revised: [Equation]

Original: Remark 2.
Revised: Remark 2.

Original: For simplicity we consider here the Euclidean square distance as a loss on the outputspace R d ′ , but our results generalize to any smooth loss satisfying a Polyak-Lojasiewicz inequality(c.f.[10]), e.g. any smooth strongly convex loss.
Revised: For simplicity we consider here the Euclidean square distance as a loss on the output space R d ′ , but our results generalize to any smooth loss satisfying a Polyak-Lojasiewicz inequality (c.f.[10]), e.g. any smooth strongly convex loss.

Original: Training the model F then amounts to finding a parameter v ∗ ∈ arg min v ∈H L ( v ) .
Revised: Training the model F then amounts to finding a parameter v ∗ ∈ arg min v ∈H L ( v ) .

Original: In order to111 perform such an empirical risk minimization (ERM) task we consider GD on v .
Revised: In order to perform such an empirical risk minimization (ERM) task we consider GD on v .

Original: For a small step sizeη , for some initialization v 0 ∈ H and for every discrete time step k ∈ N , the training dynamic reads:
Revised: For a small step size η , for some initialization v 0 ∈ H and for every discrete time step k ∈ N , the training dynamic reads:

Original: [Equation]
Revised: [Equation]

Original: Note that we do not consider any additional regularizing term on the loss.
Revised: Note that we do not consider any additional regularizing term on the loss.

Original: In classical supervisedlearning one would seek for a minimizer of the “regularized” loss L ( v ) + λ R ( v ) , with λ > 0 a constant and R a coercive regularization function.
Revised: In classical supervised learning one would seek for a minimizer of the “regularized” loss L ( v ) + λ R ( v ) , with λ > 0 a constant and R a coercive regularization function.

Original: We are here interested in the non regularized setting,i.e. λ = 0 , often used in practice.
Revised: We are here interested in the non regularized setting, i.e. λ = 0 , often used in practice.

Original: In this case, the generalization property of the computed map isargued to potentially come from the optimization method that shall select an adequate minimizer ofthe loss.
Revised: In this case, the generalization property of the computed map is argued to potentially come from the optimization method that shall select an adequate minimizer of the loss.

Original: This implicit regularization depends on the choice of the optimization method [42].
Revised: This implicit regularization depends on the choice of the optimization method [42].

Original:  Related works and contributions
Revised:  Related works and contributions

Original: Recently, several works have addressed the problem of proving convergence of (stochastic) GD inthe training of NNs.
Revised: Recently, several works have addressed the problem of proving convergence of (stochastic) GD in the training of NNs.

Revised: The convergence properties of GD are already well understood for some models, such as NNs that are linear w.r.t.

Original:  In [34, 33, 17], the authors focus on the training of “shallow” two layers fullyconnected NNs and establish convergence of GD in an overparameterized setting where width of theintermediary layer scales polynomially with the size N of the dataset.
Revised: input [24, 7, 64]. On the other hand the case of nonlinear NNs is still actively discussed. In [34, 33, 17], the authors focus on the training of “shallow” two layers fully connected NNs and establish convergence of GD in an overparameterized setting where width of the intermediary layer scales polynomially with the size N of the dataset.

Original: More recently, with the samesetup, [62] showed that the neurons of a teacher network are recovered by a student network optimized with GD as long as the width of the student network is higher than the teacher’s one.
Revised: More recently, with the same setup, [62] showed that the neurons of a teacher network are recovered by a student network optimized with GD as long as the width of the student network is higher than the teacher’s one.

Original: Formally,their analysis is similar to ours as the result holds if the loss at initialization is already sufficientlylow and the proof relies on Polyak-Lojasiewicz inequalities verified by the loss landscape.
Revised: Formally, their analysis is similar to ours as the result holds if the loss at initialization is already sufficiently low and the proof relies on Polyak-Lojasiewicz inequalities verified by the loss landscape.

Original: Infinite depth.
Revised: Infinite depth.

Original: The works of [16, 2, 64, 32, 63, 35, 12, 43] extend those results to arbitrarydeep NN in the overparameterized setting.
Revised: The works of [16, 2, 64, 32, 63, 35, 12, 43] extend those results to arbitrary deep NN in the overparameterized setting.

Original: Specifically, the results in [16, 2, 35] apply to deep
Revised: Specifically, the results in [16, 2, 35] apply to deep

Original: ResNets.
Revised: ResNets.

Original: The best result seems to be achieved in [43], with convergence as soon as the last layerhas a width m = Ω( N 3 ) and at best with linear width.
Revised: The best result seems to be achieved in [43], with convergence as soon as the last layer has a width m = Ω( N 3 ) and at best with linear width.

Original: A common feature for those works is to relyon the fact that, for a sufficiently high number of parameters, the model can be well approximatedby a linear model corresponding to its first order expansion around the initialization.
Revised: A common feature for those works is to rely on the fact that, for a sufficiently high number of parameters, the model can be well approximated by a linear model corresponding to its first order expansion around the initialization.

Original: In [15] thisphenomenon, called “lazy regime”, is attributed to an inappropriate scaling of the parameters.
Revised: In [15] this phenomenon, called “lazy regime”, is attributed to an inappropriate scaling of the parameters.

Original: Onthe other hand, [36, 35] refer to this phenomenon as “linear” or “kernel regime” and relate it theconstancy of the Neural Tangent Kernel (NTK) introduced in [26].
Revised: On the other hand, [36, 35] refer to this phenomenon as “linear” or “kernel regime” and relate it the constancy of the Neural Tangent Kernel (NTK) introduced in [26].

Original: However, in all those works thewidth of intermediary layers has to depend on the depth D of the network.
Revised: However, in all those works the width of intermediary layers has to depend on the depth D of the network.

Original: Therefore, these resultsdo not apply to the training of the model in Eq.
Revised: Therefore, these results do not apply to the training of the model in Eq.

Original: (1), corresponding to the limit D → + ∞ .
Revised: (1), corresponding to the limit D → + ∞ .

Original: Infinite width.
Revised: Infinite width.

Original: The other direction of over-parameterization, analyzed in several works [41, 14,40, 27, 38, 21, 46] is to consider the limit of infinitely wide layers.
Revised: The other direction of over-parameterization, analyzed in several works [41, 14, 40, 27, 38, 21, 46] is to consider the limit of infinitely wide layers.

Original: In such a “mean-field” setting,the model is parameterized by the distribution of the parameters at each layer.
Revised: In such a “mean-field” setting, the model is parameterized by the distribution of the parameters at each layer.

Original: In [14, 41, 40, 27] thetraining dynamic is analyzed as a gradient flow in the Wasserstein space [3], showing that the onlystationary distributions are global minimizers of the empirical risk.
Revised: In [14, 41, 40, 27] the training dynamic is analyzed as a gradient flow in the Wasserstein space [3], showing that the only stationary distributions are global minimizers of the empirical risk.

Original: In [21] a similar result is showedfor deep NN with an arbitrary number of infinitely wide layers.
Revised: In [21] a similar result is showed for deep NN with an arbitrary number of infinitely wide layers.

Original: In [13, 1], local linear convergencetowards the global optimum is shown for two layers NNs in a teacher-student setup with regularizedloss.
Revised: In [13, 1], local linear convergence towards the global optimum is shown for two layers NNs in a teacher-student setup with regularized loss.

Original: Finally, [38] analyzes the convergence of continuous ResNets with infinitely wide residuallayers and shows that every critical point is a global minimizer of the empirical risk.
Revised: Finally, [38] analyzes the convergence of continuous ResNets with infinitely wide residual layers and shows that every critical point is a global minimizer of the empirical risk.

Original: We stress outthat these results only apply to infinitely wide NNs.
Revised: We stress out that these results only apply to infinitely wide NNs.

Original: It is not clear if this mean-field limit extends tothe parametric setting of MLPs with the Euclidean metric on their parameters.
Revised: It is not clear if this mean-field limit extends to the parametric setting of MLPs with the Euclidean metric on their parameters.

Original: In contrast, a RKHSstructure naturally arises when considering a linear parameterization of the residuals.
Revised: In contrast, a RKHS structure naturally arises when considering a linear parameterization of the residuals.

Original: Assumption 1and Assumption 2 can be satisfied both in a parametric setting with a finite number of features andin a mean-field setting limit where the residuals are generated by a universal kernel.
Revised: Assumptionand Assumption 2 can be satisfied both in a parametric setting with a finite number of features and in a mean-field setting limit where the residuals are generated by a universal kernel.

Original: Contributions.
Revised: Contributions.

Original: We show convergence results for GD in the training of RKHS-NODEs (see Definition 1).
Revised: We show convergence results for GD in the training of RKHS-NODEs (see Definition 1).

Original: These correspond to infinitely deep continuous ResNets with linear parameterizationof the residuals.
Revised: These correspond to infinitely deep continuous ResNets with linear parameterization of the residuals.

Original: Our first main contribution, in Section 4, shows that under some regularity andexpressivity assumptions on the residuals, the associated empirical risk satisfies a (local) PolyakLojasiewicz Property 2.
Revised: Our first main contribution, in Section 4, shows that under some regularity and expressivity assumptions on the residuals, the associated empirical risk satisfies a (local) PolyakLojasiewicz Property 2.

Original: A consequence is Theorem 2, which states global convergence of GDtowards a global optimum (zero training loss) under the condition that the loss at initialization isalready sufficiently low.
Revised: A consequence is Theorem 2, which states global convergence of GD towards a global optimum (zero training loss) under the condition that the loss at initialization is already sufficiently low.

Original: In the limit where the loss at initialization is arbitrarily small, we recovera linear regime as described in [36, 35].
Revised: In the limit where the loss at initialization is arbitrarily small, we recover a linear regime as described in [36, 35].

Original: Our second contribution, in Section 5, shows how thiscondition for global convergence can be enforced using suitably chosen first and last linear layers.
Revised: Our second contribution, in Section 5, shows how this condition for global convergence can be enforced using suitably chosen first and last linear layers.

Original: Thereafter, we show how the assumptions of Theorem 2, can be satisfied for RKHSs generated by afinite number of Random Features, with high probability over the choice of these features.
Revised: Thereafter, we show how the assumptions of Theorem 2, can be satisfied for RKHSs generated by a finite number of Random Features, with high probability over the choice of these features.

Original: For anydataset ( x i , y i ) 1 ≤ i ≤ N ∈
Revised: For any dataset ( x i , y i ) 1 ≤ i ≤ N ∈

Original: ( R d × R d ′ )
Revised: ( R d × R d ′ )

Original: N , we conclude in Theorem 3 to convergence of GD towards aglobal minimum of Eq.
Revised: N , we conclude in Theorem 3 to convergence of GD towards a global minimum of Eq.

Original: (7) with high probability when the width of the layers scales polynomiallyw.r.t.
Revised: (7) with high probability when the width of the layers scales polynomially w.r.t.

Original: the size of the dataset N and the inverse input data separation δ − 1 .
Revised: the size of the dataset N and the inverse input data separation δ − 1 .

Revised: Finally, we point out that some of our results can be seen as a generalization of existing results concerning convergence of GD for the training of linear NNs [24, 7, 64].

Revised: We explain in Appendix E how, following the lines of our analysis, one can for example recover [64, Theorem 3.1.].

Revised: However, if Definition 1 encompasses linear ResNets as a special case, we stress that Theorem 2 applies to a way larger class of models.

Revised: In particular, it is an important feature of our work to provide an analysis structure that can be easily adapted to different models or data-structure.

Original: Notations.
Revised: Notations.

Original: In what follows ∥ .
Revised: In what follows ∥ .

Original: ∥ denotes the Euclidean ℓ 2 norm for vectors and the Frobeniusnorm for matrices.
Revised: ∥ denotes the Euclidean ℓ 2 norm for vectors and the Frobenius norm for matrices.

Original: For matrices the spectral norm is denoted ∥ .
Revised: For matrices the spectral norm is denoted ∥ .

Original: ∥ 2 , the smallest (resp. greatest)singular value is denoted σ min (resp.
Revised: ∥ 2 , the smallest (resp. greatest) singular value is denoted σ min (resp.

Original: σ max ) and for symmetric matrices the smallest (resp.
Revised: σ max ) and for symmetric matrices the smallest (resp.

Original: greatest)eigenvalue is denoted λ min (resp.
Revised: greatest) eigenvalue is denoted λ min (resp.

Original: λ max ).
Revised: λ max ).

Original: Given some Hilbert space H , the functional Hilbert space
Revised: Given some Hilbert space H , the functional Hilbert space

Original: L 2 ([0 , 1] , H ) is denoted L 2 ( H ) or L 2 when there is no ambiguity.
Revised: L 2 ([0 , 1] , H ) is denoted L 2 ( H ) or L 2 when there is no ambiguity.

Original: The notation O (resp.
Revised: The notation O (resp.

Original: Ω ) meansasymptotically inferior (resp.
Revised: Ω ) means asymptotically inferior (resp.

Original: superior)
Revised: superior)

Original: up to multiplicative constant.
Revised: up to multiplicative constant.

Original:  Analysis of convergence for overparameterized models
Revised:  Analysis of convergence for overparameterized models

Original: In this section, we review methods for analyzing the convergence of overparameterized machinelearning models based on [36, 35].
Revised: In this section, we review methods for analyzing the convergence of overparameterized machine learning models based on [36, 35].

Original: We refer to Appendix B for detailed proofs of the statements.
Revised: We refer to Appendix B for detailed proofs of the statements.

Original: As presented above, we consider an optimization over the variable v in some Hilbert space H , withfixed input and output data, say v (cid:55)→ F ( v ) :
Revised: As presented above, we consider an optimization over the variable v in some Hilbert space H , with fixed input and output data, say v (cid:55)→ F ( v ) :

Original: =
Revised: =

Original: [ F ( v, x i )]
Revised: [ F ( v, x i )]

Original: i =1 ,...,N .
Revised: i =1 ,...,N .

Original: Therefore, the empirical risk isa function of the parameters v ∈ H .
Revised: Therefore, the empirical risk is a function of the parameters v ∈ H .

Original: We say that the model is overparameterized whenever thedimension dim ( H ) of the parameter space is much larger than the dimension of the output space of
Revised: We say that the model is overparameterized whenever the dimension dim ( H ) of the parameter space is much larger than the dimension of the output space of

Original: F ( v ) , here d ′ N .
Revised: F ( v ) , here d ′ N .

Original: The RKHS-NODE model defined F in Definition 1 falls into this category as H isthe infinite dimensional functional space L 2 ([0 , 1] , V ) .
Revised: The RKHS-NODE model defined F in Definition 1 falls into this category as H is the infinite dimensional functional space L 2 ([0 , 1] , V ) .

Original: 3.1 A (local) Polyak-Lojasiewicz property
Revised: 3.1 A (local) Polyak-Lojasiewicz property

Original: When dealing with overparameterized models, one cannot expect the loss to be convex but oneexpects the model to perfectly fit the data, that is to reach the global minimum value of 0 .
Revised: When dealing with overparameterized models, one cannot expect the loss to be convex but one expects the model to perfectly fit the data, that is to reach the global minimum value of 0 .

Original: In fact, fora sufficient number of parameters, the loss landscape typically possesses a continuum of infinitelymany global minima and is non-convex in any neighbourhood of a global minima [36].
Revised: In fact, for a sufficient number of parameters, the loss landscape typically possesses a continuum of infinitely many global minima and is non-convex in any neighbourhood of a global minima [36].

Original: One thusrather needs to rely on a set of functional inequalities allowing to control the decrease rate of theloss along GD [37, 10].
Revised: One thus rather needs to rely on a set of functional inequalities allowing to control the decrease rate of the loss along GD [37, 10].

Original: Definition 2 ((local) Polyak-Lojasiewicz property) .
Revised: Definition 2 ((local) Polyak-Lojasiewicz property) .

Original: Let L : H → R + be a differentiable function.
Revised: Let L : H → R + be a differentiable function.

Original: We say that L satisfies a (local) Polyak-Lojasiewicz (PL) property if there exist positive continuousfunctions m, M : R + → R ∗ + s.t.
Revised: We say that L satisfies a (local) Polyak-Lojasiewicz (PL) property if there exist positive continuous functions m, M : R + → R ∗ + s.t.

Original: for every v ∈ H
Revised: for every v ∈ H

Original: [Equation]
Revised: [Equation]

Original: Such functional inequalities have already shown to be relevant for proving convergence guaranteesin the training of NNs [22].
Revised: Such functional inequalities have already shown to be relevant for proving convergence guarantees in the training of NNs [22].

Original: A first consequence for a loss L which satisfies the (local) PL propertyof Definition 2 is that it does not admit any spurious local minima but only global minima.
Revised: A first consequence for a loss L which satisfies the (local) PL property of Definition 2 is that it does not admit any spurious local minima but only global minima.

Original: Also, ifthe training dynamic is bounded, then m and M are uniformly lower- and upper-bounded along thedynamic, implying that L decreases at a linear rate.
Revised: Also, if the training dynamic is bounded, then m and M are uniformly lower- and upper-bounded along the dynamic, implying that L decreases at a linear rate.

Original: In most cases, m and M are degenerate when∥ v ∥ → + ∞ .
Revised: In most cases, m and M are degenerate when ∥ v ∥ → + ∞ .

Original: When the dynamic is not bounded, L can thus decrease to 0 slower than at a linear rate198 or even converge towards a strictly positive limit.
Revised: When the dynamic is not bounded, L can thus decrease to 0 slower than at a linear rate or even converge towards a strictly positive limit.

Original: 3.2 Local convergence result
Revised: 3.2 Local convergence result

Original: Because of the degeneracy of m and M , it is in general not possible to conclude an unconditionalconvergence of GD towards a global minimizer of the empirical risk.
Revised: Because of the degeneracy of m and M , it is in general not possible to conclude an unconditional convergence of GD towards a global minimizer of the empirical risk.

Original: However, PL inequalities aresufficient to prove convergence when the problem is not too hard to solve, that is when the loss atinitialization is not too high.
Revised: However, PL inequalities are sufficient to prove convergence when the problem is not too hard to solve, that is when the loss at initialization is not too high.

Original: Moreover, when using gradient descent stepping, one needs to make asupplementary smoothness assumption on the empirical risk L .
Revised: Moreover, when using gradient descent stepping, one needs to make a supplementary smoothness assumption on the empirical risk L .

Original: This ensures that the loss decreasesat each gradient step for a sufficiently small step size.
Revised: This ensures that the loss decreases at each gradient step for a sufficiently small step size.

Original: Definition 3 (Smoothness, Definition 2 of [36]) .
Revised: Definition 3 (Smoothness, Definition 2 of [36]) .

Original: Let β ≥ 0 be a constant.
Revised: Let β ≥ 0 be a constant.

Original: We say that the function
Revised: We say that the function

Original: L : R m → R is β -smooth if for every v, v ′ ∈ R m : | L 
Revised: L : H → R is β -smooth if for every v, v ′ ∈ H : | L ( v ′ )

Original: [Equation] v [Equation]
Revised: − L ( v ) − ⟨∇ L ( v ) , v ′ − v ⟩| ≤ β 2 ∥ v ′ − v ∥ 2 .

Original: The local PL property combined with this smoothness assumption then gives a local convergenceresult for the convergence of GD towards a global minimizer of the empirical risk.
Revised: The local PL property combined with this smoothness assumption then gives a local convergence result for the convergence of GD towards a global minimizer of the empirical risk.

Original: Theorem 1 (Theorem 6 of [36]) .
Revised: Theorem 1 (Theorem 6 of [36]) .

Original: Let L : R m → R + be a loss function satisfying a local PL propertywith local constants m and M .
Revised: Let L : H → R + be a loss function satisfying a local PL property with local constants m and M .

Original: Let v 0 ∈ R m and R ≥ 0 be such that
Revised: Let v 0 ∈ H and R ≥ 0 be such that

Original: [Equation]
Revised: [Equation]

Original: Furthermore, assume that L is β -smooth within the ball B ( v 0 , R ) .
Revised: Furthermore, assume that L is β -smooth within the ball B ( v 0 , R ) .

Original: Then for a step size η ≤ β − 1 ,
Revised: Then for a step size η ≤ β − 1 ,

Original: GD with initialization v 0 and step size η converges towards a global minimizer of L with a linearconvergence rate and inside a ball of radius R .
Revised: GD with initialization v 0 and step size η converges towards a global minimizer of L with a linear convergence rate and inside a ball of radius R .

Original: More precisely, for every k ≥ 0 :
Revised: More precisely, for every k ≥ 0 :

Original: [Equation]
Revised: [Equation]

Original:  Properties of RKHS-NODE
Revised:  Properties of RKHS-NODE

Original: In this section we analyze the convergence of GD in the training of the infinitely deep ResNet modelof Definition 1.
Revised: In this section we analyze the convergence of GD in the training of the infinitely deep ResNet model of Definition 1.

Original: Note that such a model is overparameterized in depth as the parameter space is theinfinite dimensional space L 2 ([0 , 1] , V ) and overparameterization can also come from width whenthe RKHS is high (or even infinite) dimensional.
Revised: Note that such a model is overparameterized in depth as the parameter space is the infinite dimensional space L 2 ([0 , 1] , V ) and overparameterization can also come from width when the RKHS is high (or even infinite) dimensional.

Original: Therefore, our proof of convergence heavily relieson a PL property verified by the empirical risk.
Revised: Therefore, our proof of convergence heavily relies on a PL property verified by the empirical risk.

Original: Recall that we consider the training of deep ResNets with a linear parameterization of the residuals.
Revised: Recall that we consider the training of deep ResNets with a linear parameterization of the residuals.

Original: The set of residuals is as in Eq.
Revised: The set of residuals is as in Eq.

Original: (3) with the metric of Eq.
Revised: (3) with the metric of Eq.

Original: (5) induced by the Frobenius metric(Eq.
Revised: (5) induced by the Frobenius metric (Eq.

Original: This provides V with a RKHS structure [4], whose associated kernel is given for anyz, z ′ ∈ R q by K ( z, z ′ ) :
Revised: This provides V with a RKHS structure [4], whose associated kernel is given for any z, z ′ ∈ R q by K ( z, z ′ ) :

Original: = ⟨ φ ( z ) , φ ( z ′ ) ⟩ Id q , and whose associated feature map is given by φ .
Revised: = ⟨ φ ( z ) , φ ( z ′ ) ⟩ Id q , and whose associated feature map is given by φ .

Original: Remark 3.
Revised: Remark 3.

Original: The definition of ⟨ ., .
Revised: The definition of ⟨ ., .

Original: ⟩ V in Eq.
Revised: ⟩ V in Eq.

Original: (5) requires Span ( φ ( R q ))
Revised: (5) requires Span ( φ ( R q ))

Original: =
Revised: =

Original: R r to associate each v ∈ Vto a unique W ∈ R q × r .
Revised: R r to associate each v ∈ V to a unique W ∈ R q × r .

Original: This is satisfied by all the feature maps φ we consider in the following.
Revised: This is satisfied by all the feature maps φ we consider in the following.

Original: Given a training dataset composed of input data points ( x i ) 1 ≤ i ≤ N ∈
Revised: Given a training dataset composed of input data points ( x i ) 1 ≤ i ≤ N ∈

Original: ( R d ) N and of target data points( y i ) 1 ≤ i ≤ N ∈
Revised: ( R d ) N and of target data points ( y i ) 1 ≤ i ≤ N ∈

Original: ( R d ′ )
Revised: ( R d ′ )

Original: N we are interested in the task of minimizing the empirical risk of Eq.
Revised: N we are interested in the task of minimizing the empirical risk of Eq.

Original: (7) by
Revised: (7) by

Original: GD over v .
Revised: GD over v .

Original: Analogously to back-propagation in discrete NNs architectures, the gradient of L canbe expressed thanks to a backward equation derived by adjoint sensitivity analysis [47].
Revised: Analogously to back-propagation in discrete NNs architectures, the gradient of L can be expressed thanks to a backward equation derived by adjoint sensitivity analysis [47].

Original: Property 1.
Revised: Property 1.

Original: Let L be the empirical risk in Eq.
Revised: Let L be the empirical risk in Eq.

Original: (7) associated with the RKHS-NODE model with aquadratic loss.
Revised: (7) associated with the RKHS-NODE model with a quadratic loss.

Original: Let K be the kernel function associated with the RKHS V .
Revised: Let K be the kernel function associated with the RKHS V .

Original: Then L is differentiableon L 2 ([0 , 1] , V ) , with for every v ∈ L 2 ([0 , 1] , V ) , ∇ L ( v ) =
Revised: Then L is differentiable on L 2 ([0 , 1] , V ) , with for every v ∈ L 2 ([0 , 1] , V ) , ∇ L ( v ) =

Original: (cid:80) Ni =1 K ( ., z i ) p
Revised: (cid:80) Ni =1 K ( ., z i ) p

Original: i , where for each indexi ∈ (cid:74) 1 , N (cid:75) ,
Revised: i , where for each index i ∈ (cid:74) 1 , N (cid:75) ,

Original: z i is the solution of Eq.
Revised: z i is the solution of Eq.

Original: (6) with initial condition
Revised: (6) with initial condition

Original: Ax i and the adjoint variable p
Revised: Ax i and the adjoint variable p

Original: i is thesolution to the backward problem :
Revised: i is the solution to the backward problem :

Original: [Equation]
Revised: [Equation]

Original: 4.1 PL property of RKHS-NODE
Revised: 4.1 PL property of RKHS-NODE

Original: Following the line of proof sketched in Section 3, we show how to derive PL inequalities of theform Eq.
Revised: Following the line of proof sketched in Section 3, we show how to derive PL inequalities of the form Eq.

Original: (8) for the empirical loss associated with the RKHS-NODE model.
Revised: (8) for the empirical loss associated with the RKHS-NODE model.

Original: For that purpose wemake a few assumptions about the RKHS V .
Revised: For that purpose we make a few assumptions about the RKHS V .

Original: The first one concerns its regularity and allows us tocontrol the solutions of Eqs.
Revised: The first one concerns its regularity and allows us to control the solutions of Eqs.

Original: (6) and (11).
Revised: (6) and (11).

Original: Assumption 1 ((strong) Admissibility) .
Revised: Assumption 1 ((strong) Admissibility) .

Original: We say that the RKHS V is (strongly) admissible if it iscontinuously embedded in W 2 , ∞ ( R q , R q ) .
Revised: We say that the RKHS V is (strongly) admissible if it is continuously embedded in W 2 , ∞ ( R q , R q ) .

Original: More precisely, there exists a constant κ > 0 s.t.
Revised: More precisely, there exists a constant κ > 0 s.t.

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Assuming V is embedded in W 1 , ∞ ( R q , R q ) is natural to ensure the regularity of the flow generatedby the control parameter [55, 58] and suffices to prove convergence of a continuous gradient flowon the parameter v .
Revised: Assuming V is embedded in W 1 , ∞ ( R q , R q ) is natural to ensure the regularity of the flow generated by the control parameter [55, 58] and suffices to prove convergence of a continuous gradient flow on the parameter v .

Original: Assumption 1 is a bit stronger because a supplementary smoothness result onthe loss landscape is necessary to prove convergence of discrete GD (c.f. Definition 3).
Revised: Assumption 1 is a bit stronger because a supplementary smoothness result on the loss landscape is necessary to prove convergence of discrete GD (c.f. Definition 3).

Original: In practice,κ can be computed for smooth kernels thanks to Property 4 in Appendix D. For example, the RKHSassociated with the Gaussian kernel k : r (cid:55)→ e − r 2 / 2 is (strongly) admissible with κ = 2 + √ 3 .
Revised: In practice, κ can be computed for smooth kernels thanks to Property 4 in Appendix D. For example, the RKHS associated with the Gaussian kernel k : r (cid:55)→ e − r 2 / 2 is (strongly) admissible with κ = 2 + √ 3 .

Original: The second assumption is related to the expressiveness of V and is a weaker form of the classicaluniversality property of RKHSs.
Revised: The second assumption is related to the expressiveness of V and is a weaker form of the classical universality property of RKHSs.

Original: Assumption 2 ( N -universality) .
Revised: Assumption 2 ( N -universality) .

Original: Let K be the kernel function associated with the RKHS V .
Revised: Let K be the kernel function associated with the RKHS V .

Original: Fora family of points ( z i ) 1 ≤ i ≤ N ∈
Revised: For a family of points ( z i ) 1 ≤ i ≤ N ∈

Original: ( R q ) N , we define the associated kernel matrix as the block matrix253 K
Revised: ( R q ) N , we define the associated kernel matrix as the block matrix K

Original: [-( z i ) i ) : = ( K ( z i , z j )) 1 ≤ i,j ≤ N . 
-]
Revised: [-( z i ) i ) : = ( K ( z i , z j )) 1 ≤ i,j ≤ N . 
-][Equation]

Original: More precisely we assume for every δ > 0 :
Revised: More precisely we assume for every δ > 0 :

Original: [Equation]
Revised: [Equation]

Original: In particular, satisfying Assumption 2 requires having V of dimension m ≥ N , but it can be satisfiedfor finite dimensional RKHSs of dimension m ≤ N q , for example by considering a polynomialkernel, or by RKHSs of dimension m ≥ poly ( N, q ) with high probability on the sampling of randomfeatures, as shown in Section 5.
Revised: In particular, satisfying Assumption 2 requires having V of dimension m ≥ N , but it can be satisfied for finite dimensional RKHSs of dimension m ≤ N q , for example by considering a polynomial kernel, or by RKHSs of dimension m ≥ poly ( N, q ) with high probability on the sampling of random features, as shown in Section 5.

Original: On the other hand, even though the existence of λ follows fromcompactness arguments, it seems to be hardly analytically tractable even for classical kernels suchas the Gaussian kernel.
Revised: On the other hand, even though the existence of λ follows from compactness arguments, it seems to be hardly analytically tractable even for classical kernels such as the Gaussian kernel.

Original: Remark 4.
Revised: Remark 4.

Original: For a RKHS V as in Eq.
Revised: For a RKHS V as in Eq.

Original: (3) , the properties of V only depend on φ .
Revised: (3) , the properties of V only depend on φ .

Original: An interesting example is when φ : z (cid:55)→ σ ( Uz ) with σ an activation function applied component-wise and U a fixedfeature matrix.
Revised: An interesting example is when φ : z (cid:55)→ σ ( Uz ) with σ an activation function applied component-wise and U a fixed feature matrix.

Original: In Section 5 we show that, when considering the complex activation σ : t (cid:55)→ e − ıt ,both assumptions can be satisfied with high probability.
Revised: In Section 5 we show that, when considering the complex activation σ : t (cid:55)→ e − ıt , both assumptions can be satisfied with high probability.

Original: On the other hand, Assumption 1 is notsatisfied when considering σ = ReLU due to its non-smoothness at 0 .
Revised: On the other hand, Assumption 1 is not satisfied when considering σ = ReLU due to its non-smoothness at 0 .

Original: Remark 5.
Revised: Remark 5.

Original: Note that Λ could also be allowed to depend on some parameters, such as max ∥
Revised: Note that Λ could also be allowed to depend on some parameters, such as max ∥

Original: z i ∥ .
Revised: z i ∥ .

Original: However, as it is a more critical aspect of our analysis, we prefer to highlight the dependency of λw.r.t.
Revised: However, as it is a more critical aspect of our analysis, we prefer to highlight the dependency of λ w.r.t.

Original: min i ̸ =
Revised: min i ̸ =

Original: j ∥ z i − z j ∥ .
Revised: j ∥ z i − z j ∥ .

Original: For all the RKHSs studied here we always have Λ ≤ N .
Revised: For all the RKHSs studied here we always have Λ ≤ N .

Original: The following PL property is satisfied by the risk L .
Revised: The following PL property is satisfied by the risk L .

Original: Property 2 is proven in Appendix C.2.
Revised: Property 2 is proven in Appendix C.2.

Original: Property 2 (RKHS-NODE satisfy PL) .
Revised: Property 2 (RKHS-NODE satisfy PL) .

Original: Assume V satisfies Assumption 1 with κ and Assumption 2with λ and Λ .
Revised: Assume V satisfies Assumption 1 with κ and Assumptionwith λ and Λ .

Original: Let L be the empirical risk in Eq.
Revised: Let L be the empirical risk in Eq.

Original: (7) associated with the RKHS-NODE modelof Definition 1.
Revised: (7) associated with the RKHS-NODE model of Definition 1.

Original: Then L satisfies the PL inequalities of Definition 2 with m and M given by:
Revised: Then L satisfies the PL inequalities of Definition 2 with m and M given by:

Original: [Equation]
Revised: [Equation]

Original: where δ :=
Revised: where δ :=

Original: min i ̸ =
Revised: min i ̸ =

Original: j ∥ x i − x j ∥ is the data separation.
Revised: j ∥ x i − x j ∥ is the data separation.

Original: Note that the degeneracy of the bounding functions M, m as R → + ∞ readily appears in Eq.
Revised: Note that the degeneracy of the bounding functions M, m as R → + ∞ readily appears in Eq.

Original: Thus one should not expect these bounds to imply global convergence of GD without making anyfurther assumption.
Revised: Thus one should not expect these bounds to imply global convergence of GD without making any further assumption.

Original: Indeed, cases where GD fails to converge towards a global optimizer of the lossare observed in [7], Section 6, with a setup corresponding to the model of Definition 1 with V asin Eq.
Revised: Indeed, cases where GD fails to converge towards a global optimizer of the loss are observed in [7], Section 6, with a setup corresponding to the model of Definition 1 with V as in Eq.

Original: (3) and φ = Id R q .
Revised: (3) and φ = Id R q .

Original: Also, note that the data separation δ plays an important role in Property 2as it intervenes in the conditioning of the kernel matrix.
Revised: Also, note that the data separation δ plays an important role in Propertyas it intervenes in the conditioning of the kernel matrix.

Original: In what follows, we always assume the datapoints to have a data separation lower-bounded by δ > 0 .
Revised: In what follows, we always assume the data points to have a data separation lower-bounded by δ > 0 .

Original: 4.2 Convergence of RKHS-NODE
Revised: 4.2 Convergence of RKHS-NODE

Original: Thanks to the convergence analysis for overparameterized models detailed in Section 3, our mainresult follows as a consequence of the previous property.
Revised: Thanks to the convergence analysis for overparameterized models detailed in Section 3, our main result follows as a consequence of the previous property.

Original: Theorem 2 is proven in Appendix C.3.
Revised: Theorem 2 is proven in Appendix C.3.

Original: Theorem 2.
Revised: Theorem 2.

Original: Let V satisfy Assumption 1 with constant κ and Assumption 2 with λ, Λ .
Revised: Let V satisfy Assumption 1 with constant κ and Assumption 2 with λ, Λ .

Original: Let v 0 besome initialization of the control parameter with ∥ v 0{+
+} ∥
Revised: Let v 0 be some initialization of the control parameter with ∥ v {+
+} ∥

Original: L 2 = R 

Revised: L 2 = R

Original: and assume there exists a positive286 radius
Revised: and assume there exists a positive radius

Original: [Equation]
Revised: [Equation]

Original: [Equation] Then, for a sufficiently small step-size η > 0 , GD with step-size η converges towards a minimizer ofthe training loss at a linear rate and inside a ball of radius R .
Revised: Then, for a sufficiently small step-size η > 0 , GD with step-size η converges towards a minimizer of the training loss at a linear rate and inside a ball of radius R .

Original: More precisely, for every k ≥ 0 :
Revised: More precisely, for every k ≥ 0 :

Original: L

Original: [Equation]
Revised: [Equation]

Original: and ∥

Original: [Equation]
Revised: [Equation]

Original: R [Equation] 16) where µ := 1 N σ 2min ( B ⊤ ) λ (cid:0) σ − 1min
Revised: where µ := 1 N σ 2min ( B ⊤ ) λ (cid:0) σ − 1min

Original: ( A ) δ − 1 e κ ( R + R 0 ) (cid:1) e − 2 κ ( R + R 0 ) .
Revised: ( A ) δ − 1 e κ ( R + R 0 ) (cid:1) e − 2 κ ( R + R 0 ) .

Original: As Theorem 1, Theorem 2 is a local convergence result in which the condition in Eq.
Revised: As Theorem 1, Theorem 2 is a local convergence result in which the condition in Eq.

Original: (15) expressesa threshold between two kinds of behaviours: (i) if L ( v 0 ) is sufficiently small, the training dynamicconverges towards a global minimizer.
Revised: (15) expresses a threshold between two kinds of behaviours: (i) if L ( v 0 ) is sufficiently small, the training dynamic converges towards a global minimizer.

Original: The limiting behaviour is when the l.h.s.
Revised: The limiting behaviour is when the l.h.s.

Original: of Eq.
Revised: of Eq.

Original: (15) tendsto 0 .
Revised: (15) tends to 0 .

Original: Because of a regularizing effect of GD (i.e. that ∥ v k − v 

Revised: Because of a regularizing effect of GD (i.e. that ∥ v k − v

Original: ∥
Revised: ∥

Original: L 2 ≤ R ), the parameter stays in aball of arbitrary small radius R all along the training dynamic.
Revised: L 2 ≤ R ), the parameter stays in a ball of arbitrary small radius R all along the training dynamic.

Original: In this limit, we recover a “linear”or “kernel” regime where the model is well approximated by its linearization at v 

Revised: In this limit, we recover a “linear” or “kernel” regime where the model is well approximated by its linearization at v

Original: (ii)
Revised: (ii)

Original: If L ( v 0 ) is too large, the result says nothing about the convergence of the GD.
Revised: If L ( v 0 ) is too large, the result says nothing about the convergence of the GD.

Original: However, it is stillobserved in practice that the training dynamic often converges towards a global minimizer of theloss [60].
Revised: However, it is still observed in practice that the training dynamic often converges towards a global minimizer of the loss [60].

Original: Explaining this phenomenon in a general setting remains a challenging open question.
Revised: Explaining this phenomenon in a general setting remains a challenging open question.

Original:  Enforcing convergence with high dimensional embedding and finite width
Revised:  Enforcing convergence with high dimensional embedding and finite width

Original: As Theorem 2 is a local convergence result, it does not allow to conclude a general convergencebehaviour of GD in the training of RKHS-NODE.
Revised: As Theorem 2 is a local convergence result, it does not allow to conclude a general convergence behaviour of GD in the training of RKHS-NODE.

Original: In the following, we show how one can enforcethe hypothesis of Theorem 2 to be verified and prove two global convergence results.
Revised: In the following, we show how one can enforce the hypothesis of Theorem 2 to be verified and prove two global convergence results.

Original: The first onerelies on suitably choosing matrices A and B in order to satisfy Eq.
Revised: The first one relies on suitably choosing matrices A and B in order to satisfy Eq.

Original: (15) and applies in the caseof infinite width, i.e. with residual layers in a universal RKHS.
Revised: (15) and applies in the case of infinite width, i.e. with residual layers in a universal RKHS.

Original: The second result recovers globalconvergence in a finite width regime, relying on a high number r of Random Fourier Features.
Revised: The second result recovers global convergence in a finite width regime, relying on a high number r of Random Fourier Features.

Original: For the sake of readability we only consider here the case where V belongs to a restricted class of
Revised: For the sake of readability we only consider here the case where V belongs to a restricted class of

Original: RKHSs and refer to Appendix D for more general results and complete proofs.
Revised: RKHSs and refer to Appendix D for more general results and complete proofs.

Original: For some positiveparameter ν > 0{+
+} we consider the Matérn kernel k defined in [57]:
Revised: For some positive parameter ν > {+
+} we consider the Matérn kernel k defined in [57]:

Original: [Equation]
Revised: [Equation]

Original: where Γ is the Gamma function and K ν is the modified Bessel function of the second kind.
Revised: where Γ is the Gamma function and K ν is the modified Bessel function of the second kind.

Original: Equivalently, k can be defined by its frequency distribution over R q as:
Revised: Equivalently, k can be defined by its frequency distribution over R q as:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: and C q,ν a normalizing constant.
Revised: and C q,ν a normalizing constant.

Original: For every q ≥ 1 , such a function is known to define a structureof vector-valued RKHS V q over R q corresponding to the Sobolev space H ν + q/ 2 ( R q , R q ) [52, 57].
Revised: For every q ≥ 1 , such a function is known to define a structure of vector-valued RKHS V q over R q corresponding to the Sobolev space H ν + q/ 2 ( R q , R q ) [52, 57].

Original: The associated kernel is given for every z, z ′ ∈ R q by: K q ( z, z ′ ) =
Revised: The associated kernel is given for every z, z ′ ∈ R q by: K q ( z, z ′ ) =

Original: k ( ∥ z − z ′ ∥ )
Revised: k ( ∥ z − z ′ ∥ )

Original: Id q .
Revised: Id q .

Original: Note that314 it is important for this RKHS to depend on the ambient dimension q .
Revised: Note that it is important for this RKHS to depend on the ambient dimension q .

Original: In particular the Sobolevspace H s ( R q , R q ) is a RKHS if and only if it has regularity s > q/ 2 .
Revised: In particular the Sobolev space H s ( R q , R q ) is a RKHS if and only if it has regularity s > q/ 2 .

Original: Assuming ν > 2 , µ q furtheradmits up to 4 finite order moment implying that k is four times differentiable at 0 [28].
Revised: Assuming ν > 2 , µ q further admits up to 4 finite order moment implying that k is four times differentiable at 0 [28].

Original: Then V qsatisfies Assumption 1 with some constant κ depending only on ν and given by Property 4:
Revised: Then V q satisfies Assumption 1 with some constant κ depending only on ν and given by Property 4:

Original: [Equation]
Revised: [Equation]

Original: Also, V q satisfies Assumption 2 with Λ ≤ N and λ depending a priori on ν , q and N .
Revised: Also, V q satisfies Assumption 2 with Λ ≤ N and λ depending a priori on ν , q and N .

Original: Note that with this choice of scaling for k and µ q , one recovers the Gaussian kernel k : r (cid:55)→ e − r 2 / 2in the limit ν → + ∞ [57].
Revised: Note that with this choice of scaling for k and µ q , one recovers the Gaussian kernel k : r (cid:55)→ e − r 2 /in the limit ν → + ∞ [57].

Original: Thereafter we consider, ν ∈
Revised: Thereafter we consider, ν ∈

Original: (2 , + ∞ ] , the case ν = + ∞ referring to321 the Gaussian kernel.
Revised: (2 , + ∞ ] , the case ν = + ∞ referring to the Gaussian kernel.

Original: We also assume that the data distribution is compactly supported.
Revised: We also assume that the data distribution is compactly supported.

Original: In particularthere exists some r 0 ≥ 

Revised: In particular there exists some r 0 ≥

Original: so that every input data x verifies ∥ x ∥ ≤ r 0 .
Revised: so that every input data x verifies ∥ x ∥ ≤ r 0 .

Original: 5.1 Global convergence with high-dimensional lifting
Revised: 5.1 Global convergence with high-dimensional lifting

Original: We first show how Eq.
Revised: We first show how Eq.

Original: (15) can be satisfied by considering appropriate embedding matrices A and
Revised: (15) can be satisfied by considering appropriate embedding matrices A and

Original: B .
Revised: B .

Original: Doing so, the square distance between the data points, i.e. the model’s loss, is preserved whereasthe conditioning of the kernel matrix can be controlled.
Revised: Doing so, the square distance between the data points, i.e. the model’s loss, is preserved whereas the conditioning of the kernel matrix can be controlled.

Original: Proposition 1.
Revised: Proposition 1.

Original: Let ν ∈
Revised: Let ν ∈

Original: (2 , + ∞ ] , let ( x i , y i )
Revised: (2 , + ∞ ] , let ( x i , y i )

Original: 1 ≤ i ≤ N ∈
Revised: 1 ≤ i ≤ N ∈

Original: ( R d × R d ′ )
Revised: ( R d × R d ′ )

Original: N be a dataset with data separationδ > 0 and let R > 0 .
Revised: N be a dataset with data separation δ > 0 and let R > 0 .

Original: There exist q ≥ 1 and matrices A ∈ R q × d , B ∈ R d ′ × q s.t.
Revised: There exist q ≥ 1 and matrices A ∈ R q × d , B ∈ R d ′ × q s.t.

Original: GD initialized atv 0 = 0 converges towards a zero-training-loss optimum in the training of RKHS-NODE.
Revised: GD initialized at v 0 = 0 converges towards a zero-training-loss optimum in the training of RKHS-NODE.

Original: In particular, Eq.
Revised: In particular, Eq.

Original: (15) holds with radius R and κ, λ, Λ associated with the RKHS V q .
Revised: (15) holds with radius R and κ, λ, Λ associated with the RKHS V q .

Original: As shown in the proof in Appendix D.1, Proposition 1 still holds for small but non-zero initialization.
Revised: As shown in the proof in Appendix D.1, Proposition 1 still holds for small but non-zero initialization.

Original: We present here two ways of obtaining matrices A and B satisfying Eq.
Revised: We present here two ways of obtaining matrices A and B satisfying Eq.

Original: Scaling Consider A
Revised: Scaling Consider A

Original: = α (Id d , 0)
Revised: = α (Id d , 0)

Original: ⊤ ∈ R ( d + d ′ )
Revised: ⊤ ∈ R ( d + d ′ )

Original: ×
Revised: ×

Original: d and B = α (0 , Id d ′ ) ∈ R d ′ ×
Revised: d and B = α (0 , Id d ′ ) ∈ R d ′ ×

Original: d + d ′ ) , for α > 0 .
Revised: d + d ′ ) , for α > 0 .

Original: We show in Appendix D.1.2 that, in this setting, the l.h.s.
Revised: We show in Appendix D.1.2 that, in this setting, the l.h.s.

Original: of Eq.
Revised: of Eq.

Original: (15) scales as O (1 /α ) and thus Theorem 2 holds for large enough α .
Revised: (15) scales as O (1 /α ) and thus Theorem 2 holds for large enough α .

Original: Moreover, observe that q = d + d ′ is independent of N and δand such a regime can easily be implemented in practice.
Revised: Moreover, observe that q = d + d ′ is independent of N and δ and such a regime can easily be implemented in practice.

Original: However, it has been shown that, althoughinterpolation of the training data can be achieved as a consequence of a suitable rescaling of theparameters, this “lazy regime” can also lead to bad generalization properties [15].
Revised: However, it has been shown that, although interpolation of the training data can be achieved as a consequence of a suitable rescaling of the parameters, this “lazy regime” can also lead to bad generalization properties [15].

Original: Lifting
Revised: Lifting

Original: Consider for q ≥ 1 the matrices: A q :=
Revised: Consider for q ≥ 1 the matrices: A q :=

Original: q − 1 / 4 (Id d , ..., Id d , 0)
Revised: q − 1 / 4 (Id d , ..., Id d , 0)

Original: ⊤ ∈ R q × d
Revised: ⊤ ∈ R q × d

Original: and
Revised: and

Original: B q :=
Revised: B q :=

Original: q 1 / 4 (Id d ′ , 0 ... 0)
Revised: q 1 / 4 (Id d ′ , 0 ... 0)

Original: ∈ R d ′ × q , with ⌊ q/d ⌋ copies of Id d in A q .
Revised: ∈ R d ′ × q , with ⌊ q/d ⌋ copies of Id d in A q .

Original: This choice is motivated bythe intention for these matrices to produce a high-dimensional lifting, which has been shown to improve on the expressiveness of ResNets [18].
Revised: This choice is motivated by the intention for these matrices to produce a high-dimensional lifting, which has been shown to improve on the expressivity of ResNets [18].

Original: We then show in Appendix D.1.1 that Eq.
Revised: We then show in Appendix D.1.1 that Eq.

Original: (15) can besatisfied for q = Ω( N 4 + δ − 4 log( N ) 4 ) .
Revised: (15) can be satisfied for q = Ω( N 4 + δ − 4 log( N ) 4 ) .

Original: We do not expect our condition on q to be optimal as weobserve in experiments (see Appendix A) that a regime of linear convergence can be obtained forq ≪ N 4 + δ − 4 log( N ) 4 .
Revised: We do not expect our condition on q to be optimal as we observe in experiments (see Appendix A) that a regime of linear convergence can be obtained for q ≪ N 4 + δ − 4 log( N ) 4 .

Original: However, we observe that increasing q does improve on the convergenceand generalization properties of our model (Fig. 2).
Revised: However, we observe that increasing q does improve on the convergence and generalization properties of our model (Fig. 2).

Original: 5.2 Global convergence with finite width
Revised: 5.2 Global convergence with finite width

Original: In the preceding we showed that, in the case of an RKHS defined by a Matérn kernel, convergenceof GD can be ensured for well-chosen matrices A and B .
Revised: In the preceding we showed that, in the case of an RKHS defined by a Matérn kernel, convergence of GD can be ensured for well-chosen matrices A and B .

Original: However, for practical implementations,the form of the residual in Eq.
Revised: However, for practical implementations, the form of the residual in Eq.

Original: (3) forces us to consider RKHSs defined by feature maps.
Revised: (3) forces us to consider RKHSs defined by feature maps.

Original: A wayto overcome this difficulty and to benefit from the properties of a wide range of kernel functions isto consider an approximation by Random Fourier Features (RFF) [48, 49].
Revised: A way to overcome this difficulty and to benefit from the properties of a wide range of kernel functions is to consider an approximation by Random Fourier Features (RFF) [48, 49].

Original: More precisely, given q ≥ 1 , recall the definition of the Matérn kernel k in terms of its frequency distribution µ q over R qin Eq.
Revised: More precisely, given q ≥ 1 , recall the definition of the Matérn kernel k in terms of its frequency distribution µ q over R q in Eq.

Original: (18) and for any sampling ω 1 , ..., ω r iid ∼ µ q of size r , consider the feature map:
Revised: (18) and for any sampling ω 1 , ..., ω r iid ∼ µ q of size r , consider the feature map:

Original: [Equation]
Revised: [Equation]

Original: In other words, considering the complex activation σ : t (cid:55)→ e ıt applied component-wise and
Revised: In other words, considering the complex activation σ : t (cid:55)→ e ıt applied component-wise and

Original: U :=
Revised: U :=

Original: ( ω 1 | . . .
Revised: ( ω 1 | . . .

Original: | ω r ) ∈ R q × r the feature matrix, we have φ ( z ) =
Revised: | ω r ) ∈ R q × r the feature matrix, we have φ ( z ) =

Original: r − 1 / 2 σ ( U ⊤ z ) .
Revised: r − 1 / 2 σ ( U ⊤ z ) .

Original: Recall that such afeature map defines a structure of RKHS on ˆ V q :=
Revised: Recall that such a feature map defines a structure of RKHS on ˆ V q :=

Original: { Wφ ( . )
Revised: { Wφ ( . )

Original: | W ∈ R q × r } .
Revised: | W ∈ R q × r } .

Original: Such a ˆ V q can be viewedas a finite-dimensional approximation of the universal RKHS V q as it is associated with the kernelfunction
Revised: Such a ˆ V q can be viewed as a finite-dimensional approximation of the universal RKHS V q as it is associated with the kernel function

Original: ˆ K q ( z, z ′ ) :
Revised: ˆ K q ( z, z ′ ) :

Original: =
Revised: =

Original: ˆ k
Revised: ˆ k

Original: ( z, z ′ )
Revised: ( z, z ′ )

Original: Id q , with:
Revised: Id q , with:

Original: [Equation]
Revised: [Equation]

Original: Given any q ≥ 1 , we show that, with high probability over the choice of features, ˆ V q recovers theproperties of admissibility and universality of V q as soon as r is sufficiently high w.r.t.
Revised: Given any q ≥ 1 , we show that, with high probability over the choice of features, ˆ V q recovers the properties of admissibility and universality of V q as soon as r is sufficiently high w.r.t.

Original: q and N .
Revised: q and N .

Original: Thefollowing is a particular case of Proposition 5 in Appendix D.2.
Revised: The following is a particular case of Proposition 5 in Appendix D.2.

Original: Proposition 2.
Revised: Proposition 2.

Original: Consider any q, N ≥ 2 and any ϵ, τ, R > 0 .
Revised: Consider any q, N ≥ 2 and any ϵ, τ, R > 0 .

Original: Assume ν > 4 .
Revised: Assume ν > 4 .

Original: (i) For r ≥ Ω( τq 8 ) , with probability greater than 1 − τ − 1 , ˆ V q satisfies Assumption 1 with ˆ κ ≤ κ +1 .
Revised: (i) For r ≥ Ω( τq 8 ) , with probability greater than 1 − τ − 1 , ˆ V q satisfies Assumption 1 with ˆ κ ≤ κ +1 .

Original: (ii) For r ≥ Ω( ϵ − 2 N 2 ( q log( ∥ A ∥ 2 r 0 + R ) + τ )) , with probability greater than 1 − e − τ , for anyv ∈ L 2 ( ˆ V q ) s.t.
Revised: (ii) For r ≥ Ω( ϵ − 2 N 2 ( q log( ∥ A ∥ 2 r 0 + R ) + τ )) , with probability greater than 1 − e − τ , for any v ∈ L 2 ( ˆ V q ) s.t.

Original: ∥ v ∥ L 2 ≤ R and any time t ∈ [0 , 1] : λ min (ˆ K (( z it ) i )) ≥ λ min ( K (( z it ) i ))
Revised: ∥ v ∥ L 2 ≤ R and any time t ∈ [0 , 1] : λ min (ˆ K (( z it ) i )) ≥ λ min ( K (( z it ) i ))

Original: − ϵ , where( z i )
Revised: − ϵ , where ( z i )

Original: i are the solutions to Eq.
Revised: i are the solutions to Eq.

Original: (6) and ˆ K , K are the kernel matrices of ˆ k and k respectively.
Revised: (6) and ˆ K , K are the kernel matrices of ˆ k and k respectively.

Original: Finally, combining Proposition 1 and Proposition 2, we obtain a global convergence result.
Revised: Finally, combining Proposition 1 and Proposition 2, we obtain a global convergence result.

Original: Theorem 3 states convergence, with high probability over a choice of features, of GD towards a zerotraining-loss optimum for infinitely deep ResNets of finite width.
Revised: Theorem 3 states convergence, with high probability over a choice of features, of GD towards a zerotraining-loss optimum for infinitely deep ResNets of finite width.

Original: Theorem 3 (Global convergence) .
Revised: Theorem 3 (Global convergence) .

Original: Assume ν > 4 and let ( x i , y i ) ∈
Revised: Assume ν > 4 and let ( x i , y i ) ∈

Original: ( R d × R d ′ )
Revised: ( R d × R d ′ )

Original: N be a compactlysupported dataset with input data separation δ > 0 .
Revised: N be a compactly supported dataset with input data separation δ > 0 .

Original: There exist matrices A ∈ R q × d and B ∈ R d ′ × qs.t.
Revised: There exist matrices A ∈ R q × d and B ∈ R d ′ × q s.t.

Original: for any τ > 0 , with probability at least 1 − τ − 1 w.r.t.
Revised: for any τ > 0 , with probability at least 1 − τ − 1 w.r.t.

Original: the choices of features, GD initialized atv 0 = 0 converges towards a zero training loss optimum in the training of the RKHS-NODE modelof Definition 1 with the feature map φ of Eq.
Revised: the choices of features, GD initialized at v 0 = 0 converges towards a zero training loss optimum in the training of the RKHS-NODE model of Definition 1 with the feature map φ of Eq.

Original: (20) as soon as r ≥ Ω( τ ( q 8 + qN 2 log( ∥
Revised: (20) as soon as r ≥ Ω( τ ( q 8 + qN 2 log( ∥

Original: A ∥ 2 )) .
Revised: A ∥ 2 )) .

Original: Proof.
Revised: Proof.

Original: Consider R = 1 .
Revised: Consider R = 1 .

Original: By Proposition 1, we can have A ∈ R q × d , B ∈ R d ′ ×
Revised: By Proposition 1, we can have A ∈ R q × d , B ∈ R d ′ ×

Original: q
Revised: q

Original: so that in Eq.
Revised: so that in Eq.

Original: [Equation]
Revised: [Equation]

Original: for κ , λ and Λ associated with k .
Revised: for κ , λ and Λ associated with k .

Original: Also, by the proof of Proposition 1 we can have:λ ( σ min ( A ) − 1 δ − 1 e ( κ +1) ) ≥ 1 / 2 .
Revised: Also, by the proof of Proposition 1 we can have: λ ( σ min ( A ) − 1 δ − 1 e ( κ +1) ) ≥ 1 / 2 .

Original: Taking ϵ = 1 / 4 in Proposition 2, the condition in Eq.
Revised: Taking ϵ = 1 / 4 in Proposition 2, the condition in Eq.

Original: (15) is satisfied by ˆ V q with probability greater than 1 − τ − 1 as soon as r ≥ Ω
Revised: (15) is satisfied by ˆ V q with probability greater than 1 − τ − 1 as soon as r ≥ Ω

Original: ( τq 8 + τqN 2 log(1+ ∥
Revised: ( τq 8 + τqN 2 log(1+ ∥

Original: A ∥ 2 r 0 )) .
Revised: A ∥ 2 r 0 )) .

Original:  Conclusion
Revised:  Conclusion

Original: We have identified a relevant infinite width limit (RKHS-NODE) for a particular model of ResNet.
Revised: We have identified a relevant infinite width limit (RKHS-NODE) for a particular model of ResNet.

Original: We showed that GD converges linearly when training this model and that a network’s width polynomial w.r.t.
Revised: We showed that GD converges linearly when training this model and that a network’s width polynomial w.r.t.

Original: to the size of the dataset is sufficient to maintain this property.
Revised: to the size of the dataset is sufficient to maintain this property.

Original: A natural extension ofour result is to study the convergence of GD when also training the hidden layers of the residuals.
Revised: A natural extension of our result is to study the convergence of GD when also training the hidden layers of the residuals.

Original: Afirst step towards this general case consists in studying the corresponding mean field model wherethe residuals are parameterized by density distributions over the neurons [14, 41, 40, 27, 38, 21] foreach residual blocks.
Revised: A first step towards this general case consists in studying the corresponding mean field model where the residuals are parameterized by density distributions over the neurons [14, 41, 40, 27, 38, 21] for each residual blocks.

Original: Interestingly, such a parametrization of the residual blocks is still linear in thismeasure and thus fits into our framework of linear in parameters.
Revised: Interestingly, such a parametrization of the residual blocks is still linear in this measure and thus fits into our framework of linear in parameters.

Original: However, it would require a finermathematical analysis to obtain similar results.
Revised: However, it would require a finer mathematical analysis to obtain similar results.

Original: Potential Negative Societal Impacts.
Revised: Potential Negative Societal Impacts.

Original: Our work aims at improving the theoretical and practicalunderstanding of deep networks and therefore we do not expect a direct negative impact. 
Revised: Our work aims at improving the theoretical and practical understanding of deep networks and therefore we do not expect a direct negative impact. Acknowledgements

Revised: The work of Gabriel Peyré was supported by the French government under management of Agence

Revised: Nationale de la Recherche as part of the ”Investissements d’avenir” program, reference ANR19P3IA-0001 (PRAIRIE 3IA Institute) and by the European Research Council (ERC project NORIA).

Original: A Numerical experiments
Revised: A Numerical experiments

Original: The goal of this section is to quantify how much (in addition to interpolating the training dataset)our model is able to generalize on the test dataset.
Revised: The goal of this section is to quantify how much (in addition to interpolating the training dataset) our model is able to generalize on the test dataset.

Original: This is also useful to compare the performancesof our model with those of standard ResNet architectures (which integrate batch normalization andtraining of the hidden layers).
Revised: This is also useful to compare the performances of our model with those of standard ResNet architectures (which integrate batch normalization and training of the hidden layers).

Original: We implemented our model in Pytorch [45] and trained it on imagedatasets for classification tasks.
Revised: We implemented our model in Pytorch [45] and trained it on image datasets for classification tasks.

Revised: Source code is available at https://github.com/rbarboni/FlowResNets .

Original: Experiments were conducted using a private infrastructure, which has a carbon efficiency of0.05 kgCO 2 eq/kWh.
Revised: Experiments were conducted using a private infrastructure, which has a carbon efficiency of 0.05 kgCO 2 eq/kWh.

Original: A cumulative of (at most) 1000 hours of computation was performed onhardware of type Tesla V100-PCIE-16GB (TDP of 300W).
Revised: A cumulative of (at most) 1000 hours of computation was performed on hardware of type Tesla V100-PCIE-16GB (TDP of 300W).

Original: Total emissions are estimated to be15 kgCO 2 eq (or 60km in an average car) of which 0 percents were directly offset.
Revised: Total emissions are estimated to be 15 kgCO 2 eq (or 60km in an average car) of which 0 percents were directly offset.

Original: Estimations were conducted using the MachineLearning Impact calculator presented in [30].
Revised: Estimations were conducted using the MachineLearning Impact calculator presented in [30].

Original: Computational setup for classification tasks.
Revised: Computational setup for classification tasks.

Original: In the context of classification tasks, we use across entropy loss in place of the least square loss of Eq.
Revised: In the context of classification tasks, we use a cross entropy loss in place of the least square loss of Eq.

Original: For a problem with K classes, theoutput dimension of the model is d ′ =
Revised: For a problem with K classes, the output dimension of the model is d ′ =

Original: K and targets y ∈ R K are one-hot vector encoding the targetclasses.
Revised: K and targets y ∈ R K are one-hot vector encoding the target classes.

Original: For a batch of N predictions ( z i ) 1 ≤ i ≤ N and targets ( y i ) 1 ≤ i ≤ N in R K the Cross Entropy598 loss is defined as:
Revised: For a batch of N predictions ( z i ) 1 ≤ i ≤ N and targets ( y i ) 1 ≤ i ≤ N in R K the Cross Entropy loss is defined as:

Original: [Equation]
Revised: [Equation]

Original: where ℓ is the Binary Entropy defined for one prediction z and one target y ∈ R K by:
Revised: where ℓ is the Binary Entropy defined for one prediction z and one target y ∈ R K by:

Original: [Equation]
Revised: [Equation]

Original: Then for a model F depending on the parameters W and a training batch ( x i , y i ) 1 ≤ i ≤ N we definethe empirical risk:
Revised: Then for a model F depending on the parameters W and a training batch ( x i , y i ) 1 ≤ i ≤ N we define the empirical risk:

Original: [Equation]
Revised: [Equation]

Original: (( F ( W, x i ) i , ( y i ) i ) , and train the model by Stochastic Gradient Descent (SGD) on W .
Revised: i , ( y i ) i ) , and train the model by Stochastic Gradient Descent (SGD) on W .

Original: Finally, the performance of themodel is assessed by the Top-1 error rate on a test dataset.
Revised: Finally, the performance of the model is assessed by the Top-1 error rate on a test dataset.

Original: Note that, as explained in Remark 2, the result of Theorem 3 can be extended to this cross entropy loss.
Revised: Note that, as explained in Remark 2, the result of Theorem 3 can be extended to this cross entropy loss.

Original: Indeed, ℓ satisfies a functional inequality similar to the Polyak-Lojasiewicz inequality.
Revised: Indeed, ℓ satisfies a functional inequality similar to the Polyak-Lojasiewicz inequality.

Original: Assuming without loss of generality that y
Revised: Assuming without loss of generality that y

Original: = e 1 is the indicator of class 1 , one has:
Revised: = e 1 is the indicator of class 1 , one has:

Original: [Equation]
Revised: [Equation]

Original: Then by convexity of exponential, when ℓ ( z, y ) ≤ 1 :
Revised: Then by convexity of exponential, when ℓ ( z, y [Equation] ≤ 1 :

Original: [Equation]
Revised: [Equation]

Original: Note however that Theorem 3 is only valid for full batch gradient descent.
Revised: Note however that Theorem 3 is only valid for full batch gradient descent.

Original: We leave its extension to
Revised: We leave its extension to

Original: SGD for future works.
Revised: SGD for future works.

Original: A.1 Experiments on MNIST
Revised: A.1 Experiments on MNIST

Original: We implemented the model of Definition 1 on Pytorch using the torchdiffeq package [11] andperformed experiments on the MNIST dataset.
Revised: We implemented the model of Definition 1 on Pytorch using the torchdiffeq package [11] and performed experiments on the MNIST dataset.

Original: Implementation using torchdiffeq .
Revised: Implementation using torchdiffeq .

Original: The model of Definition 1 is implemented as a succession ofconvolutional layers.
Revised: The model of Definition 1 is implemented as a succession of convolutional layers.

Original: Given some number of layers L the trained parameters consist of convolutionmatrices W k ∈ R C
Revised: Given some number of layers L the trained parameters consist of convolution matrices W k ∈ R C

Original: ×
Revised: ×

Original: C
Revised: C

Original: int × 3 × 3 for k ∈ (cid:74) 0 , L (cid:75) , with C the number of channels of the input image and
Revised: int × 3 × 3 for k ∈ (cid:74) 0 , L (cid:75) , with C the number of channels of the input image and

Original: C int some number of channels for the hidden layers.
Revised: C int some number of channels for the hidden layers.

Original: The control parameter v is defined at discretetime steps { k/L } 0 ≤ k ≤ L by:
Revised: The control parameter v is defined at discrete time steps { k/L } 0 ≤ k ≤ L by:

Original: [Equation]
Revised: [Equation]

Original: where U ∈ R C int × C × 3 × 3 is a fixed and untrained convolution matrix.
Revised: where U ∈ R C int × C × 3 × 3 is a fixed and untrained convolution matrix.

Original: We refer to this setting as a
Revised: We refer to this setting as a

Original: ResNet with RKHS residuals.
Revised: ResNet with RKHS residuals.

Original: On the other hand, we refer to the setting 
Revised: On the other hand, we refer to the setting where U is replaced at each layer by trained convolution matrices U k as ResNet with Single Hidden Layer (SHL) residuals.

Original: Remark 6.
Revised: Remark 6.

Original: By analogy with the definition of RKHSs generated by random features (Eq.
Revised: By analogy with the definition of RKHSs generated by random features (Eq.

Original: (20) ), theratio between the number of features and the dimension is here:
Revised: (20) ), the ratio between the number of features and the dimension is here:

Original: [Equation]
Revised: [Equation]

Original: For any t ∈ [0 , 1] , v t is defined by affine interpolation:
Revised: For any t ∈ [0 , 1] , v t is defined by affine interpolation:

Original: [Equation]
Revised: [Equation]

Original: with k =
Revised: with k =

Original: ⌊ tL ⌋ .
Revised: ⌊ tL ⌋ .

Original: The forward method consists in integrating the ODE of Eq.
Revised: The forward method consists in integrating the ODE of Eq.

Original: (6) with control parameter v using the torchdiffeq .
Revised: (6) with control parameter v using the torchdiffeq .

Original: odeint method [11].
Revised: odeint method [11].

Original: For some input z 0 define:
Revised: For some input z 0 define:

Original: [Equation]
Revised: [Equation]

Original: then for an image input x the model’s output is given by:
Revised: then for an image input x the model’s output is given by:

Original: [Equation]
Revised: [Equation]

Original: where A and B are small convolutional networks, fixed during the training of F .
Revised: where A and B are small convolutional networks, fixed during the training of F .

Original: These networksplay the same role as the matrices A and B in Definition 1, that is they are used for the purpose ofadjusting the data dimension.
Revised: These networks play the same role as the matrices A and B in Definition 1, that is they are used for the purpose of adjusting the data dimension.

Original: Hyperparameter tuning.
Revised: Hyperparameter tuning.

Original: Several choices of hyperparameters can affect the performances of themodel.
Revised: Several choices of hyperparameters can affect the performances of the model.

Original: • The convolution matrix U : as detailed in Section 5, the way the weights of U are sampleddetermines to which RKHS V belongs the control parameter v .
Revised: • The convolution matrix U : as detailed in Section 5, the way the weights of U are sampled determines to which RKHS V belongs the control parameter v .

Original: For the sake of simplicitywe choose to sample the coefficients of U as i.i.d.
Revised: For the sake of simplicity we choose to sample the coefficients of U as i.i.d.

Original: Gaussians.•
Revised: Gaussians.

Original:  The initialization of ( W k ) : the weights of the convolution matrices W k are initialized to 0 .For
Revised: • The initialization of ( W k ) : the weights of the convolution matrices W k are initialized to 0 .

Original:  an input image x the output is given at initialization by F (0 , x ) =
Revised: For an input image x the output is given at initialization by F (0 , x ) =

Original: B ( A ( x )) .•
Revised: B ( A ( x )) .

Original:  The integration method: torchdiffeq .
Revised: • The integration method: torchdiffeq .

Original: odeint allows the user to choose an integrationmethod.
Revised: odeint allows the user to choose an integration method.

Original: We observed an explicit midpoint method to offer a good trade-off between per- formance and numerical stability w.r.t.
Revised: We observed an explicit midpoint method to offer a good trade-off between performance and numerical stability w.r.t.

Original: other fixed-steps methods such as explicit Euler or RK4 .
Revised: other fixed-steps methods such as explicit Euler or RK4 .

Original: • The number of layers L : we tested our model for L ∈ { 5 , 10 , 20 } .
Revised: • The number of layers L : we tested our model for L ∈ { 5 , 10 , 20 } .

Original: This parameter controlsthe total number of parameters of the model.•
Revised: This parameter controls the total number of parameters of the model.

Original:  The networks A and B : their choice defines the dimension of space in which the forward ODE Eq.
Revised: • The networks A and B : their choice defines the dimension of space in which the forward ODE Eq.

Original: (6) is integrated, which is expected to have an important impact on the performances of the model (c.f. Section 5).
Revised: (6) is integrated, which is expected to have an important impact on the performances of the model (c.f. Section 5).

Original: Moreover, as the parameters ( W k ) are initialized at 0 ,the performances of the model before training are those of the concatenation B ◦ A .
Revised: Moreover, as the parameters ( W k ) are initialized at 0 , the performances of the model before training are those of the concatenation B ◦ A .

Original: Withouttraining, the classification error of B ◦ A is of 90% while with enough training, it can be as good as 2% .
Revised: Without training, the classification error of B ◦ A is of 90% while with enough training, it can be as good as 2% .

Original: We tested our model with different levels of training of B ◦ A 
Revised: We tested our model with different levels of training of B ◦ A .

Original: Results.
Revised: Results.

Original: Figure 1 shows the evolution of the performances of RKHS-NODEs while trained on the
Revised: Figure 1 shows the evolution of the performances of RKHS-NODEs while trained on the

Original: MNIST dataset.
Revised: MNIST dataset.

Original: The decay of the Empirical Risk is directly related to the decay of the classificationerror.
Revised: The decay of the Empirical Risk is directly related to the decay of the classification error.

Original: Without pretraining A and B , our model already achieves up to 98% accuracy on the test set.
Revised: Without pretraining A and B , our model already achieves up to 98% accuracy on the test set.

Original: When A and B are pretrained RKHS-NODE still improves on the starting accuracy: in this settingmore than 99% accuracy is reached.
Revised: When A and B are pretrained RKHS-NODE still improves on the starting accuracy: in this setting more than 99% accuracy is reached.

Original: Most importantly, Fig.
Revised: Most importantly, Fig.

Original: 1 shows that not training the hidden layers inside residual blocks does not significantly hinders the performances in classification.
Revised: 1 shows that not training the hidden layers inside residual blocks does not significantly hinders the performances in classification.

Original: Indeed,comparing the performances of ResNets with RKHS residuals and SHL residuals one observes a1% accuracy drop when training RKHS-NODE from scratch (Fig. 1a) and 0 .
Revised: Indeed, comparing the performances of ResNets with RKHS residuals and SHL residuals one observes a 1% accuracy drop when training RKHS-NODE from scratch (Fig. 1a) and 0 .

Original: 5% accuracy whennetworks A and B are pretrained (Fig. 1b).
Revised: 5% accuracy when networks A and B are pretrained (Fig. 1b).

Original: Finally we showcase the relevance of the analysis of Section 5 by training our model with a varyingnumber of input channels in Fig.
Revised: Finally we showcase the relevance of the analysis of Section 5 by training our model with a varying number of input channels in Fig.

Original: We observe a significant drop in convergence of the empiricalrisk with 4 channels compared with 8 and 32 channels.
Revised: We observe a significant drop in convergence of the empirical risk with 4 channels compared with 8 and 32 channels.

Original: Non-convergence of the empirical riskalso implies poorer performances in generalization.
Revised: Non-convergence of the empirical risk also implies poorer performances in generalization.

Original: Such results are coherent with the convergencecondition of Eq.
Revised: Such results are coherent with the convergence condition of Eq.

Original: (15): augmenting the data dimension allows to have global convergence when theloss at initialization is too high.
Revised: (15): augmenting the data dimension allows to have global convergence when the loss at initialization is too high.

Original: A.2 Experiments on CIFAR10 
Revised: A.2 Experiments on CIFAR

Original: We performed experiments on the CIFAR10 dataset, using an architecture inspired from
Revised: We performed experiments on the CIFAR10 dataset, using an architecture inspired from

Original: ResNet18 [25].
Revised: ResNet18 [25].

Original: Implementation.
Revised: Implementation.

Original: Our architecture relies on the ResNet18 architecture [25] but residual blocks arechanged and simplified (by removing the final non-linearity and the batch-normalization) to matchthe definition of RKHS-NODE (Definition 1).
Revised: Our architecture relies on the ResNet18 architecture [25] but residual blocks are changed and simplified (by removing the final non-linearity and the batch-normalization) to match the definition of RKHS-NODE (Definition 1).

Original: Each residual block consists in the composition of aconvolution U , a ReLU non-linearity and a convolution W .
Revised: Each residual block consists in the composition of a convolution U , a ReLU non-linearity and a convolution W .

Original: More precisely, for an input image x ,the output of the k th layer reads:
Revised: More precisely, for an input image x , the output of the k th layer reads:

Original: [Equation]
Revised: [Equation]

Original: where U k ∈ R C int × C × 3 × 3 , W k ∈ R C × C int × 3 × 3 are convolution matrices, C is the number ofchannels of the input image and C int is the number of channels of the hidden layer.
Revised: where U k ∈ R C int × C × 3 × 3 , W k ∈ R C × C int × 3 × 3 are convolution matrices, C is the number of channels of the input image and C int is the number of channels of the hidden layer.

Original: When bothconvolutions W k and U k are trained, we refer to these residuals as Single Hidden Layer (SHL)residuals .
Revised: When both convolutions W k and U k are trained, we refer to these residuals as Single Hidden Layer (SHL) residuals .

Original: In the framework of RKHS-NODE, all convolutions U k are fixed and set to the sameconvolution U .
Revised: In the framework of RKHS-NODE, all convolutions U k are fixed and set to the same convolution U .

Original: We refer to it as RKHS residuals .
Revised: We refer to it as RKHS residuals .

Original: Finally, ResNet18 consists of 4 blocks each containing 2 residual layers.
Revised: Finally, ResNet18 consists of 4 blocks each containing 2 residual layers.

Original: We keep 2 of our residualsin the first, second and fourth block but stack an arbitrary number D of residual layers in the thirdblock.
Revised: We keep 2 of our residuals in the first, second and fourth block but stack an arbitrary number D of residual layers in the third block.

Original: Thereby we refer to this third block as the NODE block, which performs the integrationof Eq.
Revised: Thereby we refer to this third block as the NODE block, which performs the integration of Eq.

Original: Note that compared to the residuals in the original ResNet18 architecture, batch-normalization atinput and output of the residuals as well as ReLU non-linearities are removed.
Revised: Note that compared to the residuals in the original ResNet18 architecture, batch-normalization at input and output of the residuals as well as ReLU non-linearities are removed.

Original: Moreover, in orderto reproduce the framework of Random Fourier Features (Eq.
Revised: Moreover, in order to reproduce the framework of Random Fourier Features (Eq.

Original: (20)), the weights of U are sampledas i.i.d.
Revised: (20)), the weights of U are sampled as i.i.d.

Original: gaussians and rescaled by a C − 1 / 2 int factor.
Revised: gaussians and rescaled by a C − 1 / 2 int factor.

Original: Finally, the weights of the convolutions W k areinitialized at 0 .
Revised: Finally, the weights of the convolutions W k are initialized at 0 .

Original: Such an initialization corresponds in many ways to the one proposed in [61].
Revised: Such an initialization corresponds in many ways to the one proposed in [61].

Original: Results.
Revised: Results.

Original: Fig.
Revised: Fig.

Original: 3 reports the training of RKHS-NODE on the CIFAR10 dataset.
Revised: 3 reports the training of RKHS-NODE on the CIFAR10 dataset.

Original: Figure 3a shows thetraining of RKHS-NODE (RKHS residuals) and is to be compared with Fig.
Revised: Figure 3a shows the training of RKHS-NODE (RKHS residuals) and is to be compared with Fig.

Original: 3b which shows thetraining of the same model but with trained hidden layers (SHL residuals).
Revised: 3b which shows the training of the same model but with trained hidden layers (SHL residuals).

Original: Our experiments showthat similar performances can be achieved: both ResNets achieve up to 88% accuracy on the testdataset.
Revised: Our experiments show that similar performances can be achieved: both ResNets achieve up to 88% accuracy on the test dataset.

Original: As a comparison, the ResNet18 original architecture can be trained to achieve up to 94%accuracy.
Revised: As a comparison, the ResNet18 original architecture can be trained to achieve up to 94% accuracy.

Original: Finally, Fig. 3 also compares the performances of the model depending on the number of layersinside the NODE block.
Revised: Finally, Fig. 3 also compares the performances of the model depending on the number of layers inside the NODE block.

Original: One observes significantly different behavior when there is no NODE ( 1layer) and one there is ( 10 and 20 layers): more layers are related to better performances both onthe train dataset and on the test dataset and both when hidden layers are trained or not.
Revised: One observes significantly different behavior when there is no NODE (layer) and one there is ( 10 and 20 layers): more layers are related to better performances both on the train dataset and on the test dataset and both when hidden layers are trained or not.

Original: However,one sees that the improvement related to adding more layers is limited: performances with 10 and20 layers are very similar and a NODE block with 1 layers already achieves 82% accuracy RKHSresiduals and 84% accuracy with SHL residuals.
Revised: However, one sees that the improvement related to adding more layers is limited: performances with 10 and 20 layers are very similar and a NODE block with 1 layers already achieves 82% accuracy RKHS residuals and 84% accuracy with SHL residuals.

Original: B Proofs of Section
Revised: B Proofs of Section

Original: We give a proof of Theorem 1.
Revised: We give a proof of Theorem 1.

Original: This essentially follows the proof given in [36].
Revised: This essentially follows the proof given in [36].

Original: Proof of Theorem 1.
Revised: Proof of Theorem 1.

Original: Assume the loss L satisfies Definition 2 with M and m and that Eq.
Revised: Assume the loss L satisfies Definition 2 with M and m and that Eq.

Original: (9) issatisfied at initialization v 0 ∈ R m .
Revised: (9) is satisfied at initialization v 0 ∈ R m .

Original: The proof proceeds by induction over the gradient step k
Revised: The proof proceeds by induction over the gradient step k

Original: Assume the convergence rate and the regularization bound of Eq.
Revised: Assume the convergence rate and the regularization bound of Eq.

Original: (10) are satisfied for every l ≤ k .
Revised: (10) are satisfied for every l ≤ k .

Original: Then at step k + 1 :
Revised: Then at step k + 1 :

Original: [Equation]
Revised: [Equation]

Original: ∥ )

Original: L ( v l ) .

Original: 10 −10 −

Revised: 10 −10 −

Original: E m p
Revised: E m p

Original: i r i c a l R
Revised: i r i c a l R

Original: i s k
Revised: i s k

Original: L     10 −
Revised: L     10 −

Original: C l a ss i ﬁ c a t i o n e rr o r 5 layer
Revised: C l a ss i ﬁ c a t i o n e rr o r 5 layer

Original: 10 layer 20 layer
Revised: 10 layer 20 layer

Original: [Figure]
Revised: [Figure]

Original: 10 −10 −10 −
Revised: 10 −10 −10 −

Original: E m p
Revised: E m p

Original: i r i c a l R
Revised: i r i c a l R

Original: i s k
Revised: i s k

Original: L     10 −2 × 10 −3 × 10 −
Revised: L     10 −2 × 10 −3 × 10 −

Original: C l a ss i ﬁ c a t i o n e rr o r 5 layer
Revised: C l a ss i ﬁ c a t i o n e rr o r 5 layer

Original: 10 layer 20 layer
Revised: 10 layer 20 layer

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: E m p
Revised: E m p

Original: i r i c a l R
Revised: i r i c a l R

Original: i s k
Revised: i s k

Original: L     10 −
Revised: L     10 −

Original: C l a ss i ﬁ c a t i o n e r
Revised: C l a ss i ﬁ c a t i o n e r

Original: r
Revised: r

Original: o r
Revised: o r

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: E m p
Revised: m p

Original: i r i c a l R
Revised: i r i c a l R

Original: i s k L
Revised: i s k L

Original: [Figure]
Revised: [Figure]

Original: C l a ss i ﬁ c a t i o n e rr o r 1 layer
Revised: C l a ss i ﬁ c a t i o n e rr o r 1 layer

Original: 10 layer 20 layer
Revised: 10 layer 20 layer

Original: [Figure]
Revised: [Figure]

Original: E m p
Revised: E m p

Original: i r i c a l R
Revised: i r i c a l R

Original: i s k L
Revised: i s k L

Original: [Figure]
Revised: [Figure]

Original: Using the induction hypothesis and setting µ = m ( ∥ v 0 ∥ + R ) we have:
Revised: Using the induction hypothesis and setting µ = m ( ∥ v 0 ∥ + R ) we have:

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is Eq.
Revised: where the last inequality is Eq.

Original: We thus recovered the regularization bound of Eq.
Revised: We thus recovered the regularization bound of Eq.

Original: (10) at stepk + 1 .
Revised: (10) at step k + 1 .

Original: Moreover, because v k +1 is located in B ( v 0 , R ) we have thanks to the smoothness assumption:
Revised: Moreover, because v k +1 is located in B ( v 0 , R ) we have thanks to the smoothness assumption:

Original: [Equation]
Revised: [Equation]

Original: because η ≤ β − 1 .
Revised: because η ≤ β − 1 .

Original: Thus using the lower bound in the PL inequality Eq.
Revised: Thus using the lower bound in the PL inequality Eq.

Original: [Equation]
Revised: [Equation]

Original: which gives the convergence rate of Eq.
Revised: which gives the convergence rate of Eq.

Original: (10) at step k + 1 by induction on k .
Revised: (10) at step k + 1 by induction on k .

Original: C Proofs of Section 4C.1 About the definition of RKHS-NODE
Revised: C Proofs of SectionC.1 About the definition of RKHS-NODE

Original: Before deriving proofs for the properties of our RKHS-NODE model, it is interesting to study carefully the well-posedness of Definition 1.
Revised: Before deriving proofs for the properties of our RKHS-NODE model, it is interesting to study carefully the well-posedness of Definition 1.

Original: Indeed, because the control parameter v is only integrablein time and not continuous, the Cauchy-Lipschitz theorem does not ensure that there exist solutions to Eq.
Revised: Indeed, because the control parameter v is only integrable in time and not continuous, the Cauchy-Lipschitz theorem does not ensure that there exist solutions to Eq.

Original: Instead we rely on a weaker notion of solution and use a result from Carathéodory(Section I.5 in [23]).
Revised: Instead we rely on a weaker notion of solution and use a result from Carathéodory (Section I.5 in [23]).

Original: Proposition 3.
Revised: Proposition 3.

Original: Let V be some RKHS satisfying Assumption 1 and v ∈ L 2 ([0 , 1] , V ) be some controlparameter.
Revised: Let V be some RKHS satisfying Assumption 1 and v ∈ L 2 ([0 , 1] , V ) be some control parameter.

Original: Then for every x ∈ R
Revised: Then for every x ∈ R

Original: d there exists a unique solution z of Eq.
Revised: d there exists a unique solution z of Eq.

Original: (6) in the weak sense ofabsolutely continuous functions.
Revised: (6) in the weak sense of absolutely continuous functions.

Original: More precisely there exists a unique z ∈ H 1 ([0 , 1] , R q ) such thatfor every t ∈ [0 , 1] :
Revised: More precisely there exists a unique z ∈ H 1 ([0 , 1] , R q ) such that for every t ∈ [0 , 1] :

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The map ( t, z ) ∈ [0 , 1] × R q (cid:55)→ v t ( z ) is measurable and by Assumption 1 we have for everyt ∈ [0 , 1] and every z ∈ R q :
Revised: The map ( t, z ) ∈ [0 , 1] × R q (cid:55)→ v t ( z ) is measurable and by Assumption 1 we have for every t ∈ [0 , 1] and every z ∈ R q :

Original: [Equation]
Revised: [Equation]

Original: whose upper-bound is integrable w.r.t.
Revised: whose upper-bound is integrable w.r.t.

Original: t ∈ [0 , 1] .
Revised: t ∈ [0 , 1] .

Original: Then, applying Theorem 5.1 of [23] gives a uniqueabsolutely continuous solution z of Eq.
Revised: Then, applying Theorem 5.1 of [23] gives a unique absolutely continuous solution z of Eq.

Original: Applying Assumption 1 once again, we have that ˙
Revised: Applying Assumption 1 once again, we have that ˙

Original: z issquare integrable and thus z is in H 1 .
Revised: z is square integrable and thus z is in H 1 .

Original: In the paper, every equality implying derivatives has to be understood in the sense of weak derivatives of H 1 functions.
Revised: In the paper, every equality implying derivatives has to be understood in the sense of weak derivatives of H 1 functions.

Original: In particular, this notion allows to perform integration by parts, which is usedin the following proof of Property 1.
Revised: In particular, this notion allows to perform integration by parts, which is used in the following proof of Property 1.

Original: Proof Property 1.
Revised: Proof Property 1.

Original: Consider the optimization problem of minimizing the empirical risk of Eq.
Revised: Consider the optimization problem of minimizing the empirical risk of Eq.

Original: (7)with F the RKHS-NODE model of Definition 1 and a dataset ( x i , y i ) 1 ≤ i ≤ N ∈
Revised: (7) with F the RKHS-NODE model of Definition 1 and a dataset ( x i , y i ) 1 ≤ i ≤ N ∈

Original: ( R d × R d ′ ) N .
Revised: ( R d × R d ′ ) N .

Original: Introducing for every index i ∈ (cid:74) 1 , N (cid:75) the variables z i ∈ H 1 ([0 , 1] , R q ) solutions of Eq.
Revised: Introducing for every index i ∈ (cid:74) 1 , N (cid:75) the variables z i ∈ H 1 ([0 , 1] , R q ) solutions of Eq.

Original: (6), thiscan be viewed as an optimisation problem over (( z i )
Revised: (6), this can be viewed as an optimisation problem over (( z i )

Original: i , v ) under the constraint that Eq.
Revised: i , v ) under the constraint that Eq.

Original: (6) is satisfied:
Revised: (6) is satisfied:

Original: [Equation]
Revised: [Equation]

Original: i [Equation] with ∀ i ∈ (cid:74)1 , N (cid:75) , (cid:26) ˙ z it
Revised: with ∀ i ∈ (cid:74)

Original: [Equation]
Revised: [Equation]

Original: v

Original: [Equation]

Original: ( z

Original: [Equation] z [Equation] Introducing the adjoint variables ( p i ) i ∈ H 1 ( R q ) N , the Lagrangian of the optimization problem isdefined as:
Revised: Introducing the adjoint variables ( p i ) i ∈ H 1 ( R q ) N , the Lagrangian of the optimization problem is defined as:

Original: [Equation]
Revised: [Equation]

Original: where the second equality is established by integration by parts.
Revised: where the second equality is established by integration by parts.

Original: Therefore, the condition for optimality over z i is equivalent to Eq.
Revised: Therefore, the condition for optimality over z i is equivalent to Eq.

Original: For every index i :
Revised: For every index i :

Original: [Equation]
Revised: [Equation]

Revised: which has to be understand in the sense

Original: The gradient of L is obtained by differentiating over the v variable.
Revised: The gradient of L is obtained by differentiating over the v variable.

Original: Denoting δ pz the linear formv (cid:55)→ ⟨ v ( z ) , p ⟩ , we have:
Revised: Denoting δ pz the linear form v (cid:55)→ ⟨ v ( z ) , p ⟩ , we have:

Original: [Equation]
Revised: [Equation]

Original: with K the kernel function of the RKHS V and K ∗ : V ∗ → V the associated isometry 1 .
Revised: with K the kernel function of the RKHS V and K ∗ : V ∗ → V the associated isometry 1 .

Original: C.2 Proof of Property
Revised: C.2 Proof of Property

Original: We prove here that for any given dataset ( x i , y i ) 1 ≤ i ≤ N ∈
Revised: We prove here that for any given dataset ( x i , y i ) 1 ≤ i ≤ N ∈

Original: ( R d × R d ′ ) N , the empirical risk L associated with the RKHS-NODE model satisfies a (local) Polyak-Lojasiewicz property.
Revised: ( R d × R d ′ ) N , the empirical risk L associated with the RKHS-NODE model satisfies a (local) Polyak-Lojasiewicz property.

Original: As stated in
Revised: As stated in

Original: Property 2.
Revised: Property 2.

Original: The proof uses Assumption 1 to derive estimates on the solutions of Eq.
Revised: The proof uses Assumption 1 to derive estimates on the solutions of Eq.

Original: (6) and Eq.
Revised: (6) and Eq.

Original: (11),which we give in the following lemma:
Revised: (11), which we give in the following lemma:

Original: Lemma 1.
Revised: Lemma 1.

Original: Let V satisfy Assumption 1 with constant κ and let v ∈ L 2 ([0 , 1] , V ) be some controlparameter.(i)
Revised: Let V satisfy Assumption 1 with constant κ and let v ∈ L 2 ([0 , 1] , V ) be some control parameter.

Original:  Let ( z i ) 1 ≤ i ≤ N be the solutions of Eq.
Revised: (i) Let ( z i ) 1 ≤ i ≤ N be the solutions of Eq.

Original: (6) for some data inputs ( x i ) 1 ≤ i ≤ N ∈
Revised: (6) for some data inputs ( x i ) 1 ≤ i ≤ N ∈

Original: ( R d ) N .
Revised: ( R d ) N .

Original: Then for752 every indices i, j ∈ [Equation] 1 , N (cid:75) and every time t ∈ [0 , 1] :
Revised: Then for every indices i, j ∈ (cid:74) 1 , N (cid:75) and every time t ∈ [0 , 1] :

Original: [Equation]
Revised: [Equation]

Original: (22) (ii) Let ( p i ) 1 ≤ i ≤ N be the solutions of Eq.
Revised: (22) (ii) Let ( p i ) 1 ≤ i ≤ N be the solutions of Eq.

Original: (11) associated with ( z i ) 1 ≤ i ≤ N with objective outputs( y i ) 1 ≤ i ≤ N ∈
Revised: (11) associated with ( z i ) 1 ≤ i ≤ N with objective outputs ( y i ) 1 ≤ i ≤ N ∈

Original: ( R d ′ ) N .
Revised: ( R d ′ ) N .

Original: Then for every i ∈ (cid:74) 1 , N (cid:75) and every time t ∈ [0 , 1] :
Revised: Then for every i ∈ (cid:74) 1 , N (cid:75) and every time t ∈ [0 , 1] :

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma 1.
Revised: Proof of Lemma 1.

Original: Proof of (i) Let i, j ∈ (cid:74) 1 , N (cid:75) .
Revised: Proof of (i) Let i, j ∈ (cid:74) 1 , N (cid:75) .

Original: Assume by contradiction that for some timet ∈ [0 , 1] we have:
Revised: Assume by contradiction that for some time t ∈ [0 , 1] we have:

Original: [Equation]
Revised: [Equation]

Original: Then because z i and z j are absolutely continuous, ∥
Revised: Then because z i and z j are absolutely continuous, ∥

Original: z i − z j ∥ 2 is absolutely continuous and for anytime s ∈ [0 , 1] :
Revised: z i − z j ∥ 2 is absolutely continuous and for any time s ∈ [0 , 1] :

Original: [Equation]
Revised: [Equation]

Original: where the inequality follows from ∥ Dv r ∥ 2 , ∞ ≤ κ ∥ v r ∥ V .
Revised: where the inequality follows from ∥ Dv r ∥ 2 , ∞ ≤ κ ∥ v r ∥ V .

Original: Applying Grönwall’s lemma, we have:
Revised: Applying Grönwall’s lemma, we have:

Original: [Equation]
Revised: [Equation]

Original: and by setting s = 0 :
Revised: and by setting s = 0 :

Original: [Equation]
Revised: [Equation]

Original: which is a contradiction.
Revised: which is a contradiction.

Original: Therefore for any time t ∈ [0 , 1] :
Revised: Therefore for any time t ∈ [0 , 1] :

Original: [Equation]
Revised: [Equation]

Original: and the result follows by considering the initial condition z
Revised: and the result follows by considering the initial condition z

Original: i 

Revised: i

Original: = Ax i .
Revised: = Ax i .

Original: Proof of (ii) Let i ∈ (cid:74) 1 , N (cid:75) be any index and let p i be the solution of Eq.
Revised: Proof of (ii) Let i ∈ (cid:74) 1 , N (cid:75) be any index and let p i be the solution of Eq.

Original: (11) with initial conditionp i 1 = − 1 N B ⊤
Revised: (11) with initial condition p i 1 = − 1 N B ⊤

Original: Bz i 1 − y i ) .
Revised: Bz i 1 − y i ) .

Original: Then because p i is absolutely continuous, ∥ p i ∥ is absolutely continuousand for any time t ≤ s ∈ [0 , 1] :
Revised: Then because p i is absolutely continuous, ∥ p i ∥ is absolutely continuous and for any time t ≤ s ∈ [0 , 1] :

Original: [Equation]
Revised: [Equation]

Original: so that using Assumption 1 we have:
Revised: so that using Assumption 1 we have:

Original: [Equation]
Revised: [Equation]

Original: Using Grönwall’s lemma in the first inequality and setting s = 

Revised: Using Grönwall’s lemma in the first inequality and setting s =

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: and proceeding by contradiction (such as in (i))
Revised: and proceeding by contradiction (such as in (i))

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: The result follows by considering the initial condition on p i 1 .
Revised: The result follows by considering the initial condition on p i 1 .

Original: Provided those estimates on z i and p i , it remains to use Assumption 2 in order to conclude.
Revised: Provided those estimates on z i and p i , it remains to use Assumption 2 in order to conclude.

Original: Proof of Property 2.
Revised: Proof of Property 2.

Original: Let v ∈ L 2 ([0 , 1] , V ) and consider the form of the gradient of L given by Property 1 with ( z i ) 1 ≤ i ≤ N the solutions of Eq.
Revised: Let v ∈ L 2 ([0 , 1] , V ) and consider the form of the gradient of L given by Property 1 with ( z i ) 1 ≤ i ≤ N the solutions of Eq.

Original: (6) and ( p i ) 1 ≤ i ≤ N the solutions of Eq.
Revised: (6) and ( p i ) 1 ≤ i ≤ N the solutions of Eq.

Original: Let t ∈ [0 , 1] ,773 then by definition of the norm in RKHSs:
Revised: Let t ∈ [0 , 1] , then by definition of the norm in RKHSs:

Original: [Equation]
Revised: [Equation]

Original: where we recall that K is the kernel associated with V .
Revised: where we recall that K is the kernel associated with V .

Original: Noting p :
Revised: Noting p :

Original: = ( p it )
Revised: = ( p it )

Original: ∈ R Nq , the vector of thestacked ( p it ) 1 ≤ i ≤ N , and K the kernel matrix associated with the family of points ( z it )
Revised: ∈ R Nq , the vector of the stacked ( p it ) 1 ≤ i ≤ N , and K the kernel matrix associated with the family of points ( z it )

Original: i , we have:
Revised: i , we have:

Original: [Equation]
Revised: [Equation]

Original: Then by Assumption 2, there exists a non-increasing function λ and a constant Λ such that:
Revised: Then by Assumption [Equation] there exists a non-increasing function λ and a constant Λ such that:

Original: [Equation]
Revised: [Equation]

Original: Using (i) in Lemma 1 we have:
Revised: Using (i) in Lemma 1 we have:

Original: [Equation]
Revised: [Equation]

Original: where δ : =
Revised: where δ : =

Original: min 1 ≤ i,j ≤ N ∥ x
Revised: min 1 ≤ i,j ≤ N ∥ x

Original: i − x j ∥ is the data separation.
Revised: i − x j ∥ is the data separation.

Original: Finally the result follows by using (ii).
Revised: Finally the result follows by using (ii).

Original: 779 More precisely:
Revised: More precisely:

Original: [Equation]
Revised: [Equation]

Original: and in the same manner:
Revised: and in the same manner:

Original: [Equation]
Revised: [Equation]

Original: C.3 Proof of Theorem2
Revised: C.3 Proof of Theorem

Original: Theorem 2 is a direct consequence of Property 2.
Revised: Theorem 2 is a direct consequence of Property 2.

Original: In order to apply Theorem 1, it suffices to showthat L satisfies some smoothness assumption as defined in Definition 3:
Revised: In order to apply Theorem 1, it suffices to show that L satisfies some smoothness assumption as defined in Definition 3:

Original: Property 3 (Smoothness of L ) .
Revised: Property 3 (Smoothness of L ) .

Original: Let V be some RKHS satisfying Assumption 1.
Revised: Let V be some RKHS satisfying Assumption 1.

Original: Let L be the empirical risk defined on L 2 ([0 , 1] , V ) and associated with the RKHS-NODE model.
Revised: Let L be the empirical risk defined on L 2 ([0 , 1] , V ) and associated with the RKHS-NODE model.

Original: Then there exists acontinuous function C : R + → R ∗ + such that for every R ≥ 0 and every v, ¯ v ∈ L 2 ([0 , 1] , V ) with 
Revised: Then there exists a continuous function C : R + → R ∗ + such that for every R ≥ 0 and every v, ¯ v ∈ L 2 ([0 , 1] , V ) with ∥ v ∥ L 2 , ∥ ¯ v ∥ L 2 ≤ R :

Original: [Equation]
Revised: [Equation]

Original: We note κ the constant associated with Assumption 1.
Revised: We note κ the constant associated with Assumption 1.

Original: The proof of Property 3 relies on the followinglemma:
Revised: The proof of Property 3 relies on the following lemma:

Original: Lemma 2.
Revised: Lemma 2.

Original: Let v, ¯ v ∈ L 2 ([0 , 1] , V ) be some control parameters and R ≥ 0 be some radius suchthat ∥ v ∥ L 2 , ∥ ¯ v ∥
Revised: Let v, ¯ v ∈ L 2 ([0 , 1] , V ) be some control parameters and R ≥ 0 be some radius such that ∥ v ∥ L 2 , ∥ ¯ v ∥

Original: L 2 ≤ R .
Revised: L 2 ≤ R .

Original: Let ( x, y ) ∈ R d × R d ′ be some pair of data input / objective output.
Revised: Let ( x, y ) ∈ R d × R d ′ be some pair of data input / objective output.

Original: (i) Let z, ¯ z be solutions of Eq.
Revised: (i) Let z, ¯ z be solutions of Eq.

Original: (6) with parameter v and ¯ v respectively and with the same initialcondition
Revised: (6) with parameter v and ¯ v respectively and with the same initial condition

Original: Ax , then for any t ∈ [0 , 1] :
Revised: Ax , then for any t ∈ [0 , 1] :

Original: [Equation]
Revised: [Equation]

Original: (ii) Let p, ¯ p be solutions of Eq.
Revised: (ii) Let p, ¯ p be solutions of Eq.

Original: (11) with parameter v and ¯ v respectively and with initial condition
Revised: (11) with parameter v and ¯ v respectively and with initial conditionN B ⊤ ( Bz 1 − y )

Original: 1 N B ⊤ ( Bz 1 − y ) and 1 N B ⊤ ( B ¯ z 1 − y ) , then for any t ∈ [0 , 1] : 
Revised: and 1 N B ⊤ ( B ¯ z 1 − y ) , then for any t ∈ [0 , 1] : ∥ p [Equation] − ¯

Original: [Equation]
Revised: [Equation]

Original: ) (cid:3) .

Original: Proof of Lemma 2.
Revised: Proof of Lemma 2.

Original: Proof of (i)
Revised: Proof of (i)

Original: For every time t ∈ [0 , 1] we have:
Revised: For every time t ∈ [0 , 1] we have:

Original: [Equation]
Revised: [Equation]

Original: and by triangle inequality:
Revised: and by triangle inequality:

Original: [Equation]
Revised: [Equation]

Original: where we used Assumption 1 in the second inequality.
Revised: where we used Assumption 1 in the second inequality.

Original: Therefore, by Grönwall’s lemma:
Revised: Therefore, by Grönwall’s lemma:

Original: [Equation]
Revised: [Equation]

Original: Proof of (ii) For any t ∈ [0 , 1] we have:
Revised: Proof of (ii) For any t ∈ [0 , 1] we have:

Original: [Equation]
Revised: [Equation]

Original: and using the triangle inequality and Assumption 1:
Revised: and using the triangle inequality and Assumption 1:

Original: [Equation]
Revised: [Equation]

Original: Then, using Grönwall’s lemma backward in time gives:
Revised: Then, using Grönwall’s lemma backward in time gives:

Original: [Equation]
Revised: [Equation]

Original: On one hand, because of (i) we have for every s ∈ [0 , 1] :
Revised: On one hand, because of (i) we have for every s ∈ [0 , 1] :

Original: [Equation]
Revised: [Equation]

Original: and also:
Revised: and also:

Original: [Equation]
Revised: [Equation]

Original: On the other hand, recalling (ii) of Lemma 1, for every s ∈ [0 , 1] :
Revised: On the other hand, recalling (ii) of Lemma 1, for every s ∈ [0 , 1] :

Original: [Equation]
Revised: [Equation]

Original: Putting these estimates in the preceding inequality gives:
Revised: Putting these estimates in the preceding inequality gives:

Original: [Equation]
Revised: [Equation]

Original: which is the desired result.
Revised: which is the desired result.

Original: Proof of Property 3.
Revised: Proof of Property 3.

Original: Let v, ¯ v ∈ L 2 ([0 , 1] , V ) with ∥ v ∥ L 2 , ∥ ¯ v ∥ L 2 ≤ R .
Revised: Let v, ¯ v ∈ L 2 ([0 , 1] , V ) with ∥ v ∥ L 2 , ∥ ¯ v ∥ L 2 ≤ R .

Original: Then taking the same nota-810 tion as in Lemma 2, we have for any t∈ [0 , 1] : [Equation]
Revised: Then taking the same nota- tion as in Lemma 2, we have for any t

Original: [Equation]
Revised: [Equation]

Original: ¯ p it , and we can write ∥∇ L ( v ) t − ∇ L (¯ v ) t
Revised: , and we can write ∥∇ L ( v ) t − ∇ L (¯ v ) t

Original: ∥ V ≤ T 1 + T 2 with:
Revised: ∥ V ≤ T 1 + T 2 with:

Original: [Equation]

Original: (cid:88)

Original: i

Original: [Equation]
Revised: [Equation]

Original: First we consider deriving an upper bound on T 1 .
Revised: First we consider deriving an upper bound on T 1 .

Original: Note that by the definition of the norm in RKHSsand by Assumption 2 we have:
Revised: Note that by the definition of the norm in RKHSs and by Assumption 2 we have:

Original: [Equation]
Revised: [Equation]

Original: Therefore, using (ii) from Lemma 2 to bound ∥ p it − ¯ p it ∥ for every index i we get:
Revised: Therefore [Equation] using (ii) from Lemma 2 to bound ∥ p it − ¯ p it ∥ for every index i we get:

Original: [Equation]
Revised: [Equation]

Original: with:
Revised: with:

Original: [Equation]
Revised: [Equation]

Original: where we recognised L (¯ v ) in the third line.
Revised: where we recognised L (¯ v ) in the third line.

Original: By continuity of L we can define for every R ≥ 0 :
Revised: By continuity of L we can define for every R ≥ 0 :

Original: [Equation]
Revised: [Equation]

Original: And therefore:
Revised: And therefore:

Original: [Equation]
Revised: [Equation]

Original: We then consider deriving an upper-bound on T 2 .
Revised: We then consider deriving an upper-bound on T 2 .

Original: By triangle inequality:
Revised: By triangle inequality:

Original: [Equation]
Revised: [Equation]

Original: Consider any α ∈ V , then for any index i ∈ (cid:74) 1 , N (cid:75) , by the reproducing property:
Revised: Consider any α ∈ V , then for any index i ∈ (cid:74) 1 , N (cid:75) , by the reproducing property:

Original: [Equation]
Revised: [Equation]

Original: where we used the Cauchy-Schwarz inequality and Assumption 1 applied to α .
Revised: where we used the Cauchy-Schwarz inequality and Assumption 1 applied to α .

Original: Therefore, by duality:
Revised: Therefore, by duality:

Original: [Equation]
Revised: [Equation]

Original: Using the estimates of Lemma 1 and Lemma 2 we get:
Revised: Using the estimates of Lemma 1 and Lemma 2 we get:

Original: [Equation]
Revised: [Equation]

Original: And finally, using Cauchy-Schwarz inequality and recognizing L (¯ v ) we have:
Revised: And finally, using Cauchy-Schwarz inequality and recognizing L (¯ v ) we have:

Original: [Equation]
Revised: [Equation]

Original: with:
Revised: with:

Original: [Equation]
Revised: [Equation]

Original: Therefore we obtain the result by setting:
Revised: Therefore we obtain the result by setting:

Original: [Equation]
Revised: [Equation]

Original: Provided with Property 3, we can finish the proof of Theorem 2.
Revised: Provided with Property 3, we can finish the proof of Theorem 2.

Original: Proof of Theorem 2.
Revised: Proof of Theorem 2.

Original: By Property 2, L satisfies the PL inqualities of Definition 2 and the proof is adirect corollary of Theorem 1.
Revised: By Property 2, L satisfies the PL inqualities of Definition 2 and the proof is a direct corollary of Theorem 1.

Original: It only remains to show that the smoothness condition of Definition 3is verified.
Revised: It only remains to show that the smoothness condition of Definitionis verified.

Original: Let v, ¯ v ∈ L 2 ([0 , 1] , V ) such that ∥ v ∥ L 2 , ∥ ¯ v ∥ L 2 ≤ R for some radius R ≥ 0 .
Revised: Let v, ¯ v ∈ L 2 ([0 , 1] , V ) such that ∥ v ∥ L 2 , ∥ ¯ v ∥ L 2 ≤ R for some radius R ≥ 0 .

Original: Then we have:
Revised: Then we have:

Original: [Equation]
Revised: [Equation]

Original: Using Property 3, there exists some C ( R ) such that:
Revised: Using Property 3, there exists some C ( R ) such that:

Original: [Equation]
Revised: [Equation]

Original: This gives the inequality:
Revised: This gives the inequality:

Original: [Equation]
Revised: [Equation]

Original: which is the desired result.
Revised: which is the desired result.

Original: D Proofs of Section5
Revised: D Proofs of Section

Original: The results in Section 5 show how the condition for convergence in Eq.
Revised: The results in Section 5 show how the condition for convergence in Eq.

Original: (15) can be enforced by considering suitable RKHSs of vector-fields and suitable matrices A and B .
Revised: (15) can be enforced by considering suitable RKHSs of vector-fields and suitable matrices A and B .

Original: We give in Appendix D.3examples of suitable kernels.
Revised: We give in Appendix D. examples of suitable kernels.

Original: In the following, we assume that for every q ≥ 1 we are provided with a function k q : R + → Rsuch that the induced symmetric rotationally-invariant kernel K q defined by:
Revised: In the following, we assume that for every q ≥ 1 we are provided with a function k q : R + → R such that the induced symmetric rotationally-invariant kernel K q defined by:

Original: [Equation]
Revised: [Equation]

Original: (23) is a positive-definite kernel over R q .
Revised: is a positive-definite kernel over R q .

Original: Without loss of generality, one can assume k q to be normalized,that is k q (0)
Revised: Without loss of generality, one can assume k q to be normalized, that is k q (0)

Original: = 1 .
Revised: = 1 .

Original: We note V q the vector-valued RKHS associated with K q .
Revised: We note V q the vector-valued RKHS associated with K q .

Original: The properties of V q arethen entirely determined by k q .
Revised: The properties of V q are then entirely determined by k q .

Original: In particular, smoothness of the kernel at 0 implies regularity of thevector-fields in V q :
Revised: In particular, smoothness of the kernel at 0 implies regularity of the vector-fields in V q :

Original: Property 4 (Regularity of V q ) .
Revised: Property 4 (Regularity of V q ) .

Original: Let k q : R + → R be some function defining a positive symmetrickernel K q .
Revised: Let k q : R + → R be some function defining a positive symmetric kernel K q .

Original: If k q is 4 times differentiable at 0 , with k ′ q (0)
Revised: If k q is 4 times differentiable at 0 , with k ′ q (0)

Original: =
Revised: =

Original: k (3) q (0) = 0 .
Revised: k (3) q (0) = 0 .

Original: Then V q satisfies Assumption 1 with constant κ = (cid:112) [Equation]
Revised: Then V q satisfies Assumption 1 with constant κ = (cid:112) k q (0) + (cid:113) − k ′′ q (0) + (cid:113) k (4) q (0) .

Original: As a consequence, if the derivatives of k q can be bounded uniformly over q then V q satisfies Assumption 1 with some constant κ independent of q .
Revised: As a consequence, if the derivatives of k q can be bounded uniformly over q then V q satisfies Assumption 1 with some constant κ independent of q .

Original: This, is the case for the Matérn kernel k definedin Eq.
Revised: This, is the case for the Matérn kernel k defined in Eq.

Original: Proof.
Revised: Proof.

Original: The proof proceeds by duality arguments.
Revised: The proof proceeds by duality arguments.

Original: For q ≥ 1 , consider some v ∈ V q .
Revised: For q ≥ 1 , consider some v ∈ V q .

Original: Then for anyz ∈ R q and any α ∈ V q , by the reproducing properties of RKHSs:
Revised: Then for any z ∈ R q and any α ∈ V q , by the reproducing properties of RKHSs:

Original: [Equation]
Revised: [Equation]

Original: Therefore, by duality ∥ v ( z ) ∥ ≤ (cid:112) k q (0) ∥ v ∥ V q and then by taking the supremum over z ∈ R q :
Revised: Therefore, by duality ∥ v ( z ) ∥ ≤ (cid:112) k q (0) ∥ v ∥ V q and then by taking the supremum over z ∈ R q :

Original: [Equation]
Revised: [Equation]

Original: Then for any z ∈ R q any α, β ∈ R q and any h ∈ R + :
Revised: Then for any z ∈ R q any α, β ∈ R q and any h ∈ R + :

Original: [Equation]
Revised: [Equation]

Original: In the r.h.s we have using Taylor’s expansion of k q at 0 :
Revised: In the r.h.s we have using Taylor’s expansion of k q at 0 :

Original: [Equation]
Revised: [Equation]

Original: Taking the limit h → 0 :
Revised: Taking the limit h → 0 :

Original: [Equation]
Revised: [Equation]

Original: and therefore ∥ Dv ( z ) ∥ 2 ≤ (cid:113) [Table]
Revised: and therefore ∥ Dv ( z ) ∥ 2 ≤ (cid:113) − k ′′ q (0) ∥ v ∥ V q .

Original: Finally, let us bound ∥ D 2 v ∥ 2 , ∞ .
Revised: Finally, let us bound ∥ D 2 v ∥ 2 , ∞ .

Original: For any z ∈ R q any α, β, γ ∈ R q and any h, l ≥ 

Revised: For any z ∈ R q any α, β, γ ∈ R q and any h, l ≥

Original: we have in the859 same manner:
Revised: we have in the same manner:

Original: [Equation]
Revised: [Equation]

Original: where the second line is obtained by Taylor expansion of k q at 0 .
Revised: where the second line is obtained by Taylor expansion of k q at 0 .

Original: Thus, taking the limit h, l → 0 :
Revised: Thus, taking the limit h, l → 0 :

Original: [Equation]
Revised: [Equation]

Original: and therefore ∥ D 2 v ( z ) ∥ 2 ≤
Revised: and therefore ∥ D 2 v ( z ) ∥ 2 ≤

Original: [Equation]
Revised: [Equation]

Original: Setting κ = (cid:112) k q (0) + (cid:113) − k ′′ q (0) + (cid:113) k
Revised: Setting κ = (cid:112) k q (0) + (cid:113) − k ′′ q (0) + (cid:113) k

Original: (4) q (0) we obtain the result.
Revised: (4) q (0) we obtain the result.

Original: Moreover, choosing appropriatev in the above proof, inequalities become sharp and one observes that the constant κ is optimal.
Revised: Moreover, choosing appropriate v in the above proof, inequalities become sharp and one observes that the constant κ is optimal.

Original: D.1 Enforcing convergence with high dimensional lifting and universal kernels
Revised: D.1 Enforcing convergence with high dimensional lifting and universal kernels

Original: Here we investigate the dependency of Eq.
Revised: Here we investigate the dependency of Eq.

Original: (15) w.r.t.
Revised: (15) w.r.t.

Original: q , δ and N for the class of RKHS V q andthereby recover the proof of Proposition 1.
Revised: q , δ and N for the class of RKHS V q and thereby recover the proof of Proposition 1.

Original: We make the following assumption concerning the decay of k q at infinity:
Revised: We make the following assumption concerning the decay of k q at infinity:

Original: Assumption 3 (Decay of k q ) .
Revised: Assumption 3 (Decay of k q ) .

Original: For every q ≥ 1 , k q ( x ) tends to 0 when x tends to infinity and wenote β q,N > 0 s.t.:
Revised: For every q ≥ 1 , k q ( x ) tends to 0 when x tends to infinity and we note β q,N > 0 s.t.:

Original: [Equation]
Revised: [Equation]

Original: Moreover for fixed N we assume that
Revised: Moreover for fixed N we assume that

Original: [Equation]
Revised: [Equation]

Original: D.1.1 Lifting matrices
Revised: .1 Lifting matrices

Original: For any q ≥ 1 we consider here the matrices:
Revised: For any q ≥ 1 we consider here the matrices:

Original: [Equation]
Revised: [Equation]

Original: where there are ⌊ q/d ⌋ copies of Id d in A q .
Revised: where there are ⌊ q/d ⌋ copies of Id d in A q .

Original: In particular we have:
Revised: In particular we have:

Original: [Equation]
Revised: [Equation]

Original: and B q A q ∈ R d ′ ×
Revised: and B q A q ∈ R d ′ ×

Original: d is independent of q .
Revised: d is independent of q .

Original: We also consider for every q ≥ 1 some control parameterinitialization V 0 q ∈ L 2 ( V q ) such that ∥ v 0 q ∥
Revised: We also consider for every q ≥ 1 some control parameter initialization V 0 q ∈ L 2 ( V q ) such that ∥ v 0 q ∥

Original: L 2 ≤ R 0 q − 1 / 4 and assume the data distribution to be877 compactly supported.
Revised: L 2 ≤ R 0 q − 1 / 4 and assume the data distribution to be compactly supported.

Original: Proposition 4.
Revised: Proposition 4.

Original: Let R > 0 and d, d ′ ≥ 1 .
Revised: Let R > 0 and d, d ′ ≥ 1 .

Original: Assume Assumption 3 is satisfied, V q satisfies Assumption 1with constant κ independent of q and there exists R 0 > 0 s.t.
Revised: Assume Assumption 3 is satisfied, V q satisfies Assumptionwith constant κ independent of q and there exists R 0 > 0 s.t.

Original: ∥ v 0 q ∥ ≤ R 0 q − 1 / 4 for every q ≥ 1 .
Revised: ∥ v 0 q ∥ ≤ R 0 q − 1 / 4 for every q ≥ 1 .

Original: Then there exists some constant C > 0 so that for any N ≥ 2 and any δ ∈ (0 , 1] , Eq.
Revised: Then there exists some constant C > 0 so that for any N ≥ 2 and any δ ∈ (0 , 1] , Eq.

Original: (15) is satisfiedwith matrices A q , B q and κ, λ, Λ associated with the RKHS V q as soon as:
Revised: (15) is satisfied with matrices A q , B q and κ, λ, Λ associated with the RKHS V q as soon as:

Original: [Equation]
Revised: [Equation]

Original: Note that the second condition in Eq.
Revised: Note that the second condition in Eq.

Original: (24) can always be ensured for large enough q thanks to Assumption 3.
Revised: (24) can always be ensured for large enough q thanks to Assumption 3.

Original: In the case of the Matérn kernel k defined in Eq.
Revised: In the case of the Matérn kernel k defined in Eq.

Original: (17), such an assumption is verifiedbecause it has exponential decay and it is independent of q .
Revised: (17), such an assumption is verified because it has exponential decay and it is independent of q .

Original: Hence, Proposition 1 is a direct consequence of Proposition 4.
Revised: Hence, Proposition 1 is a direct consequence of Proposition 4.

Original: Proof of Proposition 4.
Revised: Proof of Proposition 4.

Original: Let q ≥ 1 .
Revised: Let q ≥ 1 .

Original: Using the fact that d 2 ⌊ q/d ⌋ 2 ≥ q ( q − 2 d ) , considering:
Revised: Using the fact that d 2 ⌊ q/d ⌋ 2 ≥ q ( q − 2 d ) , considering:

Original: [Equation]
Revised: [Equation]

Original: is enough to ensure that:
Revised: is enough to ensure that:

Original: [Equation]
Revised: [Equation]

Original: Then, by Assumption 3 for ( z i ) 1 ≤ i ≤ N ∈
Revised: Then, by Assumption 3 for ( z i ) 1 ≤ i ≤ N ∈

Original: ( R q ) N with data separation q − 1 / 4 (cid:112)
Revised: ( R q ) N with data separation q − 1 / 4 (cid:112)

Original: ⌊ q/d
Revised: ⌊ q/d

Original: ⌋ δe − κ ( R + R 0 ) wehave:
Revised: ⌋ δe − κ ( R + R 0 ) we have:

Original: [Equation]
Revised: [Equation]

Original: Thus, the kernel matrix K =
Revised: Thus, the kernel matrix K =

Original: ( k q ( ∥ z i − zj ∥ )
Revised: [Equation] q [Equation] z

Revised: [Equation]

Original:  Id q ) i,j is diagonally dominant with:
Revised: z j ∥ ) Id q ) i,j is diagonally dominant with:

Original: [Equation]
Revised: [Equation]

Original: and by definition of λ in Eq.
Revised: and by definition of λ in Eq.

Original: [Equation]
Revised: [Equation]

Original: Moreover, Λ ≤ N because k q is bounded by 1 .
Revised: Moreover, Λ ≤ N because k q is bounded by 1 .

Original: Let x ∈ B (0 , r 0 ) and assume z is a solution of Eq.
Revised: Let x ∈ B (0 , r 0 ) and assume z is a solution of Eq.

Original: (6) for the control parameter v 0 q and with initialcondition
Revised: (6) for the control parameter v 0 q and with initial condition

Original: A q x .
Revised: A q x .

Original: We have at time t = 1 :
Revised: We have at time t = 1 :

Original: [Equation]
Revised: [Equation]

Original: so that by triangle inequality and Assumption 1:
Revised: so that by triangle inequality and Assumption 1:

Original: [Equation]
Revised: [Equation]

Original: and then because ∥ v 0 q ∥ ≤ R 0 q − 1 / 4 and the dataset is compactly supported:
Revised: and then because ∥ v 0 q ∥ ≤ R 0 q − 1 / 4 and the dataset is compactly supported:

Original: [Equation]
Revised: [Equation]

Original: with B q A q independent of q .
Revised: with B q A q independent of q .

Original: Thus L ( v 0 q ) ≤ C for some constant C independent of q , N and δ .
Revised: Thus L ( v 0 q ) ≤ C for some constant C independent of q , N and δ .

Original: Finally:
Revised: Finally:

Original: [Equation]
Revised: [Equation]

Original: and putting Eq.
Revised: and putting Eq.

Original: (26) and Eq.
Revised: (26) and Eq.

Original: (27) into the l.h.s.
Revised: (27) into the l.h.s.

Original: Eq.
Revised: Eq.

Original: (15) gives:
Revised: (15) gives:

Original: [Equation]
Revised: [Equation]

Original: Considering R > 0 is fixed (c.f. Remark 7), Theorem 2 can be applied as soon as: 
Revised: Considering R > 0 is fixed (c.f. Remark 7), Theorem 2 can be applied as soon as: [Equation]

Original: [Equation]
Revised: [Equation]

Original: and combining this bound with the one in Eq.
Revised: and combining this bound with the one in Eq.

Original: (25) gives the result.
Revised: (25) gives the result.

Original: Remark 7 (Choice of R ) .
Revised: Remark 7 (Choice of R ) .

Original: The proof of Proposition 4 holds for any fixed R > 0 whose choiceimpacts the result through the constant C .
Revised: The proof of Proposition 4 holds for any fixed R > 0 whose choice impacts the result through the constant C .

Original: There is a trade-off between minimizing e 4 κR to havea better dependency of q w.r.t.
Revised: There is a trade-off between minimizing e 4 κR to have a better dependency of q w.r.t.

Original: δ − 1 log( N ) in Eq.
Revised: δ − 1 log( N ) in Eq.

Original: and minimizing R − 1 e 3 κR to have a betterdependency w.r.t.
Revised: and minimizing R − 1 e 3 κR to have a better dependency w.r.t.

Original: N in Eq.
Revised: N in Eq.

Original: However, in any case, optimizing w.r.t.
Revised: However, in any case, optimizing w.r.t.

Original: R only improves the resultup to a constant factor.
Revised: R only improves the result up to a constant factor.

Original: For α > 0 , we consider here the matrices:[Equation] and
Revised: For α > 0 , we consider here the matrices:

Original: [Equation]
Revised: [Equation]

Original: Then, in the proof of Proposition 4 one has σ min ( A )
Revised: Then, in the proof of Proposition 4 one has σ min ( A )

Original: =
Revised: =

Original: α and thus Eq.
Revised: α and thus Eq.

Original: (26) holds as soon as:
Revised: (26) holds as soon as:

Original: [Equation]
Revised: [Equation]

Original: Moreover, σ max ( B ⊤ ) = σ min ( B ⊤ ) =
Revised: Moreover, σ max ( B ⊤ ) = σ min ( B ⊤ ) =

Original: α and F (0 , x ) = 

Revised: α and F (0 , x ) =

Original: for every input x as BA = 0 .
Revised: for every input x as BA = 0 .

Original: Thus, withinitialization v 0 = 

Revised: Thus, with initialization v 0 =

Original: the l.h.s. of Eq.
Revised: the l.h.s. of Eq.

Original: (15) scales as:
Revised: (15) scales as:

Original: [Equation]
Revised: [Equation]

Original: and global convergence holds for α = Ω( δ − 1 β d + d ′ ,N + N ) .
Revised: and global convergence holds for α = Ω( δ − 1 β d + d ′ ,N + N ) .

Original: D.2 Enforcing convergence with high dimensional embedding en finite dimensional kernels
Revised: D.2 Enforcing convergence with high dimensional embedding en finite dimensional kernels

Original: We recover here the result of Proposition 2 for the more general kernel k q .
Revised: We recover here the result of Proposition 2 for the more general kernel k q .

Original: In particular notice that,as an application of Bochner’s theorem [50], for every q ≥ 1 there exists some probability measureµ q over R q such that: 
Revised: In particular notice that, as an application of Bochner’s theorem [50], for every q ≥ 1 there exists some probability measure µ q over R q such that: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Then, such as in Eq.
Revised: Then, such as in Eq.

Original: (20) for the Matérn kernel, for any independent sampling ω
Revised: (20) for the Matérn kernel, for any independent sampling ω

Original: j ∼ µ q of size rone can consider the feature map:
Revised: j ∼ µ q of size r one can consider the feature map:

Original: [Equation]
Revised: [Equation]

Original: Such a feature map induces a structure of RKHS
Revised: Such a feature map induces a structure of RKHS

Original: ˆ V q which is the set of residuals of Eq.
Revised: ˆ V q which is the set of residuals of Eq.

Original: (3) withactivation φ .
Revised: (3) with activation φ .

Original: The associated kernel is ˆ K q : ( z, z ′ )
Revised: The associated kernel is ˆ K q : ( z, z ′ )

Original: (cid:55)→ ˆ k q ( z, z ′ )
Revised: (cid:55)→ ˆ k q ( z, z ′ )

Original: Id q with:
Revised: Id q with:

Original: [Equation]
Revised: [Equation]

Original: almost surely, by the law of large numbers.
Revised: almost surely, by the law of large numbers.

Original: We make the following assumption on µ q :
Revised: We make the following assumption on µ q :

Original: Assumption 4 (Moments of µ q ) .
Revised: Assumption 4 (Moments of µ q ) .

Original: The measure µ q admits finite moments up to order 8 :
Revised: The measure µ q admits finite moments up to order 8 :

Original: [Equation]
Revised: [Equation]

Original: Moreover, we assume those moments are independent of q .
Revised: Moreover, we assume those moments are independent of q .

Original: Note that Assumption 4 implies regularity on the function k q .
Revised: Note that Assumption 4 implies regularity on the function k q .

Original: Indeed by Fourier inversion theoremwe have for every r ∈ R + and every θ ∈ S
Revised: Indeed by Fourier inversion theorem we have for every r ∈ R + and every θ ∈ S

Original: d − 1 :
Revised: d − 1 :

Original: [Equation]
Revised: [Equation]

Original: By theorems of derivation under the integral k q is 8 th -time differentiable on R + and for 0 ≤ l ≤ 8 :
Revised: By theorems of derivation under the integral k q is 8 th -time differentiable on R + and for 0 ≤ l ≤ 8 :

Original: [Equation]
Revised: [Equation]

Original: In particular, k q is four time differentiable at 0 and:
Revised: In particular, k q is four time differentiable at 0 and:

Original: [Equation]
Revised: [Equation]

Original: Therefore, k ′ q (0) and k (3) q (0) are in ı R ∩ R = { 0 } and Property 4 holds.
Revised: Therefore, k ′ q (0) and k (3) q (0) are in ı R ∩ R = { 0 } and Property 4 holds.

Original: Moreover, as the momentsare independent of q , the associated κ is also independent of q .
Revised: Moreover, as the moments are independent of q , the associated κ is also independent of q .

Original: Proposition 5.
Revised: Proposition 5.

Original: Consider q, N ≥ 1 and ϵ, τ, R > 0 .
Revised: Consider q, N ≥ 1 and ϵ, τ, R > 0 .

Original: (i) Assume Assumption 4 is satisfied.
Revised: (i) Assume Assumption 4 is satisfied.

Original: For r ≥ Ω( τq 8 ) , with probability greater than 1 − τ − 1 , ˆ V q satisfies Assumption 1 with some ˆ κ ≤ κ + 1 .
Revised: For r ≥ Ω( τq 8 ) , with probability greater than 1 − τ − 1 , ˆ V q satisfies Assumption 1 with some ˆ κ ≤ κ + 1 .

Original: (ii) For r ≥ Ω( ϵ − 2 N 2 ( q log( ∥ A ∥ 2 r 0 + R ) + τ )) , with probability greater than 1 − e − τ , for any control parameter v ∈ L 2 ([0 , 1] , ˆ V q ) s.t.
Revised: (ii) For r ≥ Ω( ϵ − 2 N 2 ( q log( ∥ A ∥ 2 r 0 + R ) + τ )) , with probability greater than 1 − e − τ , for any control parameter v ∈ L 2 ([0 , 1] , ˆ V q ) s.t.

Original: ∥ v ∥ L 2 ≤ R and any time t ∈ [0 , 1] :
Revised: ∥ v ∥ L 2 ≤ R and any time t ∈ [0 , 1] :

Original: [Equation]
Revised: [Equation]

Original: where the ( z i ) i are the solutions to Eq.
Revised: where the ( z i ) i are the solutions to Eq.

Original: (6) and ˆ K , K are the kernel matrices associated with ˆ k andk respectively.
Revised: (6) and ˆ K , K are the kernel matrices associated with ˆ k and k respectively.

Original: As Assumption 4 is satisfied for the Matérn kernel k defined in Eq.
Revised: As Assumption 4 is satisfied for the Matérn kernel k defined in Eq.

Original: (17) as soon as ν > 4 , Proposition 2 is a direct consequence of Proposition 5.
Revised: (17) as soon as ν > 4 , Proposition 2 is a direct consequence of Proposition 5.

Original: Proof of Proposition 5.
Revised: Proof of Proposition 5.

Original: Proof of (i)
Revised: Proof of (i)

Original: We already saw that thanks to the assumption on the momentsof µ q , the RKHS V q associated with k q satisfies Assumption 1 with constant κ .
Revised: We already saw that thanks to the assumption on the moments of µ q , the RKHS V q associated with k q satisfies Assumption 1 with constant κ .

Original: Then we want to prove that for sufficiently high r , the RKHS ˆ V q generated by the feature map φin Eq.
Revised: Then we want to prove that for sufficiently high r , the RKHS ˆ V q generated by the feature map φ in Eq.

Original: (20), satisfies Assumption 1.
Revised: (20), satisfies Assumption 1.

Original: Let v ∈
Revised: Let v ∈

Original: ˆ V q be of the form:
Revised: ˆ V q be of the form:

Original: [Equation]
Revised: [Equation]

Original: for some W ∈ R q × r .
Revised: for some W ∈ R q × r .

Original: For z ∈ R q , ∥ φ ( z ) ∥
Revised: For z ∈ R q , ∥ φ ( z ) ∥

Original: = 1 and thus:
Revised: = 1 and thus:

Original: [Equation]
Revised: [Equation]

Original: so that ∥ v ∥ ∞ ≤ ∥ v ∥ ˆ V q .
Revised: so that ∥ v ∥ ∞ ≤ ∥ v ∥ ˆ V q .

Original: Then Dv ( z ) =
Revised: Then Dv ( z ) =

Original: WDφ ( z ) and by the law of large number we have for any θ ∈ S q − 1 :
Revised: WDφ ( z ) and by the law of large number we have for any θ ∈ S q − 1 :

Original: [Equation]
Revised: [Equation]

Original: Because µ q admits finite fourth order moments, the rate of convergence can be controlled using
Revised: Because µ q admits finite fourth order moments, the rate of convergence can be controlled using

Original: Chebyshev’s inequality.
Revised: Chebyshev’s inequality.

Original: For every indices k, l ∈ (cid:74) 1 , q [Equation]
Revised: For every indices k, l ∈ (cid:74) 1 , q (cid:75) :

Original: [Equation]
Revised: [Equation]

Original: For r ≥ Ω( q 4 τα 2 ) we have with probability greater than 1 − τ − 1 that the above inequality is satisfiedfor every indices k, l .
Revised: For r ≥ Ω( q 4 τα 2 ) we have with probability greater than 1 − τ − 1 that the above inequality is satisfied for every indices k, l .

Original: Thus for every z ∈ R q and every θ ∈ S q − 1 : 
Revised: Thus for every z ∈ R q and every θ ∈ S q − 1 : [Equation] ≤

Original: [Equation]
Revised: [Equation]

Original: , using Chauchy-Schwarz inequality in the last line.
Revised: using Chauchy-Schwarz inequality in the last line.

Original: We can thus conclude:
Revised: We can thus conclude:

Original: [Equation]
Revised: [Equation]

Original: The same arguments holds for D 2 v ( z )
Revised: The same arguments holds for D 2 v ( z )

Original: =
Revised: =

Original: WD 2 φ ( z ) .
Revised: WD 2 φ ( z ) .

Original: For any θ ∈ S q − 1 we have:
Revised: For any θ ∈ S q − 1 we have:

Original: [Equation]
Revised: [Equation]

Original: Passing to the squared norm we get:
Revised: Passing to the squared norm we get:

Original: [Equation]
Revised: [Equation]

Original: Then because µ q admits 8 th order moments, we can control the convergence in probability by
Revised: Then because µ q admits 8 th order moments, we can control the convergence in probability by

Original: Chebyshev’s inequality.
Revised: Chebyshev’s inequality.

Original: For r ≥ Ω( q 8 τα 2 ) we have with probability greater than 1 − τ − 1 :
Revised: For r ≥ Ω( q 8 τα 2 ) we have with probability greater than 1 − τ − 1 :

Original: [Equation]
Revised: [Equation]

Original: Finally ˆ V q satisfies Assumption 1 with:
Revised: Finally ˆ V q satisfies Assumption 1 with:

Original: [Equation]
Revised: [Equation]

Original: for α sufficiently low.
Revised: for α sufficiently low.

Original: Proof of (ii).
Revised: Proof of (ii).

Original: For t ∈ [0 , 1] , we consider ( z it )
Revised: For t ∈ [0 , 1] , we consider ( z it )

Original: i the solutions of of Eq.
Revised: i the solutions of of Eq.

Original: (6) for some control parameterv ∈ L 2 ([0 , 1] , ˆ V q ) and we introduce the kernel matrices:
Revised: (6) for some control parameter v ∈ L 2 ([0 , 1] , ˆ V q ) and we introduce the kernel matrices:

Original: [Equation]
Revised: [Equation]

Original: j ≤ N . Using the first point, we know that if ∥ v ∥ L 2 ≤ R , then ∥ z it ∥ ≤ ∥
Revised: Using the first point, we know that if ∥ v ∥ L 2 ≤ R , then ∥ z it ∥ ≤ ∥

Original: Ax i ∥
Revised: Ax i ∥

Original: + ( κ + 1) R .
Revised: + ( κ + 1) R .

Original: Then, using962 Theorem 1 in [53], we have for every indices i, j and every t ∈ [0 , 1] :
Revised: Then, using Theorem 1 in [53], we have for every indices i, j and every t ∈ [0 , 1] :

Original: [Equation]
Revised: [Equation]

Original: with h ( q, R 
Revised: with h ( q, R ) : = O

Original: [Equation]
Revised: [Equation]

Revised: log( ∥

Revised: A

Original:  Thus, choosingr ≥ Ω (cid:0) ϵ − 2 N 2 ( q log( ∥ A ∥ 2 r 0 + R ) + τ ) (cid:1) , we have with probability greater than 1 − e − τ ,λ min (ˆ K t ) ≥ λ min ( K t ) − ϵ , for any t ∈ [0 , 1] .
Revised: [Equation] + [Equation] Thus, choosing r ≥ Ω (cid:0) ϵ − 2 N 2 ( q log( ∥ A ∥ 2 r 0 + R ) + τ ) (cid:1) , we have with probability greater than 1 − e − τ , λ min (ˆ K t ) ≥ λ min ( K t ) − ϵ , for any t ∈ [0 , 1] .

Original: Note that the assumption of finite 8 th moments is only needed to control the convergence rate ofˆ k q towards k q in probability.
Revised: Note that the assumption of finite 8 th moments is only needed to control the convergence rate of ˆ k q towards k q in probability.

Original: By the law of large numbers, assuming finite 4 th -order moments issufficient to have convergence almost surely.
Revised: By the law of large numbers, assuming finite 4 th -order moments is sufficient to have convergence almost surely.

Original: Also, we used the Chebyshev’s inequality in order tocontrol the convergence rate.
Revised: Also, we used the Chebyshev’s inequality in order to control the convergence rate.

Original: Making stronger assumptions on the decay of µ q (e.g. sub-gaussianity)could have led to faster convergence by using sharper concentration inequalities.
Revised: Making stronger assumptions on the decay of µ q (e.g. sub-gaussianity) could have led to faster convergence by using sharper concentration inequalities.

Original: D.3 Example of appropriate kernels
Revised: D.3 Example of appropriate kernels

Original: We show here that the Matérn kernel of parameter ν ∈ (8 , + ∞ ] satisfies Assumption 3 and Assumption 4.
Revised: We show here that the Matérn kernel of parameter ν ∈ (8 , + ∞ ] satisfies Assumption 3 and Assumption 4.

Original: Gaussian kernel.
Revised: Gaussian kernel.

Original: The Gaussian kernel defined by for some parameter σ > 0 by k q ( r ) =
Revised: The Gaussian kernel defined by for some parameter σ > 0 by k q ( r ) =

Original: e − σ 2 r 2 2 .
Revised: e − σ 2 r 2 2 .

Original: In this case the frequency distribution µ q is the multivariate normal of variance σ and has a densitygiven for every ω ∈ R q by:
Revised: In this case the frequency distribution µ q is the multivariate normal of variance σ and has a density given for every ω ∈ R q by:

Original: [Equation]
Revised: [Equation]

Original: This distribution admits finite moments of every order which are independent of q .
Revised: This distribution admits finite moments of every order which are independent of q .

Original: Also, k q is fourtimes differentiable at 0 and by Property 4 the associated V q is (strongly) admissible with κ = 2+ √[-
-]
Revised: Also, k q is four times differentiable at 0 and by Property 4 the associated V q is (strongly) admissible with κ = 2+ √[-
-]

Original: Moreover Assumption 3 as one has | k q ( x ) | ≤ 1 / 2 N if:
Revised: Moreover Assumption 3 as one has | k q ( x ) | ≤ 1 / 2 N if:

Original: [Equation]
Revised: [Equation]

Original: Matérn kernel.
Revised: Matérn kernel.

Original: Sobolev spaces H s ( R q , R q ) are RKHSs as soon as s > q/ 2 .
Revised: Sobolev spaces H s ( R q , R q ) are RKHSs as soon as s > q/ 2 .

Original: Given some ν > 0 ,the kernel k q associated with H ( q/ 2+ ν ) ( R q , R q ) is independent of q and is defined in Eq.
Revised: Given some ν > 0 , the kernel k q associated with H ( q/ 2+ ν ) ( R q , R q ) is independent of q and is defined in Eq.

Original: It isassociated with the multivariate t-distribution:
Revised: It is associated with the multivariate t-distribution:

Original: [Equation]
Revised: [Equation]

Original: for some normalising constant C ( q, ν ) .
Revised: for some normalising constant C ( q, ν ) .

Original: Therefore, µ q admits l th order moments as soon as ν ≥ l/ 2 ,and those moments are bounded independently of q (see [28] for the computation of moments).
Revised: Therefore, µ q admits l th order moments as soon as ν ≥ l/ 2 , and those moments are bounded independently of q (see [28] for the computation of moments).

Original: In particular, for ν > 2 , k q is four times differentiable at 0 with k ′′ (0) =
Revised: In particular, for ν > 2 , k q is four times differentiable at 0 with k ′′ (0) =

Original: ν/ ( ν − 1) andk (4) (0) =
Revised: ν/ ( ν − 1) and k (4) (0) =

Original: 3 ν 2 / ( ν − 1)( ν − 2) .
Revised: 3 ν 2 / ( ν − 1)( ν − 2) .

Original: Thus by Property 4, V q is (strongly) admissible with:
Revised: Thus by Property 4, V q is (strongly) admissible with:

Original: [Equation]
Revised: [Equation]

Original: Because k q has exponential decay (see [31]), there exist constants H ν , G ν such that:
Revised: Because k q has exponential decay (see [31]), there exist constants H ν , G ν such that:

Original: [Equation]
Revised: [Equation]

Original: and Assumption 3 is satisfied with
Revised: and Assumption 3 is satisfied with

Original: [Equation]
Revised: [Equation]

Original: log

Original: [Equation]

Original: N ) .

Original: Remark 8 (Sampling) .
Revised: Remark 8 (Sampling) .

Original: Sampling over µ q can be achieved using that for Y ∼ N (0 , Id q ) and for udistributed according to χ 22 ν , the chi-squared distribution with 2 ν degrees of freedom, Y/ (cid:112) u/ 2 ν isdistributed according to µ q . 
Revised: Sampling over µ q can be achieved using that for Y ∼ N (0 , Id q ) and for u distributed according to χ 22 ν , the chi-squared distribution with 2 ν degrees of freedom, Y/ (cid:112) u/ 2 ν is distributed according to µ q . E RKHS-NODE as a generalization of linear networks

Revised: In an attempt to better understand the convergence properties of GD in the training of ResNets, lots of attention has first been brought towards the study of linear models, for which the training dynamic is now well understood [24, 7, 64].

Revised: We explain here in what extent our work can be seen, at least formally, as a generalization of these results to a more general class of ResNets.

Revised: In this purpose, we highlight the similarity between Theorem 2, which applies to the whole class of models described by Definition 1, and [64, Theorem 3.1.], which only applies to linear ResNets.

Revised: More precisely, [64] studies model of the form:

Revised: [Equation]

Revised: (31) where x ∈ R

Revised: d is the input data, W =

Revised: ( W 1 , ..., W D ) ∈

Revised: ( R q × q ) D is the trained parameter and

Revised: A ∈ R q × d , B ∈ R d ′ ×

Revised: q are fixed matrices.

Revised: Taking the limit of infinite depth D → + ∞ in the above model motivates the following definition for linear Neural ODE models:

Revised: Definition 4 (Linear-NODE) .

Revised: Let A ∈ R q × d and B ∈ R d ′ ×

Revised: q be fixed matrices.

Revised: Then for

Revised: W ∈ L 2 ([0 , 1] , R q × q ) and input x ∈ R d , the Linear-NODE output is given by F ( W, x ) :

Revised: = BU 1 Ax , where U is the solution to the following forward problem :

Revised: [Equation]

Revised: One sees that the ResNet F has residual terms that are linear w.r.t.

Revised: the parameters and thus fits in the framework of our analysis.

Revised: More precisely, the Linear-NODE of Definition 4 can be seen as a special instance of RKHS-NODE of Definition 1 with space of residual defined as:

Revised: [Equation]

Revised: This corresponds to Eq.

Revised: (3) with the choice of feature map φ

Revised: =

Revised: I

Revised: d : R q → R q .

Revised: The set of residuals

Revised: V is then of course a RKHS for the Frobenius metric on matrices.

Revised: In particular V satisfies an analog of Assumption 1 in the sense that for ( v : z

Revised: (cid:55)→ Wz ) ∈ V :

Revised: [Equation]

Revised: Universality (Assumption 2) is also satisfied on full-rank data matrices.

Revised: If Z = ( z 1 | ... | z N ) ∈ R q × N then the associated kernel matrix verifies:

Revised: [Equation]

Revised: As in our above presentation we consider training Linear-NODE for the minimization of the empirical risk associated to the square euclidean distance on the output space R d ′ .

Revised: Given data matrices

Revised: X =

Revised: ( x 1 | ... | x N

Revised: ) ∈ R d × N for the input and Y =

Revised: ( y 1 | ... | y N ) ∈ R d ′ × N for the output, we aim at finding a control parameter minimizing the risk defined for every W ∈ L 2 ([0 , 1] , R q × q ) as:

Revised: [Equation]

Original: {+One difference with the previous analysis is that one can not expect the empirical risk to reach the value 
+}
Revised: {+One difference with the previous analysis is that one can not expect the empirical risk to reach the value 
+}

Revised: if the target data Y is not in the linear span of the input X .

Revised: We are thus interested in minimizing the excess risk defined as:

Revised: [Equation]

Revised: with L ∗ :

Revised: = inf U

Revised: [Equation]

Revised: Following the line of the proof of Property 2, one can then show that the excess risk ˜

Revised: L associated to our Linear-NODE model verifies the following (local) PL property:

Revised: [Equation]

Revised: where m and M are given for R ≥ 0 by:

Revised: [Equation]

Revised: with σ r ( X ) the smallest positive singular value of X .

Revised: Hence, in the same way local PL implies local convergence for a general RKHS V (Theorem 2), convergence in the linear case follows as an application of Theorem 1:

Revised: Theorem 4 (analog to Theorem 3.1.

Revised: in [64]) .

Revised: Let W 0 be some control parameter initialization with norm ∥ W 0 ∥

Revised: = R 0 and assume there exists some R > 0 s.t.:

Revised: [Equation]

Revised: then, for a sufficiently small step-size η , GD initialized at W 0 converges towards a global minimizer of L with linear convergence rate.