Original: On global convergence of ResNets: From finite to infinite width using linear parameterization
Revised: On global convergence of ResNets: From finite to infinite width using linear parameterization

Original: Abstract
Revised: Abstract

Original: Overparameterization is a key factor in the absence of convexity to explain global convergence of gradient descent (GD) for neural networks.
Revised: Overparameterization is a key factor in the absence of convexity to explain global convergence of gradient descent (GD) for neural networks.

Original: Beside the well studied lazy regime, infinite width (mean field) analysis has been developed for shallow networks, using convex optimization techniques.
Revised: Beside the well studied lazy regime, infinite width (mean field) analysis has been developed for shallow networks, using convex optimization techniques.

Original: To bridge the gap between the lazy and mean field regimes, we study Residual Networks (ResNets) in which the residual block has linear parameterization while still being nonlinear.
Revised: To bridge the gap between the lazy and mean field regimes, we study Residual Networks (ResNets) in which the residual block has linear parameterization while still being nonlinear.

Original: Such
Revised: Such

Original: ResNets admit both infinite depth and width limits, encoding residual blocks in a
Revised: ResNets admit both infinite depth and width limits, encoding residual blocks in a

Original: Reproducing Kernel Hilbert Space (RKHS).
Revised: Reproducing Kernel Hilbert Space (RKHS).

Original: In this limit, we prove a local PolyakLojasiewicz inequality.
Revised: In this limit, we prove a local PolyakLojasiewicz inequality.

Original: Thus, every critical point is a global minimizer and a local convergence result of GD holds, retrieving the lazy regime.
Revised: Thus, every critical point is a global minimizer and a local convergence result of GD holds, retrieving the lazy regime.

Original: In contrast with other mean-field studies, it applies to both parametric and non-parametric cases under an expressivity condition on the residuals.
Revised: In contrast with other mean-field studies, it applies to both parametric and non-parametric cases under an expressivity condition on the residuals.

Original: Our analysis leads to a practical and quantified recipe: starting from a universal RKHS, Random Fourier Features are applied to obtain a finite dimensional parameterization satisfying with highprobability our expressivity condition.
Revised: Our analysis leads to a practical and quantified recipe: starting from a universal RKHS, Random Fourier Features are applied to obtain a finite dimensional parameterization satisfying with highprobability our expressivity condition.

Original:  Introduction
Revised:  Introduction

Original: State of the art supervised learning methods are based on deep neural networks, sometimes heavily overparameterized, which perfectly fit training data or even noisy data while exhibiting good generalization properties.
Revised: State of the art supervised learning methods are based on deep neural networks, sometimes heavily overparameterized, which perfectly fit training data or even noisy data while exhibiting good generalization properties.

Original: Such a behaviour appears as a paradox and questions the established theory of “bias-variance trade-off” [9].
Revised: Such a behaviour appears as a paradox and questions the established theory of “bias-variance trade-off” [9].

Original: That an overparameterized model can fit data perfectly comes as no surprise but this capability does not explain the observed generalization properties.
Revised: That an overparameterized model can fit data perfectly comes as no surprise but this capability does not explain the observed generalization properties.

Original: Towards a better understanding of it, one first needs to understand the optimization procedure in the parameter space that selects the interpolation map.
Revised: Towards a better understanding of it, one first needs to understand the optimization procedure in the parameter space that selects the interpolation map.

Original: This question is tightly linked with the parameterization of the space of maps that are explored and state of the art parameterizations have emerged in the past years.
Revised: This question is tightly linked with the parameterization of the space of maps that are explored and state of the art parameterizations have emerged in the past years.

Original: One key architecture that is ubiquitous in deep learning are skip connections, heavily used in Residual Neural Networks (ResNets) [25] and it has led to state of the art results in supervised learning.
Revised: One key architecture that is ubiquitous in deep learning are skip connections, heavily used in Residual Neural Networks (ResNets) [25] and it has led to state of the art results in supervised learning.

Original: ResNets actually allow one to consider a very large number of layers [59].
Revised: ResNets actually allow one to consider a very large number of layers [59].

Original: Continuous models.
Revised: Continuous models.

Original: Passing to the limit of infinite depth allows the connection with continuous models (Neural ODE) for which theoretical methods and new algorithms can be designed [11, 56].
Revised: Passing to the limit of infinite depth allows the connection with continuous models (Neural ODE) for which theoretical methods and new algorithms can be designed [11, 56].

Original: Indeed, the similarities between ResNet architectures and discrete numerical schemes motivated the
Revised: Indeed, the similarities between ResNet architectures and discrete numerical schemes motivated the

Original: introduction
Revised: introduction

Original: of
Revised: of

Original: a continuous neural ODE
Revised: a continuous neural ODE

Original: [Equation]
Revised: [Equation]

Original: where W ∈ L 2 ([0 , 1] , R m ) is the parameter of the model and v : R m × R q → R q is a residual transformation whose output is the residual term .
Revised: where W ∈ L 2 ([0 , 1] , R m ) is the parameter of the model and v : R m × R q → R q is a residual transformation whose output is the residual term .

Original: These models correspond to limiting models of a discrete ResNet whose depth D tends to infinity.
Revised: These models correspond to limiting models of a discrete ResNet whose depth D tends to infinity.

Original: Therefore, their study brings a theoretical framework for understanding deep ResNet architectures, and more generally very deep NNs [19, 20].
Revised: Therefore, their study brings a theoretical framework for understanding deep ResNet architectures, and more generally very deep NNs [19, 20].

Original: Moreover, their mathematical analysis is facilitated since it allows one to leverage a large body of works and tools from analysis and in particular the theory of optimal control [47].
Revised: Moreover, their mathematical analysis is facilitated since it allows one to leverage a large body of works and tools from analysis and in particular the theory of optimal control [47].

Original: Conversely, methods from numerical analysis can bring inspiration for designing new architectures and new optimization algorithms [39].
Revised: Conversely, methods from numerical analysis can bring inspiration for designing new architectures and new optimization algorithms [39].

Original: RKHS parameterization.
Revised: RKHS parameterization.

Original: Most often in the literature studying the training properties of ResNets, the considered residual transformations are Multi-Layer Perceptrons (MLP)
Revised: Most often in the literature studying the training properties of ResNets, the considered residual transformations are Multi-Layer Perceptrons (MLP)

Original: Those consist in the composition of several trained linear layers alternatively composed with a non-linear activation function.
Revised: Those consist in the composition of several trained linear layers alternatively composed with a non-linear activation function.

Original: A 2-layer MLP with width r reads:
Revised: A 2-layer MLP with width r reads:

Original: [Equation]
Revised: [Equation]

Original: where U ∈ R r × q and W ∈ R q × r are the parameters for the “hidden” and the “visible” layer respectively and σ : R → R is a non-linear activation function applied component-wise.
Revised: where U ∈ R r × q and W ∈ R q × r are the parameters for the “hidden” and the “visible” layer respectively and σ : R → R is a non-linear activation function applied component-wise.

Original: Popular activation functions are for example the ReLU or the Swish function.
Revised: Popular activation functions are for example the ReLU or the Swish function.

Original: Provided with these activations, MLPs enjoy a nice universal approximation property as shown in the seminal work of Barron [6].
Revised: Provided with these activations, MLPs enjoy a nice universal approximation property as shown in the seminal work of Barron [6].

Original: In contrast, we consider here a setting where the residual term is linear w.r.t.
Revised: In contrast, we consider here a setting where the residual term is linear w.r.t.

Original: the parameters while still being nonlinear w.r.t the inputs.
Revised: the parameters while still being nonlinear w.r.t the inputs.

Original: Given a feature map φ : R q → R r , we consider as space of residuals the space:
Revised: Given a feature map φ : R q → R r , we consider as space of residuals the space:

Original: [Equation]
Revised: [Equation]

Original: where the matrices W ∈ R q × r are the trained parameters.
Revised: where the matrices W ∈ R q × r are the trained parameters.

Original: Compared to Eq.
Revised: Compared to Eq.

Original: (2), this can be seen as an MLP where the hidden layer is fixed by introducing the feature map φ : z (cid:55)→ σ
Revised: (2), this can be seen as an MLP where the hidden layer is fixed by introducing the feature map φ : z (cid:55)→ σ

Original: [Equation]
Revised: [Equation]

Original: Uz
Revised: Uz

Original: [Equation]
Revised: [Equation]

Original: for some feature matrix U .
Revised: for some feature matrix U .

Original: As is standard, the gradient of some loss L w.r.t.
Revised: As is standard, the gradient of some loss L w.r.t.

Original: W is computed in the sense of the Frobenius metric on the set of matrices:
Revised: W is computed in the sense of the Frobenius metric on the set of matrices:

Original: [Equation]
Revised: [Equation]

Original: Such an L 2 penalization induces a metric structure on V through the identification v ↔ W in Eq.
Revised: Such an L 2 penalization induces a metric structure on V through the identification v ↔ W in Eq.

Original: [Equation]
Revised: [Equation]

Original: As a finite dimensional space of continuous maps, V has the structure of Reproducing Kernel Hilbert
Revised: As a finite dimensional space of continuous maps, V has the structure of Reproducing Kernel Hilbert

Original: Space (RKHS).
Revised: Space (RKHS).

Original: Moreover, as pointed out in [5], the space V has a natural infinite width limit or mean field limit which is an infinite dimensional RKHS.
Revised: Moreover, as pointed out in [5], the space V has a natural infinite width limit or mean field limit which is an infinite dimensional RKHS.

Original: In this paper, we are interested in understanding the convergence properties of Gradient Descent (GD) on a ResNet model for which the residual layers are encoded in a – possibly infinitedimensional – vector-valued RKHS V .
Revised: In this paper, we are interested in understanding the convergence properties of Gradient Descent (GD) on a ResNet model for which the residual layers are encoded in a – possibly infinitedimensional – vector-valued RKHS V .

Original: For V as in Eq.
Revised: For V as in Eq.

Original: (3), we stress out that, as the metric on
Revised: (3), we stress out that, as the metric on

Original: V is induced by the one on R q × r , GD on V for this metric is strictly equivalent to GD on R q × r with the Frobenius metric.
Revised: V is induced by the one on R q × r , GD on V for this metric is strictly equivalent to GD on R q × r with the Frobenius metric.

Original: Our model is defined as follows:
Revised: Our model is defined as follows:

Original: Definition 1 (RKHS Neural ODE (RKHS-NODE)) .
Revised: Definition 1 (RKHS Neural ODE (RKHS-NODE)) .

Original: Let V be a RKHS of vector-fields over R q and
Revised: Let V be a RKHS of vector-fields over R q and

Original: A ∈ R q × d , B ∈ R d ′ × q .
Revised: A ∈ R q × d , B ∈ R d ′ × q .

Original: Then for v ∈ L 2 ([0 , 1] , V ) and a data input x ∈ R d , the RKHS-NODE’s output is F ( v, x ) :
Revised: Then for v ∈ L 2 ([0 , 1] , V ) and a data input x ∈ R d , the RKHS-NODE’s output is F ( v, x ) :

Original: =
Revised: =

Original: Bz 1 , where z is the solution to the forward problem
Revised: Bz 1 , where z is the solution to the forward problem

Original: [Equation]
Revised: [Equation]

Original: The variable v will thereafter be called control parameter .
Revised: The variable v will thereafter be called control parameter .

Original: Remark 1.
Revised: Remark 1.

Original: Note that the matrices A and B are fixed and only the control parameter v is trained.
Revised: Note that the matrices A and B are fixed and only the control parameter v is trained.

Original: However, we argue that our approach can be simply adapted to the case where B is trained, following for example the proof of [43].
Revised: However, we argue that our approach can be simply adapted to the case where B is trained, following for example the proof of [43].

Original: Training A seems more challenging as the model is highly non-linear w.r.t.
Revised: Training A seems more challenging as the model is highly non-linear w.r.t.

Original: this parameter.
Revised: this parameter.

Original: Relevance of the RKHS model.
Revised: Relevance of the RKHS model.

Original: The main difference between the model of Definition 1 and standard ResNets is linearity in the parameters of the residual blocks.
Revised: The main difference between the model of Definition 1 and standard ResNets is linearity in the parameters of the residual blocks.

Original: As a comparison, a 2-layer
Revised: As a comparison, a 2-layer

Original: MLP is nonlinear w.r.t.
Revised: MLP is nonlinear w.r.t.

Original: the parameters of the hidden layers.
Revised: the parameters of the hidden layers.

Original: However, this linearity assumption does not impact the expressivity of the model, but only its training dynamic.
Revised: However, this linearity assumption does not impact the expressivity of the model, but only its training dynamic.

Original: (i)
Revised: (i)

Original: Indeed, considering V to be a Random Features approximation (c.f. Eq.
Revised: Indeed, considering V to be a Random Features approximation (c.f. Eq.

Original: (20)) of some universal RKHS, the residual blocks are as expressive as a 2-layer MLP since both are dense in the space of continuous functions.
Revised: (20)) of some universal RKHS, the residual blocks are as expressive as a 2-layer MLP since both are dense in the space of continuous functions.

Original: (ii)
Revised: (ii)

Original: Up to the cost of adding a supplementary variable, the dynamical system parameterized by a 2-layer
Revised: Up to the cost of adding a supplementary variable, the dynamical system parameterized by a 2-layer

Original: MLP can be expressed as a model which is linear w.r.t.
Revised: MLP can be expressed as a model which is linear w.r.t.

Original: its parameters [56, Section 3.2].
Revised: its parameters [56, Section 3.2].

Original: Only the training dynamic between these two architectures differs.
Revised: Only the training dynamic between these two architectures differs.

Original: Also, this assumption of linearity in the parameters also prevents the use of normalization layers.
Revised: Also, this assumption of linearity in the parameters also prevents the use of normalization layers.

Original: In this direction, [61] has shown that
Revised: In this direction, [61] has shown that

Original: ResNets without normalization but proper initialization of the weights can lead to robust training and similar performance on the train set than standard ResNets.
Revised: ResNets without normalization but proper initialization of the weights can lead to robust training and similar performance on the train set than standard ResNets.

Original: Finally, the model of Definitionstill retains the effect of depth and the nonlinearity w.r.t.
Revised: Finally, the model of Definitionstill retains the effect of depth and the nonlinearity w.r.t.

Original: the input.
Revised: the input.

Original: Due to composition of these residual blocks the model’s output is still highly non-linear w.r.t.
Revised: Due to composition of these residual blocks the model’s output is still highly non-linear w.r.t.

Original: parameters.
Revised: parameters.

Original: For these reasons, we consider this model as an important step towards the study of the general case.
Revised: For these reasons, we consider this model as an important step towards the study of the general case.

Original: In turn, this linearity in parameters naturally leads to an RKHS parameterization which has two important benefits on the theoretical side: (i) Flows of vector-fields as implemented by our model in Eq.
Revised: In turn, this linearity in parameters naturally leads to an RKHS parameterization which has two important benefits on the theoretical side: (i) Flows of vector-fields as implemented by our model in Eq.

Original: (6) have already been studied theoretically and for applications in image registration problems [58, 8, 44].
Revised: (6) have already been studied theoretically and for applications in image registration problems [58, 8, 44].

Original: Under some regularity assumptions on the considered RKHS V , one can show that the model’s output corresponds to the invertible action of a diffeomorphism by composition on the input [55].
Revised: Under some regularity assumptions on the considered RKHS V , one can show that the model’s output corresponds to the invertible action of a diffeomorphism by composition on the input [55].

Original: This property was already used in [51] to implement models of Normalizing Flows [29] with applications in generative modeling.
Revised: This property was already used in [51] to implement models of Normalizing Flows [29] with applications in generative modeling.

Original: (ii)
Revised: (ii)

Original: There is an important literature in Machine Learning about Kernel methods [52].
Revised: There is an important literature in Machine Learning about Kernel methods [52].

Original: In practice, various sub-sampling methods exist in order to approximate infinite-dimensional RKHSs with finite-dimensional spaces generated by Random Fourier Features (RFF) [48, 49].
Revised: In practice, various sub-sampling methods exist in order to approximate infinite-dimensional RKHSs with finite-dimensional spaces generated by Random Fourier Features (RFF) [48, 49].

Original: Thereby, leveraging results on the approximation bound for RFF [54, 53], we show that the expressiveness properties of universal kernels, such as the Gaussian kernel, can be efficiently recovered using residuals of the form Eq.
Revised: Thereby, leveraging results on the approximation bound for RFF [54, 53], we show that the expressiveness properties of universal kernels, such as the Gaussian kernel, can be efficiently recovered using residuals of the form Eq.

Original: (3) with a finite number of neurons.
Revised: (3) with a finite number of neurons.

Original: To further support the practical applicability and the relevance of our model in comparison with stan dard architectures, we report in the supplementary material (Appendix A) numerical experiments on
Revised: To further support the practical applicability and the relevance of our model in comparison with stan dard architectures, we report in the supplementary material (Appendix A) numerical experiments on

Original: MNIST and
Revised: MNIST and

Original: CIFAR datasets.
Revised: CIFAR datasets.

Original: They show that – as predicted by our theory
Revised: They show that – as predicted by our theory

Original: – our model can be trained in these cases to almost zero loss.
Revised: – our model can be trained in these cases to almost zero loss.

Original: But more importantly, they show that our model is able to generalize well on the test dataset with performances that are similar to those of classical ResNets.
Revised: But more importantly, they show that our model is able to generalize well on the test dataset with performances that are similar to those of classical ResNets.

Original: Supervised learning.
Revised: Supervised learning.

Original: We consider a map F from H× R d to R d ′ for some Hilbert space of parameter
Revised: We consider a map F from H× R d to R d ′ for some Hilbert space of parameter

Original: H (e.g. the model of Definition 1 with H =
Revised: H (e.g. the model of Definition 1 with H =

Original: L 2 ([0 , 1] , V ) ) and a training dataset consisting on a family of inputs ( x i ) 1 ≤ i ≤ N ∈
Revised: L 2 ([0 , 1] , V ) ) and a training dataset consisting on a family of inputs ( x i ) 1 ≤ i ≤ N ∈

Original: ( R d ) N and target outputs ( y i )
Revised: ( R d ) N and target outputs ( y i )

Original: 1 ≤ i ≤ N ∈
Revised: 1 ≤ i ≤ N ∈

Original: ( R d ′ ) N .
Revised: ( R d ′ ) N .

Original: Then for every parameter v ∈ H , we define the associated Empirical Risk as:
Revised: Then for every parameter v ∈ H , we define the associated Empirical Risk as:

Original: [Equation]
Revised: [Equation]

Original: Remark 2.
Revised: Remark 2.

Original: For simplicity we consider here the Euclidean square distance as a loss on the output space R d ′ , but our results generalize to any smooth loss satisfying a Polyak-Lojasiewicz inequality (c.f.[10]), e.g. any smooth strongly convex loss.
Revised: For simplicity we consider here the Euclidean square distance as a loss on the output space R d ′ , but our results generalize to any smooth loss satisfying a Polyak-Lojasiewicz inequality (c.f.[10]), e.g. any smooth strongly convex loss.

Original: Training the model F then amounts to finding a parameter v ∗ ∈ arg min v ∈H L ( v ) .
Revised: Training the model F then amounts to finding a parameter v ∗ ∈ arg min v ∈H L ( v ) .

Original: In order to perform such an empirical risk minimization (ERM) task we consider GD on v .
Revised: In order to perform such an empirical risk minimization (ERM) task we consider GD on v .

Original: For a small step size η , for some initialization v 0 ∈ H and for every discrete time step k ∈ N , the training dynamic reads:
Revised: For a small step size η , for some initialization v 0 ∈ H and for every discrete time step k ∈ N , the training dynamic reads:

Original: [Equation]
Revised: [Equation]

Original: Note that we do not consider any additional regularizing term on the loss.
Revised: Note that we do not consider any additional regularizing term on the loss.

Original: In classical supervised learning one would seek for a minimizer of the “regularized” loss L ( v ) + λ R ( v ) , with λ > 0 a constant and R a coercive regularization function.
Revised: In classical supervised learning one would seek for a minimizer of the “regularized” loss L ( v ) + λ R ( v ) , with λ > 0 a constant and R a coercive regularization function.

Original: We are here interested in the non regularized setting, i.e. λ = 0 , often used in practice.
Revised: We are here interested in the non regularized setting, i.e. λ = 0 , often used in practice.

Original: In this case, the generalization property of the computed map is argued to potentially come from the optimization method that shall select an adequate minimizer of the loss.
Revised: In this case, the generalization property of the computed map is argued to potentially come from the optimization method that shall select an adequate minimizer of the loss.

Original: This implicit regularization depends on the choice of the optimization method [42].
Revised: This implicit regularization depends on the choice of the optimization method [42].

Original:  Related works and contributions
Revised:  Related works and contributions

Original: Recently, several works have addressed the problem of proving convergence of (stochastic) GD in the training of NNs.
Revised: Recently, several works have addressed the problem of proving convergence of (stochastic) GD in the training of NNs.

Original: The convergence properties of GD arealready well understood forsome models, such as NNs that are linear w.r.t.
Revised: If the convergence properties of GD are well understood for NNs that are linear w.r.t.

Original: input [24, 7, 64]. On the other hand the case of nonlinear NNs is still actively discussed.
Revised: input [24, 7, 64], it is not the case for non-linear NNs.

Original: In [34, 33, 17], the authors focus on the training of “shallow” two layers fully connected NNs and establish convergence of GD in an overparameterized setting where width of the intermediary layer scales polynomially with the size N of the dataset.
Revised: In [34, 33, 17], the authors focus on the training of “shallow” two layers fully connected NNs and establish convergence of GD in an overparameterized setting where width of the intermediary layer scales polynomially with the size N of the dataset.

Original: More recently, with the same setup, [62] showed that the neurons of a teacher network are recovered by a student network optimized with GD as long as the width of the student network is higher than the teacher’s one.
Revised: More recently, with the same setup, [62] showed that the neurons of a teacher network are recovered by a student network optimized with GD as long as the width of the student network is higher than the teacher’s one.

Original: Formally, their analysis is similar to ours as the result holds if the loss at initialization is already sufficiently low and the proof relies on Polyak-Lojasiewicz inequalities verified by the loss landscape.
Revised: Formally, their analysis is similar to ours as the result holds if the loss at initialization is already sufficiently low and the proof relies on Polyak-Lojasiewicz inequalities verified by the loss landscape.

Original: Infinite depth.
Revised: Infinite depth.

Original: The works of [16, 2, 64, 32, 63, 35, 12, 43] extend those results to arbitrary deep NN in the overparameterized setting.
Revised: The works of [16, 2, 64, 32, 63, 35, 12, 43] extend those results to arbitrary deep NN in the overparameterized setting.

Original: Specifically, the results in [16, 2, 35] apply to deep
Revised: Specifically, the results in [16, 2, 35] apply to deep

Original: ResNets.
Revised: ResNets.

Original: The best result seems to be achieved in [43], with convergence as soon as the last layer has a width m = Ω( N 3 ) and at best with linear width.
Revised: The best result seems to be achieved in [43], with convergence as soon as the last layer has a width m = Ω( N 3 ) and at best with linear width.

Original: A common feature for those works is to rely on the fact that, for a sufficiently high number of parameters, the model can be well approximated by a linear model corresponding to its first order expansion around the initialization.
Revised: A common feature for those works is to rely on the fact that, for a sufficiently high number of parameters, the model can be well approximated by a linear model corresponding to its first order expansion around the initialization.

Original: In [15] this phenomenon, called “lazy regime”, is attributed to an inappropriate scaling of the parameters.
Revised: In [15] this phenomenon, called “lazy regime”, is attributed to an inappropriate scaling of the parameters.

Original: On the other hand, [36, 35] refer to this phenomenon as “linear” or “kernel regime” and relate it the constancy of the Neural Tangent Kernel (NTK) introduced in [26].
Revised: On the other hand, [36, 35] refer to this phenomenon as “linear” or “kernel regime” and relate it the constancy of the Neural Tangent Kernel (NTK) introduced in [26].

Original: However, in all those works the width of intermediary layers has to depend on the depth D of the network.
Revised: However, in all those works the width of intermediary layers has to depend on the depth D of the network.

Original: Therefore, these results do not apply to the training of the model in Eq.
Revised: Therefore, these results do not apply to the training of the model in Eq.

Original: (1), corresponding to the limit D → + ∞ .
Revised: (1), corresponding to the limit D → + ∞ .

Original: Infinite width.
Revised: Infinite width.

Original: The other direction of over-parameterization, analyzed in several works [41, 14, 40, 27, 38, 21, 46] is to consider the limit of infinitely wide layers.
Revised: The other direction of over-parameterization, analyzed in several works [41, 14, 40, 27, 38, 21, 46] is to consider the limit of infinitely wide layers.

Original: In such a “mean-field” setting, the model is parameterized by the distribution of the parameters at each layer.
Revised: In such a “mean-field” setting, the model is parameterized by the distribution of the parameters at each layer.

Original: In [14, 41, 40, 27] the training dynamic is analyzed as a gradient flow in the Wasserstein space [3], showing that the only stationary distributions are global minimizers of the empirical risk.
Revised: In [14, 41, 40, 27] the training dynamic is analyzed as a gradient flow in the Wasserstein space [3], showing that the only stationary distributions are global minimizers of the empirical risk.

Original: In [21] a similar result is showed for deep NN with an arbitrary number of infinitely wide layers.
Revised: In [21] a similar result is showed for deep NN with an arbitrary number of infinitely wide layers.

Original: In [13, 1], local linear convergence towards the global optimum is shown for two layers NNs in a teacher-student setup with regularized loss.
Revised: In [13, 1], local linear convergence towards the global optimum is shown for two layers NNs in a teacher-student setup with regularized loss.

Original: Finally, [38] analyzes the convergence of continuous ResNets with infinitely wide residual layers and shows that every critical point is a global minimizer of the empirical risk.
Revised: Finally, [38] analyzes the convergence of continuous ResNets with infinitely wide residual layers and shows that every critical point is a global minimizer of the empirical risk.

Original: We stress out that these results only apply to infinitely wide NNs.
Revised: We stress out that these results only apply to infinitely wide NNs.

Original: It is not clear if this mean-field limit extends to the parametric setting of MLPs with the Euclidean metric on their parameters.
Revised: It is not clear if this mean-field limit extends to the parametric setting of MLPs with the Euclidean metric on their parameters.

Original: In contrast, a RKHS structure naturally arises when considering a linear parameterization of the residuals.
Revised: In contrast, a RKHS structure naturally arises when considering a linear parameterization of the residuals.

Original: Assumptionand Assumption 2 can be satisfied both in a parametric setting with a finite number of features and in a mean-field setting limit where the residuals are generated by a universal kernel.
Revised: Assumptionand Assumption 2 can be satisfied both in a parametric setting with a finite number of features and in a mean-field setting limit where the residuals are generated by a universal kernel.

Original: Contributions.
Revised: Contributions.

Original: We show convergence results for GD in the training of RKHS-NODEs (see Definition 1).
Revised: We show convergence results for GD in the training of RKHS-NODEs (see Definition 1).

Original: These correspond to infinitely deep continuous ResNets with linear parameterization of the residuals.
Revised: These correspond to infinitely deep continuous ResNets with linear parameterization of the residuals.

Original: Our first main contribution, in Section 4, shows that under some regularity and expressivity assumptions on the residuals, the associated empirical risk satisfies a (local) PolyakLojasiewicz Property 2.
Revised: Our first main contribution, in Section 4, shows that under some regularity and expressivity assumptions on the residuals, the associated empirical risk satisfies a (local) PolyakLojasiewicz Property 2.

Original: A consequence is Theorem 2, which states global convergence of GD towards a global optimum (zero training loss) under the condition that the loss at initialization is already sufficiently low.
Revised: A consequence is Theorem 2, which states global convergence of GD towards a global optimum (zero training loss) under the condition that the loss at initialization is already sufficiently low.

Original: In the limit where the loss at initialization is arbitrarily small, we recover a linear regime as described in [36, 35].
Revised: In the limit where the loss at initialization is arbitrarily small, we recover a linear regime as described in [36, 35].

Original: Our second contribution, in Section 5, shows how this condition for global convergence can be enforced using suitably chosen first and last linear layers.
Revised: Our second contribution, in Section 5, shows how this condition for global convergence can be enforced using suitably chosen first and last linear layers.

Original: Thereafter, we show how the assumptions of Theorem 2, can be satisfied for RKHSs generated by a finite number of Random Features, with high probability over the choice of these features.
Revised: Thereafter, we show how the assumptions of Theorem 2, can be satisfied for RKHSs generated by a finite number of Random Features, with high probability over the choice of these features.

Original: For any dataset ( x i , y i ) 1 ≤ i ≤ N ∈
Revised: For any dataset ( x i , y i ) 1 ≤ i ≤ N ∈

Original: ( R d × R d ′ )
Revised: ( R d × R d ′ )

Original: N , we conclude in Theorem 3 to convergence of GD towards a global minimum of Eq.
Revised: N , we conclude in Theorem 3 to convergence of GD towards a global minimum of Eq.

Original: (7) with high probability when the width of the layers scales polynomially w.r.t.
Revised: (7) with high probability when the width of the layers scales polynomially w.r.t.

Original: the size of the dataset N and the inverse input data separation δ − 1 .
Revised: the size of the dataset N and the inverse input data separation δ − 1 .

Original: Finally, we point out that some of our results can be seen as a generalization of existing results concerning convergence of GD for the training of linear NNs [24, 7, 64].
Revised: Finally, we point out that some of our results can be seen as a generalization of existing results concerning convergence of GD for the training of linear NNs [24, 7, 64].

Original: We explain in Appendix E how, following the lines of our analysis, one can for example recover [64, Theorem 3.1.].
Revised: We explain in Appendix E how, following the line of our analysis, one can for example recover [64, Theorem 3.1.].

Original: However, if Definition 1 encompasses linear ResNets as a special case, we stress that Theorem 2 applies to a way larger class of models.
Revised: However, if Definition 1 encompasses linear ResNets as a special case, we stress that Theorem 2 applies to a way larger class of models.

Original: In particular, it is an important feature of our work to provide an analysis structure that can be easily adapted to different models or data-structure.

Original: Notations.
Revised: Notations.

Original: In what follows ∥ .
Revised: In what follows ∥ .

Original: ∥ denotes the Euclidean ℓ 2 norm for vectors and the Frobenius norm for matrices.
Revised: ∥ denotes the Euclidean ℓ 2 norm for vectors and the Frobenius norm for matrices.

Original: For matrices the spectral norm is denoted ∥ .
Revised: For matrices the spectral norm is denoted ∥ .

Original: ∥ 2 , the smallest (resp. greatest) singular value is denoted σ min (resp.
Revised: ∥ 2 , the smallest (resp. greatest) singular value is denoted σ min (resp.

Original: σ max ) and for symmetric matrices the smallest (resp.
Revised: σ max ) and for symmetric matrices the smallest (resp.

Original: greatest) eigenvalue is denoted λ min (resp.
Revised: greatest) eigenvalue is denoted λ min (resp.

Original: λ max ).
Revised: λ max ).

Original: Given some Hilbert space H , the functional Hilbert space
Revised: Given some Hilbert space H , the functional Hilbert space

Original: L 2 ([0 , 1] , H ) is denoted L 2 ( H ) or L 2 when there is no ambiguity.
Revised: L 2 ([0 , 1] , H ) is denoted L 2 ( H ) or L 2 when there is no ambiguity.

Original: The notation O (resp.
Revised: The notation O (resp.

Original: Ω ) means asymptotically inferior (resp.
Revised: Ω ) means asymptotically inferior (resp.

Original: superior)
Revised: superior)

Original: up to multiplicative constant.
Revised: up to multiplicative constant.

Original:  Analysis of convergence for overparameterized models
Revised:  Analysis of convergence for overparameterized models

Original: In this section, we review methods for analyzing the convergence of overparameterized machine learning models based on [36, 35].
Revised: In this section, we review methods for analyzing the convergence of overparameterized machine learning models based on [36, 35].

Original: We refer to Appendix B for detailed proofs of the statements.
Revised: We refer to Appendix B for detailed proofs of the statements.

Original: As presented above, we consider an optimization over the variable v in some Hilbert space H , with fixed input and output data, say v (cid:55)→ F ( v ) :
Revised: As presented above, we consider an optimization over the variable v in some Hilbert space H , with fixed input and output data, say v (cid:55)→ F ( v ) :

Original: =
Revised: =

Original: [ F ( v, x i )]
Revised: [ F ( v, x i )]

Original: i =1 ,...,N .
Revised: i =1 ,...,N .

Original: Therefore, the empirical risk is a function of the parameters v ∈ H .
Revised: Therefore, the empirical risk is a function of the parameters v ∈ H .

Original: We say that the model is overparameterized whenever the dimension dim ( H ) of the parameter space is much larger than the dimension of the output space of
Revised: We say that the model is overparameterized whenever the dimension dim ( H ) of the parameter space is much larger than the dimension of the output space of

Original: F ( v ) , here d ′ N .
Revised: F ( v ) , here d ′ N .

Original: The RKHS-NODE model defined F in Definition 1 falls into this category as H is the infinite dimensional functional space
Revised: The RKHS-NODE model defined F in Definition 1 falls into this category as H is the infinite dimensional functional space

Original: L 2 ([0 , 1] , V ) .
Revised: L 2 ([0 , 1] , V ) .

Original: 3.1 A (local) Polyak-Lojasiewicz property
Revised: 3.1 A (local) Polyak-Lojasiewicz property

Original: When dealing with overparameterized models, one cannot expect the loss to be convex but one expects the model to perfectly fit the data, that is to reach the global minimum value of 0 .
Revised: When dealing with overparameterized models, one cannot expect the loss to be convex but one expects the model to perfectly fit the data, that is to reach the global minimum value of 0 .

Original: In fact, for a sufficient number of parameters, the loss landscape typically possesses a continuum of infinitely many global minima and is non-convex in any neighbourhood of a global minima [36].
Revised: In fact, for a sufficient number of parameters, the loss landscape typically possesses a continuum of infinitely many global minima and is non-convex in any neighbourhood of a global minima [36].

Original: One thus rather needs to rely on a set of functional inequalities allowing to control the decrease rate of the loss along GD [37, 10].
Revised: One thus rather needs to rely on a set of functional inequalities allowing to control the decrease rate of the loss along GD [37, 10].

Original: Definition 2 ((local) Polyak-Lojasiewicz property) .
Revised: Definition 2 ((local) Polyak-Lojasiewicz property) .

Original: Let L : H → R + be a differentiable function.
Revised: Let L : H → R + be a differentiable function.

Original: We say that L satisfies a (local) Polyak-Lojasiewicz (PL) property if there exist positive continuous functions m, M : R + → R ∗ + s.t.
Revised: We say that L satisfies a (local) Polyak-Lojasiewicz (PL) property if there exist positive continuous functions m, M : R + → R ∗ + s.t.

Original: for every v ∈ H
Revised: for every v ∈ H

Original: [Equation]
Revised: [Equation]

Original: Such functional inequalities have already shown to be relevant for proving convergence guarantees in the training of NNs [22].
Revised: Such functional inequalities have already shown to be relevant for proving convergence guarantees in the training of NNs [22].

Original: A first consequence for a loss L which satisfies the (local) PL property of Definition 2 is that it does not admit any spurious local minima but only global minima.
Revised: A first consequence for a loss L which satisfies the (local) PL property of Definition 2 is that it does not admit any spurious local minima but only global minima.

Original: Also, if the training dynamic is bounded, then m and M are uniformly lower- and upper-bounded along the dynamic, implying that L decreases at a linear rate.
Revised: Also, if the training dynamic is bounded, then m and M are uniformly lower- and upper-bounded along the dynamic, implying that L decreases at a linear rate.

Original: In most cases, m and M are degenerate when ∥ v ∥ → + ∞ .
Revised: In most cases, m and M are degenerate when ∥ v ∥ → + ∞ .

Original: When the dynamic is not bounded, L can thus decrease to 0 slower than at a linear rate or even converge towards a strictly positive limit.
Revised: When the dynamic is not bounded, L can thus decrease to 0 slower than at a linear rate or even converge towards a strictly positive limit.

Original: 3.2 Local convergence result
Revised: 3.2 Local convergence result

Original: Because of the degeneracy of m and M , it is in general not possible to conclude an unconditional convergence of GD towards a global minimizer of the empirical risk.
Revised: Because of the degeneracy of m and M , it is in general not possible to conclude an unconditional convergence of GD towards a global minimizer of the empirical risk.

Original: However, PL inequalities are sufficient to prove convergence when the problem is not too hard to solve, that is when the loss at initialization is not too high.
Revised: However, PL inequalities are sufficient to prove convergence when the problem is not too hard to solve, that is when the loss at initialization is not too high.

Original: Moreover, when using gradient descent stepping, one needs to make a supplementary smoothness assumption on the empirical risk L .
Revised: Moreover, when using gradient descent stepping, one needs to make a supplementary smoothness assumption on the empirical risk L .

Original: This ensures that the loss decreases at each gradient step for a sufficiently small step size.
Revised: This ensures that the loss decreases at each gradient step for a sufficiently small step size.

Original: Definition 3 (Smoothness, Definition 2 of [36]) .
Revised: Definition 3 (Smoothness, Definition 2 of [36]) .

Original: Let β ≥ 0 be a constant.
Revised: Let β ≥ 0 be a constant.

Original: We say that the function
Revised: We say that the function

Original: L : H → R is β -smooth if for every v, v ′ ∈ H : | L ( v ′ )
Revised: L : H → R is β -smooth if for every v, v ′ ∈ H : | L ( v ′ )

Original: − L ( v ) − ⟨∇ L ( v ) , v ′ − v ⟩| ≤ β 2 ∥ v ′ − v ∥ 2 .
Revised: − L ( v ) − ⟨∇ L ( v ) , v ′ − v ⟩| ≤ β 2 ∥ v ′ − v ∥ 2 .

Original: The local PL property combined with this smoothness assumption then gives a local convergence result for the convergence of GD towards a global minimizer of the empirical risk.
Revised: The local PL property combined with this smoothness assumption then gives a local convergence result for the convergence of GD towards a global minimizer of the empirical risk.

Original: Theorem 1 (Theorem 6 of [36]) .
Revised: Theorem 1 (Theorem 6 of [36]) .

Original: Let L : H → R + be a loss function satisfying a local PL property with local constants m and M .
Revised: Let L : H → R + be a loss function satisfying a local PL property with local constants m and M .

Original: Let v 0 ∈ H and R ≥ 0 be such that
Revised: Let v 0 ∈ H and R ≥ 0 be such that

Original: [Equation]
Revised: [Equation]

Original:  Furthermore, assume that L is β -smooth within the ball B ( v 0 , R ) .
Revised: [Equation] Furthermore, assume that L is β -smooth within the ball B ( v 0 , R ) .

Original: Then for a step size η ≤ β − 1 ,
Revised: Then for a step size η ≤ β − 1 ,

Original: GD with initialization v 0 and step size η converges towards a global minimizer of L with a linear convergence rate and inside a ball of radius R .
Revised: GD with initialization v 0 and step size η converges towards a global minimizer of L with a linear convergence rate and inside a ball of radius R .

Original: More precisely, for every k ≥ 0 :
Revised: More precisely, for every k ≥ 0 :

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  Properties of RKHS-NODE
Revised:  Properties of RKHS-NODE

Original: In this section we analyze the convergence of GD in the training of the infinitely deep ResNet model of Definition 1.
Revised: In this section we analyze the convergence of GD in the training of the infinitely deep ResNet model of Definition 1.

Original: Note that such a model is overparameterized in depth as the parameter space is the infinite dimensional space L 2 ([0 , 1] , V ) and overparameterization can also come from width when the RKHS is high (or even infinite) dimensional.
Revised: Note that such a model is overparameterized in depth as the parameter space is the infinite dimensional space L 2 ([0 , 1] , V ) and overparameterization can also come from width when the RKHS is high (or even infinite) dimensional.

Original: Therefore, our proof of convergence heavily relies on a PL property verified by the empirical risk.
Revised: Therefore, our proof of convergence heavily relies on a PL property verified by the empirical risk.

Original: Recall that we consider the training of deep ResNets with a linear parameterization of the residuals.
Revised: Recall that we consider the training of deep ResNets with a linear parameterization of the residuals.

Original: The set of residuals is as in Eq.
Revised: The set of residuals is as in Eq.

Original: (3) with the metric of Eq.
Revised: (3) with the metric of Eq.

Original: (5) induced by the Frobenius metric (Eq.
Revised: (5) induced by the Frobenius metric (Eq.

Original: This provides V with a RKHS structure [4], whose associated kernel is given for any z, z ′ ∈ R q by K ( z, z ′ ) :
Revised: This provides V with a RKHS structure [4], whose associated kernel is given for any z, z ′ ∈ R q by K ( z, z ′ ) :

Original: = ⟨ φ ( z ) , φ ( z ′ ) ⟩ Id q , and whose associated feature map is given by φ .
Revised: = ⟨ φ ( z ) , φ ( z ′ ) ⟩ Id q , and whose associated feature map is given by φ .

Original: Remark 3.
Revised: Remark 3.

Original: The definition of ⟨ ., .
Revised: The definition of ⟨ ., .

Original: ⟩ V in Eq.
Revised: ⟩ V in Eq.

Original: (5) requires Span ( φ ( R q ))
Revised: (5) requires Span ( φ ( R q ))

Original: =
Revised: =

Original: R r to associate each v ∈ V to a unique W ∈ R q × r .
Revised: R r to associate each v ∈ V to a unique W ∈ R q × r .

Original: This is satisfied by all the feature maps φ we consider in the following.
Revised: This is satisfied by all the feature maps φ we consider in the following.

Original: Given a training dataset composed of input data points ( x i ) 1 ≤ i ≤ N ∈
Revised: Given a training dataset composed of input data points ( x i ) 1 ≤ i ≤ N ∈

Original: ( R d ) N and of target data points ( y i ) 1 ≤ i ≤ N ∈
Revised: ( R d ) N and of target data points ( y i ) 1 ≤ i ≤ N ∈

Original: ( R d ′ )
Revised: ( R d ′ )

Original: N we are interested in the task of minimizing the empirical risk of Eq.
Revised: N we are interested in the task of minimizing the empirical risk of Eq.

Original: (7) by
Revised: (7) by

Original: GD over v .
Revised: GD over v .

Original: Analogously to back-propagation in discrete NNs architectures, the gradient of L can be expressed thanks to a backward equation derived by adjoint sensitivity analysis [47].
Revised: Analogously to back-propagation in discrete NNs architectures, the gradient of L can be expressed thanks to a backward equation derived by adjoint sensitivity analysis [47].

Original: Property 1.
Revised: Property 1.

Original: Let L be the empirical risk in Eq.
Revised: Let L be the empirical risk in Eq.

Original: (7) associated with the RKHS-NODE model with a quadratic loss.
Revised: (7) associated with the RKHS-NODE model with a quadratic loss.

Original: Let K be the kernel function associated with the RKHS V .
Revised: Let K be the kernel function associated with the RKHS V .

Original: Then L is differentiable on L 2 ([0 , 1] , V ) , with for every v ∈ L 2 ([0 , 1] , V ) , ∇ L ( v ) =
Revised: Then L is differentiable on L 2 ([0 , 1] , V ) , with for every v ∈ L 2 ([0 , 1] , V ) , ∇ L ( v ) =

Original: (cid:80) Ni =1 K ( ., z i ) p
Revised: (cid:80) Ni =1 K ( ., z i ) p

Original: i , where for each index i ∈ (cid:74) 1 , N (cid:75) ,
Revised: i , where for each index i ∈ (cid:74) 1 , N (cid:75) ,

Original: z i is the solution of Eq.
Revised: z i is the solution of Eq.

Original: (6) with initial condition
Revised: (6) with initial condition

Original: Ax i and the adjoint variable p
Revised: Ax i and the adjoint variable p

Original: i is the solution to the backward problem :
Revised: i is the solution to the backward problem :

Original: [Equation]
Revised: [Equation]

Original: 4.1 PL property of RKHS-NODE
Revised: 4.1 PL property of RKHS-NODE

Original: Following the line of proof sketched in Section 3, we show how to derive PL inequalities of the form Eq.
Revised: Following the line of proof sketched in Section 3, we show how to derive PL inequalities of the form Eq.

Original: (8) for the empirical loss associated with the RKHS-NODE model.
Revised: (8) for the empirical loss associated with the RKHS-NODE model.

Original: For that purpose we make a few assumptions about the RKHS V .
Revised: For that purpose we make a few assumptions about the RKHS V .

Original: The first one concerns its regularity and allows us to control the solutions of Eqs.
Revised: The first one concerns its regularity and allows us to control the solutions of Eqs.

Original: (6) and (11).
Revised: (6) and (11).

Original: Assumption 1 ((strong) Admissibility) .
Revised: Assumption 1 ((strong) Admissibility) .

Original: We say that the RKHS V is (strongly) admissible if it is continuously embedded in W 2 , ∞ ( R q , R q ) .
Revised: We say that the RKHS V is (strongly) admissible if it is continuously embedded in W 2 , ∞ ( R q , R q ) .

Original: More precisely, there exists a constant κ > 0 s.t.
Revised: More precisely, there exists a constant κ > 0 s.t.

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Assuming V is embedded in W 1 , ∞ ( R q , R q ) is natural to ensure the regularity of the flow generated by the control parameter [55, 58] and suffices to prove convergence of a continuous gradient flow on the parameter v .
Revised: Assuming V is embedded in W 1 , ∞ ( R q , R q ) is natural to ensure the regularity of the flow generated by the control parameter [55, 58] and suffices to prove convergence of a continuous gradient flow on the parameter v .

Original: Assumption 1 is a bit stronger because a supplementary smoothness result on the loss landscape is necessary to prove convergence of discrete GD (c.f. Definition 3).
Revised: Assumption 1 is a bit stronger because a supplementary smoothness result on the loss landscape is necessary to prove convergence of discrete GD (c.f. Definition 3).

Original: In practice, κ can be computed for smooth kernels thanks to Property 4 in Appendix D. For example, the RKHS associated with the Gaussian kernel k : r
Revised: In practice, κ can be computed for smooth kernels thanks to Property 4 in Appendix D. For example, the RKHS associated with the Gaussian kernel k : r

Original: (cid:55)→ e − r 2 / 2 is (strongly) admissible with κ = 2 + √ 3 .
Revised: (cid:55)→ e − r 2 / 2 is (strongly) admissible with κ = 2 + √ 3 .

Original: The second assumption is related to the expressiveness of V and is a weaker form of the classical universality property of RKHSs.
Revised: The second assumption is related to the expressiveness of V and is a weaker form of the classical universality property of RKHSs.

Original: Assumption 2 ( N -universality) .
Revised: Assumption 2 ( N -universality) .

Original: Let K be the kernel function associated with the RKHS V .
Revised: Let K be the kernel function associated with the RKHS V .

Original: For a family of points ( z i ) 1 ≤ i ≤ N ∈
Revised: For a family of points ( z i ) 1 ≤ i ≤ N ∈

Original: ( R q ) N , we define the associated kernel matrix as the block matrix K
Revised: ( R q ) N , we define the associated kernel matrix as the block matrix K

Original: [Equation]
Revised: [Equation]

Original: More precisely we assume for every δ > 0 :
Revised: More precisely we assume for every δ > 0 :

Original: [Equation]
Revised: [Equation]

Revised: Assumption 2 is required in order to ensure the expressivity of our model, quantified by the conditioning of the kernel matrix K and by Λ and λ .

Revised: The choice of the RKHS V may thus have a significant impact on training.

Original: In particular, satisfying Assumption 2 requires having V of dimension m ≥ N , but it can be satisfied for finite dimensional RKHSs of dimension m ≤ N q , for example by considering a polynomial kernel, or by RKHSs of dimension m ≥ poly ( N, q ) with high probability on the sampling of random features, as shown in Section 5.
Revised: In particular, satisfying Assumption 2 requires having V of dimension m ≥ N , but it can be satisfied for finite dimensional RKHSs of dimension m ≤ N q , for example by considering a polynomial kernel, or by RKHSs of dimension m ≥ poly ( N, q ) with high probability on the sampling of random features, as shown in Section 5.

Original: On the other hand, even though the existence of λ follows from compactness arguments, it seems to be hardly analytically tractable even for classical kernels such as the Gaussian kernel.
Revised: On the other hand, even though the existence of λ follows from compactness arguments, it seems to be hardly analytically tractable even for classical kernels such as the Gaussian kernel.

Revised: Therefore, if, in theory, prior knowledge of the data distribution might allow to optimize the choice of kernel, we expect the selection of an optimal kernel to be an intractable problem in practice.

Revised: Instead, cross-validation techniques can be used to select a suitable kernel.

Original: Remark 4.
Revised: Remark 4.

Original: For a RKHS V as in Eq.
Revised: For a RKHS V as in Eq.

Original: (3) , the properties of V only depend on φ .
Revised: (3) , the properties of V only depend on φ .

Original: An interesting example is when φ : z (cid:55)→ σ ( Uz ) with σ an activation function applied component-wise and U a fixed feature matrix.
Revised: An interesting example is when φ : z (cid:55)→ σ ( Uz ) with σ an activation function applied component-wise and U a fixed feature matrix.

Original: In Section 5 we show that, when considering the complex activation σ : t (cid:55)→ e − ıt , both assumptions can be satisfied with high probability.
Revised: In Section 5 we show that, when considering the complex activation σ : t (cid:55)→ e − ıt , both assumptions can be satisfied with high probability.

Original: On the other hand, Assumption 1 is not satisfied when considering σ = ReLU due to its non-smoothness at 0 .
Revised: On the other hand, Assumption 1 is not satisfied when considering σ = ReLU due to its non-smoothness at 0 .

Original: Remark 5.
Revised: Remark 5.

Original: Note that Λ could also be allowed to depend on some parameters, such as max ∥
Revised: Note that Λ could also be allowed to depend on some parameters, such as max ∥

Original: z i ∥ .
Revised: z i ∥ .

Original: However, as it is a more critical aspect of our analysis, we prefer to highlight the dependency of λ w.r.t.
Revised: However, as it is a more critical aspect of our analysis, we prefer to highlight the dependency of λ w.r.t.

Original: min i ̸ =
Revised: min i ̸ =

Original: j ∥ z i − z j ∥ .
Revised: j ∥ z i − z j ∥ .

Original: For all the RKHSs studied here we always have Λ ≤ N .
Revised: For all the RKHSs studied here we always have Λ ≤ N .

Original: The following PL property is satisfied by the risk L .
Revised: The following PL property is satisfied by the risk L .

Original: Property 2 is proven in Appendix C.2.
Revised: Property 2 is proven in Appendix C.2.

Original: Property 2 (RKHS-NODE satisfy PL) .
Revised: Property 2 (RKHS-NODE satisfy PL) .

Original: Assume V satisfies Assumption 1 with κ and Assumptionwith λ and Λ .
Revised: Assume V satisfies Assumption 1 with κ and Assumptionwith λ and Λ .

Original: Let L be the empirical risk in Eq.
Revised: Let L be the empirical risk in Eq.

Original: (7) associated with the RKHS-NODE model of Definition 1.
Revised: (7) associated with the RKHS-NODE model of Definition 1.

Original: Then L satisfies the PL inequalities of Definition 2 with m and M given by:
Revised: Then L satisfies the PL inequalities of Definition 2 with m and M given by:

Original: [Equation]
Revised: [Equation]

Original: where δ :=
Revised: where δ :=

Original: min i ̸ =
Revised: min i ̸ =

Original: j ∥ x i − x j ∥ is the data separation.
Revised: j ∥ x i − x j ∥ is the data separation.

Revised: Sketch of proof.

Revised: Assumption 1 can be used to have estimates on the solutions

Revised: z i of the forward problem Eq.

Revised: (6) and on the solutions p

Original: Note that the degeneracy of the bounding functions M, m as R → + ∞ readily appears in Eq.
Revised: i of backward problem Eq. This gives for every indices i, j ∈ (cid:74) 1 , N (cid:75) and every t ∈ [0 , 1] : [Equation] where z i solves Eq.

Revised: (6) with initial condition

Revised: Ax i , and:

Revised: [Equation]

Revised: Moreover using the initial condition p i 1 = − 1 N B ⊤ ( BZ i 1 − y i ) we have:

Revised: [Equation]

Revised: Then denoting ˜ p t the vector of stacked p it and using properties of RKHSs, we have for t ∈ [0 , 1] :

Revised: [Equation]

Revised: where K is the kernel matrix associated with the points ( z it ) i .

Revised: This last equality gives the result using Assumption 2 and the previous estimates on p i .

Revised: Note that the degeneracy of the bounding functions M, m as R → + ∞ readily appears in Eq.

Original: Thus one should not expect these bounds to imply global convergence of GD without making any further assumption.
Revised: Thus one should not expect these bounds to imply global convergence of GD without making any further assumption.

Original: Indeed, cases where GD fails to converge towards a global optimizer of the loss are observed in [7], Section 6, with a setup corresponding to the model of Definition 1 with V as in Eq.
Revised: Indeed, cases where GD fails to converge towards a global optimizer of the loss are observed in [7], Section 6, with a setup corresponding to the model of Definition 1 with V as in Eq.

Original: (3) and φ = Id R q .
Revised: (3) and φ = Id R q .

Original: Also, note that the data separation δ plays an important role in Propertyas it intervenes in the conditioning of the kernel matrix.
Revised: Also, note that the data separation δ plays an important role in Propertyas it intervenes in the conditioning of the kernel matrix.

Original: In what follows, we always assume the data points to have a data separation lower-bounded by δ > 0 .
Revised: In what follows, we always assume the data points to have a data separation lower-bounded by δ > 0 .

Original: 4.2 Convergence of RKHS-NODE
Revised: 4.2 Convergence of RKHS-NODE

Original: Thanks to the convergence analysis for overparameterized models detailed in Section 3, our main result follows as a consequence of the previous property.
Revised: Thanks to the convergence analysis for overparameterized models detailed in Section 3, our main result follows as a consequence of the previous property.

Original: Theorem 2 is proven in Appendix C.3.
Revised: Theorem 2 is proven in Appendix C.3.

Original: Theorem 2.
Revised: Theorem 2.

Original: Let V satisfy Assumption 1 with constant κ and Assumption 2 with λ, Λ .
Revised: Let V satisfy Assumption 1 with constant κ and Assumption 2 with λ, Λ .

Original: Let v 0 be some initialization of the control parameter with ∥ v 

Revised: Let v 0 be some initialization of the control parameter with ∥ v

Original: ∥
Revised: ∥

Original: L 2 = R 

Revised: L 2 = R

Original: and assume there exists a positive radius
Revised: and assume there exists a positive radius

Revised: R

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Then, for a sufficiently small step-size η > 0 , GD with step-size η converges towards a minimizer of the training loss at a linear rate and inside a ball of radius R .
Revised: Then, for a sufficiently small step-size η > 0 , GD with step-size η converges towards a minimizer of the training loss at a linear rate and inside a ball of radius R .

Original: More precisely, for every k ≥ 0 :
Revised: More precisely, for every k ≥ 0 :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where µ := 1 N σ 2min ( B ⊤ ) λ (cid:0) σ − 1min
Revised: where µ := 1 N σ 2min ( B ⊤ ) λ (cid:0) σ − 1min

Original: ( A ) δ − 1 e κ ( R + R 0 ) (cid:1) e − 2 κ ( R + R 0 ) .
Revised: ( A ) δ − 1 e κ ( R + R 0 ) (cid:1) e − 2 κ ( R + R 0 ) .

Original: As Theorem 1, Theorem 2 is a local convergence result in which the condition in Eq.
Revised: As Theorem 1, Theorem 2 is a local convergence result in which the condition in Eq.

Original: (15) expresses a threshold between two kinds of behaviours: (i) if L ( v 0 ) is sufficiently small, the training dynamic converges towards a global minimizer.
Revised: (15) expresses a threshold between two kinds of behaviours: (i) if L ( v 0 ) is sufficiently small, the training dynamic converges towards a global minimizer.

Original: The limiting behaviour is when the l.h.s.
Revised: The limiting behaviour is when the l.h.s.

Original: of Eq.
Revised: of Eq.

Original: (15) tends to 0 .
Revised: (15) tends to 0 .

Original: Because of a regularizing effect of GD (i.e. that ∥ v k − v 

Revised: Because of a regularizing effect of GD (i.e. that ∥ v k − v

Original: ∥
Revised: ∥

Original: L 2 ≤ R ), the parameter stays in a ball of arbitrary small radius R all along the training dynamic.
Revised: L 2 ≤ R ), the parameter stays in a ball of arbitrary small radius R all along the training dynamic.

Original: In this limit, we recover a “linear” or “kernel” regime where the model is well approximated by its linearization at v 

Revised: In this limit, we recover a “linear” or “kernel” regime where the model is well approximated by its linearization at v

Original: (ii)
Revised: (ii)

Original: If L ( v 0 ) is too large, the result says nothing about the convergence of the GD.
Revised: If L ( v 0 ) is too large, the result says nothing about the convergence of the GD.

Original: However, it is still observed in practice that the training dynamic often converges towards a global minimizer of the loss [60].
Revised: However, it is still observed in practice that the training dynamic often converges towards a global minimizer of the loss [60].

Original: Explaining this phenomenon in a general setting remains a challenging open question.
Revised: Explaining this phenomenon in a general setting remains a challenging open question.

Original:  Enforcing convergence with high dimensional embedding and finite width
Revised:  Enforcing convergence with high dimensional embedding and finite width

Original: As Theorem 2 is a local convergence result, it does not allow to conclude a general convergence behaviour of GD in the training of RKHS-NODE.
Revised: As Theorem 2 is a local convergence result, it does not allow to conclude a general convergence behaviour of GD in the training of RKHS-NODE.

Original: In the following, we show how one can enforce the hypothesis of Theorem 2 to be verified and prove two global convergence results.
Revised: In the following, we show how one can enforce the hypothesis of Theorem 2 to be verified and prove two global convergence results.

Original: The first one relies on suitably choosing matrices A and B in order to satisfy Eq.
Revised: The first one relies on suitably choosing matrices A and B in order to satisfy Eq.

Original: (15) and applies in the case of infinite width, i.e. with residual layers in a universal RKHS.
Revised: (15) and applies in the case of infinite width, i.e. with residual layers in a universal RKHS.

Original: The second result recovers global convergence in a finite width regime, relying on a high number r of Random Fourier Features.
Revised: The second result recovers global convergence in a finite width regime, relying on a high number r of Random Fourier Features.

Original: For the sake of readability we only consider here the case where V belongs to a restricted class of
Revised: For the sake of readability we only consider here the case where V belongs to a restricted class of

Original: RKHSs and refer to Appendix D for more general results and complete proofs.
Revised: RKHSs and refer to Appendix D for more general results and complete proofs.

Original: For some positive parameter ν > 

Revised: For some positive parameter ν >

Original: we consider the Matérn kernel k defined in [57]:
Revised: we consider the Matérn kernel k defined in [57]:

Original: [Equation]
Revised: [Equation]

Original: where Γ is the Gamma function and K ν is the modified Bessel function of the second kind.
Revised: where Γ is the Gamma function and K ν is the modified Bessel function of the second kind.

Original: Equivalently, k can be defined by its frequency distribution over R q as: 
Revised: Equivalently, k can be defined by its frequency distribution over R q as: [Equation]

Original: [Equation]
Revised: [Equation]

Original: and C q,ν a normalizing constant.
Revised: and C q,ν a normalizing constant.

Original: For every q ≥ 1 , such a function is known to define a structure of vector-valued RKHS V q over R q corresponding to the Sobolev space H ν + q/ 2 ( R q , R q ) [52, 57].
Revised: For every q ≥ 1 , such a function is known to define a structure of vector-valued RKHS V q over R q corresponding to the Sobolev space H ν + q/ 2 ( R q , R q ) [52, 57].

Original: The associated kernel is given for every z, z ′ ∈ R q by: K q ( z, z ′ ) =
Revised: The associated kernel is given for every z, z ′ ∈ R q by: K q ( z, z ′ ) =

Original: k ( ∥ z − z ′ ∥ )
Revised: k ( ∥ z − z ′ ∥ )

Original: Id q .
Revised: Id q .

Original: Note that it is important for this RKHS to depend on the ambient dimension q .
Revised: Note that it is important for this RKHS to depend on the ambient dimension q .

Original: In particular the Sobolev space H s ( R q , R q ) is a RKHS if and only if it has regularity s > q/ 2 .
Revised: In particular the Sobolev space H s ( R q , R q ) is a RKHS if and only if it has regularity s > q/ 2 .

Original: Assuming ν > 2 , µ q further admits up to 4 finite order moment implying that k is four times differentiable at 0 [28].
Revised: Assuming ν > 2 , µ q further admits up to 4 finite order moment implying that k is four times differentiable at 0 [28].

Original: Then V q satisfies Assumption 1 with some constant κ depending only on ν and given by Property 4:
Revised: Then V q satisfies Assumption 1 with some constant κ depending only on ν and given by Property 4:

Original: [Equation]
Revised: [Equation]

Original: Also, V q satisfies Assumption 2 with Λ ≤ N and λ depending a priori on ν , q and N .
Revised: Also, V q satisfies Assumption 2 with Λ ≤ N and λ depending a priori on ν , q and N .

Original: Note that with this choice of scaling for k and µ q , one recovers the Gaussian kernel k : r (cid:55)→ e − r 2 /in the limit ν → + ∞ [57].
Revised: Note that with this choice of scaling for k and µ q , one recovers the Gaussian kernel k : r (cid:55)→ e − r 2 /in the limit ν → + ∞ [57].

Original: Thereafter we consider, ν ∈
Revised: Thereafter we consider, ν ∈

Original: (2 , + ∞ ] , the case ν = + ∞ referring to the Gaussian kernel.
Revised: (2 , + ∞ ] , the case ν = + ∞ referring to the Gaussian kernel.

Original: We also assume that the data distribution is compactly supported.
Revised: We also assume that the data distribution is compactly supported.

Original: In particular there exists some r 0 ≥ 

Revised: In particular there exists some r 0 ≥

Original: so that every input data x verifies ∥ x ∥ ≤ r 0 .
Revised: so that every input data x verifies ∥ x ∥ ≤ r 0 .

Original: 5.1 Global convergence with high-dimensional lifting
Revised: 5.1 Global convergence with high-dimensional lifting

Original: We first show how Eq.
Revised: We first show how Eq.

Original: (15) can be satisfied by considering appropriate embedding matrices A and
Revised: (15) can be satisfied by considering appropriate embedding matrices A and

Original: B .
Revised: B .

Original: Doing so, the square distance between the data points, i.e. the model’s loss, is preserved whereas the conditioning of the kernel matrix can be controlled.
Revised: Doing so, the square distance between the data points, i.e. the model’s loss, is preserved whereas the conditioning of the kernel matrix can be controlled.

Original: Proposition 1.
Revised: Proposition 1.

Original: Let ν ∈
Revised: Let ν ∈

Original: (2 , + ∞ ] , let ( x i , y i )
Revised: (2 , + ∞ ] , let ( x i , y i )

Original: 1 ≤ i ≤ N ∈
Revised: 1 ≤ i ≤ N ∈

Original: ( R d × R d ′ )
Revised: ( R d × R d ′ )

Original: N be a dataset with data separation δ > 0 and let R > 0 .
Revised: N be a dataset with data separation δ > 0 and let R > 0 .

Original: There exist q ≥ 1 and matrices A ∈ R q × d , B ∈ R d ′ × q s.t.
Revised: There exist q ≥ 1 and matrices A ∈ R q × d , B ∈ R d ′ × q s.t.

Original: GD initialized at v 0 = 0 converges towards a zero-training-loss optimum in the training of RKHS-NODE.
Revised: GD initialized at v 0 = 0 converges towards a zero-training-loss optimum in the training of RKHS-NODE.

Original: In particular, Eq.
Revised: In particular, Eq.

Original: (15) holds with radius R and κ, λ, Λ associated with the RKHS V q .
Revised: (15) holds with radius R and κ, λ, Λ associated with the RKHS V q .

Original: As shown in the proof in Appendix D.1, Proposition 1 still holds for small but non-zero initialization.
Revised: As shown in the proof in Appendix D.1, Proposition 1 still holds for small but non-zero initialization.

Original: We present here two ways of obtaining matrices A and B satisfying Eq.
Revised: We present here two ways of obtaining matrices A and B satisfying Eq.

Original: Scaling Consider A
Revised: Scaling Consider A

Original: = α (Id d , 0)
Revised: = α (Id d , 0)

Original: ⊤ ∈ R ( d + d ′ )
Revised: ⊤ ∈ R ( d + d ′ )

Original: ×
Revised: ×

Original: d and B = α (0 , Id d ′ ) ∈ R d ′ ×
Revised: d and B = α (0 , Id d ′ ) ∈ R d ′ ×

Original: d + d ′ ) , for α > 0 .
Revised: d + d ′ ) , for α > 0 .

Original: We show in Appendix D.1.2 that, in this setting, the l.h.s.
Revised: We show in Appendix D.1.2 that, in this setting, the l.h.s.

Original: of Eq.
Revised: of Eq.

Original: (15) scales as O (1 /α ) and thus Theorem 2 holds for large enough α .
Revised: (15) scales as O (1 /α ) and thus Theorem 2 holds for large enough α .

Original: Moreover, observe that q = d + d ′ is independent of N and δ and such a regime can easily be implemented in practice.
Revised: Moreover, observe that q = d + d ′ is independent of N and δ and such a regime can easily be implemented in practice.

Original: However, it has been shown that, although interpolation of the training data can be achieved as a consequence of a suitable rescaling of the parameters, this “lazy regime” can also lead to bad generalization properties [15].
Revised: However, it has been shown that, although interpolation of the training data can be achieved as a consequence of a suitable rescaling of the parameters, this “lazy regime” can also lead to bad generalization properties [15].

Original: Lifting
Revised: Lifting

Original: Consider for q ≥ 1 the matrices: A q :=
Revised: Consider for q ≥ 1 the matrices: A q :=

Original: q − 1 / 4 (Id d , ..., Id d , 0)
Revised: q − 1 / 4 (Id d , ..., Id d , 0)

Original: ⊤ ∈ R q × d
Revised: ⊤ ∈ R q × d

Original: and
Revised: and

Original: B q :=
Revised: B q :=

Original: q 1 / 4 (Id d ′ , 0 ... 0)
Revised: q 1 / 4 (Id d ′ , 0 ... 0)

Original: ∈ R d ′ × q , with ⌊ q/d ⌋ copies of Id d in A q .
Revised: ∈ R d ′ × q , with ⌊ q/d ⌋ copies of Id d in A q .

Original: This choice is motivated by the intention for these matrices to produce a high-dimensional lifting, which has been shown to improve on the expressivity of ResNets [18].
Revised: This choice is motivated by the intention for these matrices to produce a high-dimensional lifting, which has been shown to improve on the expressivity of ResNets [18].

Original: We then show in Appendix D.1.1 that Eq.
Revised: We then show in Appendix D.1.1 that Eq.

Original: (15) can be satisfied for q = Ω( N 4 + δ − 4 log( N ) 4 ) .
Revised: (15) can be satisfied for q = Ω( N 4 + δ − 4 log( N ) 4 ) .

Original: We do not expect our condition on q to be optimal as we observe in experiments (see Appendix A) that a regime of linear convergence can be obtained for q ≪ N 4 + δ − 4 log( N ) 4 .
Revised: We do not expect our condition on q to be optimal as we observe in experiments (see Appendix A) that a regime of linear convergence can be obtained for q ≪ N 4 + δ − 4 log( N ) 4 .

Original: However, we observe that increasing q does improve on the convergence and generalization properties of our model (Fig. 2).
Revised: However, we observe that increasing q does improve on the convergence and generalization properties of our model (Fig. 2).

Original: 5.2 Global convergence with finite width
Revised: 5.2 Global convergence with finite width

Original: In the preceding we showed that, in the case of an RKHS defined by a Matérn kernel, convergence of GD can be ensured for well-chosen matrices A and B .
Revised: In the preceding we showed that, in the case of an RKHS defined by a Matérn kernel, convergence of GD can be ensured for well-chosen matrices A and B .

Original: However, for practical implementations, the form of the residual in Eq.
Revised: However, for practical implementations, the form of the residual in Eq.

Original: (3) forces us to consider RKHSs defined by feature maps.
Revised: (3) forces us to consider RKHSs defined by feature maps.

Original: A way to overcome this difficulty and to benefit from the properties of a wide range of kernel functions is to consider an approximation by Random Fourier Features (RFF) [48, 49].
Revised: A way to overcome this difficulty and to benefit from the properties of a wide range of kernel functions is to consider an approximation by Random Fourier Features (RFF) [48, 49].

Original: More precisely, given q ≥ 1 , recall the definition of the Matérn kernel k in terms of its frequency distribution µ q over R q in Eq.
Revised: More precisely, given q ≥ 1 , recall the definition of the Matérn kernel k in terms of its frequency distribution µ q over R q in Eq.

Original: (18) and for any sampling ω 1 , ..., ω r iid ∼ µ q of size r , consider the feature map:
Revised: (18) and for any sampling ω 1 , ..., ω r iid ∼ µ q of size r , consider the feature map:

Original: [Equation]
Revised: [Equation]

Original: In other words, considering the complex activation σ : t (cid:55)→ e ıt applied component-wise and
Revised: In other words, considering the complex activation σ : t (cid:55)→ e ıt applied component-wise and

Original: U :=
Revised: U :=

Original: ( ω 1 | . . .
Revised: ( ω 1 | . . .

Original: | ω r ) ∈ R q × r the feature matrix, we have φ ( z ) =
Revised: | ω r ) ∈ R q × r the feature matrix, we have φ ( z ) =

Original: r − 1 / 2 σ ( U ⊤ z ) .
Revised: r − 1 / 2 σ ( U ⊤ z ) .

Original: Recall that such a feature map defines a structure of RKHS on ˆ V q :=
Revised: Recall that such a feature map defines a structure of RKHS on ˆ V q :=

Original: { Wφ ( . )
Revised: { Wφ ( . )

Original: | W ∈ R q × r } .
Revised: | W ∈ R q × r } .

Original: Such a ˆ V q can be viewed as a finite-dimensional approximation of the universal RKHS V q as it is associated with the kernel function
Revised: Such a ˆ V q can be viewed as a finite-dimensional approximation of the universal RKHS V q as it is associated with the kernel function

Original: ˆ K q ( z, z ′ ) :
Revised: ˆ K q ( z, z ′ ) :

Original: =
Revised: =

Original: ˆ k
Revised: ˆ k

Original: ( z, z ′ )
Revised: ( z, z ′ )

Original: Id q , with:
Revised: Id q , with:

Original: [Equation]
Revised: [Equation]

Original: Given any q ≥ 1 , we show that, with high probability over the choice of features, ˆ V q recovers the properties of admissibility and universality of V q as soon as r is sufficiently high w.r.t.
Revised: Given any q ≥ 1 , we show that, with high probability over the choice of features, ˆ V q recovers the properties of admissibility and universality of V q as soon as r is sufficiently high w.r.t.

Original: q and N .
Revised: q and N .

Original: The following is a particular case of Proposition 5 in Appendix D.2.
Revised: The following is a particular case of Proposition 5 in Appendix D.2.

Original: Proposition 2.
Revised: Proposition 2.

Original: Consider any q, N ≥ 2 and any ϵ, τ, R > 0 .
Revised: Consider any q, N ≥ 2 and any ϵ, τ, R > 0 .

Original: Assume ν > 4 .
Revised: Assume ν > 4 .

Original: (i) For r ≥ Ω( τq 8 ) , with probability greater than 1 − τ − 1 , ˆ V q satisfies Assumption 1 with ˆ κ ≤ κ +1 .
Revised: (i) For r ≥ Ω( τq 8 ) , with probability greater than 1 − τ − 1 , ˆ V q satisfies Assumption 1 with ˆ κ ≤ κ +1 .

Original: (ii) For r ≥ Ω( ϵ − 2 N 2 ( q log( ∥ A ∥ 2 r 0 + R ) + τ )) , with probability greater than 1 − e − τ , for any v ∈ L 2 ( ˆ V q ) s.t.
Revised: (ii) For r ≥ Ω( ϵ − 2 N 2 ( q log( ∥ A ∥ 2 r 0 + R ) + τ )) , with probability greater than 1 − e − τ , for any v ∈ L 2 ( ˆ V q ) s.t.

Original: ∥ v ∥ L 2 ≤ R and any time t ∈ [0 , 1] : λ min (ˆ K (( z it ) i )) ≥ λ min ( K (( z it ) i ))
Revised: ∥ v ∥ L 2 ≤ R and any time t ∈ [0 , 1] : λ min (ˆ K (( z it ) i )) ≥ λ min ( K (( z it ) i ))

Original: − ϵ , where ( z i )
Revised: − ϵ , where ( z i )

Original: i are the solutions to Eq.
Revised: i are the solutions to Eq.

Original: (6) and ˆ K , K are the kernel matrices of ˆ k and k respectively.
Revised: (6) and ˆ K , K are the kernel matrices of ˆ k and k respectively.

Revised: Sketch of proof for (i).

Original: Finally, combining Proposition 1 and Proposition 2, we obtain a global convergence result.
Revised: First note that for ν > 4 , µ q admits up to 8 th -order finite moments and these can be bounded uniformly in q [28]. Let φ be the feature map of Eq. Then for every z ∈ R q , ∥ φ ( z ) ∥ ≤ 1 so that for every v ∈ ˆ V q , ∥ v ∥ ∞ ≤ ∥ W ∥∥ φ ∥ ∞ ≤ ∥ v ∥ V . For the differential Dv we have for every z

Revised: [Equation]

Revised: Then, by the Bienayme-Chebyshev inequality, Dφ ( z ) ∗ Dφ ( z ) = 1 r (cid:80) rj =1 ω j ( ω j ) ⊤ converges in probability to − k ′′ (0) Id q as r → + ∞ .

Revised: Thus, for α > 0 and r sufficiently high w.r.t.

Revised: q , α and τ , ∥ Dv ∥ 2 , ∞ =

Revised: ∥ WDφ ∥ 2 , ∞ ≤ (cid:112) − k ′′ (0) + α ∥ v ∥ ˆ V q , with probability greater than 1 − τ − 1 .

Revised: The same idea applies to bound ∥ D 2 v ∥ 2 , ∞ and the result follows using that κ is given by Eq.

Revised: Finally, combining Proposition 1 and Proposition 2, we obtain a global convergence result.

Original: Theorem 3 states convergence, with high probability over a choice of features, of GD towards a zerotraining-loss optimum for infinitely deep ResNets of finite width.
Revised: Theorem 3 states convergence, with high probability over a choice of features, of GD towards a zerotraining-loss optimum for infinitely deep ResNets of finite width.

Original: Theorem 3 (Global convergence) .
Revised: Theorem 3 (Global convergence) .

Original: Assume ν > 4 and let ( x i , y i ) ∈
Revised: Assume ν > 4 and let ( x i , y i ) ∈

Original: ( R d × R d ′ )
Revised: ( R d × R d ′ )

Original: N be a compactly supported dataset with input data separation δ > 0 .
Revised: N be a compactly supported dataset with input data separation δ > 0 .

Original: There exist matrices A ∈ R q × d and B ∈ R d ′ × q s.t.
Revised: There exist matrices A ∈ R q × d and B ∈ R d ′ × q s.t.

Original: for any τ > 0 , with probability at least 1 − τ − 1 w.r.t.
Revised: for any τ > 0 , with probability at least 1 − τ − 1 w.r.t.

Original: the choices of features, GD initialized at v 0 = 0 converges towards a zero training loss optimum in the training of the RKHS-NODE model of Definition 1 with the feature map φ of Eq.
Revised: the choices of features, GD initialized at v 0 = 0 converges towards a zero training loss optimum in the training of the RKHS-NODE model of Definition 1 with the feature map φ of Eq.

Original: (20) as soon as r ≥ Ω( τ ( q 8 + qN 2 log( ∥
Revised: (20) as soon as r ≥ Ω( τ ( q 8 + qN 2 log( ∥

Original: A ∥ 2 )) .
Revised: A ∥ 2 )) .

Original: Proof.
Revised: Proof.

Original: Consider R = 1 .
Revised: Consider R = 1 .

Original: By Proposition 1, we can have A ∈ R q × d , B ∈ R d ′ ×
Revised: By Proposition 1, we can have A ∈ R q × d , B ∈ R d ′ ×

Original: q
Revised: q

Original: so that in Eq.
Revised: so that in Eq.

Original: [Equation]
Revised: [Equation]

Original: for κ , λ and Λ associated with k .
Revised: for κ , λ and Λ associated with k .

Original: Also, by the proof of Proposition 1 we can have: λ ( σ min ( A ) − 1 δ − 1 e ( κ +1) ) ≥ 1 / 2 .
Revised: Also, by the proof of Proposition 1 we can have: λ ( σ min ( A ) − 1 δ − 1 e ( κ +1) ) ≥ 1 / 2 .

Original: Taking ϵ = 1 / 4 in Proposition 2, the condition in Eq.
Revised: Taking ϵ = 1 / 4 in Proposition 2, the condition in Eq.

Original: (15) is satisfied by ˆ V q with probability greater than 1 − τ − 1 as soon as r ≥ Ω
Revised: (15) is satisfied by ˆ V q with probability greater than 1 − τ − 1 as soon as r ≥ Ω

Original: ( τq 8 + τqN 2 log(1+ ∥
Revised: ( τq 8 + τqN 2 log(1+ ∥

Original: A ∥ 2 r 0 )) .
Revised: A ∥ 2 r 0 )) .

Original:  Conclusion
Revised:  Conclusion

Original: We have identified a relevant infinite width limit (RKHS-NODE) for a particular model of ResNet.
Revised: We have identified a relevant infinite width limit (RKHS-NODE) for a particular model of ResNet.

Original: We showed that GD converges linearly when training this model and that a network’s width polynomial w.r.t.
Revised: We showed that GD converges linearly when training this model and that a network’s width polynomial w.r.t.

Original: to the size of the dataset is sufficient to maintain this property.
Revised: to the size of the dataset is sufficient to maintain this property.

Original: A natural extension of our result is to study the convergence of GD when also training the hidden layers of the residuals.
Revised: A natural extension of our result is to study the convergence of GD when also training the hidden layers of the residuals.

Original: A first step towards this general case consists in studying the corresponding mean field model where the residuals are parameterized by density distributions over the neurons [14, 41, 40, 27, 38, 21] for each residual blocks.
Revised: A first step towards this general case consists in studying the corresponding mean field model where the residuals are parameterized by density distributions over the neurons [14, 41, 40, 27, 38, 21] for each residual blocks.

Original: Interestingly, such a parametrization of the residual blocks is still linear in this measure and thus fits into our framework of linear in parameters.
Revised: Interestingly, such a parametrization of the residual blocks is still linear in this measure and thus fits into our framework of linear in parameters.

Original: However, it would require a finer mathematical analysis to obtain similar results.
Revised: However, it would require a finer mathematical analysis to obtain similar results.

Original: Potential Negative Societal Impacts.
Revised: Potential Negative Societal Impacts.

Original: Our work aims at improving the theoretical and practical understanding of deep networks and therefore we do not expect a direct negative impact.
Revised: Our work aims at improving the theoretical and practical understanding of deep networks and therefore we do not expect a direct negative impact.

Original: Acknowledgements
Revised: Acknowledgements

Original: The work of Gabriel Peyré was supported by the French government under management of Agence
Revised: The work of Gabriel Peyré was supported by the French government under management of Agence

Original: Nationale de la Recherche as part of the ”Investissements d’avenir” program, reference ANR19P3IA-0001 (PRAIRIE 3IA Institute) and by the European Research Council (ERC project NORIA).
Revised: Nationale de la Recherche as part of the ”Investissements d’avenir” program, reference ANR19P3IA-0001 (PRAIRIE 3IA Institute) and by the European Research Council (ERC project NORIA).

Original: A Numerical experiments
Revised: A Numerical experiments

Original: The goal of this section is to quantify how much (in addition to interpolating the training dataset) our model is able to generalize on the test dataset.
Revised: The goal of this section is to quantify how much (in addition to interpolating the training dataset) our model is able to generalize on the test dataset.

Original: This is also useful to compare the performances of our model with those of standard ResNet architectures (which integrate batch normalization and training of the hidden layers).
Revised: This is also useful to compare the performances of our model with those of standard ResNet architectures (which integrate batch normalization and training of the hidden layers).

Original: We implemented our model in Pytorch [45] and trained it on image datasets for classification tasks.
Revised: We implemented our model in Pytorch [45] and trained it on image datasets for classification tasks.

Original: Source code is available at https://github.com/rbarboni/FlowResNets .
Revised: Source code is available at https://github.com/rbarboni/FlowResNets .

Original: Experiments were conducted using a private infrastructure, which has a carbon efficiency of 0.05 kgCO 2 eq/kWh.
Revised: Experiments were conducted using a private infrastructure, which has a carbon efficiency of 0.05 kgCO 2 eq/kWh.

Original: A cumulative of (at most) 1000 hours of computation was performed on hardware of type Tesla V100-PCIE-16GB (TDP of 300W).
Revised: A cumulative of (at most) 1000 hours of computation was performed on hardware of type Tesla V100-PCIE-16GB (TDP of 300W).

Original: Total emissions are estimated to be 15 kgCO 2 eq (or 60km in an average car) of which 0 percents were directly offset.
Revised: Total emissions are estimated to be 15 kgCO 2 eq (or 60km in an average car) of which 0 percents were directly offset.

Original: Estimations were conducted using the MachineLearning Impact calculator presented in [30].
Revised: Estimations were conducted using the MachineLearning Impact calculator presented in [30].

Original: Computational setup for classification tasks.
Revised: Computational setup for classification tasks.

Original: In the context of classification tasks, we use a cross entropy loss in place of the least square loss of Eq.
Revised: In the context of classification tasks, we use a cross entropy loss in place of the least square loss of Eq.

Original: For a problem with K classes, the output dimension of the model is d ′ =
Revised: For a problem with K classes, the output dimension of the model is d ′ =

Original: K and targets y ∈ R K are one-hot vector encoding the target classes.
Revised: K and targets y ∈ R K are one-hot vector encoding the target classes.

Original: For a batch of N predictions ( z i ) 1 ≤ i ≤ N and targets ( y i ) 1 ≤ i ≤ N in R K the Cross Entropy loss is defined as:
Revised: For a batch of N predictions ( z i ) 1 ≤ i ≤ N and targets ( y i ) 1 ≤ i ≤ N in R K the Cross Entropy loss is defined as:

Original: [Equation]
Revised: [Equation]

Original: where ℓ is the Binary Entropy defined for one prediction z and one target y ∈ R K by:
Revised: where ℓ is the Binary Entropy defined for one prediction z and one target y ∈ R K by:

Original: [Equation]
Revised: [Equation]

Original: Then for a model F depending on the parameters W and a training batch ( x i , y i ) 1 ≤ i ≤ N we define the empirical risk:
Revised: Then for a model F depending on the parameters W and a training batch ( x i , y i ) 1 ≤ i ≤ N we define the empirical risk:

Original: [Equation]
Revised: [Equation]

Original: i , ( y i ) i ) , and train the model by Stochastic Gradient Descent (SGD) on W .
Revised: i , ( y i ) i ) , and train the model by Stochastic Gradient Descent (SGD) on W .

Original: Finally, the performance of the model is assessed by the Top-1 error rate on a test dataset.
Revised: Finally, the performance of the model is assessed by the Top-1 error rate on a test dataset.

Original: Note that, as explained in Remark 2, the result of Theorem 3 can be extended to this cross entropy loss.
Revised: Note that, as explained in Remark 2, the result of Theorem 3 can be extended to this cross entropy loss.

Original: Indeed, ℓ satisfies a functional inequality similar to the Polyak-Lojasiewicz inequality.
Revised: Indeed, ℓ satisfies a functional inequality similar to the Polyak-Lojasiewicz inequality.

Original: Assuming without loss of generality that y
Revised: Assuming without loss of generality that y

Original: = e 1 is the indicator of class 1 , one has:
Revised: = e 1 is the indicator of class 1 , one has:

Original: [Equation]
Revised: [Equation]

Original: Then by convexity of exponential, when ℓ ( z, y
Revised: Then by convexity of exponential, when ℓ ( z, y

Original: [Equation]
Revised: [Equation]

Original: ≤ 1 :
Revised: ≤ 1 :

Original: [Equation]
Revised: [Equation]

Original: Note however that Theorem 3 is only valid for full batch gradient descent.
Revised: Note however that Theorem 3 is only valid for full batch gradient descent.

Original: We leave its extension to
Revised: We leave its extension to

Original: SGD for future works.
Revised: SGD for future works.

Original: A.1 Experiments on MNIST
Revised: A.1 Experiments on MNIST

Original: We implemented the model of Definition 1 on Pytorch using the torchdiffeq package [11] and performed experiments on the MNIST dataset.
Revised: We implemented the model of Definition 1 on Pytorch using the torchdiffeq package [11] and performed experiments on the MNIST dataset.

Original: Implementation using torchdiffeq .
Revised: Implementation using torchdiffeq .

Original: The model of Definition 1 is implemented as a succession of convolutional layers.
Revised: The model of Definition 1 is implemented as a succession of convolutional layers.

Original: Given some number of layers L the trained parameters consist of convolution matrices W k ∈ R C
Revised: Given some number of layers L the trained parameters consist of convolution matrices W k ∈ R C

Original: ×
Revised: ×

Original: C
Revised: C

Original: int × 3 × 3 for k ∈ (cid:74) 0 , L (cid:75) , with C the number of channels of the input image and
Revised: int × 3 × 3 for k ∈ (cid:74) 0 , L (cid:75) , with C the number of channels of the input image and

Original: C int some number of channels for the hidden layers.
Revised: C int some number of channels for the hidden layers.

Original: The control parameter v is defined at discrete time steps { k/L } 0 ≤ k ≤ L by:
Revised: The control parameter v is defined at discrete time steps { k/L } 0 ≤ k ≤ L by:

Original: [Equation]
Revised: [Equation]

Original: where U ∈ R C int × C × 3 × 3 is a fixed and untrained convolution matrix.
Revised: where U ∈ R C int × C × 3 × 3 is a fixed and untrained convolution matrix.

Original: We refer to this setting as a
Revised: We refer to this setting as a

Original: ResNet with RKHS residuals.
Revised: ResNet with RKHS residuals.

Original: On the other hand, we refer to the setting where U is replaced at each layer by trained convolution matrices U k as ResNet with Single Hidden Layer (SHL) residuals.
Revised: On the other hand, we refer to the setting where U is replaced at each layer by trained convolution matrices U k as ResNet with Single Hidden Layer (SHL) residuals.

Original: Remark 6.
Revised: Remark 6.

Original: By analogy with the definition of RKHSs generated by random features (Eq.
Revised: By analogy with the definition of RKHSs generated by random features (Eq.

Original: (20) ), the ratio between the number of features and the dimension is here:
Revised: (20) ), the ratio between the number of features and the dimension is here:

Original: [Equation]
Revised: [Equation]

Original: For any t ∈ [0 , 1] , v t is defined by affine interpolation:
Revised: For any t ∈ [0 , 1] , v t is defined by affine interpolation:

Original: [Equation]
Revised: [Equation]

Original: with k =
Revised: with k =

Original: ⌊ tL ⌋ .
Revised: ⌊ tL ⌋ .

Original: The forward method consists in integrating the ODE of Eq.
Revised: The forward method consists in integrating the ODE of Eq.

Original: (6) with control parameter v using the torchdiffeq .
Revised: (6) with control parameter v using the torchdiffeq .

Original: odeint method [11].
Revised: odeint method [11].

Original: For some input z 0 define:
Revised: For some input z 0 define:

Original: [Equation]
Revised: [Equation]

Original: then for an image input x the model’s output is given by:
Revised: then for an image input x the model’s output is given by:

Original: [Equation]
Revised: [Equation]

Original: where A and B are small convolutional networks, fixed during the training of F .
Revised: where A and B are small convolutional networks, fixed during the training of F .

Original: These networks play the same role as the matrices A and B in Definition 1, that is they are used for the purpose of adjusting the data dimension.
Revised: These networks play the same role as the matrices A and B in Definition 1, that is they are used for the purpose of adjusting the data dimension.

Original: Hyperparameter tuning.
Revised: Hyperparameter tuning.

Original: Several choices of hyperparameters can affect the performances of the model.
Revised: Several choices of hyperparameters can affect the performances of the model.

Original: • The convolution matrix U : as detailed in Section 5, the way the weights of U are sampled determines to which RKHS V belongs the control parameter v .
Revised: • The convolution matrix U : as detailed in Section 5, the way the weights of U are sampled determines to which RKHS V belongs the control parameter v .

Original: For the sake of simplicity we choose to sample the coefficients of U as i.i.d.
Revised: For the sake of simplicity we choose to sample the coefficients of U as i.i.d.

Original: Gaussians.
Revised: Gaussians.

Original: • The initialization of ( W k ) : the weights of the convolution matrices W k are initialized to 0 .
Revised: • The initialization of ( W k ) : the weights of the convolution matrices W k are initialized to 0 .

Original: For an input image x the output is given at initialization by F (0 , x ) =
Revised: For an input image x the output is given at initialization by F (0 , x ) =

Original: B ( A ( x )) .
Revised: B ( A ( x )) .

Original: • The integration method: torchdiffeq .
Revised: • The integration method: torchdiffeq .

Original: odeint allows the user to choose an integration method.
Revised: odeint allows the user to choose an integration method.

Original: We observed an explicit midpoint method to offer a good trade-off between performance and numerical stability w.r.t.
Revised: We observed an explicit midpoint method to offer a good trade-off between performance and numerical stability w.r.t.

Original: other fixed-steps methods such as explicit Euler or RK4 .
Revised: other fixed-steps methods such as explicit Euler or RK4 .

Original: • The number of layers L : we tested our model for L ∈ { 5 , 10 , 20 } .
Revised: • The number of layers L : we tested our model for L ∈ { 5 , 10 , 20 } .

Original: This parameter controls the total number of parameters of the model.
Revised: This parameter controls the total number of parameters of the model.

Original: • The networks A and B : their choice defines the dimension of space in which the forward ODE Eq.
Revised: • The networks A and B : their choice defines the dimension of space in which the forward ODE Eq.

Original: (6) is integrated, which is expected to have an important impact on the performances of the model (c.f. Section 5).
Revised: (6) is integrated, which is expected to have an important impact on the performances of the model (c.f. Section 5).

Original: Moreover, as the parameters ( W k ) are initialized at 0 , the performances of the model before training are those of the concatenation B ◦ A .
Revised: Moreover, as the parameters ( W k ) are initialized at 0 , the performances of the model before training are those of the concatenation B ◦ A .

Original: Without training, the classification error of B ◦ A is of 90% while with enough training, it can be as good as 2% .
Revised: Without training, the classification error of B ◦ A is of 90% while with enough training, it can be as good as 2% .

Original: We tested our model with different levels of training of B ◦ A .
Revised: We tested our model with different levels of training of B ◦ A .

Original: Results.
Revised: Results.

Original: Figure 1 shows the evolution of the performances of RKHS-NODEs while trained on the
Revised: Figure 1 shows the evolution of the performances of RKHS-NODEs while trained on the

Original: MNIST dataset.
Revised: MNIST dataset.

Original: The decay of the Empirical Risk is directly related to the decay of the classification error.
Revised: The decay of the Empirical Risk is directly related to the decay of the classification error.

Original: Without pretraining A and B , our model already achieves up to 98% accuracy on the test set.
Revised: Without pretraining A and B , our model already achieves up to 98% accuracy on the test set.

Original: When A and B are pretrained RKHS-NODE still improves on the starting accuracy: in this setting more than 99% accuracy is reached.
Revised: When A and B are pretrained RKHS-NODE still improves on the starting accuracy: in this setting more than 99% accuracy is reached.

Original: Most importantly, Fig.
Revised: Most importantly, Fig.

Original: 1 shows that not training the hidden layers inside residual blocks does not significantly hinders the performances in classification.
Revised: 1 shows that not training the hidden layers inside residual blocks does not significantly hinders the performances in classification.

Original: Indeed, comparing the performances of ResNets with RKHS residuals and SHL residuals one observes a 1% accuracy drop when training RKHS-NODE from scratch (Fig. 1a) and 0 .
Revised: Indeed, comparing the performances of ResNets with RKHS residuals and SHL residuals one observes a 1% accuracy drop when training RKHS-NODE from scratch (Fig. 1a) and 0 .

Original: 5% accuracy when networks A and B are pretrained (Fig. 1b).
Revised: 5% accuracy when networks A and B are pretrained (Fig. 1b).

Original: Finally we showcase the relevance of the analysis of Section 5 by training our model with a varying number of input channels in Fig.
Revised: Finally we showcase the relevance of the analysis of Section 5 by training our model with a varying number of input channels in Fig.

Original: We observe a significant drop in convergence of the empirical risk with 4 channels compared with 8 and 32 channels.
Revised: We observe a significant drop in convergence of the empirical risk with 4 channels compared with 8 and 32 channels.

Original: Non-convergence of the empirical risk also implies poorer performances in generalization.
Revised: Non-convergence of the empirical risk also implies poorer performances in generalization.

Original: Such results are coherent with the convergence condition of Eq.
Revised: Such results are coherent with the convergence condition of Eq.

Original: (15): augmenting the data dimension allows to have global convergence when the loss at initialization is too high.
Revised: (15): augmenting the data dimension allows to have global convergence when the loss at initialization is too high.

Original: A.2 Experiments on CIFAR 
Revised: A.2 Experiments on CIFAR

Original: We performed experiments on the CIFAR10 dataset, using an architecture inspired from
Revised: We performed experiments on the CIFAR10 dataset, using an architecture inspired from

Original: ResNet18 [25].
Revised: ResNet18 [25].

Original: Implementation.
Revised: Implementation.

Original: Our architecture relies on the ResNet18 architecture [25] but residual blocks are changed and simplified (by removing the final non-linearity and the batch-normalization) to match the definition of RKHS-NODE (Definition 1).
Revised: Our architecture relies on the ResNet18 architecture [25] but residual blocks are changed and simplified (by removing the final non-linearity and the batch-normalization) to match the definition of RKHS-NODE (Definition 1).

Original: Each residual block consists in the composition of a convolution U , a ReLU non-linearity and a convolution W .
Revised: Each residual block consists in the composition of a convolution U , a ReLU non-linearity and a convolution W .

Original: More precisely, for an input image x , the output of the k th layer reads:
Revised: More precisely, for an input image x , the output of the k th layer reads:

Original: [Equation]
Revised: [Equation]

Original: where U k ∈ R C int × C × 3 × 3 , W k ∈ R C × C int × 3 × 3 are convolution matrices, C is the number of channels of the input image and C int is the number of channels of the hidden layer.
Revised: where U k ∈ R C int × C × 3 × 3 , W k ∈ R C × C int × 3 × 3 are convolution matrices, C is the number of channels of the input image and C int is the number of channels of the hidden layer.

Original: When both convolutions W k and U k are trained, we refer to these residuals as Single Hidden Layer (SHL) residuals .
Revised: When both convolutions W k and U k are trained, we refer to these residuals as Single Hidden Layer (SHL) residuals .

Original: In the framework of RKHS-NODE, all convolutions U k are fixed and set to the same convolution U .
Revised: In the framework of RKHS-NODE, all convolutions U k are fixed and set to the same convolution U .

Original: We refer to it as RKHS residuals .
Revised: We refer to it as RKHS residuals .

Original: Finally, ResNet18 consists of 4 blocks each containing 2 residual layers.
Revised: Finally, ResNet18 consists of 4 blocks each containing 2 residual layers.

Original: We keep 2 of our residuals in the first, second and fourth block but stack an arbitrary number D of residual layers in the third block.
Revised: We keep 2 of our residuals in the first, second and fourth block but stack an arbitrary number D of residual layers in the third block.

Original: Thereby we refer to this third block as the NODE block, which performs the integration of Eq.
Revised: Thereby we refer to this third block as the NODE block, which performs the integration of Eq.

Original: Note that compared to the residuals in the original ResNet18 architecture, batch-normalization at input and output of the residuals as well as ReLU non-linearities are removed.
Revised: Note that compared to the residuals in the original ResNet18 architecture, batch-normalization at input and output of the residuals as well as ReLU non-linearities are removed.

Original: Moreover, in order to reproduce the framework of Random Fourier Features (Eq.
Revised: Moreover, in order to reproduce the framework of Random Fourier Features (Eq.

Original: (20)), the weights of U are sampled as i.i.d.
Revised: (20)), the weights of U are sampled as i.i.d.

Original: gaussians and rescaled by a C − 1 / 2 int factor.
Revised: gaussians and rescaled by a C − 1 / 2 int factor.

Original: Finally, the weights of the convolutions W k are initialized at 0 .
Revised: Finally, the weights of the convolutions W k are initialized at 0 .

Original: Such an initialization corresponds in many ways to the one proposed in [61].
Revised: Such an initialization corresponds in many ways to the one proposed in [61].

Original: Results.
Revised: Results.

Original: Fig.
Revised: Fig.

Original: 3 reports the training of RKHS-NODE on the CIFAR10 dataset.
Revised: 3 reports the training of RKHS-NODE on the CIFAR10 dataset.

Original: Figure 3a shows the training of RKHS-NODE (RKHS residuals) and is to be compared with Fig.
Revised: Figure 3a shows the training of RKHS-NODE (RKHS residuals) and is to be compared with Fig.

Original: 3b which shows the training of the same model but with trained hidden layers (SHL residuals).
Revised: 3b which shows the training of the same model but with trained hidden layers (SHL residuals).

Original: Our experiments show that similar performances can be achieved: both ResNets achieve up to 88% accuracy on the test dataset.
Revised: Our experiments show that similar performances can be achieved: both ResNets achieve up to 88% accuracy on the test dataset.

Original: As a comparison, the ResNet18 original architecture can be trained to achieve up to 94% accuracy.
Revised: As a comparison, the ResNet18 original architecture can be trained to achieve up to 94% accuracy.

Original: Finally, Fig. 3 also compares the performances of the model depending on the number of layers inside the NODE block.
Revised: Finally, Fig. 3 also compares the performances of the model depending on the number of layers inside the NODE block.

Original: One observes significantly different behavior when there is no NODE (layer) and one there is ( 10 and 20 layers): more layers are related to better performances both on the train dataset and on the test dataset and both when hidden layers are trained or not.
Revised: One observes significantly different behavior when there is no NODE (layer) and one there is ( 10 and 20 layers): more layers are related to better performances both on the train dataset and on the test dataset and both when hidden layers are trained or not.

Original: However, one sees that the improvement related to adding more layers is limited: performances with 10 and 20 layers are very similar and a NODE block with 1 layers already achieves 82% accuracy RKHS residuals and 84% accuracy with SHL residuals.
Revised: However, one sees that the improvement related to adding more layers is limited: performances with 10 and 20 layers are very similar and a NODE block with 1 layers already achieves 82% accuracy RKHS residuals and 84% accuracy with SHL residuals.

Original: B Proofs of Section
Revised: B Proofs of Section

Original: We give a proof of Theorem 1.
Revised: We give a proof of Theorem 1.

Original: This essentially follows the proof given in [36].
Revised: This essentially follows the proof given in [36].

Original: Proof of Theorem 1.
Revised: Proof of Theorem 1.

Original: Assume the loss L satisfies Definition 2 with M and m and that Eq.
Revised: Assume the loss L satisfies Definition 2 with M and m and that Eq.

Original: (9) is satisfied at initialization v 0 ∈ R m .
Revised: (9) is satisfied at initialization v 0 ∈ R m .

Original: The proof proceeds by induction over the gradient step k
Revised: The proof proceeds by induction over the gradient step k

Original: Assume the convergence rate and the regularization bound of Eq.
Revised: Assume the convergence rate and the regularization bound of Eq.

Original: (10) are satisfied for every l ≤ k .
Revised: (10) are satisfied for every l ≤ k .

Original: Then at step k + 1 :
Revised: Then at step k + 1 :

Original: [Equation]
Revised: [Equation]

Original: 10 −10 −

Revised: 10 −10 −

Original: E m p
Revised: E m p

Original: i r i c a l R
Revised: i r i c a l R

Original: i s k
Revised: i s k

Original: L     10 −
Revised: L     10 −

Original: C l a ss i ﬁ c a t i o n e rr o r 5 layer
Revised: C l a ss i ﬁ c a t i o n e rr o r 5 layer

Original: 10 layer 20 layer
Revised: 10 layer 20 layer

Original: [Figure]
Revised: [Figure]

Original: 10 −10 −10 −
Revised: 10 −10 −10 −

Original: E m p
Revised: E m p

Original: i r i c a l R
Revised: i r i c a l R

Original: i s k
Revised: i s k

Original: L     10 −2 × 10 −3 × 10 −
Revised: L     10 −2 × 10 −3 × 10 −

Original: C l a ss i ﬁ c a t i o n e rr o r 5 layer
Revised: C l a ss i ﬁ c a t i o n e rr o r 5 layer

Original: 10 layer 20 layer
Revised: 10 layer 20 layer

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: E m p
Revised: E m p

Original: i r i c a l R
Revised: i r i c a l R

Original: i s k
Revised: i s k

Original: L     10 −
Revised: L     10 −

Original: C l a ss i ﬁ c a t i o n e r
Revised: C l a ss i ﬁ c a t i o n e r

Original: r
Revised: r

Original: o r
Revised: o r

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: m p
Revised: m p

Original: i r i c a l R
Revised: i r i c a l R

Original: i s k L
Revised: i s k L

Original: [Figure]
Revised: [Figure]

Original: C l a ss i ﬁ c a t i o n e rr o r 1 layer
Revised: C l a ss i ﬁ c a t i o n e rr o r 1 layer

Original: 10 layer 20 layer
Revised: 10 layer 20 layer

Original: [Figure]
Revised: [Figure]

Original: E m p
Revised: E m p

Original: i r i c a l R
Revised: i r i c a l R

Original: i s k L
Revised: i s k L

Original: [Figure]
Revised: [Figure]

Original: Using the induction hypothesis and setting µ = m ( ∥ v 0 ∥ + R ) we have:
Revised: Using the induction hypothesis and setting µ = m ( ∥ v 0 ∥ + R ) we have:

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is Eq.
Revised: where the last inequality is Eq.

Original: We thus recovered the regularization bound of Eq.
Revised: We thus recovered the regularization bound of Eq.

Original: (10) at step k + 1 .
Revised: (10) at step k + 1 .

Original: Moreover, because v k +1 is located in B ( v 0 , R ) we have thanks to the smoothness assumption:
Revised: Moreover, because v k +1 is located in B ( v 0 , R ) we have thanks to the smoothness assumption:

Original: [Equation]
Revised: [Equation]

Original: because η ≤ β − 1 .
Revised: because η ≤ β − 1 .

Original: Thus using the lower bound in the PL inequality Eq.
Revised: Thus using the lower bound in the PL inequality Eq.

Original: [Equation]
Revised: [Equation]

Original: which gives the convergence rate of Eq.
Revised: which gives the convergence rate of Eq.

Original: (10) at step k + 1 by induction on k .
Revised: (10) at step k + 1 by induction on k .

Original: C Proofs of SectionC.1 About the definition of RKHS-NODE
Revised: C Proofs of SectionC.1 About the definition of RKHS-NODE

Original: Before deriving proofs for the properties of our RKHS-NODE model, it is interesting to study carefully the well-posedness of Definition 1.
Revised: Before deriving proofs for the properties of our RKHS-NODE model, it is interesting to study carefully the well-posedness of Definition 1.

Original: Indeed, because the control parameter v is only integrable in time and not continuous, the Cauchy-Lipschitz theorem does not ensure that there exist solutions to Eq.
Revised: Indeed, because the control parameter v is only integrable in time and not continuous, the Cauchy-Lipschitz theorem does not ensure that there exist solutions to Eq.

Original: Instead we rely on a weaker notion of solution and use a result from Carathéodory (Section I.5 in [23]).
Revised: Instead we rely on a weaker notion of solution and use a result from Carathéodory (Section I.5 in [23]).

Original: Proposition 3.
Revised: Proposition 3.

Original: Let V be some RKHS satisfying Assumption 1 and v ∈ L 2 ([0 , 1] , V ) be some control parameter.
Revised: Let V be some RKHS satisfying Assumption 1 and v ∈ L 2 ([0 , 1] , V ) be some control parameter.

Original: Then for every x ∈ R
Revised: Then for every x ∈ R

Original: d there exists a unique solution z of Eq.
Revised: d there exists a unique solution z of Eq.

Original: (6) in the weak sense of absolutely continuous functions.
Revised: (6) in the weak sense of absolutely continuous functions.

Original: More precisely there exists a unique z ∈ H 1 ([0 , 1] , R q ) such that for every t ∈ [0 , 1] :
Revised: More precisely there exists a unique z ∈ H 1 ([0 , 1] , R q ) such that for every t ∈ [0 , 1] :

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The map ( t, z ) ∈ [0 , 1] × R q (cid:55)→ v t ( z ) is measurable and by Assumption 1 we have for every t ∈ [0 , 1] and every z ∈ R q :
Revised: The map ( t, z ) ∈ [0 , 1] × R q (cid:55)→ v t ( z ) is measurable and by Assumption 1 we have for every t ∈ [0 , 1] and every z ∈ R q :

Original: [Equation]
Revised: [Equation]

Original: whose upper-bound is integrable w.r.t.
Revised: whose upper-bound is integrable w.r.t.

Original: t ∈ [0 , 1] .
Revised: t ∈ [0 , 1] .

Original: Then, applying Theorem 5.1 of [23] gives a unique absolutely continuous solution z of Eq.
Revised: Then, applying Theorem 5.1 of [23] gives a unique absolutely continuous solution z of Eq.

Original: Applying Assumption 1 once again, we have that ˙
Revised: Applying Assumption 1 once again, we have that ˙

Original: z is square integrable and thus z is in H 1 .
Revised: z is square integrable and thus z is in H 1 .

Original: In the paper, every equality implying derivatives has to be understood in the sense of weak derivatives of H 1 functions.
Revised: In the paper, every equality implying derivatives has to be understood in the sense of weak derivatives of H 1 functions.

Original: In particular, this notion allows to perform integration by parts, which is used in the following proof of Property 1.
Revised: In particular, this notion allows to perform integration by parts, which is used in the following proof of Property 1.

Original: Proof Property 1.
Revised: Proof Property 1.

Original: Consider the optimization problem of minimizing the empirical risk of Eq.
Revised: Consider the optimization problem of minimizing the empirical risk of Eq.

Original: (7) with F the RKHS-NODE model of Definition 1 and a dataset ( x i , y i ) 1 ≤ i ≤ N ∈
Revised: (7) with F the RKHS-NODE model of Definition 1 and a dataset ( x i , y i ) 1 ≤ i ≤ N ∈

Original: ( R d × R d ′ ) N .
Revised: ( R d × R d ′ ) N .

Original: Introducing for every index i ∈ (cid:74) 1 , N (cid:75) the variables z i ∈ H 1 ([0 , 1] , R q ) solutions of Eq.
Revised: Introducing for every index i ∈ (cid:74) 1 , N (cid:75) the variables z i ∈ H 1 ([0 , 1] , R q ) solutions of Eq.

Original: (6), this can be viewed as an optimisation problem over (( z i )
Revised: (6), this can be viewed as an optimisation problem over (( z i )

Original: i , v ) under the constraint that Eq.
Revised: i , v ) under the constraint that Eq.

Original: (6) is satisfied:
Revised: (6) is satisfied:

Original: [Equation]
Revised: [Equation]

Original: with ∀ i ∈ (cid:74)
Revised: with ∀ i ∈ (cid:74)

Original: [Equation]
Revised: [Equation]

Original: Introducing the adjoint variables ( p i ) i ∈ H 1 ( R q ) N , the Lagrangian of the optimization problem is defined as:
Revised: Introducing the adjoint variables ( p i ) i ∈ H 1 ( R q ) N , the Lagrangian of the optimization problem is defined as:

Original: [Equation]
Revised: [Equation]

Original: where the second equality is established by integration by parts.
Revised: where the second equality is established by integration by parts.

Original: Therefore, the condition for optimality over z i is equivalent to Eq.
Revised: Therefore, the condition for optimality over z i is equivalent to Eq.

Original: For every index i :
Revised: For every index i :

Original: [Equation]
Revised: [Equation]

Original: which has to be understand in the sense
Revised: which has to be understand in the sense

Original: The gradient of L is obtained by differentiating over the v variable.
Revised: The gradient of L is obtained by differentiating over the v variable.

Original: Denoting δ pz the linear form v (cid:55)→ ⟨ v ( z ) , p ⟩ , we have:
Revised: Denoting δ pz the linear form v (cid:55)→ ⟨ v ( z ) , p ⟩ , we have:

Original: [Equation]
Revised: [Equation]

Original: with K the kernel function of the RKHS V and K ∗ : V ∗ → V the associated isometry 1 .
Revised: with K the kernel function of the RKHS V and K ∗ : V ∗ → V the associated isometry 1 .

Original: C.2 Proof of Property
Revised: C.2 Proof of Property

Original: We prove here that for any given dataset ( x i , y i ) 1 ≤ i ≤ N ∈
Revised: We prove here that for any given dataset ( x i , y i ) 1 ≤ i ≤ N ∈

Original: ( R d × R d ′ ) N , the empirical risk L associated with the RKHS-NODE model satisfies a (local) Polyak-Lojasiewicz property.
Revised: ( R d × R d ′ ) N , the empirical risk L associated with the RKHS-NODE model satisfies a (local) Polyak-Lojasiewicz property.

Original: As stated in
Revised: As stated in

Original: Property 2.
Revised: Property 2.

Original: The proof uses Assumption 1 to derive estimates on the solutions of Eq.
Revised: The proof uses Assumption 1 to derive estimates on the solutions of Eq.

Original: (6) and Eq.
Revised: (6) and Eq.

Original: (11), which we give in the following lemma:
Revised: (11), which we give in the following lemma:

Original: Lemma 1.
Revised: Lemma 1.

Original: Let V satisfy Assumption 1 with constant κ and let v ∈ L 2 ([0 , 1] , V ) be some control parameter.
Revised: Let V satisfy Assumption 1 with constant κ and let v ∈ L 2 ([0 , 1] , V ) be some control parameter.

Original: (i) Let ( z i ) 1 ≤ i ≤ N be the solutions of Eq.
Revised: (i) Let ( z i ) 1 ≤ i ≤ N be the solutions of Eq.

Original: (6) for some data inputs ( x i ) 1 ≤ i ≤ N ∈
Revised: (6) for some data inputs ( x i ) 1 ≤ i ≤ N ∈

Original: ( R d ) N .
Revised: ( R d ) N .

Original: Then for every indices i, j ∈ (cid:74) 1 , N (cid:75) and every time t ∈ [0 , 1] :
Revised: Then for every indices i, j ∈ (cid:74) 1 , N (cid:75) and every time t ∈ [0 , 1] :

Original: [Equation]
Revised: [Equation]

Original: (22) (ii) Let ( p i ) 1 ≤ i ≤ N be the solutions of Eq.
Revised: (22) (ii) Let ( p i ) 1 ≤ i ≤ N be the solutions of Eq.

Original: (11) associated with ( z i ) 1 ≤ i ≤ N with objective outputs ( y i ) 1 ≤ i ≤ N ∈
Revised: (11) associated with ( z i ) 1 ≤ i ≤ N with objective outputs ( y i ) 1 ≤ i ≤ N ∈

Original: ( R d ′ ) N .
Revised: ( R d ′ ) N .

Original: Then for every i ∈ (cid:74) 1 , N (cid:75) and every time t ∈ [0 , 1] :
Revised: Then for every i ∈ (cid:74) 1 , N (cid:75) and every time t ∈ [0 , 1] :

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma 1.
Revised: Proof of Lemma 1.

Original: Proof of (i) Let i, j ∈ (cid:74) 1 , N (cid:75) .
Revised: Proof of (i) Let i, j ∈ (cid:74) 1 , N (cid:75) .

Original: Assume by contradiction that for some time t ∈ [0 , 1] we have:
Revised: Assume by contradiction that for some time t ∈ [0 , 1] we have:

Original: [Equation]
Revised: [Equation]

Original: Then because z i and z j are absolutely continuous, ∥
Revised: Then because z i and z j are absolutely continuous, ∥

Original: z i − z j ∥ 2 is absolutely continuous and for any time s ∈ [0 , 1] :
Revised: z i − z j ∥ 2 is absolutely continuous and for any time s ∈ [0 , 1] :

Original: [Equation]
Revised: [Equation]

Original: where the inequality follows from ∥ Dv r ∥ 2 , ∞ ≤ κ ∥ v r ∥ V .
Revised: where the inequality follows from ∥ Dv r ∥ 2 , ∞ ≤ κ ∥ v r ∥ V .

Original: Applying Grönwall’s lemma, we have:
Revised: Applying Grönwall’s lemma, we have:

Original: [Equation]
Revised: [Equation]

Original: and by setting s = 0 :
Revised: and by setting s = 0 :

Original: [Equation]
Revised: [Equation]

Original: which is a contradiction.
Revised: which is a contradiction.

Original: Therefore for any time t ∈ [0 , 1] :
Revised: Therefore for any time t ∈ [0 , 1] :

Original: [Equation]
Revised: [Equation]

Original: and the result follows by considering the initial condition z
Revised: and the result follows by considering the initial condition z

Original: i 

Revised: i

Original: = Ax i .
Revised: = Ax i .

Original: Proof of (ii) Let i ∈ (cid:74) 1 , N (cid:75) be any index and let p i be the solution of Eq.
Revised: Proof of (ii) Let i ∈ (cid:74) 1 , N (cid:75) be any index and let p i be the solution of Eq.

Original: (11) with initial condition p i 1 = − 1 N B ⊤
Revised: (11) with initial condition p i 1 = − 1 N B ⊤

Original: Bz i 1 − y i ) .
Revised: Bz i 1 − y i ) .

Original: Then because p i is absolutely continuous, ∥ p i ∥ is absolutely continuous and for any time t ≤ s ∈ [0 , 1] :
Revised: Then because p i is absolutely continuous, ∥ p i ∥ is absolutely continuous and for any time t ≤ s ∈ [0 , 1] :

Original: [Equation]
Revised: [Equation]

Original: so that using Assumption 1 we have:
Revised: so that using Assumption 1 we have:

Original: [Equation]
Revised: [Equation]

Original: Using Grönwall’s lemma in the first inequality and setting s = 

Revised: Using Grönwall’s lemma in the first inequality and setting s =

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: and proceeding by contradiction (such as in (i))
Revised: and proceeding by contradiction (such as in (i))

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: The result follows by considering the initial condition on p i 1 .
Revised: The result follows by considering the initial condition on p i 1 .

Original: Provided those estimates on z i and p i , it remains to use Assumption 2 in order to conclude.
Revised: Provided those estimates on z i and p i , it remains to use Assumption 2 in order to conclude.

Original: Proof of Property 2.
Revised: Proof of Property 2.

Original: Let v ∈ L 2 ([0 , 1] , V ) and consider the form of the gradient of L given by Property 1 with ( z i ) 1 ≤ i ≤ N the solutions of Eq.
Revised: Let v ∈ L 2 ([0 , 1] , V ) and consider the form of the gradient of L given by Property 1 with ( z i ) 1 ≤ i ≤ N the solutions of Eq.

Original: (6) and ( p i ) 1 ≤ i ≤ N the solutions of Eq.
Revised: (6) and ( p i ) 1 ≤ i ≤ N the solutions of Eq.

Original: Let t ∈ [0 , 1] , then by definition of the norm in RKHSs:
Revised: Let t ∈ [0 , 1] , then by definition of the norm in RKHSs:

Original: [Equation]
Revised: [Equation]

Original: where we recall that K is the kernel associated with V .
Revised: where we recall that K is the kernel associated with V .

Original: Noting p :
Revised: Noting p :

Original: = ( p it )
Revised: = ( p it )

Original: ∈ R Nq , the vector of the stacked ( p it ) 1 ≤ i ≤ N , and K the kernel matrix associated with the family of points ( z it )
Revised: ∈ R Nq , the vector of the stacked ( p it ) 1 ≤ i ≤ N , and K the kernel matrix associated with the family of points ( z it )

Original: i , we have:
Revised: i , we have:

Original: [Equation]
Revised: [Equation]

Original: Then by Assumption
Revised: Then by Assumption

Original: [Equation]
Revised: [Equation]

Original: there exists a non-increasing function λ and a constant Λ such that:
Revised: there exists a non-increasing function λ and a constant Λ such that:

Original: [Equation]
Revised: [Equation]

Original: Using (i) in Lemma 1 we have:
Revised: Using (i) in Lemma 1 we have:

Original: [Equation]
Revised: [Equation]

Original: where δ : =
Revised: where δ : =

Original: min 1 ≤ i,j ≤ N ∥ x
Revised: min 1 ≤ i,j ≤ N ∥ x

Original: i − x j ∥ is the data separation.
Revised: i − x j ∥ is the data separation.

Original: Finally the result follows by using (ii).
Revised: Finally the result follows by using (ii).

Original: More precisely:
Revised: More precisely:

Original: [Equation]
Revised: [Equation]

Original: and in the same manner:
Revised: and in the same manner:

Original: [Equation]
Revised: [Equation]

Original: C.3 Proof of Theorem
Revised: C.3 Proof of Theorem

Original: Theorem 2 is a direct consequence of Property 2.
Revised: Theorem 2 is a direct consequence of Property 2.

Original: In order to apply Theorem 1, it suffices to show that L satisfies some smoothness assumption as defined in Definition 3:
Revised: In order to apply Theorem 1, it suffices to show that L satisfies some smoothness assumption as defined in Definition 3:

Original: Property 3 (Smoothness of L ) .
Revised: Property 3 (Smoothness of L ) .

Original: Let V be some RKHS satisfying Assumption 1.
Revised: Let V be some RKHS satisfying Assumption 1.

Original: Let L be the empirical risk defined on L 2 ([0 , 1] , V ) and associated with the RKHS-NODE model.
Revised: Let L be the empirical risk defined on L 2 ([0 , 1] , V ) and associated with the RKHS-NODE model.

Original: Then there exists a continuous function C : R + → R ∗ + such that for every R ≥ 0 and every v, ¯ v ∈ L 2 ([0 , 1] , V ) with ∥ v ∥ L 2 , ∥ ¯ v ∥ L 2 ≤ R :
Revised: Then there exists a continuous function C : R + → R ∗ + such that for every R ≥ 0 and every v, ¯ v ∈ L 2 ([0 , 1] , V ) with ∥ v ∥ L 2 , ∥ ¯ v ∥ L 2 ≤ R :

Original: [Equation]
Revised: [Equation]

Original: We note κ the constant associated with Assumption 1.
Revised: We note κ the constant associated with Assumption 1.

Original: The proof of Property 3 relies on the following lemma:
Revised: The proof of Property 3 relies on the following lemma:

Original: Lemma 2.
Revised: Lemma 2.

Original: Let v, ¯ v ∈ L 2 ([0 , 1] , V ) be some control parameters and R ≥ 0 be some radius such that ∥ v ∥ L 2 , ∥ ¯ v ∥
Revised: Let v, ¯ v ∈ L 2 ([0 , 1] , V ) be some control parameters and R ≥ 0 be some radius such that ∥ v ∥ L 2 , ∥ ¯ v ∥

Original: L 2 ≤ R .
Revised: L 2 ≤ R .

Original: Let ( x, y ) ∈ R d × R d ′ be some pair of data input / objective output.
Revised: Let ( x, y ) ∈ R d × R d ′ be some pair of data input / objective output.

Original: (i) Let z, ¯ z be solutions of Eq.
Revised: (i) Let z, ¯ z be solutions of Eq.

Original: (6) with parameter v and ¯ v respectively and with the same initial condition
Revised: (6) with parameter v and ¯ v respectively and with the same initial condition

Original: Ax , then for any t ∈ [0 , 1] :
Revised: Ax , then for any t ∈ [0 , 1] :

Original: [Equation]
Revised: [Equation]

Original: (ii) Let p, ¯ p be solutions of Eq.
Revised: (ii) Let p, ¯ p be solutions of Eq.

Original: (11) with parameter v and ¯ v respectively and with initial conditionN B ⊤ ( Bz 1 − y )
Revised: (11) with parameter v and ¯ v respectively and with initial conditionN B ⊤ ( Bz 1 − y )

Original: and 1 N B ⊤ ( B ¯ z 1 − y ) , then for any t ∈ [0 , 1] : ∥ p
Revised: and 1 N B ⊤ ( B ¯ z 1 − y ) , then for any t ∈ [0 , 1] : ∥ p

Original: [Equation]
Revised: [Equation]

Original: − ¯
Revised: − ¯

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma 2.
Revised: Proof of Lemma 2.

Original: Proof of (i)
Revised: Proof of (i)

Original: For every time t ∈ [0 , 1] we have:
Revised: For every time t ∈ [0 , 1] we have:

Original: [Equation]
Revised: [Equation]

Original: and by triangle inequality:
Revised: and by triangle inequality:

Original: [Equation]
Revised: [Equation]

Original: where we used Assumption 1 in the second inequality.
Revised: where we used Assumption 1 in the second inequality.

Original: Therefore, by Grönwall’s lemma:
Revised: Therefore, by Grönwall’s lemma:

Original: [Equation]
Revised: [Equation]

Original: Proof of (ii) For any t ∈ [0 , 1] we have:
Revised: Proof of (ii) For any t ∈ [0 , 1] we have:

Original: [Equation]
Revised: [Equation]

Original: and using the triangle inequality and Assumption 1:
Revised: and using the triangle inequality and Assumption 1:

Original: [Equation]
Revised: [Equation]

Original: Then, using Grönwall’s lemma backward in time gives:
Revised: Then, using Grönwall’s lemma backward in time gives:

Original: [Equation]
Revised: [Equation]

Original: On one hand, because of (i) we have for every s ∈ [0 , 1] :
Revised: On one hand, because of (i) we have for every s ∈ [0 , 1] :

Original: [Equation]
Revised: [Equation]

Original: and also:
Revised: and also:

Original: [Equation]
Revised: [Equation]

Original: On the other hand, recalling (ii) of Lemma 1, for every s ∈ [0 , 1] :
Revised: On the other hand, recalling (ii) of Lemma 1, for every s ∈ [0 , 1] :

Original: [Equation]
Revised: [Equation]

Original: Putting these estimates in the preceding inequality gives:
Revised: Putting these estimates in the preceding inequality gives:

Original: [Equation]
Revised: [Equation]

Original: which is the desired result.
Revised: which is the desired result.

Original: Proof of Property 3.
Revised: Proof of Property 3.

Original: Let v, ¯ v ∈ L 2 ([0 , 1] , V ) with ∥ v ∥ L 2 , ∥ ¯ v ∥ L 2 ≤ R .
Revised: Let v, ¯ v ∈ L 2 ([0 , 1] , V ) with ∥ v ∥ L 2 , ∥ ¯ v ∥ L 2 ≤ R .

Original: Then taking the same nota- tion as in Lemma 2, we have for any t
Revised: Then taking the same nota- tion as in Lemma 2, we have for any t

Original: [Equation]
Revised: [Equation]

Original: , and we can write ∥∇ L ( v ) t − ∇ L (¯ v ) t
Revised: , and we can write ∥∇ L ( v ) t − ∇ L (¯ v ) t

Original: ∥ V ≤ T 1 + T 2 with:
Revised: ∥ V ≤ T 1 + T 2 with:

Original: [Equation]
Revised: [Equation]

Original: First we consider deriving an upper bound on T 1 .
Revised: First we consider deriving an upper bound on T 1 .

Original: Note that by the definition of the norm in RKHSs and by Assumption 2 we have:
Revised: Note that by the definition of the norm in RKHSs and by Assumption 2 we have:

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: using (ii) from Lemma 2 to bound ∥ p it − ¯ p it ∥ for every index i we get:
Revised: using (ii) from Lemma 2 to bound ∥ p it − ¯ p it ∥ for every index i we get:

Original: [Equation]
Revised: [Equation]

Original: with:
Revised: with:

Original: [Equation]
Revised: [Equation]

Original: where we recognised L (¯ v ) in the third line.
Revised: where we recognised L (¯ v ) in the third line.

Original: By continuity of L we can define for every R ≥ 0 :
Revised: By continuity of L we can define for every R ≥ 0 :

Original: [Equation]
Revised: [Equation]

Original: And therefore:
Revised: And therefore:

Original: [Equation]
Revised: [Equation]

Original: We then consider deriving an upper-bound on T 2 .
Revised: We then consider deriving an upper-bound on T 2 .

Original: By triangle inequality:
Revised: By triangle inequality:

Original: [Equation]
Revised: [Equation]

Original: Consider any α ∈ V , then for any index i ∈ (cid:74) 1 , N (cid:75) , by the reproducing property:
Revised: Consider any α ∈ V , then for any index i ∈ (cid:74) 1 , N (cid:75) , by the reproducing property:

Original: [Equation]
Revised: [Equation]

Original: where we used the Cauchy-Schwarz inequality and Assumption 1 applied to α .
Revised: where we used the Cauchy-Schwarz inequality and Assumption 1 applied to α .

Original: Therefore, by duality:
Revised: Therefore, by duality:

Original: [Equation]
Revised: [Equation]

Original: Using the estimates of Lemma 1 and Lemma 2 we get:
Revised: Using the estimates of Lemma 1 and Lemma 2 we get:

Original: [Equation]
Revised: [Equation]

Original: And finally, using Cauchy-Schwarz inequality and recognizing L (¯ v ) we have:
Revised: And finally, using Cauchy-Schwarz inequality and recognizing L (¯ v ) we have:

Original: [Equation]
Revised: [Equation]

Original: with:
Revised: with:

Original: [Equation]
Revised: [Equation]

Original: Therefore we obtain the result by setting:
Revised: Therefore we obtain the result by setting:

Original: [Equation]
Revised: [Equation]

Original: Provided with Property 3, we can finish the proof of Theorem 2.
Revised: Provided with Property 3, we can finish the proof of Theorem 2.

Original: Proof of Theorem 2.
Revised: Proof of Theorem 2.

Original: By Property 2, L satisfies the PL inqualities of Definition 2 and the proof is a direct corollary of Theorem 1.
Revised: By Property 2, L satisfies the PL inqualities of Definition 2 and the proof is a direct corollary of Theorem 1.

Original: It only remains to show that the smoothness condition of Definitionis verified.
Revised: It only remains to show that the smoothness condition of Definitionis verified.

Original: Let v, ¯ v ∈ L 2 ([0 , 1] , V ) such that ∥ v ∥ L 2 , ∥ ¯ v ∥ L 2 ≤ R for some radius R ≥ 0 .
Revised: Let v, ¯ v ∈ L 2 ([0 , 1] , V ) such that ∥ v ∥ L 2 , ∥ ¯ v ∥ L 2 ≤ R for some radius R ≥ 0 .

Original: Then we have:
Revised: Then we have:

Original: [Equation]
Revised: [Equation]

Original: Using Property 3, there exists some C ( R ) such that:
Revised: Using Property 3, there exists some C ( R ) such that:

Original: [Equation]
Revised: [Equation]

Original: This gives the inequality:
Revised: This gives the inequality:

Original: [Equation]
Revised: [Equation]

Original: which is the desired result.
Revised: which is the desired result.

Original: D Proofs of Section
Revised: D Proofs of Section

Original: The results in Section 5 show how the condition for convergence in Eq.
Revised: The results in Section 5 show how the condition for convergence in Eq.

Original: (15) can be enforced by considering suitable RKHSs of vector-fields and suitable matrices A and B .
Revised: (15) can be enforced by considering suitable RKHSs of vector-fields and suitable matrices A and B .

Original: We give in Appendix D. examples of suitable kernels.
Revised: We give in Appendix D. examples of suitable kernels.

Original: In the following, we assume that for every q ≥ 1 we are provided with a function k q : R + → R such that the induced symmetric rotationally-invariant kernel K q defined by:
Revised: In the following, we assume that for every q ≥ 1 we are provided with a function k q : R + → R such that the induced symmetric rotationally-invariant kernel K q defined by:

Original: [Equation]
Revised: [Equation]

Original: is a positive-definite kernel over R q .
Revised: is a positive-definite kernel over R q .

Original: Without loss of generality, one can assume k q to be normalized, that is k q (0)
Revised: Without loss of generality, one can assume k q to be normalized, that is k q (0)

Original: = 1 .
Revised: = 1 .

Original: We note V q the vector-valued RKHS associated with K q .
Revised: We note V q the vector-valued RKHS associated with K q .

Original: The properties of V q are then entirely determined by k q .
Revised: The properties of V q are then entirely determined by k q .

Original: In particular, smoothness of the kernel at 0 implies regularity of the vector-fields in V q :
Revised: In particular, smoothness of the kernel at 0 implies regularity of the vector-fields in V q :

Original: Property 4 (Regularity of V q ) .
Revised: Property 4 (Regularity of V q ) .

Original: Let k q : R + → R be some function defining a positive symmetric kernel K q .
Revised: Let k q : R + → R be some function defining a positive symmetric kernel K q .

Original: If k q is 4 times differentiable at 0 , with k ′ q (0)
Revised: If k q is 4 times differentiable at 0 , with k ′ q (0)

Original: =
Revised: =

Original: k (3) q (0) = 0 .
Revised: k (3) q (0) = 0 .

Original: Then V q satisfies Assumption 1 with constant κ = (cid:112) k q (0) + (cid:113) − k ′′ q (0) + (cid:113) k (4) q (0) .
Revised: Then V q satisfies Assumption 1 with constant κ = (cid:112) k q (0) + (cid:113) − k ′′ q (0) + (cid:113) k (4) q (0) .

Original: As a consequence, if the derivatives of k q can be bounded uniformly over q then V q satisfies Assumption 1 with some constant κ independent of q .
Revised: As a consequence, if the derivatives of k q can be bounded uniformly over q then V q satisfies Assumption 1 with some constant κ independent of q .

Original: This, is the case for the Matérn kernel k defined in Eq.
Revised: This, is the case for the Matérn kernel k defined in Eq.

Original: Proof.
Revised: Proof.

Original: The proof proceeds by duality arguments.
Revised: The proof proceeds by duality arguments.

Original: For q ≥ 1 , consider some v ∈ V q .
Revised: For q ≥ 1 , consider some v ∈ V q .

Original: Then for any z ∈ R q and any α ∈ V q , by the reproducing properties of RKHSs:
Revised: Then for any z ∈ R q and any α ∈ V q , by the reproducing properties of RKHSs:

Original: [Equation]
Revised: [Equation]

Original: Therefore, by duality ∥ v ( z ) ∥ ≤ (cid:112) k q (0) ∥ v ∥ V q and then by taking the supremum over z ∈ R q :
Revised: Therefore, by duality ∥ v ( z ) ∥ ≤ (cid:112) k q (0) ∥ v ∥ V q and then by taking the supremum over z ∈ R q :

Original: [Equation]
Revised: [Equation]

Original: Then for any z ∈ R q any α, β ∈ R q and any h ∈ R + :
Revised: Then for any z ∈ R q any α, β ∈ R q and any h ∈ R + :

Original: [Equation]
Revised: [Equation]

Original: In the r.h.s we have using Taylor’s expansion of k q at 0 :
Revised: In the r.h.s we have using Taylor’s expansion of k q at 0 :

Original: [Equation]
Revised: [Equation]

Original: Taking the limit h → 0 :
Revised: Taking the limit h → 0 :

Original: [Equation]
Revised: [Equation]

Original: and therefore ∥ Dv ( z ) ∥ 2 ≤ (cid:113) − k ′′ q (0) ∥ v ∥ V q .
Revised: and therefore ∥ Dv ( z ) ∥ 2 ≤ (cid:113) − k ′′ q (0) ∥ v ∥ V q .

Original: Finally, let us bound ∥ D 2 v ∥ 2 , ∞ .
Revised: Finally, let us bound ∥ D 2 v ∥ 2 , ∞ .

Original: For any z ∈ R q any α, β, γ ∈ R q and any h, l ≥ 

Revised: For any z ∈ R q any α, β, γ ∈ R q and any h, l ≥

Original: we have in the same manner:
Revised: we have in the same manner:

Original: [Equation]
Revised: [Equation]

Original: where the second line is obtained by Taylor expansion of k q at 0 .
Revised: where the second line is obtained by Taylor expansion of k q at 0 .

Original: Thus, taking the limit h, l → 0 :
Revised: Thus, taking the limit h, l → 0 :

Original: [Equation]
Revised: [Equation]

Original: and therefore ∥ D 2 v ( z ) ∥ 2 ≤
Revised: and therefore ∥ D 2 v ( z ) ∥ 2 ≤

Original: [Equation]
Revised: [Equation]

Original: Setting κ = (cid:112) k q (0) + (cid:113) − k ′′ q (0) + (cid:113) k
Revised: Setting κ = (cid:112) k q (0) + (cid:113) − k ′′ q (0) + (cid:113) k

Original: (4) q (0) we obtain the result.
Revised: (4) q (0) we obtain the result.

Original: Moreover, choosing appropriate v in the above proof, inequalities become sharp and one observes that the constant κ is optimal.
Revised: Moreover, choosing appropriate v in the above proof, inequalities become sharp and one observes that the constant κ is optimal.

Original: D.1 Enforcing convergence with high dimensional lifting and universal kernels
Revised: D.1 Enforcing convergence with high dimensional lifting and universal kernels

Original: Here we investigate the dependency of Eq.
Revised: Here we investigate the dependency of Eq.

Original: (15) w.r.t.
Revised: (15) w.r.t.

Original: q , δ and N for the class of RKHS V q and thereby recover the proof of Proposition 1.
Revised: q , δ and N for the class of RKHS V q and thereby recover the proof of Proposition 1.

Original: We make the following assumption concerning the decay of k q at infinity:
Revised: We make the following assumption concerning the decay of k q at infinity:

Original: Assumption 3 (Decay of k q ) .
Revised: Assumption 3 (Decay of k q ) .

Original: For every q ≥ 1 , k q ( x ) tends to 0 when x tends to infinity and we note β q,N > 0 s.t.:
Revised: For every q ≥ 1 , k q ( x ) tends to 0 when x tends to infinity and we note β q,N > 0 s.t.:

Original: [Equation]
Revised: [Equation]

Original: Moreover for fixed N we assume that
Revised: Moreover for fixed N we assume that

Original: [Equation]
Revised: [Equation]

Original: .1 Lifting matrices
Revised: .1 Lifting matrices

Original: For any q ≥ 1 we consider here the matrices:
Revised: For any q ≥ 1 we consider here the matrices:

Original: [Equation]
Revised: [Equation]

Original: where there are ⌊ q/d ⌋ copies of Id d in A q .
Revised: where there are ⌊ q/d ⌋ copies of Id d in A q .

Original: In particular we have:
Revised: In particular we have:

Original: [Equation]
Revised: [Equation]

Original: and B q A q ∈ R d ′ ×
Revised: and B q A q ∈ R d ′ ×

Original: d is independent of q .
Revised: d is independent of q .

Original: We also consider for every q ≥ 1 some control parameter initialization V 0 q ∈ L 2 ( V q ) such that ∥ v 0 q ∥
Revised: We also consider for every q ≥ 1 some control parameter initialization V 0 q ∈ L 2 ( V q ) such that ∥ v 0 q ∥

Original: L 2 ≤ R 0 q − 1 / 4 and assume the data distribution to be compactly supported.
Revised: L 2 ≤ R 0 q − 1 / 4 and assume the data distribution to be compactly supported.

Original: Proposition 4.
Revised: Proposition 4.

Original: Let R > 0 and d, d ′ ≥ 1 .
Revised: Let R > 0 and d, d ′ ≥ 1 .

Original: Assume Assumption 3 is satisfied, V q satisfies Assumptionwith constant κ independent of q and there exists R 0 > 0 s.t.
Revised: Assume Assumption 3 is satisfied, V q satisfies Assumptionwith constant κ independent of q and there exists R 0 > 0 s.t.

Original: ∥ v 0 q ∥ ≤ R 0 q − 1 / 4 for every q ≥ 1 .
Revised: ∥ v 0 q ∥ ≤ R 0 q − 1 / 4 for every q ≥ 1 .

Original: Then there exists some constant C > 0 so that for any N ≥ 2 and any δ ∈ (0 , 1] , Eq.
Revised: Then there exists some constant C > 0 so that for any N ≥ 2 and any δ ∈ (0 , 1] , Eq.

Original: (15) is satisfied with matrices A q , B q and κ, λ, Λ associated with the RKHS V q as soon as:
Revised: (15) is satisfied with matrices A q , B q and κ, λ, Λ associated with the RKHS V q as soon as:

Original: [Equation]
Revised: [Equation]

Original: Note that the second condition in Eq.
Revised: Note that the second condition in Eq.

Original: (24) can always be ensured for large enough q thanks to Assumption 3.
Revised: (24) can always be ensured for large enough q thanks to Assumption 3.

Original: In the case of the Matérn kernel k defined in Eq.
Revised: In the case of the Matérn kernel k defined in Eq.

Original: (17), such an assumption is verified because it has exponential decay and it is independent of q .
Revised: (17), such an assumption is verified because it has exponential decay and it is independent of q .

Original: Hence, Proposition 1 is a direct consequence of Proposition 4.
Revised: Hence, Proposition 1 is a direct consequence of Proposition 4.

Original: Proof of Proposition 4.
Revised: Proof of Proposition 4.

Original: Let q ≥ 1 .
Revised: Let q ≥ 1 .

Original: Using the fact that d 2 ⌊ q/d ⌋ 2 ≥ q ( q − 2 d ) , considering:
Revised: Using the fact that d 2 ⌊ q/d ⌋ 2 ≥ q ( q − 2 d ) , considering:

Original: [Equation]
Revised: [Equation]

Original: is enough to ensure that:
Revised: is enough to ensure that:

Original: [Equation]
Revised: [Equation]

Original: Then, by Assumption 3 for ( z i ) 1 ≤ i ≤ N ∈
Revised: Then, by Assumption 3 for ( z i ) 1 ≤ i ≤ N ∈

Original: ( R q ) N with data separation q − 1 / 4 (cid:112)
Revised: ( R q ) N with data separation q − 1 / 4 (cid:112)

Original: ⌊ q/d
Revised: ⌊ q/d

Original: ⌋ δe − κ ( R + R 0 ) we have:
Revised: ⌋ δe − κ ( R + R 0 ) we have:

Original: [Equation]
Revised: [Equation]

Original: Thus, the kernel matrix K =
Revised: Thus, the kernel matrix K =

Original: [Equation]
Revised: [Equation]

Original: q
Revised: q

Original: [Equation]
Revised: [Equation]

Original: z
Revised: z

Original: [Equation]
Revised: [Equation]

Original: z j ∥ ) Id q ) i,j is diagonally dominant with:
Revised: z j ∥ ) Id q ) i,j is diagonally dominant with:

Original: [Equation]
Revised: [Equation]

Original: and by definition of λ in Eq.
Revised: and by definition of λ in Eq.

Original: [Equation]
Revised: [Equation]

Original: Moreover, Λ ≤ N because k q is bounded by 1 .
Revised: Moreover, Λ ≤ N because k q is bounded by 1 .

Original: Let x ∈ B (0 , r 0 ) and assume z is a solution of Eq.
Revised: Let x ∈ B (0 , r 0 ) and assume z is a solution of Eq.

Original: (6) for the control parameter v 0 q and with initial condition
Revised: (6) for the control parameter v 0 q and with initial condition

Original: A q x .
Revised: A q x .

Original: We have at time t = 1 :
Revised: We have at time t = 1 :

Original: [Equation]
Revised: [Equation]

Original: so that by triangle inequality and Assumption 1:
Revised: so that by triangle inequality and Assumption 1:

Original: [Equation]
Revised: [Equation]

Original: and then because ∥ v 0 q ∥ ≤ R 0 q − 1 / 4 and the dataset is compactly supported:
Revised: and then because ∥ v 0 q ∥ ≤ R 0 q − 1 / 4 and the dataset is compactly supported:

Original: [Equation]
Revised: [Equation]

Original: with B q A q independent of q .
Revised: with B q A q independent of q .

Original: Thus L ( v 0 q ) ≤ C for some constant C independent of q , N and δ .
Revised: Thus L ( v 0 q ) ≤ C for some constant C independent of q , N and δ .

Original: Finally:
Revised: Finally:

Original: [Equation]
Revised: [Equation]

Original: and putting Eq.
Revised: and putting Eq.

Original: (26) and Eq.
Revised: (26) and Eq.

Original: (27) into the l.h.s.
Revised: (27) into the l.h.s.

Original: Eq.
Revised: Eq.

Original: (15) gives:
Revised: (15) gives:

Original: [Equation]
Revised: [Equation]

Original: Considering R > 0 is fixed (c.f. Remark 7), Theorem 2 can be applied as soon as:
Revised: Considering R > 0 is fixed (c.f. Remark 7), Theorem 2 can be applied as soon as:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: and combining this bound with the one in Eq.
Revised: and combining this bound with the one in Eq.

Original: (25) gives the result.
Revised: (25) gives the result.

Original: Remark 7 (Choice of R ) .
Revised: Remark 7 (Choice of R ) .

Original: The proof of Proposition 4 holds for any fixed R > 0 whose choice impacts the result through the constant C .
Revised: The proof of Proposition 4 holds for any fixed R > 0 whose choice impacts the result through the constant C .

Original: There is a trade-off between minimizing e 4 κR to have a better dependency of q w.r.t.
Revised: There is a trade-off between minimizing e 4 κR to have a better dependency of q w.r.t.

Original: δ − 1 log( N ) in Eq.
Revised: δ − 1 log( N ) in Eq.

Original: and minimizing R − 1 e 3 κR to have a better dependency w.r.t.
Revised: and minimizing R − 1 e 3 κR to have a better dependency w.r.t.

Original: N in Eq.
Revised: N in Eq.

Original: However, in any case, optimizing w.r.t.
Revised: However, in any case, optimizing w.r.t.

Original: R only improves the result up to a constant factor.
Revised: R only improves the result up to a constant factor.

Original: D.1.2 Scaling matrices
Revised: D.1.2 Scaling matrices

Original: For α > 0 , we consider here the matrices:
Revised: For α > 0 , we consider here the matrices:

Original: [Equation]
Revised: [Equation]

Original: Then, in the proof of Proposition 4 one has σ min ( A )
Revised: Then, in the proof of Proposition 4 one has σ min ( A )

Original: =
Revised: =

Original: α and thus Eq.
Revised: α and thus Eq.

Original: (26) holds as soon as:
Revised: (26) holds as soon as:

Original: [Equation]
Revised: [Equation]

Original: Moreover, σ max ( B ⊤ ) = σ min ( B ⊤ ) =
Revised: Moreover, σ max ( B ⊤ ) = σ min ( B ⊤ ) =

Original: α and F (0 , x ) = 

Revised: α and F (0 , x ) =

Original: for every input x as BA = 0 .
Revised: for every input x as BA = 0 .

Original: Thus, with initialization v 0 = 

Revised: Thus, with initialization v 0 =

Original: the l.h.s. of Eq.
Revised: the l.h.s. of Eq.

Original: (15) scales as:
Revised: (15) scales as:

Original: [Equation]
Revised: [Equation]

Original: and global convergence holds for α = Ω( δ − 1 β d + d ′ ,N + N ) .
Revised: and global convergence holds for α = Ω( δ − 1 β d + d ′ ,N + N ) .

Original: D.2 Enforcing convergence with high dimensional embedding en finite dimensional kernels
Revised: D.2 Enforcing convergence with high dimensional embedding en finite dimensional kernels

Original: We recover here the result of Proposition 2 for the more general kernel k q .
Revised: We recover here the result of Proposition 2 for the more general kernel k q .

Original: In particular notice that, as an application of Bochner’s theorem [50], for every q ≥ 1 there exists some probability measure µ q over R q such that:
Revised: In particular notice that, as an application of Bochner’s theorem [50], for every q ≥ 1 there exists some probability measure µ q over R q such that:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Then, such as in Eq.
Revised: Then, such as in Eq.

Original: (20) for the Matérn kernel, for any independent sampling ω
Revised: (20) for the Matérn kernel, for any independent sampling ω

Original: j ∼ µ q of size r one can consider the feature map:
Revised: j ∼ µ q of size r one can consider the feature map:

Original: [Equation]
Revised: [Equation]

Original: Such a feature map induces a structure of RKHS
Revised: Such a feature map induces a structure of RKHS

Original: ˆ V q which is the set of residuals of Eq.
Revised: ˆ V q which is the set of residuals of Eq.

Original: (3) with activation φ .
Revised: (3) with activation φ .

Original: The associated kernel is ˆ K q : ( z, z ′ )
Revised: The associated kernel is ˆ K q : ( z, z ′ )

Original: (cid:55)→ ˆ k q ( z, z ′ )
Revised: (cid:55)→ ˆ k q ( z, z ′ )

Original: Id q with:
Revised: Id q with:

Original: [Equation]
Revised: [Equation]

Original: almost surely, by the law of large numbers.
Revised: almost surely, by the law of large numbers.

Original: We make the following assumption on µ q :
Revised: We make the following assumption on µ q :

Original: Assumption 4 (Moments of µ q ) .
Revised: Assumption 4 (Moments of µ q ) .

Original: The measure µ q admits finite moments up to order 8 :
Revised: The measure µ q admits finite moments up to order 8 :

Original: [Equation]
Revised: [Equation]

Original: Moreover, we assume those moments are independent of q .
Revised: Moreover, we assume those moments are independent of q .

Original: Note that Assumption 4 implies regularity on the function k q .
Revised: Note that Assumption 4 implies regularity on the function k q .

Original: Indeed by Fourier inversion theorem we have for every r ∈ R + and every θ ∈ S
Revised: Indeed by Fourier inversion theorem we have for every r ∈ R + and every θ ∈ S

Original: d − 1 :
Revised: d − 1 :

Original: [Equation]
Revised: [Equation]

Original: By theorems of derivation under the integral k q is 8 th -time differentiable on R + and for 0 ≤ l ≤ 8 :
Revised: By theorems of derivation under the integral k q is 8 th -time differentiable on R + and for 0 ≤ l ≤ 8 :

Original: [Equation]
Revised: [Equation]

Original: In particular, k q is four time differentiable at 0 and:
Revised: In particular, k q is four time differentiable at 0 and:

Original: [Equation]
Revised: [Equation]

Original: Therefore, k ′ q (0) and k (3) q (0) are in ı R ∩ R = { 0 } and Property 4 holds.
Revised: Therefore, k ′ q (0) and k (3) q (0) are in ı R ∩ R = { 0 } and Property 4 holds.

Original: Moreover, as the moments are independent of q , the associated κ is also independent of q .
Revised: Moreover, as the moments are independent of q , the associated κ is also independent of q .

Original: Proposition 5.
Revised: Proposition 5.

Original: Consider q, N ≥ 1 and ϵ, τ, R > 0 .
Revised: Consider q, N ≥ 1 and ϵ, τ, R > 0 .

Original: (i) Assume Assumption 4 is satisfied.
Revised: (i) Assume Assumption 4 is satisfied.

Original: For r ≥ Ω( τq 8 ) , with probability greater than 1 − τ − 1 , ˆ V q satisfies Assumption 1 with some ˆ κ ≤ κ + 1 .
Revised: For r ≥ Ω( τq 8 ) , with probability greater than 1 − τ − 1 , ˆ V q satisfies Assumption 1 with some ˆ κ ≤ κ + 1 .

Original: (ii) For r ≥ Ω( ϵ − 2 N 2 ( q log( ∥ A ∥ 2 r 0 + R ) + τ )) , with probability greater than 1 − e − τ , for any control parameter v ∈ L 2 ([0 , 1] , ˆ V q ) s.t.
Revised: (ii) For r ≥ Ω( ϵ − 2 N 2 ( q log( ∥ A ∥ 2 r 0 + R ) + τ )) , with probability greater than 1 − e − τ , for any control parameter v ∈ L 2 ([0 , 1] , ˆ V q ) s.t.

Original: ∥ v ∥ L 2 ≤ R and any time t ∈ [0 , 1] :
Revised: ∥ v ∥ L 2 ≤ R and any time t ∈ [0 , 1] :

Original: [Equation]
Revised: [Equation]

Original: where the ( z i ) i are the solutions to Eq.
Revised: where the ( z i ) i are the solutions to Eq.

Original: (6) and ˆ K , K are the kernel matrices associated with ˆ k and k respectively.
Revised: (6) and ˆ K , K are the kernel matrices associated with ˆ k and k respectively.

Original: As Assumption 4 is satisfied for the Matérn kernel k defined in Eq.
Revised: As Assumption 4 is satisfied for the Matérn kernel k defined in Eq.

Original: (17) as soon as ν > 4 , Proposition 2 is a direct consequence of Proposition 5.
Revised: (17) as soon as ν > 4 , Proposition 2 is a direct consequence of Proposition 5.

Original: Proof of Proposition 5.
Revised: Proof of Proposition 5.

Original: Proof of (i)
Revised: Proof of (i)

Original: We already saw that thanks to the assumption on the moments of µ q , the RKHS V q associated with k q satisfies Assumption 1 with constant κ .
Revised: We already saw that thanks to the assumption on the moments of µ q , the RKHS V q associated with k q satisfies Assumption 1 with constant κ .

Original: Then we want to prove that for sufficiently high r , the RKHS ˆ V q generated by the feature map φ in Eq.
Revised: Then we want to prove that for sufficiently high r , the RKHS ˆ V q generated by the feature map φ in Eq.

Original: (20), satisfies Assumption 1.
Revised: (20), satisfies Assumption 1.

Original: Let v ∈
Revised: Let v ∈

Original: ˆ V q be of the form:
Revised: ˆ V q be of the form:

Original: [Equation]
Revised: [Equation]

Original: for some W ∈ R q × r .
Revised: for some W ∈ R q × r .

Original: For z ∈ R q , ∥ φ ( z ) ∥
Revised: For z ∈ R q , ∥ φ ( z ) ∥

Original: = 1 and thus:
Revised: = 1 and thus:

Original: [Equation]
Revised: [Equation]

Original: so that ∥ v ∥ ∞ ≤ ∥ v ∥ ˆ V q .
Revised: so that ∥ v ∥ ∞ ≤ ∥ v ∥ ˆ V q .

Original: Then Dv ( z ) =
Revised: Then Dv ( z ) =

Original: WDφ ( z ) and by the law of large number we have for any θ ∈ S q − 1 :
Revised: WDφ ( z ) and by the law of large number we have for any θ ∈ S q − 1 :

Original: [Equation]
Revised: [Equation]

Original: Because µ q admits finite fourth order moments, the rate of convergence can be controlled using
Revised: Because µ q admits finite fourth order moments, the rate of convergence can be controlled using

Original: Chebyshev’s inequality.
Revised: Chebyshev’s inequality.

Original: For every indices k, l ∈ (cid:74) 1 , q (cid:75) :
Revised: For every indices k, l ∈ (cid:74) 1 , q (cid:75) :

Original: [Equation]
Revised: [Equation]

Original: For r ≥ Ω( q 4 τα 2 ) we have with probability greater than 1 − τ − 1 that the above inequality is satisfied for every indices k, l .
Revised: For r ≥ Ω( q 4 τα 2 ) we have with probability greater than 1 − τ − 1 that the above inequality is satisfied for every indices k, l .

Original: Thus for every z ∈ R q and every θ ∈ S q − 1 :
Revised: Thus for every z ∈ R q and every θ ∈ S q − 1 :

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: using Chauchy-Schwarz inequality in the last line.
Revised: using Chauchy-Schwarz inequality in the last line.

Original: We can thus conclude:
Revised: We can thus conclude:

Original: [Equation]
Revised: [Equation]

Original: The same arguments holds for D 2 v ( z )
Revised: The same arguments holds for D 2 v ( z )

Original: =
Revised: =

Original: WD 2 φ ( z ) .
Revised: WD 2 φ ( z ) .

Original: For any θ ∈ S q − 1 we have:
Revised: For any θ ∈ S q − 1 we have:

Original: [Equation]
Revised: [Equation]

Original: Passing to the squared norm we get:
Revised: Passing to the squared norm we get:

Original: [Equation]
Revised: [Equation]

Original: Then because µ q admits 8 th order moments, we can control the convergence in probability by
Revised: Then because µ q admits 8 th order moments, we can control the convergence in probability by

Original: Chebyshev’s inequality.
Revised: Chebyshev’s inequality.

Original: For r ≥ Ω( q 8 τα 2 ) we have with probability greater than 1 − τ − 1 :
Revised: For r ≥ Ω( q 8 τα 2 ) we have with probability greater than 1 − τ − 1 :

Original: [Equation]
Revised: [Equation]

Original: Finally ˆ V q satisfies Assumption 1 with:
Revised: Finally ˆ V q satisfies Assumption 1 with:

Original: [Equation]
Revised: [Equation]

Original: for α sufficiently low.
Revised: for α sufficiently low.

Original: Proof of (ii).
Revised: Proof of (ii).

Original: For t ∈ [0 , 1] , we consider ( z it )
Revised: For t ∈ [0 , 1] , we consider ( z it )

Original: i the solutions of of Eq.
Revised: i the solutions of of Eq.

Original: (6) for some control parameter v ∈ L 2 ([0 , 1] , ˆ V q )
Revised: (6) for some control parameter v ∈ L 2 ([0 , 1] , ˆ V q )

Original: and we introduce the kernel matrices:
Revised: and we introduce the kernel matrices:

Original: [Equation]
Revised: [Equation]

Original: Using the first point, we know that if ∥ v ∥ L 2 ≤ R , then ∥ z it ∥ ≤ ∥
Revised: Using the first point, we know that if ∥ v ∥ L 2 ≤ R , then ∥ z it ∥ ≤ ∥

Original: Ax i ∥
Revised: Ax i ∥

Original: + ( κ + 1) R .
Revised: + ( κ + 1) R .

Original: Then, using Theorem 1 in [53], we have for every indices i, j and every t ∈ [0 , 1] :
Revised: Then, using Theorem 1 in [53], we have for every indices i, j and every t ∈ [0 , 1] :

Original: [Equation]
Revised: [Equation]

Original: with h ( q, R ) :
Revised: with h ( q, R ) :

Original: =
Revised: =

Original: O
Revised: O

Original: [Equation]
Revised: [Equation]

Original: log( ∥
Revised: log( ∥

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: Thus, choosing r ≥ Ω (cid:0) ϵ − 2 N 2 ( q log( ∥ A ∥ 2 r 0 + R ) + τ ) (cid:1) , we have with probability greater than 1 − e − τ , λ min (ˆ K t ) ≥ λ min ( K t ) − ϵ , for any t ∈ [0 , 1] .
Revised: Thus, choosing r ≥ Ω (cid:0) ϵ − 2 N 2 ( q log( ∥ A ∥ 2 r 0 + R ) + τ ) (cid:1) , we have with probability greater than 1 − e − τ , λ min (ˆ K t ) ≥ λ min ( K t ) − ϵ , for any t ∈ [0 , 1] .

Original: Note that the assumption of finite 8 th moments is only needed to control the convergence rate of ˆ k q towards k q in probability.
Revised: Note that the assumption of finite 8 th moments is only needed to control the convergence rate of ˆ k q towards k q in probability.

Original: By the law of large numbers, assuming finite 4 th -order moments is sufficient to have convergence almost surely.
Revised: By the law of large numbers, assuming finite 4 th -order moments is sufficient to have convergence almost surely.

Original: Also, we used the Chebyshev’s inequality in order to control the convergence rate.
Revised: Also, we used the Chebyshev’s inequality in order to control the convergence rate.

Original: Making stronger assumptions on the decay of µ q (e.g. sub-gaussianity) could have led to faster convergence by using sharper concentration inequalities.
Revised: Making stronger assumptions on the decay of µ q (e.g. sub-gaussianity) could have led to faster convergence by using sharper concentration inequalities.

Original: D.3 Example of appropriate kernels
Revised: D.3 Example of appropriate kernels

Original: We show here that the Matérn kernel of parameter ν ∈ (8 , + ∞ ] satisfies Assumption 3 and Assumption 4.
Revised: We show here that the Matérn kernel of parameter ν ∈ (8 , + ∞ ] satisfies Assumption 3 and Assumption 4.

Original: Gaussian kernel.
Revised: Gaussian kernel.

Original: The Gaussian kernel defined by for some parameter σ > 0 by k q ( r ) =
Revised: The Gaussian kernel defined by for some parameter σ > 0 by k q ( r ) =

Original: e − σ 2 r 2 2 .
Revised: e − σ 2 r 2 2 .

Original: In this case the frequency distribution µ q is the multivariate normal of variance σ and has a density given for every ω ∈ R q by:
Revised: In this case the frequency distribution µ q is the multivariate normal of variance σ and has a density given for every ω ∈ R q by:

Original: [Equation]
Revised: [Equation]

Original: This distribution admits finite moments of every order which are independent of q .
Revised: This distribution admits finite moments of every order which are independent of q .

Original: Also, k q is four times differentiable at 0 and by Property 4 the associated V q is (strongly) admissible with κ = 2+ √
Revised: Also, k q is four times differentiable at 0 and by Property 4 the associated V q is (strongly) admissible with κ = 2+ √

Original: Moreover Assumption 3 as one has | k q ( x ) | ≤ 1 / 2 N if:
Revised: Moreover Assumption 3 as one has | k q ( x ) | ≤ 1 / 2 N if:

Original: [Equation]
Revised: [Equation]

Original: Matérn kernel.
Revised: Matérn kernel.

Original: Sobolev spaces H s ( R q , R q ) are RKHSs as soon as s > q/ 2 .
Revised: Sobolev spaces H s ( R q , R q ) are RKHSs as soon as s > q/ 2 .

Original: Given some ν > 0 , the kernel k q associated with H ( q/ 2+ ν ) ( R q , R q ) is independent of q and is defined in Eq.
Revised: Given some ν > 0 , the kernel k q associated with H ( q/ 2+ ν ) ( R q , R q ) is independent of q and is defined in Eq.

Original: It is associated with the multivariate t-distribution:
Revised: It is associated with the multivariate t-distribution:

Original: [Equation]
Revised: [Equation]

Original: for some normalising constant C ( q, ν ) .
Revised: for some normalising constant C ( q, ν ) .

Original: Therefore, µ q admits l th order moments as soon as ν ≥ l/ 2 , and those moments are bounded independently of q (see [28] for the computation of moments).
Revised: Therefore, µ q admits l th order moments as soon as ν ≥ l/ 2 , and those moments are bounded independently of q (see [28] for the computation of moments).

Original: In particular, for ν > 2 , k q is four times differentiable at 0 with k ′′ (0) =
Revised: In particular, for ν > 2 , k q is four times differentiable at 0 with k ′′ (0) =

Original: ν/ ( ν − 1) and k (4) (0) =
Revised: ν/ ( ν − 1) and k (4) (0) =

Original: 3 ν 2 / ( ν − 1)( ν − 2) .
Revised: 3 ν 2 / ( ν − 1)( ν − 2) .

Original: Thus by Property 4, V q is (strongly) admissible with:
Revised: Thus by Property 4, V q is (strongly) admissible with:

Original: [Equation]
Revised: [Equation]

Original: Because k q has exponential decay (see [31]), there exist constants H ν , G ν such that:
Revised: Because k q has exponential decay (see [31]), there exist constants H ν , G ν such that:

Original: [Equation]
Revised: [Equation]

Original: and Assumption 3 is satisfied with
Revised: and Assumption 3 is satisfied with

Original: [Equation]
Revised: [Equation]

Original: Remark 8 (Sampling) .
Revised: Remark 8 (Sampling) .

Original: Sampling over µ q can be achieved using that for Y ∼ N (0 , Id q ) and for u distributed according to χ 22 ν , the chi-squared distribution with 2 ν degrees of freedom, Y/ (cid:112) u/ 2 ν is distributed according to µ q .
Revised: Sampling over µ q can be achieved using that for Y ∼ N (0 , Id q ) and for u distributed according to χ 22 ν , the chi-squared distribution with 2 ν degrees of freedom, Y/ (cid:112) u/ 2 ν is distributed according to µ q .

Original: E RKHS-NODE as a generalization of linear networks
Revised: E RKHS-NODE as a generalization of linear networks

Original: In an attempt to better understand the convergence properties of GD in the training of ResNets, lots of attention has first been brought towards the study of linear models, for which the training dynamic is now well understood [24, 7, 64].
Revised: In an attempt to better understand the convergence properties of GD in the training of ResNets, lots of attention has first been brought towards the study of linear models, for which the training dynamic is now well understood [24, 7, 64].

Original: We explain here in what extent our work can be seen, at least formally, as a generalization of these results to a more general class of ResNets.
Revised: We explain here in what extent our work can be seen, at least formally, as a generalization of these results to a more general class of ResNets.

Original: In this purpose, we highlight the similarity between Theorem 2, which applies to the whole class of models described by Definition 1, and [64, Theorem 3.1.], which only applies to linear ResNets.
Revised: In this purpose, we highlight the similarity between Theorem 2, which applies to the whole class of models described by Definition 1, and [64, Theorem 3.1.], which only applies to linear ResNets.

Original: More precisely, [64] studies model of the form:
Revised: More precisely, [64] studies model of the form:

Original: [Equation]
Revised: [Equation]

Original: (31) where x ∈ R
Revised: (31) where x ∈ R

Original: d is the input data, W =
Revised: d is the input data, W =

Original: ( W 1 , ..., W D ) ∈
Revised: ( W 1 , ..., W D ) ∈

Original: ( R q × q ) D is the trained parameter and
Revised: ( R q × q ) D is the trained parameter and

Original: A ∈ R q × d , B ∈ R d ′ ×
Revised: A ∈ R q × d , B ∈ R d ′ ×

Original: q are fixed matrices.
Revised: q are fixed matrices.

Original: Taking the limit of infinite depth D → + ∞ in the above model motivates the following definition for linear Neural ODE models:
Revised: Taking the limit of infinite depth D → + ∞ in the above model motivates the following definition for linear Neural ODE models:

Original: Definition 4 (Linear-NODE) .
Revised: Definition 4 (Linear-NODE) .

Original: Let A ∈ R q × d and B ∈ R d ′ ×
Revised: Let A ∈ R q × d and B ∈ R d ′ ×

Original: q be fixed matrices.
Revised: q be fixed matrices.

Original: Then for
Revised: Then for

Original: W ∈ L 2 ([0 , 1] , R q × q ) and input x ∈ R d , the Linear-NODE output is given by F ( W, x ) :
Revised: W ∈ L 2 ([0 , 1] , R q × q ) and input x ∈ R d , the Linear-NODE output is given by F ( W, x ) :

Original: = BU 1 Ax , where U is the solution to the following forward problem :
Revised: = BU 1 Ax , where U is the solution to the following forward problem :

Original: [Equation]
Revised: [Equation]

Original: One sees that the ResNet F has residual terms that are linear w.r.t.
Revised: One sees that the ResNet F has residual terms that are linear w.r.t.

Original: the parameters and thus fits in the framework of our analysis.
Revised: the parameters and thus fits in the framework of our analysis.

Original: More precisely, the Linear-NODE of Definition 4 can be seen as a special instance of RKHS-NODE of Definition 1 with space of residual defined as:
Revised: More precisely, the Linear-NODE of Definition 4 can be seen as a special instance of RKHS-NODE of Definition 1 with space of residual defined as:

Original: [Equation]
Revised: [Equation]

Original: This corresponds to Eq.
Revised: This corresponds to Eq.

Original: (3) with the choice of feature map φ
Revised: (3) with the choice of feature map φ

Original: =
Revised: =

Original: I
Revised: I

Original: d : R q → R q .
Revised: d : R q → R q .

Original: The set of residuals
Revised: The set of residuals

Original: V is then of course a RKHS for the Frobenius metric on matrices.
Revised: V is then of course a RKHS for the Frobenius metric on matrices.

Original: In particular V satisfies an analog of Assumption 1 in the sense that for ( v : z
Revised: In particular V satisfies an analog of Assumption 1 in the sense that for ( v : z

Original: (cid:55)→ Wz ) ∈ V :
Revised: (cid:55)→ Wz ) ∈ V :

Original: [Equation]
Revised: [Equation]

Original: Universality (Assumption 2) is also satisfied on full-rank data matrices.
Revised: Universality (Assumption 2) is also satisfied on full-rank data matrices.

Original: If Z = ( z 1 | ... | z N ) ∈ R q × N then the associated kernel matrix verifies:
Revised: If Z = ( z 1 | ... | z N ) ∈ R q × N then the associated kernel matrix verifies:

Original: [Equation]
Revised: [Equation]

Original: As in our above presentation we consider training Linear-NODE for the minimization of the empirical risk associated to the square euclidean distance on the output space R d ′ .
Revised: As in our above presentation we consider training Linear-NODE for the minimization of the empirical risk associated to the square euclidean distance on the output space R d ′ .

Original: Given data matrices
Revised: Given data matrices

Original: X =
Revised: X =

Original: ( x 1 | ... | x N
Revised: ( x 1 | ... | x N

Original: ) ∈ R d × N for the input and Y =
Revised: ) ∈ R d × N for the input and Y =

Original: ( y 1 | ... | y N ) ∈ R d ′ × N for the output, we aim at finding a control parameter minimizing the risk defined for every W ∈ L 2 ([0 , 1] , R q × q ) as:
Revised: ( y 1 | ... | y N ) ∈ R d ′ × N for the output, we aim at finding a control parameter minimizing the risk defined for every W ∈ L 2 ([0 , 1] , R q × q ) as:

Original: [Equation]
Revised: [Equation]

Original: One difference with the previous analysis is that one can not expect the empirical risk to reach the value 

Revised: One difference with the previous analysis is that one can not expect the empirical risk to reach the value

Original: if the target data Y is not in the linear span of the input X .
Revised: if the target data Y is not in the linear span of the input X .

Original: We are thus interested in minimizing the excess risk defined as:
Revised: We are thus interested in minimizing the excess risk defined as:

Original: [Equation]
Revised: [Equation]

Original: with L ∗ :
Revised: with L ∗ :

Original: = inf U
Revised: = inf U

Original: [Equation]
Revised: [Equation]

Original: Following the line of the proof of Property 2, one can then show that the excess risk ˜
Revised: Following the line of the proof of Property 2, one can then show that the excess risk ˜

Original: L associated to our Linear-NODE model verifies the following (local) PL property:
Revised: L associated to our Linear-NODE model verifies the following (local) PL property:

Original: [Equation]
Revised: [Equation]

Original: where m and M are given for R ≥ 0 by:
Revised: where m and M are given for R ≥ 0 by:

Original: [Equation]
Revised: [Equation]

Original: with σ r ( X ) the smallest positive singular value of X .
Revised: with σ r ( X ) the smallest positive singular value of X .

Original: Hence, in the same way local PL implies local convergence for a general RKHS V (Theorem 2), convergence in the linear case follows as an application of Theorem 1:
Revised: Hence, in the same way local PL implies local convergence for a general RKHS V (Theorem 2), convergence in the linear case follows as an application of Theorem 1:

Original: Theorem 4 (analog to Theorem 3.1.
Revised: Theorem 4 (analog to Theorem 3.1.

Original: in [64]) .
Revised: in [64]) .

Original: Let W 0 be some control parameter initialization with norm ∥ W 0 ∥
Revised: Let W 0 be some control parameter initialization with norm ∥ W 0 ∥

Original: = R 0 and assume there exists some R > 0 s.t.:
Revised: = R 0 and assume there exists some R > 0 s.t.:

Original: [Equation]
Revised: [Equation]

Original: then, for a sufficiently small step-size η , GD initialized at W 0 converges towards a global minimizer of L with linear convergence rate.
Revised: then, for a sufficiently small step-size η , GD initialized at W 0 converges towards a global minimizer of L with linear convergence rate.