Original: Do Residual Neural Networks discretize Neural Ordinary Differential Equations?
Revised: Do Residual Neural Networks discretize Neural Ordinary Differential Equations?

Original: Author(s

Original: Abstract
Revised: Abstract

Original: Neural Ordinary Differential Equations (Neural ODEs) are the continuous analogof Residual Neural Networks (ResNets).
Revised: Neural Ordinary Differential Equations (Neural ODEs) are the continuous analog of Residual Neural Networks (ResNets).

Original: We investigate whether the discretedynamics defined by a ResNet are close to the continuous one of a Neural ODE.
Revised: We investigate whether the discrete dynamics defined by a ResNet are close to the continuous one of a Neural ODE.

Original: We first quantify the distance between the ResNet’s hidden state trajectory and thesolution of its corresponding Neural ODE.
Revised: We first quantify the distance between the ResNet’s hidden state trajectory and the solution of its corresponding Neural ODE.

Original: Our bound is tight and, on the negativeside, does not go to 0 with depth N if the residual functions are not smooth withdepth.
Revised: Our bound is tight and, on the negative side, does not go to 0 with depth N if the residual functions are not smooth with depth.

Original: On the positive side, we show that this smoothness is preserved by gradientdescent for a ResNet with linear residual functions and small enough initial loss.
Revised: On the positive side, we show that this smoothness is preserved by gradient descent for a ResNet with linear residual functions and small enough initial loss.

Original: Itensures an implicit regularization towards a limit Neural ODE at rate 1 N , uniformlywith depth and optimization time.
Revised: It ensures an implicit regularization towards a limit Neural ODE at rate 1 N , uniformly with depth and optimization time.

Original: As a byproduct of our analysis, we consider theuse of a memory-free discrete adjoint method to train a ResNet by recovering theactivations on the fly through a backward pass of the network, and show that thismethod theoretically succeeds at large depth if the residual functions are Lipschitzwith the input.
Revised: As a byproduct of our analysis, we consider the use of a memory-free discrete adjoint method to train a ResNet by recovering the activations on the fly through a backward pass of the network, and show that this method theoretically succeeds at large depth if the residual functions are Lipschitz with the input.

Original: We then show that Heun’s method, a second order ODE integrationscheme, allows for better gradient estimation with the adjoint method when theresidual functions are smooth with depth.
Revised: We then show that Heun’s method, a second order ODE integration scheme, allows for better gradient estimation with the adjoint method when the residual functions are smooth with depth.

Original: We experimentally validate that ouradjoint method succeeds at large depth, and that Heun’s method needs fewer layersto succeed.
Revised: We experimentally validate that our adjoint method succeeds at large depth, and that Heun’s method needs fewer layers to succeed.

Original: We finally use the adjoint method successfully for fine-tuning verydeep ResNets without memory consumption in the residual layers.
Revised: We finally use the adjoint method successfully for fine-tuning very deep ResNets without memory consumption in the residual layers.

Original:  Introduction
Revised:  Introduction

Original: Problem setup Residual Neural Networks (ResNets) [20, 21] keep on outperforming state of the artin computer vision [45, 6], and more generally skip connections are widely used in a various range ofapplications [41, 14].
Revised: Problem setup. Residual Neural Networks (ResNets) [20, 21] keep on outperforming state of the art in computer vision [46, 6], and more generally skip connections are widely used in a various range of applications [42, 14].

Original: A ResNet of depth N iterates, starting from x 0 ∈ R d , x n +1 = x n + f ( x n , θ Nn )and outputs a final value x N ∈ R d where f is a neural network called residual function.
Revised: A ResNet of depth N iterates, starting from x 0 ∈ R d , x n +1 = x n + f ( x n , θ Nn ) and outputs a final value x N ∈ R d where f is a neural network called residual function.

Original: In this work,we consider a simple modification of this forward rule by letting explicitly the residual mappingdepend on the depth of the network:
Revised: In this work, we consider a simple modification of this forward rule by letting explicitly the residual mapping depend on the depth of the network:

Original: [Equation]
Revised: [Equation]

Original: On the other hand, a Neural ODE [8] uses a neural network φ Θ ( x, s ) , that takes time s into account,to parameterise a vector field [24] in a differential equation, as follows,
Revised: On the other hand, a Neural ODE [8] uses a neural network φ Θ ( x, s ) , that takes time s into account, to parameterise a vector field [24] in a differential equation, as follows,

Original: [Equation]
Revised: [Equation]

Original:  with x
Revised: ) with x

Original: [Equation] 
Revised: [Equation] =

Revised: x

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) and outputs a final value x (1) ∈ R d , the solution of Eq.(2).
Revised: and outputs a final value x (1) ∈ R d , the solution of Eq.(2).

Original: The Neural ODE framework enables learning without storing activations (the x n ’s) using the adjoint state method, hence significantly reducingthe memory usage for backpropagation that can be a bottleneck during training [42, 32, 47, 16].
Revised: The Neural ODE framework enables learning without storing activations (the x n ’s) using the adjoint state method, hence significantly reducing the memory usage for backpropagation that can be a bottleneck during training [43, 34, 48, 16].

Original: Input of the network
Revised: Input of the network

Original: N u m b e r o f l a y e r s
Revised: N u m b e r o f l a y e r s

Original: Neural ODE unlike
Revised: Neural ODE unlike

Original: Input of the network
Revised: Input of the network

Original: N u m b e r o f l a y e r s
Revised: N u m b e r o f l a y e r s

Original: Neural ODE like
Revised: Neural ODE like

Original: Input of the network
Revised: Input of the network

Original: N u m b e r o f l a y e r s
Revised: N u m b e r o f l a y e r s

Original: NeuralODE unlike
Revised: Neural unlike

Original: Input of the network
Revised: Input of the network

Original: N u m b e r o f l a y e r s
Revised: N u m b e r o f l a y e r s

Original: Neural ODEs also provide a theoretical frameworkto study deep learning models from the continuous viewpoint, using the arsenal of ODE theory[39, 25, 40].
Revised: Neural ODEs also provide a theoretical framework to study deep learning models from the continuous viewpoint, using the arsenal of ODE theory [40, 25, 41].

Original: Importantly, they can also be seenas the continuous analog of ResNets.
Revised: Importantly, they can also be seen as the continuous analog of ResNets.

Original: Indeed, consider for N an integer, the Euler scheme for solving
Revised: Indeed, consider for N an integer, the Euler scheme for solving

Original: Eq.
Revised: Eq.

Original: (2) with time step 1 N starting from x 0 and iterating x n +1 = x n + 1 N φ Θ ( x n , nN ) .
Revised: (2) with time step 1 N starting from x 0 and iterating x n +1 = x n + 1 N φ Θ ( x n , nN ) .

Original: Under mild assumptions on φ Θ , this scheme is known to convergeto the true solution of Eq.
Revised: Under mild assumptions on φ Θ , this scheme is known to converge to the true solution of Eq.

Original: (2) as N goes to + ∞ .
Revised: (2) as N goes to + ∞ .

Original: Also,if Θ = ( θ Nn ) i ∈
Revised: Also, if Θ = ( θ Nn ) i ∈

Original: [ N − 1] and φ Θ ( ., nN ) =
Revised: [ N − 1] and φ Θ ( ., nN ) =

Original: f ( ., θ Nn ) , thenthe ResNet equation Eq.
Revised: f ( ., θ Nn ) , then the ResNet equation Eq.

Original: (1) corresponds to a Eulerdiscretization with time step 1 N of Eq.
Revised: (1) corresponds to a Euler discretization with time step 1 N of Eq.

Original: However,for a given ResNet with fixed depth N and weights,the activations in Eq.
Revised: However, for a given ResNet with fixed depth N and weights, the activations in Eq.

Original: (1) can be far from the solutionof Eq.
Revised: (1) can be far from the solution of Eq.

Original: This is illustrated in Figure 1 where weshow that a deep ResNet can easily break the topology of the input space, which is impossible fora Neural ODE.
Revised: This is illustrated in Figure 1 where we show that a deep ResNet can easily break the topology of the input space, which is impossible for a Neural ODE.

Original: In this paper, we study the link between ResNets and Neural ODEs.
Revised: In this paper, we study the link between ResNets and Neural ODEs.

Original: We make thefollowing contributions:
Revised: We make the following contributions:

Original: • In Section 3, we propose a framework to define a set of associated Neural ODEs for a given ResNet.
Revised: • In Section 3, we propose a framework to define a set of associated Neural ODEs for a given ResNet.

Original: We control the error between the discrete and the continuous trajectory.
Revised: We control the error between the discrete and the continuous trajectory.

Original: We show that withoutadditional assumptions on the smoothness with depth of the residual functions, this error does notgo to 0 as N → ∞ (Prop. 1).
Revised: We show that without additional assumptions on the smoothness with depth of the residual functions, this error does not go to 0 as N → ∞ (Prop. 1).

Original: However, we show that under some assumptions on the weightinitialization, the trained parameters of a deep linear ResNet uniformly (with respect to both depthand training time) approach a Lipschitz function as the depth N of the network goes to infinity,at speed 1 N (Prop. 2 and Th. 1).
Revised: However, we show that under some assumptions on the weight initialization, the trained parameters of a deep linear ResNet uniformly (with respect to both depth and training time) approach a Lipschitz function as the depth N of the network goes to infinity, at speed 1 N (Prop. 2 and Th. 1).

Original: This result highlights an implicit regularization towards a limitNeural ODE.•
Revised: This result highlights an implicit regularization towards a limit Neural ODE.

Original:  In Section 4, we investigate a simple technique to train ResNets without storing activations.
Revised: • In Section 4, we investigate a simple technique to train ResNets without storing activations.

Original: Inspiredby the adjoint method, we propose to recover the approximated activations during the backward pass by using a reverse-time Euler scheme.
Revised: Inspired by the adjoint method, we propose to recover the approximated activations during the backward pass by using a reverse-time Euler scheme.

Original: We control the error for recovering the activations andgradients with this method.
Revised: We control the error for recovering the activations and gradients with this method.

Original: We show that if the residuals of the ResNet are bounded and Lipschitzcontinuous, with constants independent of N , then this error scales in O ( 1 N ) (Prop. 3).
Revised: We show that if the residuals of the ResNet are bounded and Lipschitz continuous, with constants independent of N , then this error scales in O ( 1 N ) (Prop. 3).

Original: Hence,the adjoint method needs a large number of layers to lead to correct gradients (Prop. 4).
Revised: Hence, the adjoint method needs a large number of layers to lead to correct gradients (Prop. 4).

Original: We thenconsider a smoothness-dependent reconstruction with Heun’s method to bound the error betweenthe true and approximated gradient by a term that depends on 1 N times the smoothness in depthof the residual functions, hence guaranteeing a better approximation when successive weights areclose one to another (Prop. 5 and 6).
Revised: We then consider a smoothness-dependent reconstruction with Heun’s method to bound the error between the true and approximated gradient by a term that depends on 1 N times the smoothness in depth of the residual functions, hence guaranteeing a better approximation when successive weights are close one to another (Prop. 5 and 6).

Original: • In Section 5, on the experimental side, we show that the adjoint method fails when training aResNet 101 on ImageNet.
Revised: • In Section 5, on the experimental side, we show that the adjoint method fails when training a ResNet 101 on ImageNet.

Original: Nevertheless, we empirically show that very deep ResNets pretrainedwith tied weights (constant weights: θ Nn = θ ∀ n ) can be refined -using our adjoint method- onCIFAR-10 and ImageNet by untying their weights, leading to a better test accuracy.
Revised: Nevertheless, we empirically show that very deep ResNets pretrained with tied weights (constant weights: θ Nn = θ ∀ n ) can be refined -using our adjoint method- on CIFAR-10 and ImageNet by untying their weights, leading to a better test accuracy.

Original: Last, but notleast, we show using a ResNet architecture with heavy downsampling in the first layer that ouradjoint method succeeds at large depth and that Heun’s method leads to a better behaved training,hence confirming our theoretical results.
Revised: Last, but not least, we show using a ResNet architecture with heavy downsampling in the first layer that our adjoint method succeeds at large depth and that Heun’s method leads to a better behaved training, hence confirming our theoretical results.

Original:  Background and related work
Revised:  Background and related work

Original: Neural ODEs.
Revised: Neural ODEs.

Original: Neural ODEs are a class of implicit deep learning models defined by an ODEwhere a neural network parameterises the vector field [43, 8,  39, 38, 44, 29, 34, 24].
Revised: Neural ODEs are a class of implicit deep learning models defined by an ODE where a neural network parameterises the vector field [44, 8, 40, 39, 45, 29, 36, 24].

Original: Given an inputx 0 , the output of the model is the solution of the ODE (2) at time 1 .
Revised: Given an input x 0 , the output of the model is the solution of the ODE (2) at time 1 .

Original: From a theoretical viewpoint,the expression capabilities of Neural ODEs have been investigated in [12, 40, 25] and the Neural
Revised: From a theoretical viewpoint, the expression capabilities of Neural ODEs have been investigated in [12, 41, 25] and the Neural

Original: ODE framework has been used to better understand the dynamics of more general architecturesthat include residual connections such as Transformers [36, 27].
Revised: ODE framework has been used to better understand the dynamics of more general architectures that include residual connections such as Transformers [38, 27].

Original: Experimentaly, Neural ODEs havebeen successful in a various range of applications, among which physical modelling [18, 11] andgenerative modeling [8, 17].
Revised: Experimentaly, Neural ODEs have been successful in a various range of applications, among which physical modelling [18, 11] and generative modeling [8, 17].

Original: However, there are many areas where Neural ODEs have failed to replace ResNets, for instance for building computer vision classification models.
Revised: However, there are many areas where Neural ODEs have failed to replace ResNets, for instance for building computer vision classification models.

Original: Neural ODEs fail tocompete with ResNets on ImageNet, and to the best of our knowledge, previous works using Neural 
Revised: Neural ODEs fail to compete with ResNets on ImageNet, and to the best of our knowledge, previous works using Neural ODEs on

Original: ODEs on ImageNet consider weight-tied architectures and only achieves the same accuracy as a ResNet18 [49].
Revised: ImageNet consider weight-tied architectures and only achieves the same accuracy as a ResNet18 [50].

Revised: It has also been shown that Neural ODEs sometimes do not admit a continuous interpretation at all [32].

Original: Implicit Regularization of ResNets towards ODEs.
Revised: Implicit Regularization of ResNets towards ODEs.

Original: Recent works have studied the link between
Revised: Recent works have studied the link between

Original: ResNets and Neural ODEs.
Revised: ResNets and Neural ODEs.

Original: In [9], the authors carry experiments to better understand the scalingbehavior of weights in ResNets as a function of the depth.
Revised: In [9], the authors carry experiments to better understand the scaling behavior of weights in ResNets as a function of the depth.

Original: They show that under the assumption thatthere exists a scaling limit θ ( s ) =
Revised: They show that under the assumption that there exists a scaling limit θ ( s ) =

Original: N β lim θ N ⌊ Ns ⌋ for the weights of the ResNets (with 0 < β < 1 )and if the scale of the ResNet is 1 N α with 0 < α < 1 and α
Revised: N β lim θ N ⌊ Ns ⌋ for the weights of the ResNets (with 0 < β < 1 ) and if the scale of the ResNet is 1 N α with 0 < α < 1 and α

Original: + β = 1 , then the hidden state of the
Revised: + β = 1 , then the hidden state of the

Original: ResNet converges to a solution of a linear ODE.
Revised: ResNet converges to a solution of a linear ODE.

Original: In this paper, we are interested in the case whereα = 1 , which seems more natural since it is the scaling that appears in Euler’s method with step1 N .
Revised: In this paper, we are interested in the case where α = 1 , which seems more natural since it is the scaling that appears in Euler’s method with step N .

Original: In addition, we do not assume the existence of a scaling limit θ ( s ) =
Revised: In addition, we do not assume the existence of a scaling limit θ ( s ) =

Original: lim
Revised: lim

Original: θ N
Revised: θ N

Original: ⌊ Ns ⌋ .
Revised: ⌊ Ns ⌋ .

Original: In subsection3.2, we demonstrate the existence of this scaling limit in the linear setting, under some assumptions.
Revised: In subsection 3.2, we demonstrate the existence of this scaling limit in the linear setting, under some assumptions.

Original: The recent work [10] shows results regarding linear convergence of gradient descent in ResNets andprove the existence of an 12 -Hölder continuous scaling limit as N → ∞ with a scaling factor forthe residuals in 1 √ N which is different from ours.
Revised: The recent work [10] shows results regarding linear convergence of gradient descent in ResNets and prove the existence of an 12 -Hölder continuous scaling limit as N → ∞ with a scaling factor for the residuals in 1 √ N which is different from ours ( 1 N ).

Original: In contrast, we show that our limit function is Lipschitz continuous, which is a stronger regularity.
Revised: In contrast, we show that our limit function is Lipschitz continuous, which is a stronger regularity.

Original: We also show that our convergence is uniform indepth and optimization time.
Revised: This is to be linked with the recent work of [31], where the authors show that whereas the 1 √ N scaling corresponds to the proper one for standard i.i.d. initializations, 1 N is the proper scaling for smooth initialization to obtain non-trivial behaviour (other choices lead to explosion or to identity [31]).

Original: More generally, recent works have proved the convergence of gradientdescent training of ResNet when the initial loss is small enough.
Revised: More generally, recent works have proved the convergence of gradient descent training of ResNet when the initial loss is small enough.

Original: This include ResNet with finitewidth but arbitrary large depth [15, 26] and ResNet with both infinite width and depth [28, 3].
Revised: This include ResNet with finite width but arbitrary large depth [15, 26] and ResNet with both infinite width and depth [28, 3].

Original: Theseconvergence proofs leverage an implicit bias toward weights with small amplitudes.
Revised: These convergence proofs leverage an implicit bias toward weights with small amplitudes.

Original: They howeverleave open the question of convergence of individual weights as depth increases, which we tackle inthis work in the linear case.
Revised: They however leave open the question of convergence of individual weights as depth increases, which we tackle in this work in the linear case.

Original: This requires showing an extra bias toward weights with small variationsacross depth.
Revised: This requires showing an extra bias toward weights with small variations across depth.

Original: Memory bottleneck in ResNets.
Revised: Memory bottleneck in ResNets.

Original: Training deep learning models involve graphics processing units(GPUs) where memory is a practical bottleneck [42, 32, 47].
Revised: Training deep learning models involve graphics processing units (GPUs) where memory is a practical bottleneck [43, 34, 48].

Original: Indeed, backpropagation requires tostore activations at each layer during the forward pass.
Revised: Indeed, backpropagation requires to store activations at each layer during the forward pass.

Original: Since samples are processed using minibatches, this storage can be important.
Revised: Since samples are processed using mini batches, this storage can be important.

Original: For instance, with batches of size 128, the memory needed tocompute gradients for a ResNet 152 on ImageNet is about 22 GiB. Note that the memory neededto store the parameters of the model is only 220 MiB, which is negligible compared to the memoryneeded to store the activations.
Revised: For instance, with batches of size 128, the memory needed to compute gradients for a ResNet 152 on ImageNet is about 22 GiB. Note that the memory needed to store the parameters of the model is only 220 MiB, which is negligible compared to the memory needed to store the activations.

Original: Thus, designing deep invertible architectures where one can recoverthe activations on the fly during the backpropagation iterations has been an active field in recent years[16, 37, 23].
Revised: Thus, designing deep invertible architectures where one can recover the activations on the fly during the backpropagation iterations has been an active field in recent years [16, 37, 23].

Original: In this work, we propose to approximate activations using a reverse-time Euler scheme,as we detail in the next subsection.
Revised: In this work, we propose to approximate activations using a reverse-time Euler scheme, as we detail in the next subsection.

Original: Adjoint Method.
Revised: Adjoint Method.

Original: Consider a loss function L ( x N ) for the ResNet (1).
Revised: Consider a loss function L ( x N ) for the ResNet (1).

Original: The backpropagationequations [5] are
Revised: The backpropagation equations [5] are

Original: [Equation]
Revised: [Equation]

Original: Now, consider a loss function L ( x (1)) for the Neural ODE (2).
Revised: Now, consider a loss function L ( x (1)) for the Neural ODE (2).

Original: The adjoint state method [33, 8] gives
Revised: The adjoint state method [35, 8] gives

Original: [Equation]
Revised: [Equation]

Original: Note that if Θ = ( θ Nn ) n ∈ [ N − 1] and φ Θ ( ., nN ) =
Revised: Note that if Θ = ( θ Nn ) n ∈ [ N − 1] and φ Θ ( ., nN ) =

Original: f ( ., θ Nn ) , then Eq.
Revised: f ( ., θ Nn ) , then Eq.

Original: (3) corresponds to a Eulerdiscretization with time step 1 N of Eq.
Revised: (3) corresponds to a Euler discretization with time step 1 N of Eq.

Original: The key advantage of using Eq.
Revised: The key advantage of using Eq.

Original: (4) is that one can recoverx ( s ) on the fly by solving the Neural ODE (2) backward in time starting from x (1) .
Revised: (4) is that one can recover x ( s ) on the fly by solving the Neural ODE (2) backward in time starting from x (1) .

Original: This strategyavoids storing the forward trajectory ( x ( s )) s ∈ [0 , 1] and leads to a O (1) memory footprint [8].
Revised: This strategy avoids storing the forward trajectory ( x ( s )) s ∈ [0 , 1] and leads to a O (1) memory footprint [8].

Original: Inthis work, we propose to use a discrete adjoint method by using a reverse-time Euler scheme forapproximately recovering the activations in a ResNet (Section 4).
Revised: In this work, we propose to use a discrete adjoint method by using a reverse-time Euler scheme for approximately recovering the activations in a ResNet (Section 4).

Original: Contrarily to other models such as
Revised: Contrarily to other models such as

Original: RevNets [16] (architecture change) or Momentum ResNets [35] (forward rule modification) whichrely on an exactly invertible forward rule, the proposed method requires no change at all in thenetwork, but gives approximate gradients.
Revised: RevNets [16] (architecture change) or Momentum ResNets [37] (forward rule modification) which rely on an exactly invertible forward rule, the proposed method requires no change at all in the network, but gives approximate gradients.

Original: Notations.
Revised: Notations.

Original: For k ∈ N , C k is the set of functions f : R d → R d k times differentiable with continuousk th differential.
Revised: For k ∈ N , C k is the set of functions f : R d → R d k times differentiable with continuous k th differential.

Original: If f ∈ C 1 , ∂ x f ( x )[ y ] is the differential of f at x evaluated in y .
Revised: If f ∈ C 1 , ∂ x f ( x )[ y ] is the differential of f at x evaluated in y .

Original: For K ⊂ R d compact,∥ .
Revised: For K ⊂ R d compact, ∥ .

Original: ∥ a norm and f a continuous function on R d , we denote ∥ f ∥ K ∞ = sup x ∈ K ∥
Revised: ∥ a norm and f a continuous function on R d , we denote ∥ f ∥ K ∞ = sup x ∈ K ∥

Original: f ( x ) ∥ .
Revised: f ( x ) ∥ .

Original:  ResNets as discretization of Neural ODEs
Revised:  ResNets as discretization of Neural ODEs

Original: In this section we first show that without further assumptions, the distance between the discretetrajectory and the solution of associated ODEs can be constant with respect to the depth of thenetwork if the residual functions lack smoothness with depth.
Revised: In this section we first show that without further assumptions, the distance between the discrete trajectory and the solution of associated ODEs can be constant with respect to the depth of the network if the residual functions lack smoothness with depth.

Original: We then present a positive result bystudying the linear case where we show that, under some hypothesis (small loss initialization andinitial smoothness with depth), the ResNet converges to a Neural ODE as the number of layers goesto infinity.
Revised: We then present a positive result by studying the linear case where we show that, under some hypothesis (small loss initialization and initial smoothness with depth), the ResNet converges to a Neural ODE as the number of layers goes to infinity.

Original: We show that this convergence is uniform with depth and optimization time.
Revised: We show that this convergence is uniform with depth and optimization time.

Original: 3.1 Distance to an ODE
Revised: 3.1 Distance to an ODE

Original: We first define associated Neural ODEs for a given ResNet.
Revised: We first define associated Neural ODEs for a given ResNet.

Original: Definition 1.
Revised: Definition 1.

Original: We say that a neural network φ Θ : R d × R → R d smoothly interpolates the ResNet
Revised: We say that a neural network φ Θ : R d × R → R d smoothly interpolates the ResNet

Original: Eq.
Revised: Eq.

Original: if φ Θ is smooth and ∀ n ∈ { 0 , ..., N − 1 } , φ Θ ( ., nN ) =
Revised: if φ Θ is smooth and ∀ n ∈ { 0 , ..., N − 1 } , φ Θ ( ., n N ) =

Original: f ( ., θ Nn ) .
Revised: f ( ., θ N n ) .

Original: Note that we omit the dependency of Θ in N to simplify notations.
Revised: Note that we omit the dependency of Θ in N to simplify notations.

Original: For example, for a given
Revised: For example, for a given

Original: ResNet, there are two natural ways to interpolate it with a Neural ODE, either by interpolating theresiduals , or by interpolating the weights .
Revised: ResNet, there are two natural ways to interpolate it with a Neural ODE, either by interpolating the residuals , or by interpolating the weights .

Original: Indeed, one can interpolate the residuals with φ Θ ( · , s ) =( n + 1 − Ns ) f ( ., θ Nn ) + ( Ns − n ) f ( · , θ Nn +1 ) when s ∈ [ nN , n +1 N ] , or interpolate the weights withφ Θ ( · , s )=
Revised: Indeed, one can interpolate the residuals with φ Θ ( · , s ) = ( n + 1 − Ns ) f ( ., θ Nn ) + ( Ns − n ) f ( · , θ Nn +1 ) when s ∈ [ nN , n +1 N ] , or interpolate the weights with φ Θ ( · , s )

Revised: =

Original: f ( · , ( n +1 − Ns ) θ N n +( Ns − n ) θ Nn +1 ) for s ∈ [ nN , n +1 N ] .
Revised: f ( · , ( n +1 − Ns ) θ Nn +( Ns − n ) θ Nn +1 ) for s ∈ [ nN , n +1 N ] .

Original: If θ Nn = θ N does not depend onn , then both interpolations are identical and one can simply consider φ Θ ( x, s ) =
Revised: If θ Nn = θ N does not depend on n , then both interpolations are identical and one can simply consider φ Θ ( x, s ) =

Original: f ( x, θ N ) , ∀ ( x, s ) .
Revised: f ( x, θ N ) , ∀ ( x, s ) .

Original: We now consider any smooth interpolation φ Θ for the ResNet (1) and a Euler scheme for the Neural
Revised: We now consider any smooth interpolation φ Θ for the ResNet (1) and a Euler scheme for the Neural

Original: ODE (2) with time step 1 N .
Revised: ODE (2) with time step 1 N .

Original: Proposition 1 (Approximation error) .
Revised: Proposition 1 (Approximation error) .

Original: We suppose that φ Θ is C 1 , and L -Lipschitz with respect to x ,uniformly in s .
Revised: We suppose that φ Θ is C 1 , and L -Lipschitz with respect to x , uniformly in s .

Original: Note that this implies that the solution of Eq.
Revised: Note that this implies that the solution of Eq.

Original: (2) is well defined, unique, C 2 , and thatthe trajectory is included in some compact K ⊂ R d .
Revised: (2) is well defined, unique, C 2 , and that the trajectory is included in some compact K ⊂ R d .

Original: Denote C N := ∥ ∂
Revised: Denote C N := ∥ ∂

Original: s φ Θ + ∂ x φ Θ [ φ Θ ] ∥ K × [0 , 1] ∞ .
Revised: s φ Θ + ∂ x φ Θ [ φ Θ ] ∥ K × [0 , 1] ∞ .

Original: Then one has for all n : ∥ x n − x ( nN ) ∥
Revised: Then one has for all n : ∥ x n − x ( nN ) ∥

Original: ≤ e L − 1 2 NL C N if L > 0 and ∥ x n − x ( nN )
Revised: ≤ e L − 1 2 NL C N if L > 0 and ∥ x n − x ( nN )

Original: ∥ ≤ C N 2 N if L = 0 .
Revised: ∥ ≤ C N 2 N if L = 0 .

Original: For a full proof, see appendix A.1.
Revised: For a full proof, see appendix A.1.

Original: Note that this result extends Theorem 3.2 from [48] to the nonautonomous case: our bound depends on ∂ s φ Θ .
Revised: Note that this result extends Theorem 3.2 from [49] to the nonautonomous case: our bound depends on ∂ s φ Θ .

Original: Finally, our bound is tight.
Revised: Finally, our bound is tight.

Original: Indeed, for φ Θ ( x, s ) =
Revised: Indeed, for φ Θ ( x, s ) =

Original: as + b for a, b ∈ R d , we get ∂
Revised: as + b for a, b ∈ R d , we get ∂

Original: s φ Θ + ∂ x φ Θ [ φ Θ ] = a , L = 0 and ∥ x (1) − x N ∥ =
Revised: s φ Θ + ∂ x φ Θ [ φ Θ ] = a , L = 0 and ∥ x (1) − x N ∥ =

Original: ∥ a ∥ 2 N .
Revised: ∥ a ∥ 2 N .

Original: Implication.
Revised: Implication.

Original: The tightness of our bound shows that closeness to the ODE solution is not guaranteed, because we do not know whether C N /N → 0 .
Revised: The tightness of our bound shows that closeness to the ODE solution is not guaranteed, because we do not know whether C N /N → 0 .

Original: Indeed, consider first the residual interpolation φ Θ ( x, s ) = (cid:0) ( n + 1 − Ns ) f
Revised: Indeed, consider first the residual interpolation φ Θ ( x, s ) = (cid:0) ( n + 1 − Ns ) f

Original: ( x, θ Nn )
Revised: ( x, θ Nn )

Original: + ( Ns − n ) f
Revised: + ( Ns − n ) f

Original: ( x, θ Nn +1 ) (cid:1) )
Revised: ( x, θ Nn +1 ) (cid:1) )

Original: 1 s ∈ [ nN , n +1 N ] and the simplecase where ∂ x φ Θ [ φ Θ ] = 0 .
Revised: 1 s ∈ [ nN , n +1 N ] and the simple case where ∂ x φ Θ [ φ Θ ] = 0 .

Original: We get ∂
Revised: We get ∂

Original: s φ Θ ( x, s ) =
Revised: s φ Θ ( x, s ) =

Original: N ( f ( x, θ Nn +1 ) − f ( x, θ Nn ))
Revised: N ( f ( x, θ Nn +1 ) − f ( x, θ N n ))

Original: 1 s ∈ [ nN , n +1 N ] , whichcorresponds to the discrete derivative.
Revised: 1 s ∈ [ nN , n +1 N ] , which corresponds to the discrete derivative.

Original: It means that although there is a 1 N factor in our bound, thetime derivative term – without further regularity with depth of the weights, which is at the heart ofsubection 3.2 – usually scales with N : C N N =
Revised: It means that although there is a 1 N factor in our bound, the time derivative term – without further regularity with depth of the weights, which is at the heart of subection 3.2 – usually scales with N : C N N =

Original: O (1) .
Revised: O (1) .

Original: As a first example, consider the simple casewhere f ( x, θ N n ) 
Revised: As a first example, consider the simple case where f ( x, θ Nn ) =

Original: = n.
Revised: n.

Original: This gives x N = x 0 + ( N − 1)while the integration of the Neural ODE (2)leads to x (1) = x 0 + N 2 because φ Θ ( x, s ) =
Revised: This gives x N = x 0 + ( N − 1)while the integration of the Neural ODE (2) leads to x (1) = x 0 + N 2 because φ Θ ( x, s ) =

Original: Ns , so the ∥ x N − x (1) ∥ = 12 is not small.
Revised: Ns , so the ∥ x N − x (1) ∥ = 12 is not small.

Original: Intuitively,this shows that weights cannot scale with depth when using the residual interpolation.
Revised: Intuitively, this shows that weights cannot scale with depth when using the residual interpolation.

Original: Now, consider the weight interpolation, θ Nn = ( − 1) n and suppose f is written as f ( x, θ )
Revised: Now, consider the weight interpolation, θ Nn = ( − 1) n and suppose f is written as f ( x, θ )

Original: =
Revised: =

Original: θ 2 .
Revised: θ 2 .

Original: This givesφ Θ ( ., s ) 
Revised: This gives φ Θ ( ., s ) =

Original: = (2 Ns − (2 n + 1)) 2 when s ∈ [ nN , n +1 N ] .
Revised: (2 Ns − (2 n + 1)) 2 when s ∈ [ nN , n +1 N ] .

Original: Integrating, we get x (1) =
Revised: Integrating, we get x (1) =

Original: while the outputof the ResNet is x N = 1 .
Revised: while the output of the ResNet is x N = 1 .

Original: Hence ∥ x N − x (1) ∥ = 23 is also not small, even though the weights arebounded.
Revised: Hence ∥ x N − x (1) ∥ = 23 is also not small, even though the weights are bounded.

Original: Thus, one needs additional regularity assumptions on the weights of the ResNet to obtain a
Revised: Thus, one needs additional regularity assumptions on the weights of the ResNet to obtain a

Original: Neural ODE in the large depth limit.
Revised: Neural ODE in the large depth limit.

Original: Intuitively if the weights are initialized close from one anotherand they are updated using gradient descent, they should stay close from one another during training,since the gradients in two consecutive layers will be similar, as highlighted in Eq.
Revised: Intuitively if the weights are initialized close from one another and they are updated using gradient descent, they should stay close from one another during training, since the gradients in two consecutive layers will be similar, as highlighted in Eq.

Original: Indeed, wesee that if x n and x n +1 are close, then ∇ x n L and ∇ x n +1 L are close, and then if θ Nn and θ Nn +1 areclose, ∇ θ Nn L and ∇ θ Nn +1 L are also close.
Revised: Indeed, we see that if x n and x n +1 are close, then ∇ x n L and ∇ x n +1 L are close, and then if θ Nn and θ Nn +1 are close, ∇ θ Nn L and ∇ θ Nn +1 L are also close.

Original: In subsection 3.2, we formalize this intuition and present apositive result for ResNets with linear residual functions.
Revised: In subsection 3.2, we formalize this intuition and present a positive result for ResNets with linear residual functions.

Original: More precisely, we show that with properinitialization, the difference between two successive parameters is in 1 N during the entire training.
Revised: More precisely, we show that with proper initialization, the difference between two successive parameters is in 1 N during the entire training.

Original: Furthermore, we show that the weights of the network converge to a smooth function, hence defininga limit Neural ODE.
Revised: Furthermore, we show that the weights of the network converge to a smooth function, hence defining a limit Neural ODE.

Original: 3.2 Linear Case
Revised: 3.2 Linear Case

Original: As a further step towards a theoretical understanding of the connections between ResNets and Neural
Revised: As a further step towards a theoretical understanding of the connections between ResNets and Neural

Original: ODEs we investigate the linear setting, where the residual functions are written f ( x, θ ) =
Revised: ODEs we investigate the linear setting, where the residual functions are written f ( x, θ ) =

Original: θx for anyθ ∈ R d × d .
Revised: θx for any θ ∈ R d × d .

Original: It corresponds to a deep matrix factorization problem [50, 4, 2, 1].
Revised: It corresponds to a deep matrix factorization problem [51, 4, 2, 1].

Original: As opposed to theseprevious works, we study the infinite depth limit of these linear ResNets with a focus on the learnedweights.
Revised: As opposed to these previous works, we study the infinite depth limit of these linear ResNets with a focus on the learned weights.

Original: We show that, if the weights are initialized close one to another, then at any training time,the weights stay close one to another (Prop. 2) and importantly, they converge to a smooth functionof the continuous depth s as N → ∞ (Th. 1).
Revised: We show that, if the weights are initialized close one to another, then at any training time, the weights stay close one to another (Prop. 2) and importantly, they converge to a smooth function of the continuous depth s as N → ∞ (Th. 1).

Original: All the proofs are available in appendix A.
Revised: All the proofs are available in appendix A.

Original: Setting.
Revised: Setting.

Original: Given a training set ( x k , y k ) k ∈ [ n ] in R d , we solve the regression problem of mapping x kto y k with a linear ResNet, i.e. f ( x, θ ) = θx , of depth N and parameters ( θ N 1 , . . .
Revised: Given a training set ( x k , y k ) k ∈ [ n ] in R d , we solve the regression problem of mapping x k to y k with a linear ResNet, i.e. f ( x, θ ) = θx , of depth N and parameters ( θ N 1 , . . .

Original: , θ NN ) .
Revised: , θ NN ) .

Original: The ResNettherefore maps x k to Π N x kwhere Π N :=
Revised: The ResNet therefore maps x k to Π N x k

Original:  (cid:81) Nn =

Revised: where Π N := (cid:81) N n =

Original: ( I d + θ Nn N ) =
Revised: ( I d + θ Nn N ) =

Original: ( I d + θ NN N ) · · · ( I d + θ N 1 N ) .
Revised: ( I d + θ NN N ) · · · ( I d + θ N 1 N ) .

Original: It is trainedby minimizing the average errors ∥ Π N x k − y k ∥ 22 , which is equivalent to the deep matrix factorizationproblem:
Revised: It is trained by minimizing the average errors ∥ Π N x k − y k ∥ 22 , which is equivalent to the deep matrix factorization problem:

Original: [Equation]
Revised: [Equation]

Original: (5) where
Revised: (5) where

Original: ∥ A ∥ 2Σ =
Revised: ∥ A ∥ 2Σ =

Original: Tr( A Σ A T ) , Σ is the empirical covariance matrix of the data: Σ := 1 n (cid:80) nk
Revised: Tr( A Σ A T ) , Σ is the empirical covariance matrix of the data: Σ := 1 n (cid:80) nk

Original: =1 x k x ⊤ k ,and B := 1 n (cid:80) nk =1 y k x ⊤ k Σ − 1 .
Revised: =1 x k x ⊤ k , and B := 1 n (cid:80) nk =1 y k x ⊤ k Σ − 1 .

Original: As is standard, we suppose that Σ is non degenerated.
Revised: As is standard, we suppose that Σ is non degenerated.

Original: We denote by
Revised: We denote by

Original: M > 0 (resp. m > 0 ) its largest (resp.
Revised: M > 0 (resp. m > 0 ) its largest (resp.

Original: smallest) eigenvalue.
Revised: smallest) eigenvalue.

Original: Gradient.
Revised: Gradient.

Original: We denote Π N : n :=
Revised: We denote Π N : n :=

Original: ( I d + θ NN N ) · · · ( I d + θ Nn +1 N ) and Π Nn : := ( I d + θ Nn − 1 N ) · · · ( I d + θ N 1 N )and write the gradient ∇ Nn ( t ) =
Revised: ( I d + θ NN N ) · · · ( I d + θ Nn +1 N ) and Π Nn : := ( I d + θ Nn − 1 N ) · · · ( I d + θ N 1 N ) and write the gradient ∇ Nn ( t ) =

Original: ∇ θ Nn L ( θ N 1 ( t ) , ..
Revised: ∇ θ Nn L ( θ N 1 ( t ) , ..

Original: θ Nn ( t ) , .., θ NN ( t )) .
Revised: θ Nn ( t ) , .., θ NN ( t )) .

Original: The chain rule gives N ∇ Nn =Π N ⊤ : n (Π N − B )ΣΠ N ⊤ n : .
Revised: The chain rule gives N ∇ Nn = Π N ⊤ : n (Π N − B )ΣΠ N ⊤ n : .

Original: Intuitively, as N goes to + ∞ , the products Π N , Π N : n and Π Nn : should convergeto some limit, hence we see that N ∇ Nn scale as 1 .
Revised: Intuitively, as N goes to + ∞ , the products Π N , Π N : n and Π Nn : should converge to some limit, hence we see that N ∇ Nn scale as 1 .

Original: Therefore, we train θ Nn by the rescaled gradientflow dθ Nn dt ( t ) =
Revised: Therefore, we train θ Nn by the rescaled gradient flow dθ Nn dt ( t ) =

Original: − N ∇ Nn ( t ) to minimize L and denote ℓ N ( t ) =
Revised: − N ∇ Nn ( t ) to minimize L and denote ℓ N ( t ) =

Original: L ( θ N 1 ( t ) , . .
Revised: L ( θ N 1 ( t ) , . .

Original: , θ NN ( t )) .
Revised: , θ NN ( t )) .

Original: Two continuous variables involved.
Revised: Two continuous variables involved.

Original: Our results involve two continuous variables: s ∈ [0 , 1] is thedepth of the limit network and corresponds to the time variable in the Neural ODE, whereas t ∈ R
Revised: Our results involve two continuous variables: s ∈ [0 , 1] is the depth of the limit network and corresponds to the time variable in the Neural ODE, whereas t ∈ R

Original: +is the gradient flow time variable.
Revised: + is the gradient flow time variable.

Original: As is standard in the analysis of convergence of gradient descentfor linear networks, we consider the following assumption:
Revised: As is standard in the analysis of convergence of gradient descent for linear networks, we consider the following assumption:

Original: Assumption 1.
Revised: Assumption 1.

Original: Suppose that at initialisation one has (cid:112) ℓ N (0) <
Revised: Suppose that at initialisation one has (cid:112) ℓ N (0) <

Original: m 4 √ 2 Me 3 and ∥ θ Nn (0) ∥ ≤ 14 .
Revised: [Equation]

Original: Assumption 1 is the classical assumption in the literature [50, 3] to prove linear convergence of ourloss and that the θ Nn ( t ) ’s stay bounded with t .
Revised: Assumption 1 is the classical assumption in the literature [51, 3] to prove linear convergence of our loss and that the θ Nn ( t ) ’s stay bounded with t .

Original: Note that this bounded norm assumption implies that1 N θ n (0) =
Revised: Note that this bounded norm assumption implies that N θ n (0) =

Original: O ( 1 N ) .
Revised: O ( 1 N ) .

Original: This is in contrast with classical initialization scales in the feedforward case wherethe initialization only depends on width [19].
Revised: This is in contrast with classical initialization scales in the feedforward case where the initialization only depends on width [19].

Original: However this initialization scale is coherent with thoseof ResNets for which the scale has to depend on depth [46].
Revised: However this initialization scale is coherent with those of ResNets for which the scale has to depend on depth [47, 31].

Original: In addition, the experimental findings in[9] suggest that the weights in ResNets scale in 1 N β with β > 0 .
Revised: In addition, the experimental findings in [9] suggest that the weights in ResNets scale in 1 N β with β > 0 .

Original: We now prove an implicit regularization result showing that if at initialization, in addition to assumption 1, the weights are close from one another ( O ( 1 N ) ), they will stay at distance O ( 1 N ) : the discretederivative stay in O ( 1 N ) , which is a central result to consider the infinite depth limit in our Th. 1.
Revised: We now prove an implicit regularization result showing that if at initialization, in addition to assumption 1, the weights are close from one another ( O ( 1 N ) ), they will stay at distance O ( 1 N ) : the discrete derivative stay in O ( 1 N ) , which is a central result to consider the infinite depth limit in our Th. 1.

Original: Proposition 2 (Smoothness in depth of the weights) .
Revised: Proposition 2 (Smoothness in depth of the weights) .

Original: Suppose assumption 1.
Revised: Suppose assumption 1.

Original: Suppose that thereexists C 0 > 0 independent of n and N such that ∥ θ Nn +1 (0) − θ Nn (0) ∥ ≤ C 0 N .
Revised: Suppose that there exists C 0 > 0 independent of n and N such that ∥ θ Nn +1 (0) − θ Nn (0) ∥ ≤ C 0 N .

Original: Then, ∀ t ∈ R + ,∥ θ Nn ( t ) ∥ < 12 , and θ Nn ( t ) admits a limit ψ Nn as t → + ∞ .
Revised: Then, ∀ t ∈ R + , ∥ θ N n ( t ) ∥ < 1 2 , and θ N n ( t ) admits a limit ψ N n as t → + ∞ .

Original: Moreover, there exists C > 0 such that∀ t ∈ R + , ∥ θ Nn +1 ( t ) − θ Nn ( t ) ∥ ≤ CN .
Revised: Moreover, there exists C > 0 such that ∀ t ∈ R + , ∥ θ Nn +1 ( t ) − θ Nn ( t ) ∥ ≤ CN .

Original: For a full proof, see appendix A.2.
Revised: For a full proof, see appendix A.2.

Original: The inequality ∥
Revised: The inequality ∥

Original: θ Nn +1 ( t ) − θ Nn ( t ) ∥ ≤ CN corresponds to a discrete
Revised: θ Nn +1 ( t ) − θ Nn ( t ) ∥ ≤ CN corresponds to a discrete

Original: Lipschitz property in depth.
Revised: Lipschitz property in depth.

Original: Indeed, for s ∈ [0 , 1] and t ∈ R + , let ψ N ( s, t )
Revised: Indeed, for s ∈ [0 , 1] and t ∈ R + , let ψ N ( s, t )

Original: =
Revised: =

Original: θ N ⌊ Ns ⌋ ( t ) .
Revised: θ N ⌊ Ns ⌋ ( t ) .

Original: Thenour result gives ∥ ψ N ( n +1 N , t ) − ψ N ( nN , t ) ∥ ≤ CN which implies that ∥ ψ N ( s 1 , t ) − ψ N ( s 2 , t ) ∥ ≤
Revised: Then our result gives ∥ ψ N ( n +1 N , t ) − ψ N ( nN , t ) ∥ ≤ CN which implies that ∥ ψ N ( s 1 , t ) − ψ N ( s 2 , t ) ∥ ≤

Original: C | s 1 − s 2 | + CN .
Revised: C | s 1 − s 2 | + CN .

Original: We now turn to the infinite depth limit N → ∞ .
Revised: We now turn to the infinite depth limit N → ∞ .

Original: Th.
Revised: Th.

Original: 1 shows that there exists a limitfunction ψ such that ψ N converges uniformly to ψ in depth s and optimization time t .
Revised: 1 shows that there exists a limit function ψ such that ψ N converges uniformly to ψ in depth s and optimization time t .

Original: Furthermore,this limit is Lipschitz continuous in ( s, t ) .
Revised: Furthermore, this limit is Lipschitz continuous in ( s, t ) .

Original: In addition, we show that the ResNet Π N converges to thelimit Neural ODE defined by ψ that is preserved along the optimization flow, exhibiting an implicitregularization property of deep linear ResNets towards Neural ODEs.
Revised: In addition, we show that the ResNet Π N converges to the limit Neural ODE defined by ψ that is preserved along the optimization flow, exhibiting an implicit regularization property of deep linear ResNets towards Neural ODEs.

Original: Theorem 1 (Existence of a limit map) .
Revised: Theorem 1 (Existence of a limit map) .

Original: Suppose assumption 1, ∥ θ Nn +1 (0) − θ Nn (0)
Revised: Suppose assumption 1, ∥ θ Nn +1 (0) − θ Nn (0)

Original: ∥ ≤ C 0 N for some
Revised: ∥ ≤ C 0 N for some

Original: C 0 > 0 and that there exists a function ψ init : [0 , 1]
Revised: C 0 > 0 and that there exists a function ψ init : [0 , 1]

Original: → R d × d such that ψ N ( s, 0) → ψ init ( s ) in ∥ .
Revised: → R d × d such that ψ N ( s, 0) → ψ init ( s ) in ∥ .

Original: ∥ ∞uniformly in s as N → ∞ , at speed 1 N .
Revised: ∥ ∞ uniformly in s as N → ∞ , at speed 1 N .

Original: Then the sequence ( ψ N ) N ∈ N uniformly converges (in ∥ .
Revised: Then the sequence ( ψ N ) N ∈ N uniformly converges (in ∥ .

Original: ∥ ∞w.r.t ( s, t ) ) to a limit ψ Lipschitz continuous in ( s, t ) and ∥ ψ − ψ N ∥ ∞
Revised: ∥ ∞ w.r.t ( s, t ) ) to a limit ψ Lipschitz continuous in ( s, t ) and ∥ ψ − ψ N ∥ ∞

Original: = O ( 1 N ) .
Revised: = O ( 1 N ) .

Original: Furthermore, Π Nuniformly converges as N → ∞ to the mapping x 0 → x 1 where x 1 is the solution at time 1 of the
Revised: Furthermore, Π N uniformly converges as N → ∞ to the mapping x 0 → x 1 where x 1 is the solution at time 1 of the

Original: Neural ODE d x d s ( s ) =
Revised: Neural ODE d x d s ( s ) =

Original: ψ ( s, t ) x
Revised: ψ ( s, t ) x

Original: ( s ) with initial condition x 0 .
Revised: ( s ) with initial condition x 0 .

Original: [Figure]
Revised: [Figure]

Revised: As predicted by Th.

Revised: 1, this distance goes to 0 as

Revised: N → + ∞ .

Original: We illustrate Th.
Revised: We illustrate Th.

Original: 1 in Figure 2.
Revised: 1 in Figure 2.

Original: The assumption onthe existence of ψ init ensures a convergence at speed1 N to a Neural ODE at optimization time 0 .
Revised: The assumption on the existence of ψ init ensures a convergence at speed N to a Neural ODE at optimization time 0 .

Original: Note thatfor instance, the constant initialization θ Nn (0)
Revised: Note that for instance, the constant initialization θ Nn (0)

Original: =
Revised: =

Original: θ 0 ∈
Revised: θ 0 ∈

Original: R d × d satisfies this hypothesis.
Revised: R d × d satisfies this hypothesis.

Original: In order to prove Th.1, for which a full proof is presented in appendix A.5,we first present a useful lemma: the weights of thenetwork have at least one accumulation point.
Revised: In order to prove Th. 1, for which a full proof is presented in appendix A.5, we first present a useful lemma: the weights of the network have at least one accumulation point.

Original: Lemma 1 (Existence of limit functions) .
Revised: Lemma 1 (Existence of limit functions) .

Original: For s ∈[0 , 1] and t ∈ R + , let ψ N ( s, t )
Revised: For s ∈ [0 , 1] and t ∈ R + , let ψ N ( s, t )

Original: =
Revised: =

Original: θ N
Revised: θ N

Original: ⌊ Ns ⌋ ( t ) .
Revised: ⌊ Ns ⌋ ( t ) .

Original: Underthe assumptions of Prop. 2, there exists a subsequence ψ σ ( N ) and ψ σ : [0 , 1] × R + → R d × d Lipschitz continuous with respect to both parameters sand t such that ψ σ ( N ) → ψ σ uniformly (in ∥ .
Revised: Under the assumptions of Prop. 2, there exists a subsequence ψ σ ( N ) and ψ σ : [0 , 1] × R + → R d × d Lipschitz continuous with respect to both parameters s and t such that ψ σ ( N ) → ψ σ uniformly (in ∥ .

Original: ∥ ∞ w.r.t( s, t ) ).
Revised: ∥ ∞ w.r.t ( s, t ) ).

Original: Lemma 1 is proved in appendix A.3, and gives us the existence of a Lipschitz continuous accumulationpoint, but not the uniqueness nor the convergence speed.
Revised: Lemma 1 is proved in appendix A.3, and gives us the existence of a Lipschitz continuous accumulation point, but not the uniqueness nor the convergence speed.

Original: For the uniqueness, we show in appendix
Revised: For the uniqueness, we show in appendix

Original: A.5 that, under the assumptions of Th.
Revised: A.5 that, under the assumptions of Th.

Original: 1, one has that any accumulation point of ψ σ satisfies the limit
Revised: 1, one has that any accumulation point of ψ σ satisfies the limit

Original: Neural ODE
Revised: Neural ODE

Original: [Equation]
Revised: [Equation]

Original: and show that F satisfies the hypothesis of the Picard–Lindelöf theorem, hence showing the256 uniqueness of ψ .
Revised: and show that F satisfies the hypothesis of the Picard–Lindelöf theorem, hence showing the uniqueness of ψ .

Original: We finally show that, as intuitively expected, trajectories of the weights of our linear
Revised: We finally show that, as intuitively expected, trajectories of the weights of our linear

Original: ResNets of depth N and 2 N remain close one to each other.
Revised: ResNets of depth N and 2 N remain close one to each other.

Original: This gives the convergence speed in
Revised: This gives the convergence speed in

Original: Th.
Revised: Th.

Original: See appendix A.4 for a proof.
Revised: See appendix A.4 for a proof.

Original: Lemma 2 (Closeness of trajectories) .
Revised: Lemma 2 (Closeness of trajectories) .

Original: Suppose asumption 1, ∥ θ Nn +1 (0) − θ Nn (0)
Revised: Suppose asumption 1, ∥ θ Nn +1 (0) − θ Nn (0)

Original: ∥ ≤ C 0 N for some
Revised: ∥ ≤ C 0 N for some

Original: C 0 > 0 and that ∥ θ Nn (0) − θ 2 N 2 n (0) ∥
Revised: C 0 > 0 and that ∥ θ Nn (0) − θ 2 N 2 n (0) ∥

Original: =
Revised: =

Original: O ( 1 N ) .
Revised: O ( 1 N ) .

Original: Then ∀ t ∈ R + , ∥ θ Nn ( t ) − θ 2 N 2 n ( t ) ∥
Revised: Then ∀ t ∈ R + , ∥ θ Nn ( t ) − θ 2 N 2 n ( t ) ∥

Original: =
Revised: =

Original: O ( 1 N ) .
Revised: O ( 1 N ) .

Original:  Adjoint Method in Residual Networks
Revised:  Adjoint Method in Residual Networks

Original: In this section, we focus on a particularly useful feature of Neural ODEs and its applicability to
Revised: In this section, we focus on a particularly useful feature of Neural ODEs and its applicability to

Original: ResNets: their memory free backpropagation thanks to the adjoint method.
Revised: ResNets: their memory free backpropagation thanks to the adjoint method.

Original: We consider a ResNet(1) and try to invert it using reverse mode Euler discretization of the Neural ODE (2) when φ Θ isany smooth interpolation of the ResNet.
Revised: We consider a ResNet (1) and try to invert it using reverse mode Euler discretization of the Neural ODE (2) when φ Θ is any smooth interpolation of the ResNet.

Original: This corresponds to defining ˜ x N = x N and iterate forn ∈ { N − 1 , . . . 0 } :
Revised: This corresponds to defining ˜ x N = x N and iterate for n ∈ { N − 1 , . . . 0 } :

Original: [Equation]
Revised: [Equation]

Original: We then use the approximated activations (˜ x n ) n ∈ [ N ] as a proxy for the true activations ( x n ) n ∈ [ N ] tocompute gradients without storing the activations:
Revised: We then use the approximated activations (˜ x n ) n ∈ [ N ] as a proxy for the true activations ( x n ) n ∈ [ N ] to compute gradients without storing the activations:

Original: [Equation]
Revised: [Equation]

Original: The approximate recovery of the activations in Eq.
Revised: The approximate recovery of the activations in Eq.

Original: (6) is implementable for any ResNet: there is noneed for particular architecture or forward rule modification.
Revised: (6) is implementable for any ResNet: there is no need for particular architecture or forward rule modification.

Original: The drawback is that the recovery isonly approximate.
Revised: The drawback is that the recovery is only approximate.

Original: We devote the remainder of the section to the study of the corresponding errors andto error reduction using second order
Revised: We devote the remainder of the section to the study of the corresponding errors and to error reduction using second order

Original: Heun’s method.
Revised: Heun’s method.

Original: We first show that, if f ( ., θ Nn ) and its derivativeare bounded by a constant independent of N , then the error for reconstructing the activations in thebackward scheme (6) is O ( 1 N ) .
Revised: We first show that, if f ( ., θ Nn ) and its derivative are bounded by a constant independent of N , then the error for reconstructing the activations in the backward scheme (6) is O ( 1 N ) .

Original: Proofs of the theoretical results are in appendix A.
Revised: Proofs of the theoretical results are in appendix A.

Original: Error for reconstructing activations.
Revised: Error for reconstructing activations.

Original: We consider the following assumption:
Revised: We consider the following assumption:

Original: Assumption 2.
Revised: Assumption 2.

Original: There exists constants C f and L f such that ∀ N ∈ N , ∀ n ∈
Revised: There exists constants C f and L f such that ∀ N ∈ N , ∀ n ∈

Original: [ N − 1] , ∥ f ( ., θ Nn ) ∥ ∞ ≤
Revised: [ N − 1] , ∥ f ( ., θ Nn ) ∥ ∞ ≤

Original: C f and ∥ ∂ x [ f ( ., θ Nn )] ∥ ∞ ≤ L f .
Revised: C f and ∥ ∂ x [ f ( ., θ Nn )] ∥ ∞ ≤ L f .

Original: Then the error made by reconstructing the activations is in O ( 1 N ) .
Revised: Then the error made by reconstructing the activations is in O ( 1 N ) .

Original: Proposition 3 (Reconstruction error) .
Revised: Proposition 3 (Reconstruction error) .

Original: With assumption 2, one has ∥ x n − ˜ x n ∥ ≤ ( e Lf − 1) C f N + O ( 1 N 2 ) .
Revised: With assumption 2, one has ∥ x n − ˜ x n ∥ ≤ ( e Lf − 1) C f N [Equation]

Original: Prop. 3 shows a slow convergence of the error for recovering activations.
Revised: Prop. 3 shows a slow convergence of the error for recovering activations.

Original: This bound does notdepend on the discrete derivative f ( ., θ Nn +1 )
Revised: This bound does not depend on the discrete derivative f ( ., θ Nn +1 )

Original: − f ( ., θ Nn ) , contrarily to the errors between the ResNetactivations and the trajectory of the interpolating Neural ODE in Prop 1.
Revised: − f ( ., θ Nn ) , contrarily to the errors between the ResNet activations and the trajectory of the interpolating Neural ODE in Prop 1.

Original: In summary, even thoughregularity in depth is necessary to imply closeness to a Neural ODE, it is not necessary to recoveractivations, and neither gradients, as we now show.
Revised: In summary, even though regularity in depth is necessary to imply closeness to a Neural ODE, it is not necessary to recover activations, and neither gradients, as we now show.

Original: Error in gradients when using the adjoint method.
Revised: Error in gradients when using the adjoint method.

Original: We use the result obtained in Prop. 3 toderive a bound in O ( 1 N 2 ) on the error made for computing gradients using formulas (7).
Revised: We use the result obtained in Prop. 3 to derive a bound in O ( 1 N 2 ) on the error made for computing gradients using formulas (7).

Original: Proposition 4 (Gradient error) .
Revised: Proposition 4 (Gradient error) .

Original: Suppose assumption 2.
Revised: Suppose assumption 2.

Original: Suppose in addition that ∂ x f ( ., θ ) admits a
Revised: Suppose in addition that ∂ x f ( ., θ ) admits a

Original: Lipschitz constant L df , ∂ θ f ( ., θ ) admits a Lipschitz constant ∆ , and an upper bound Ω , all of whichare independent of θ .
Revised: Lipschitz constant L df , ∂ θ f ( ., θ ) admits a Lipschitz constant ∆ , and an upper bound Ω , all of which are independent of θ .

Original: Then one has ∥ ˜ ∇ θ Nn L − ∇ θ Nn L ∥ =
Revised: Then one has ∥ ˜ ∇ θ Nn L − ∇ θ Nn L ∥ =

Original: O ( 1 N 2 ) .
Revised: O ( 1 N 2 ) .

Original: For a proof, see appendix A.7, where we give the dependency of our upper bound as a function of∆ , L f , C f , Ω and L df .
Revised: For a proof, see appendix A.7, where we give the dependency of our upper bound as a function of ∆ , L f , C f , Ω and L df .

Original: Smoothness-dependent reconstruction with Heun’s method.
Revised: Smoothness-dependent reconstruction with Heun’s method.

Original: The bounds in Prop. 3 and 4 do notdepend on the smoothness with respect to the weights of the f ( ., θ Nn ) .
Revised: The bounds in Prop. 3 and 4 do not depend on the smoothness with respect to the weights of the f ( ., θ Nn ) .

Original: Only the magnitude of theresiduals plays a role in the correct recovery of the activations and estimation of the gradient.
Revised: Only the magnitude of the residuals plays a role in the correct recovery of the activations and estimation of the gradient.

Original: Hence,there is no apparent benefit of having such a network behave like a Neural ODE.
Revised: Hence, there is no apparent benefit of having such a network behave like a Neural ODE.

Original: We now turn to
Revised: We now turn to

Original: Heun’s method, a second order integration scheme, and show that in this case smoothness in depth ofthe network improves activation recovery.
Revised: Heun’s method, a second order integration scheme, and show that in this case smoothness in depth of the network improves activation recovery.

Original: A HeunNet [30] of depth N with parameters θ N 1 , . .
Revised: A HeunNet [30] of depth N with parameters θ N 1 , . .

Original: , θ NNiterates for n = 0 , . . .
Revised: , θ NN iterates for n = 0 , . . .

Original: , N − 1 :
Revised: , N − 1 :

Revised: [Equation]

Revised: and

Original: [Equation]
Revised: [Equation]

Original: These forward iterations can once again be approximately reversed by doing for n = N − 1 , . . . 
Revised: These forward iterations can once again be approximately reversed by doing for n = N − 1 , . . . , 0 : ˜ y n = ˜ x

Original: [Equation]
Revised: [Equation]

Revised: N f

Revised: [Equation]

Revised: [Equation]

Revised: and

Revised: [Equation]

Original: which also enables approximated backpropagation without storing activations.
Revised: which also enables approximated backpropagation without storing activations.

Original: When discretizing an
Revised: When discretizing an

Original: ODE, Heun’s method has a better O ( 1 N 2 ) error, hence we expect a better recovery than in Prop. 3.
Revised: ODE, Heun’s method has a better O ( 1 N 2 ) error, hence we expect a better recovery than in Prop. 3.

Original: Indeed, we have:
Revised: Indeed, we have:

Original: Proposition 5 (Reconstruction error - Heun’s method) .
Revised: Proposition 5 (Reconstruction error - Heun’s method) .

Original: Assume assumption 2.
Revised: Assume assumption 2.

Original: Denote by L ′ f
Revised: Denote by L ′ f

Original: the
Revised: the

Original: Lipschitz constant of x (cid:55)→ 12 ( f ( x, θ Nn +1 ) + f ( x − 1 N f ( x, θ Nn +1 ) , θ Nn )) , by L θ the Lispchitz constantof θ (cid:55)→ f ( · , θ ) and by L ′ θ that of θ (cid:55)→ ∂ x f ( ., θ ) .
Revised: Lipschitz constant of x (cid:55)→ 12 ( f ( x, θ Nn +1 ) + f ( x − 1 N f ( x, θ Nn +1 ) , θ Nn )) , by L θ the Lispchitz constant of θ (cid:55)→ f ( · , θ ) and by L ′ θ that of θ (cid:55)→ ∂ x f ( ., θ ) .

Original: Let C ′ f
Revised: Let C ′ f

Original: = 14 L ′ θ L θ .
Revised: = 14 L ′ θ L θ .

Original: Finally, define ∆ Nθ :=max n ∥ θ Nn +1 − θ Nn ∥ 2 .
Revised: Finally, define ∆ Nθ := max n ∥ θ Nn +1 − θ Nn ∥ 2 .

Original: Using Heun’s method, we have: ∥ x n − ˜ x n ∥ ≤ ( e L ′ f − 1) C ′ f
Revised: Using Heun’s method, we have: ∥ x n − ˜ x n ∥ ≤ ( e L ′ f − 1) C ′ f

Original: L ′ f N × ∆ Nθ + O ( 1 N 2 ) .
Revised: L ′ f N × ∆ Nθ + O ( 1 N 2 ) .

Original: This bound is very similar to that in proposition 3, with an additional factor ∆ Nθ .
Revised: This bound is very similar to that in proposition 3, with an additional factor ∆ Nθ .

Original: Hence, we see thatunder the condition that ∆ Nθ
Revised: Hence, we see that under the condition that ∆ Nθ

Original: = O ( 1 N ) , the reconstruction error ∥
Revised: = O ( 1 N ) , the reconstruction error ∥

Original: x n − ˜ x n ∥ is in O ( 1 N 2 ) .
Revised: x n − ˜ x n ∥ is in O ( 1 N 2 ) .

Original: In the linearcase, we have proven under some hypothesis in Prop. 2 that such a condition on ∆ Nθ holds duringtraining.
Revised: In the linear case, we have proven under some hypothesis in Prop. 2 that such a condition on ∆ Nθ holds during training.

Original: Consequently, the smoothness of the weights of a HeunNet in turns helps it recover theactivations, while it is not true for a ResNet.
Revised: Consequently, the smoothness of the weights of a HeunNet in turns helps it recover the activations, while it is not true for a ResNet.

Original: This provides better guarantees on the error on gradients:
Revised: This provides better guarantees on the error on gradients:

Original: Proposition 6 (Gradient error - Heun’s method) .
Revised: Proposition 6 (Gradient error - Heun’s method) .

Original: Suppose assumption 2.
Revised: Suppose assumption 2.

Original: Suppose in addition that∂ x f ( ., θ ) admits Lipschitz constant, ∂ θ f ( ., θ ) admits a Lipschitz constant and an upper bound, all ofwhich are independent of θ .
Revised: Suppose in addition that ∂ x f ( ., θ ) admits Lipschitz constant, ∂ θ f ( ., θ ) admits a Lipschitz constant and an upper bound, all of which are independent of θ .

Original: Then one has ∥ ˜ ∇ θ Nn L − ∇ θ Nn L ∥ =
Revised: Then one has ∥ ˜ ∇ θ Nn L − ∇ θ Nn L ∥ =

Original: O ( ∆ Nθ N 2 + 1 N 3 ) .
Revised: O ( ∆ Nθ N 2 + 1 N 3 ) .

Original: Just like with activation, we see that Heun’s method allows for a better gradient estimation whenthe weights are smooth with depth.
Revised: Just like with activation, we see that Heun’s method allows for a better gradient estimation when the weights are smooth with depth.

Original: Equivalently, for a fixed depth, this proposition indicates that
Revised: Equivalently, for a fixed depth, this proposition indicates that

Original: HeunNets have a better estimation of the gradient with the adjoint method than ResNets whichultimately leads to better training and overall better performances by such memory-free model.
Revised: HeunNets have a better estimation of the gradient with the adjoint method than ResNets which ultimately leads to better training and overall better performances by such memory-free model.

Original:  Experiments
Revised:  Experiments

Original: [Figure]

Original: We now present experiments to investigate the applicability of the results presented in this paper.
Revised: We now present experiments to investigate the applicability of the results presented in this paper.

Original: Weuse Pytorch [31] and Nvidia Tesla V100 GPUs.
Revised: We use Pytorch [33] and Nvidia Tesla V100 GPUs.

Original: Our code will be open-sourced.
Revised: Our code is available on GitHub.

Original: All the experimentaldetails are given in appendix B, and we provide a recap on ResNet architectures in appendix C.
Revised: All the experimental details are given in appendix B, and we provide a recap on ResNet architectures in appendix C.

Original: [Figure]
Revised: [Figure]

Original:  i n
Revised: T r a i n

Original: [Figure]
Revised: [Figure]

Revised: rr o r

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Original: 5.1 Validation of our model with step size 1 N
Revised: 5.1 Validation of our model with step size 1 N

Original: The
Revised: The

Original: ResNet model (1) is different from the classical
Revised: ResNet model (1) is different from the classical

Original: ResNet because ofthe 1 N term.
Revised: ResNet because of the 1 N term.

Original: This makes the model depth aware, and we want to studythe impact of this modification on the accuracy on
Revised: This makes the model depth aware, and we want to study the impact of this modification on the accuracy on

Original: CIFAR and
Revised: CIFAR and

Original: ImageNet.
Revised: ImageNet.

Original: [Table]
Revised: [Table]

Original: We first train a ResNet-101 [20] on CIFAR-10 and ImageNet using the same hyper-parameters.
Revised: We first train a ResNet-101 [20] on CIFAR-10 and ImageNet using the same hyper-parameters.

Original: Experimentaldetails are in appendix B and results are summarized intable 1, showing that the explicit addition of the step size1 N does not affect accuracy.
Revised: Experimental details are in appendix B and results are summarized in table 1, showing that the explicit addition of the step size N does not affect accuracy.

Original: In strike contrast, the classical
Revised: In strike contrast, the classical

Original: ResNet rule without the scaling 1 N makes the network behave badly at large depth, while it still workswell with our scaling 1 N , as shown in Figure 3 (a).
Revised: ResNet rule without the scaling 1 N makes the network behave badly at large depth, while it still works well with our scaling 1 N , as shown in Figure 3 (a).

Original: On ImageNet, the scaling 1 N also leads to similartest accuracy in the weight tied setting: 72 . 5% with 4 blocks per layer, 73 .
Revised: On ImageNet, the scaling 1 N also leads to similar test accuracy in the weight tied setting: 72 . 5% with 4 blocks per layer, 73 .

Original: 2% with 8 blocks per layerand 72% with 16 blocks per layer (mean over 2 runs).
Revised: 2% with 8 blocks per layer and 72% with 16 blocks per layer (mean over 2 runs).

Original: 5.2 Adjoint method
Revised: 5.2 Adjoint method

Original: New training strategy.
Revised: New training strategy.

Original: Our results in Prop.
Revised: Our results in Prop.

Original: 3 and 4 assume uniform bounds in N onour residual functions and their derivatives.
Revised: 3 and 4 assume uniform bounds in N on our residual functions and their derivatives.

Original: We also formally proved in the linear settingthat these assumptions hold during the whole learning process if the initial loss is small.
Revised: We also formally proved in the linear setting that these assumptions hold during the whole learning process if the initial loss is small.

Revised: A natural idea to start from a small loss is to consider a pretrained model.

Original: [Table]
Revised: [Table]

Original: A natural idea to start from a small loss is to considera pretrained model.

Original: In addition, we also want our pretrained model to verify assumption 

Revised: In addition, we also want our pretrained model to verify assumption

Original: so we considerthe following setup.
Revised: so we consider the following setup.

Original: On CIFAR (resp.
Revised: On CIFAR (resp.

Original: ImageNet) wetrain a ResNet with 4 (resp.
Revised: ImageNet) we train a ResNet with 4 (resp.

Original: 8) blocks in each layer,where weights are tied within each layer.
Revised: 8) blocks in each layer, where weights are tied within each layer.

Original: A first observation is that one can transfer these weightsto deeper ResNets without significantly affecting the test accuracy of the model: it remains above94 .
Revised: A first observation is that one can transfer these weights to deeper ResNets without significantly affecting the test accuracy of the model: it remains above 94 .

Original: 5% on CIFAR-10 and 72% on ImageNet.
Revised: 5% on CIFAR-10 and 72% on ImageNet.

Original: We then untie the weights of our models and refine them.
Revised: We then untie the weights of our models and refine them.

Original: More precisely, for CIFAR, we then transfer the weights of our model to a ResNet with 4 , 4 , 64 and 4 blocks within each layer and fine-tune it only by refining the third layer, using our adjoint method.
Revised: More precisely, for CIFAR, we then transfer the weights of our model to a ResNet with 4 , 4 , 64 and 4 blocks within each layer and fine-tune it only by refining the third layer, using our adjoint method.

Original: We display in table 2 the median of the new test accuracy, over 5 runs for the initial pretraining of themodel.
Revised: We display in table 2 the median of the new test accuracy, overruns for the initial pretraining of the model.

Original: For ImageNet, we transfer the weights to a ResNet with 100 blocks per layer and fine-tunethe whole model with our adjoint method for the residual layers.
Revised: For ImageNet, we transfer the weights to a ResNet with 100 blocks per layer and fine-tune the whole model with our adjoint method for the residual layers.

Original: Results are summarized in table 2.
Revised: Results are summarized in table 2.

Original: To the best of our knowledge, this is the first time a Neural-ODE like ResNet achieves a test-accuracyof 75 .
Revised: To the best of our knowledge, this is the first time a Neural-ODE like ResNet achieves a test-accuracy of 75 .

Original: 1% on ImageNet.
Revised: 1% on ImageNet.

Original: [Figure]

Original: Failure in usual settings.
Revised: Failure in usual settings.

Original: In Prop. 3 we showed underassumption 2, that is if the residuals are bounded and Lipschitz continuous with constant independent of the depth N , then the error for computing the activations backwardwould scale in 1 N as well as the error for the gradients(Prop. 4).
Revised: In Prop. 3 we showed under assumption 2, that is if the residuals are bounded and Lipschitz continuous with constant independent of the depth N , then the error for computing the activations backward would scale in 1 N as well as the error for the gradients (Prop. 4).

Original: First, this results shows that the architectureneeds to be deep enough, because it scales in 1 N : for instance, we fail to train a ResNet-101 [20] on the ImageNetdataset using the adjoint method on its third layer (depth23 ), as shown in Figure 3 (b).
Revised: First, this results shows that the architecture needs to be deep enough, because it scales in 1 N : for instance, we fail to train a ResNet-101 [20] on the ImageNet dataset using the adjoint method on its third layer (depth 23 ), as shown in Figure 3 (b).

Original: Success at large depth.
Revised: Success at large depth.

Original: To further investigate the applicability of the adjoint method for training deeper ResNets,we train a simple ResNet model on the CIFAR data set.
Revised: To further investigate the applicability of the adjoint method for training deeper ResNets, we train a simple ResNet model on the CIFAR data set.

Original: First, the input is processed by a 5 × 5 convolution with16 out channels, and the image is down-sampled to a size10 × 10 . 
Revised: First, the input is processed by a 5 × 5 convolution with 16 out channels, and the image is down-sampled to a size 10 × 10 . [Figure]

Revised: a function depth when using or Heun’s discretization method with or without the adjoint method.

Original: We then apply a batch norm, a ReLU and iteraterelation (1) where f is a pre-activation basic block [21].
Revised: We then apply a batch norm, a ReLU and iterate relation (1) where f is a pre-activation basic block [21].

Original: We consider the zero residual initialisation: the last batchnorm of each basic block is initialized to zero.
Revised: We consider the zero residual initialisation: the last batch norm of each basic block is initialized to zero.

Original: We consider different values for the depth N and notice that in this setup, the deeper our model is, the betterit performs in term of test accuracy.
Revised: We consider different values for the depth N and notice that in this setup, the deeper our model is, the better it performs in term of test accuracy.

Original: We then compare the performance of our model using a ResNet(forward rule (1)) or a HeunNet (forward rule (8)).
Revised: We then compare the performance of our model using a ResNet (forward rule (1)) or a HeunNet (forward rule (8)).

Original: We train our networks using either the classicalbackpropagation or our corresponding proxys using the adjoint method (formulas (6) and (9)).
Revised: We train our networks using either the classical backpropagation or our corresponding proxys using the adjoint method (formulas (6) and (9)).

Original: Wedisplay the final test accuracy (median over 5 runs) for different values of the depth N in Figure 4.
Revised: We display the final test accuracy (median over 5 runs) for different values of the depth N in Figure 4.

Original: The true backpropagation gives the same curves for the ResNet and the HeunNet.
Revised: The true backpropagation gives the same curves for the ResNet and the HeunNet.

Original: Approximatedgradients, however, lead to a large test error at small depth, but give the same performance at largedepth, hence confirming our results in Prop. 4 and 6.
Revised: Approximated gradients, however, lead to a large test error at small depth, but give the same performance at large depth, hence confirming our results in Prop. 4 and 6.

Original: In addition, at fixed depth, the accuracy whentraining a HeunNet with the adjoint method is better (or similar at depths 2 , 32 and 64 ) than forthe ResNet with the adjoint method.
Revised: In addition, at fixed depth, the accuracy when training a HeunNet with the adjoint method is better (or similar at depths 2 , 32 and 64 ) than for the ResNet with the adjoint method.

Original: This is to be linked with the two different bounds in Prop. 4and 6: for the HeunNet, smoothness with depth, which is expected at large depth, according to thetheoretical results for the linear case (Prop. 2), implies a faster convergence to the true gradients forthe HeunNet than for the ResNet.
Revised: This is to be linked with the two different bounds in Prop. 4 and 6: for the HeunNet, smoothness with depth, which is expected at large depth, according to the theoretical results for the linear case (Prop. 2), implies a faster convergence to the true gradients for the HeunNet than for the ResNet.

Original: We finally validate this convergence in Figure 3 (c): the deeper thearchitecture, the better the approximation on the gradients.
Revised: We finally validate this convergence in Figure 3 (c): the deeper the architecture, the better the approximation on the gradients.

Original: In addition, the HeunNet approximatesthe true gradient better than the ResNet.
Revised: In addition, the HeunNet approximates the true gradient better than the ResNet.

Original: Conclusion, limitations and future works
Revised: Conclusion, limitations and future works

Original: We propose a methodology to analyze how well a ResNet discretizes a Neural ODE.
Revised: We propose a methodology to analyze how well a ResNet discretizes a Neural ODE.

Original: The positiveresults predicted by our theory in the linear case are also observed in practice with real architectures:one can successfully use the adjoint method to train ResNets (or even more effectively HeunNets)using very deep architectures on CIFAR, or fine-tune them on ImageNet, without memory cost in theresidual layers.
Revised: The positive results predicted by our theory in the linear case are also observed in practice with real architectures: one can successfully use the adjoint method to train ResNets (or even more effectively HeunNets) using very deep architectures on CIFAR, or fine-tune them on ImageNet, without memory cost in the residual layers.

Original: However, we also show that for large scale problems such as ImageNet classificationfrom scratch, the adjoint method fails at usual depths.
Revised: However, we also show that for large scale problems such as ImageNet classification from scratch, the adjoint method fails at usual depths.

Original: Our work provides a theoretical guarantee for the convergence to a Neural ODE in the linear settingunder a small loss initialization.
Revised: Our work provides a theoretical guarantee for the convergence to a Neural ODE in the linear setting under a small loss initialization.

Original: A natural extension would be to study the non-linear case.
Revised: A natural extension would be to study the non-linear case.

Original: Inaddition, the adjoint method is time consuming, and an improvement would be to propose a cheapermethod than a reverse mode traversal of the architecture for approximating the activations.Checklist For all authors...
Revised: In addition, the adjoint method is time consuming, and an improvement would be to propose a cheaper method than a reverse mode traversal of the architecture for approximating the activations.

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? [Yes] (b) Did you describe the limitations of your work?
Revised: Acknowledgments

Original: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work? [No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: This work was granted access to the HPC resources of IDRIS under the allocation 2020-

Original: [Yes]

Original: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results? [Yes] , all the assumptions are stated. (b) Did you include complete proofs of all theoretical results?
Revised: [AD011012073] made by GENCI.

Original: [Yes] , the proofs are in the

Original: supplementary material.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in  the supplemental material or as a URL)? [Yes] , in appendix
Revised: This work was supported in part by the French government under management of Agence Nationale de la Recherche as part of the “Investissements d’avenir” program, reference ANR19-P3IA-0001 (PRAIRIE 3IA Institute).

Original: B

Original: (b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
Revised: This work was supported in part by the European Research Council (ERC project NORIA).

Original: [Yes] , in appendix B (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] , see Figure 4 and table
Revised: M. S. thanks Mathieu Blondel

Original: (d) Did you include the total amount of compute andthe type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] If you are using existing assets (e
Revised: and

Original: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators? [Yes] for Pytorch, CIFAR, ImageNet (b) Did you mention the license of the assets?
Revised: Zaccharie

Original: [Yes]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?

Original: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?

Original: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable? [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: Ramzi for helpful discussions

Original: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?

Original: [N/A]