Original: Do Residual Neural Networks discretize Neural Ordinary Differential Equations?
Revised: Do Residual Neural Networks discretize Neural Ordinary Differential Equations?

Original: Anonymous Author(s
Revised: Author(s

Original: Abstract
Revised: Abstract

Original: Neural Ordinary Differential Equations (Neural ODEs) are the continuous analogof Residual Neural Networks (ResNets).
Revised: Neural Ordinary Differential Equations (Neural ODEs) are the continuous analogof Residual Neural Networks (ResNets).

Original: We investigate whether the discretedynamics defined by a ResNet are close to the continuous one of a Neural ODE.
Revised: We investigate whether the discretedynamics defined by a ResNet are close to the continuous one of a Neural ODE.

Original: We first quantify the distance between the ResNet’s hidden state trajectory and thesolution of its corresponding Neural ODE.
Revised: We first quantify the distance between the ResNet’s hidden state trajectory and thesolution of its corresponding Neural ODE.

Original: Our bound is tight and, on the negativeside, does not go to 0 with depth N if the residual functions are not smooth withdepth.
Revised: Our bound is tight and, on the negativeside, does not go to 0 with depth N if the residual functions are not smooth withdepth.

Original: On the positive side, we show that this smoothness is preserved by gradientdescent for a ResNet with linear residual functions and small enough initial loss.
Revised: On the positive side, we show that this smoothness is preserved by gradientdescent for a ResNet with linear residual functions and small enough initial loss.

Original: Itensures an implicit regularization towards a limit Neural ODE at rate 1 N , uniformlywith depth and optimization time.
Revised: Itensures an implicit regularization towards a limit Neural ODE at rate 1 N , uniformlywith depth and optimization time.

Original: As a byproduct of our analysis, we consider theuse of a memory-free discrete adjoint method to train a ResNet by recovering theactivations on the fly through a backward pass of the network, and show that thismethod theoretically succeeds at large depth if the residual functions are Lipschitzwith the input.
Revised: As a byproduct of our analysis, we consider theuse of a memory-free discrete adjoint method to train a ResNet by recovering theactivations on the fly through a backward pass of the network, and show that thismethod theoretically succeeds at large depth if the residual functions are Lipschitzwith the input.

Original: We then show that Heun’s method, a second order ODE integrationscheme, allows for better gradient estimation with the adjoint method when theresidual functions are smooth with depth.
Revised: We then show that Heun’s method, a second order ODE integrationscheme, allows for better gradient estimation with the adjoint method when theresidual functions are smooth with depth.

Original: We experimentally validate that ouradjoint method succeeds at large depth, and that Heun’s method needs fewer layersto succeed.
Revised: We experimentally validate that ouradjoint method succeeds at large depth, and that Heun’s method needs fewer layersto succeed.

Original: We finally use the adjoint method successfully for fine-tuning verydeep ResNets without memory consumption in the residual layers.
Revised: We finally use the adjoint method successfully for fine-tuning verydeep ResNets without memory consumption in the residual layers.

Original:  Introduction
Revised:  Introduction

Original: Problem setup Residual Neural Networks (ResNets) [18, 19] keep on outperforming state of the artin computer vision [40, 6], and more generally skip connections are widely used in a various range ofapplications [36, 13].
Revised: Problem setup Residual Neural Networks (ResNets) [20, 21] keep on outperforming state of the artin computer vision [45, 6], and more generally skip connections are widely used in a various range ofapplications [41, 14].

Original: A ResNet of depth N iterates, starting from x 0 ∈ R d , x n +1 = x n + f ( x n , θ Nn )and outputs a final value x N ∈ R d where f is a neural network called residual function.
Revised: A ResNet of depth N iterates, starting from x 0 ∈ R d , x n +1 = x n + f ( x n , θ Nn )and outputs a final value x N ∈ R d where f is a neural network called residual function.

Original: In this work,we consider a simple modification of this forward rule by letting explicitly the residual mappingdepend on the depth of the network:
Revised: In this work,we consider a simple modification of this forward rule by letting explicitly the residual mappingdepend on the depth of the network:

Original: [Equation]
Revised: [Equation]

Original: On the other hand, a Neural ODE [7] uses a neural network φ Θ ( x, s ) , that takes time s into account,to parameterise a vector field [20] in a differential equation, as follows,
Revised: On the other hand, a Neural ODE [8] uses a neural network φ Θ ( x, s ) , that takes time s into account,to parameterise a vector field [24] in a differential equation, as follows,

Original: [Equation]
Revised: [Equation]

Original: with x
Revised: with x

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original:  and outputs a final value x (1) ∈ R d , the solution of Eq.(2).
Revised: ) and outputs a final value x (1) ∈ R d , the solution of Eq.(2).

Original: The Neural ODE framework enableslearning without storing activations (the x n ’s) using the adjoint state method, hence significantlyreducing the memory usage for backpropagation that can be a bottleneck during training [37, 28, 41].
Revised: The Neural ODE framework enables learning without storing activations (the x n ’s) using the adjoint state method, hence significantly reducingthe memory usage for backpropagation that can be a bottleneck during training [42, 32, 47, 16].

Original: Input of the network
Revised: Input of the network

Original: N u m b e r o f l a y e r s
Revised: N u m b e r o f l a y e r s

Original: Neural ODE unlike
Revised: Neural ODE unlike

Original: Input of the network
Revised: Input of the network

Original: N u m b e r o f l a y e r s
Revised: N u m b e r o f l a y e r s

Original: Neural ODE like
Revised: Neural ODE like

Original: Input of the network
Revised: Input of the network

Original: N u m b e r o f l a y e r s
Revised: N u m b e r o f l a y e r s

Original: Neural ODE unlike
Revised: Neural ODE unlike

Original: Input of the network
Revised: Input of the network

Original: N u m b e r o f l a y e r s
Revised: N u m b e r o f l a y e r s

Original: Neural ODEs also provide a theoretical frameworkto study deep learning models from the continuous viewpoint, using the arsenal of ODE theory[34, 21, 35].
Revised: Neural ODEs also provide a theoretical frameworkto study deep learning models from the continuous viewpoint, using the arsenal of ODE theory[39, 25, 40].

Original: Importantly, they can also be seenas the continuous analog of ResNets.
Revised: Importantly, they can also be seenas the continuous analog of ResNets.

Original: Indeed, consider for N an integer, the Euler scheme for solving
Revised: Indeed, consider for N an integer, the Euler scheme for solving

Original: Eq.
Revised: Eq.

Original: (2) with time step 1 N starting from x 0 and iterating x n +1 = x n + 1 N φ Θ ( x n , nN ) .
Revised: (2) with time step 1 N starting from x 0 and iterating x n +1 = x n + 1 N φ Θ ( x n , nN ) .

Original: Under mild assumptions on φ Θ , this scheme is known to convergeto the true solution of Eq.
Revised: Under mild assumptions on φ Θ , this scheme is known to convergeto the true solution of Eq.

Original: (2) as N goes to + ∞ .
Revised: (2) as N goes to + ∞ .

Original: Also,if Θ = ( θ Nn ) i ∈
Revised: Also,if Θ = ( θ Nn ) i ∈

Original: [ N − 1] and φ Θ ( ., nN ) =
Revised: [ N − 1] and φ Θ ( ., nN ) =

Original: f ( ., θ Nn ) , thenthe ResNet equation Eq.
Revised: f ( ., θ Nn ) , thenthe ResNet equation Eq.

Original: (1) corresponds to a Eulerdiscretization with time step 1 N of Eq.
Revised: (1) corresponds to a Eulerdiscretization with time step 1 N of Eq.

Original: However,for a given ResNet with fixed depth N and weights,the activations in Eq.
Revised: However,for a given ResNet with fixed depth N and weights,the activations in Eq.

Original: (1) can be far from the solutionof Eq.
Revised: (1) can be far from the solutionof Eq.

Original: This is illustrated in Figure 1 where weshow that a deep ResNet can easily break the topology of the input space, which is impossible fora Neural ODE.
Revised: This is illustrated in Figure 1 where weshow that a deep ResNet can easily break the topology of the input space, which is impossible fora Neural ODE.

Original: In this paper, we study the link between ResNets and Neural ODEs.
Revised: In this paper, we study the link between ResNets and Neural ODEs.

Original: We make thefollowing contributions:
Revised: We make thefollowing contributions:

Original: • In Section 3, we propose a framework to define a set of associated Neural ODEs for a given ResNet.
Revised: • In Section 3, we propose a framework to define a set of associated Neural ODEs for a given ResNet.

Original: We control the error between the discrete and the continuous trajectory.
Revised: We control the error between the discrete and the continuous trajectory.

Original: We show that withoutadditional assumptions on the smoothness with depth of the residual functions, this error does notgo to 0 as N → ∞ (Prop. 1).
Revised: We show that withoutadditional assumptions on the smoothness with depth of the residual functions, this error does notgo to 0 as N → ∞ (Prop. 1).

Original: However, we show that under some assumptions on the weightinitialization, the trained parameters of a deep linear ResNet uniformly (with respect to both depthand training time) approach a Lipschitz function as the depth N of the network goes to infinity,at speed 1 N (Prop. 2 and Th. 1).
Revised: However, we show that under some assumptions on the weightinitialization, the trained parameters of a deep linear ResNet uniformly (with respect to both depthand training time) approach a Lipschitz function as the depth N of the network goes to infinity,at speed 1 N (Prop. 2 and Th. 1).

Original: This result highlights an implicit regularization towards a limitNeural ODE.•
Revised: This result highlights an implicit regularization towards a limitNeural ODE.•

Original: In Section 4, we investigate a simple technique to train ResNets without storing activations.
Revised: In Section 4, we investigate a simple technique to train ResNets without storing activations.

Original: Inspiredby the adjoint method, we propose to recover the approximated activations during the backwardpass by using a reverse-time Euler scheme.
Revised: Inspiredby the adjoint method, we propose to recover the approximated activations during the backward pass by using a reverse-time Euler scheme.

Original: We control the error for recovering the activations andgradients with this method.
Revised: We control the error for recovering the activations andgradients with this method.

Original: We show that if the residuals of the ResNet are bounded and Lipschitzcontinuous, with constants independent of N , then this error scales in O ( 1 N ) (Prop. 3).
Revised: We show that if the residuals of the ResNet are bounded and Lipschitzcontinuous, with constants independent of N , then this error scales in O ( 1 N ) (Prop. 3).

Original: Hence,the adjoint method needs a large number of layers to lead to correct gradients (Prop. 4).
Revised: Hence,the adjoint method needs a large number of layers to lead to correct gradients (Prop. 4).

Original: We thenconsider a smoothness-dependent reconstruction with Heun’s method to bound the error betweenthe true and approximated gradient by a term that depends on 1 N times the smoothness in depthof the residual functions, hence guaranteeing a better approximation when successive weights areclose one to another (Prop. 5 and 6).
Revised: We thenconsider a smoothness-dependent reconstruction with Heun’s method to bound the error betweenthe true and approximated gradient by a term that depends on 1 N times the smoothness in depthof the residual functions, hence guaranteeing a better approximation when successive weights areclose one to another (Prop. 5 and 6).

Original: • In Section 5, on the experimental side, we show that the adjoint method fails when training a ResNet101 on ImageNet.
Revised: • In Section 5, on the experimental side, we show that the adjoint method fails when training aResNet 101 on ImageNet.

Original: Nevertheless, we empirically show that very deep ResNets pretrained with tiedweights can be refined -using our adjoint method- on CIFAR-10 and ImageNet by untying theirweights, leading to a better test accuracy.
Revised: Nevertheless, we empirically show that very deep ResNets pretrainedwith tied weights (constant weights: θ Nn = θ ∀ n ) can be refined -using our adjoint method- onCIFAR-10 and ImageNet by untying their weights, leading to a better test accuracy.

Original: Last, but not least, we show using a ResNet architecturewith heavy downsampling in the first layer that our adjoint method succeeds at largedepth and thatHeun’s method leads to a better
Revised: Last, but notleast, we show using a ResNet architecture with heavy downsampling in the first layer that ouradjoint method succeeds at large

Original:  behaved training, hence confirming our theoretical results.
Revised: depth and that Heun’s method leads to a better behaved training,hence confirming our theoretical results.

Original:  Background and related work
Revised:  Background and related work

Original: Neural ODEs.
Revised: Neural ODEs.

Original: Neural ODEs are a class of implicit deep learning models defined by an ODEwhere a neural network parameterises the vector field [38, 7, 34, 33, 39, 25, 30, 20].
Revised: Neural ODEs are a class of implicit deep learning models defined by an ODEwhere a neural network parameterises the vector field [43, 8, 39, 38, 44, 29, 34, 24].

Original: Given an inputx 0 , the output of the model is the solution of the ODE (2) at time 1 .
Revised: Given an inputx 0 , the output of the model is the solution of the ODE (2) at time 1 .

Original: From a theoretical viewpoint,the expression capabilities of Neural ODEs have been investigated in [11, 35, 21] and the Neural
Revised: From a theoretical viewpoint,the expression capabilities of Neural ODEs have been investigated in [12, 40, 25] and the Neural

Original: ODE framework has been used to better understand the dynamics of more general architecturesthat include residual connections such as Transformers [32, 23].
Revised: ODE framework has been used to better understand the dynamics of more general architecturesthat include residual connections such as Transformers [36, 27].

Original: Experimentaly, Neural ODEs havebeen successful in a various range of applications, among which physical modelling [17, 10] andgenerative modeling [7, 16].
Revised: Experimentaly, Neural ODEs havebeen successful in a various range of applications, among which physical modelling [18, 11] andgenerative modeling [8, 17].

Original: However, there are many areas where Neural ODEs have failed to replace ResNets, for instance for building computer vision classification models.
Revised: However, there are many areas where Neural ODEs have failed to replace ResNets, for instance for building computer vision classification models.

Original: Neural ODEs fail tocompete with ResNets on ImageNet, and to the best of our knowledge, previous works using Neural
Revised: Neural ODEs fail tocompete with ResNets on ImageNet, and to the best of our knowledge, previous works using Neural

Original: ODEs on ImageNet consider weight-tied architectures and only achieves the same accuracy as a
Revised: ODEs on ImageNet consider weight-tied architectures and only achieves the same accuracy as a

Original: ResNet18 [43].
Revised: ResNet18 [49].

Original: Implicit Regularization of ResNets towards ODEs.
Revised: Implicit Regularization of ResNets towards ODEs.

Original: Recent works have studied the link between
Revised: Recent works have studied the link between

Original: ResNets and Neural ODEs.
Revised: ResNets and Neural ODEs.

Original: In [8], the authors carry experiments to better understand the scalingbehavior of weights in ResNets as a function of the depth.
Revised: In [9], the authors carry experiments to better understand the scalingbehavior of weights in ResNets as a function of the depth.

Original: They show that under the assumption thatthere exists a scaling limit θ ( s ) =
Revised: They show that under the assumption thatthere exists a scaling limit θ ( s ) =

Original: N β lim θ N ⌊ Ns ⌋ for the weights of the ResNets (with 0 < β < 1 )and if the scale of the ResNet is 1 N α with 0 < α < 1 and α
Revised: N β lim θ N ⌊ Ns ⌋ for the weights of the ResNets (with 0 < β < 1 )and if the scale of the ResNet is 1 N α with 0 < α < 1 and α

Original: + β = 1 , then the hidden state of the
Revised: + β = 1 , then the hidden state of the

Original: ResNet converges to a solution of a linear ODE.
Revised: ResNet converges to a solution of a linear ODE.

Original: In this paper, we are interested in the case whereα = 1 , which seems more natural since it is the scaling that appears in Euler’s method with step1 N .
Revised: In this paper, we are interested in the case whereα = 1 , which seems more natural since it is the scaling that appears in Euler’s method with step1 N .

Original: In addition, we do not assume the existence of a scaling limit θ ( s ) =
Revised: In addition, we do not assume the existence of a scaling limit θ ( s ) =

Original: lim
Revised: lim

Original: θ N
Revised: θ N

Original: ⌊ Ns ⌋ .
Revised: ⌊ Ns ⌋ .

Original: In subsection3.2, we demonstrate the existence of this scaling limit in the linear setting, under some assumptions.
Revised: In subsection3.2, we demonstrate the existence of this scaling limit in the linear setting, under some assumptions.

Original: The recent work [9] shows results regarding linear convergence of gradient descent in ResNets andprove the existence of an 12 -Hölder continuous scaling limit as N → ∞ with a scaling factor forthe residuals in 1 √ N which is different from ours.
Revised: The recent work [10] shows results regarding linear convergence of gradient descent in ResNets andprove the existence of an 12 -Hölder continuous scaling limit as N → ∞ with a scaling factor forthe residuals in 1 √ N which is different from ours.

Original: In contrast, we show that our limit function is
Revised: In contrast, we show that our limit function is

Original: Lipschitz continuous, which is a stronger regularity.
Revised: Lipschitz continuous, which is a stronger regularity.

Original: We also show that our convergence is uniform indepth and optimization time.
Revised: We also show that our convergence is uniform indepth and optimization time.

Original: More generally, recent works have proved the convergence of gradientdescent training of ResNet when the initial loss is small enough.
Revised: More generally, recent works have proved the convergence of gradientdescent training of ResNet when the initial loss is small enough.

Original: This include ResNet with finitewidth but arbitrary large depth [14, 22] and ResNet with both infinite width and depth [24, 3].
Revised: This include ResNet with finitewidth but arbitrary large depth [15, 26] and ResNet with both infinite width and depth [28, 3].

Original: Theseconvergence proofs leverage an implicit bias toward weights with small amplitudes.
Revised: Theseconvergence proofs leverage an implicit bias toward weights with small amplitudes.

Original: They howeverleave open the question of convergence of individual weights as depth increases, which we tackle inthis work in the linear case.
Revised: They howeverleave open the question of convergence of individual weights as depth increases, which we tackle inthis work in the linear case.

Original: This requires showing an extra bias toward weights with small variationsacross depth.
Revised: This requires showing an extra bias toward weights with small variationsacross depth.

Revised: Memory bottleneck in ResNets.

Revised: Training deep learning models involve graphics processing units(GPUs) where memory is a practical bottleneck [42, 32, 47].

Revised: Indeed, backpropagation requires tostore activations at each layer during the forward pass.

Revised: Since samples are processed using minibatches, this storage can be important.

Original:  Adjoint Method.
Revised: For instance, with batches of size 128, the memory needed tocompute gradients for a ResNet 152 on ImageNet is about 22 GiB. Note that the memory neededto store the parameters of the model is only 220 MiB, which is negligible compared to the memoryneeded to store the activations. Thus, designing deep invertible architectures where one can recoverthe activations on the fly during the backpropagation iterations has been an active field in recent years[16, 37, 23]. In this work, we propose to approximate activations using a reverse-time Euler scheme,as we detail in the next subsection. Adjoint Method.

Original: Consider a loss function L ( x N ) for the ResNet (1).
Revised: Consider a loss function L ( x N ) for the ResNet (1).

Original: The backpropagationequations [5] are
Revised: The backpropagationequations [5] are

Original: [Equation]
Revised: [Equation]

Original: Now, consider a loss function L ( x (1)) for the Neural ODE (2).
Revised: Now, consider a loss function L ( x (1)) for the Neural ODE (2).

Original: The adjoint state method [29, 7] gives
Revised: The adjoint state method [33, 8] gives

Original: [Equation]
Revised: [Equation]

Original: Note that if Θ = ( θ Ni ) i ∈ [ N − 1] and φ Θ ( ., nN ) =
Revised: Note that if Θ = ( θ Nn ) n ∈ [ N − 1] and φ Θ ( ., nN ) =

Original: f ( ., θ Nn ) , then Eq.
Revised: f ( ., θ Nn ) , then Eq.

Original: (3) corresponds to a Eulerdiscretization with time step 1 N of Eq.
Revised: (3) corresponds to a Eulerdiscretization with time step 1 N of Eq.

Original: The key advantage of using Eq.
Revised: The key advantage of using Eq.

Original: (4) is that one can recoverx ( s ) on the fly by solving the Neural ODE (2) backward in time starting from x (1) .
Revised: (4) is that one can recoverx ( s ) on the fly by solving the Neural ODE (2) backward in time starting from x (1) .

Original: This strategyavoids storing the forward trajectory ( x ( s )) s ∈ [0 , 1] and leads to a O (1) memory footprint [7].
Revised: This strategyavoids storing the forward trajectory ( x ( s )) s ∈ [0 , 1] and leads to a O (1) memory footprint [8].

Original: Inthis work, we propose to use a discrete adjoint method by using a reverse-time Euler scheme forapproximately recovering the activations in a ResNet (Section 4).
Revised: Inthis work, we propose to use a discrete adjoint method by using a reverse-time Euler scheme forapproximately recovering the activations in a ResNet (Section 4).

Original: Contrarily to other models such as
Revised: Contrarily to other models such as

Original: RevNets [15] (architecture change) or Momentum ResNets [31] (forward rule modification) whichrely on an exactly invertible forward rule, the proposed method requires no change at all in thenetwork, but gives approximate gradients. 
Revised: RevNets [16] (architecture change) or Momentum ResNets [35] (forward rule modification) whichrely on an exactly invertible forward rule, the proposed method requires no change at all in thenetwork, but gives approximate gradients. Notations.

Revised: For k ∈ N , C k is the set of functions f : R d → R d that are k times differentiable withtheir k th differential being continuous.

Revised: For f ∈ C 1 , ∂ x f ( x )[ y ] is the differential of f at x ∈ R devaluated in y ∈ R d .

Revised: For A and B two sets, A × B = { ( a, b ) | a ∈ A , b ∈ B } .

Original:  ResNets as discretization of Neural ODEs
Revised:  ResNets as discretization of Neural ODEs

Original: In this section we first show that without further assumptions, the distance between the discretetrajectory and the solution of associated ODEs can be constant with respect to the depth of thenetwork if the residual functions lack smoothness with depth.
Revised: In this section we first show that without further assumptions, the distance between the discretetrajectory and the solution of associated ODEs can be constant with respect to the depth of thenetwork if the residual functions lack smoothness with depth.

Original: We then present a positive result bystudying the linear case where we show that, under some hypothesis (small loss initialization andinitial smoothness with depth), the ResNet converges to a Neural ODE as the number of layers goesto infinity.
Revised: We then present a positive result bystudying the linear case where we show that, under some hypothesis (small loss initialization andinitial smoothness with depth), the ResNet converges to a Neural ODE as the number of layers goesto infinity.

Original: We show that this convergence is uniform with depth and optimization time.
Revised: We show that this convergence is uniform with depth and optimization time.

Original: 3.1 Distance to an ODE
Revised: 3.1 Distance to an ODE

Original: We first define associated Neural ODEs for a given ResNet.
Revised: We first define associated Neural ODEs for a given ResNet.

Original: Definition 1.
Revised: Definition 1.

Original: We say that a neural network φ Θ : R d × R → R d smoothly interpolates the ResNet
Revised: We say that a neural network φ Θ : R d × R → R d smoothly interpolates the ResNet

Original: Eq.
Revised: Eq.

Original: if φ Θ is smooth and ∀ n ∈ { 0 , ..., N − 1 } , φ Θ ( ., nN ) =
Revised: if φ Θ is smooth and ∀ n ∈ { 0 , ..., N − 1 } , φ Θ ( ., nN ) =

Original: f ( ., θ Nn ) .
Revised: f ( ., θ Nn ) .

Original: Note that we omit the dependency of Θ in N to simplify notations.
Revised: Note that we omit the dependency of Θ in N to simplify notations.

Original: For example, for a given
Revised: For example, for a given

Original: ResNet, there are two natural ways to interpolate it with a Neural ODE, either by interpolating theresiduals , or by interpolating the weights .
Revised: ResNet, there are two natural ways to interpolate it with a Neural ODE, either by interpolating theresiduals , or by interpolating the weights .

Original: Indeed, one can interpolate the residuals with φ Θ ( · , s ) =( n + 1 − Ns ) f ( ., θ Nn ) + ( Ns − n ) f ( · , θ Nn +1 ) when s ∈ [ nN , n +1 N ] , or interpolate the weights withφ Θ ( · , s ) =
Revised: Indeed, one can interpolate the residuals with φ Θ ( · , s ) =( n + 1 − Ns ) f ( ., θ Nn ) + ( Ns − n ) f ( · , θ N n +1 ) when s ∈ [ nN , n +1 N ] , or interpolate the weights withφ Θ ( · , s ) =

Original: f ( · , ( n +1 − Ns ) θ
Revised: f ( · , ( n +1 − Ns ) θ

Original: Nn +( Ns − n ) θ Nn +1 ) for s ∈ [ nN , n +1 N ] .
Revised: Nn +( Ns − n ) θ Nn +1 ) for s ∈ [ nN , n +1 N ] .

Original: If θ Nn = θ N does not depend onn , then both interpolations are identical and one can simply consider φ Θ ( x, s ) =
Revised: If θ Nn = θ N does not depend onn , then both interpolations are identical and one can simply consider φ Θ ( x, s ) =

Original: f ( x, θ N ) , ∀ ( x, s ) .
Revised: f ( x, θ N ) , ∀ ( x, s ) .

Original: We now consider a smooth interpolation φ Θ for the ResNet (1) and a Euler scheme for the Neural
Revised: We now consider any smooth interpolation φ Θ for the ResNet (1) and a Euler scheme for the Neural

Original: ODE (2) with time step 1 N .
Revised: ODE (2) with time step 1 N .

Original: Proposition 1 (Approximation error) .
Revised: Proposition 1 (Approximation error) .

Original: We suppose that φ Θ is C 1 , and L -Lipschitz with respect to x ,uniformly in s .
Revised: We suppose that φ Θ is C 1 , and L -Lipschitz with respect to x ,uniformly in s .

Original: Note that this implies that the solution of Eq.
Revised: Note that this implies that the solution of Eq.

Original: (2) is well defined, unique, C 2 , and thatthe trajectory is included in some compact K ⊂ R d .
Revised: (2) is well defined, unique, C 2 , and thatthe trajectory is included in some compact K ⊂ R d .

Original: Denote C N := ∥ ∂
Revised: Denote C N := ∥ ∂

Original: s φ Θ + ∂ x φ Θ [ φ Θ ] ∥ K × [0 , 1] ∞ .
Revised: s φ Θ + ∂ x φ Θ [ φ Θ ] ∥ K × [0 , 1] ∞ .

Original: Then one has for all n : ∥ x n − x ( nN ) ∥
Revised: Then one has for all n : ∥ x n − x ( nN ) ∥

Original: ≤ e L − 1 2 NL C N .
Revised: ≤ e L − 1 2 NL C N .

Original: For a full proof, see appendix A.1.
Revised: For a full proof, see appendix A.1.

Original: Note that this result extends Theorem 3.2 from [42] to thenon-autonomous case: our bound depends on ∂ s φ Θ .
Revised: Note that this result extends Theorem 3.2 from [48] to thenon-autonomous case: our bound depends on ∂ s φ Θ .

Revised: This quantity is also well defined at L = 0 sincee L − 1 L → 1 as L → 0 .

Original: Note also that our bound is tight.
Revised: Finally, our bound is tight.

Original: Indeed, forφ Θ ( x, s ) = as + b for a, b ∈ R d , we get ∂ s φ Θ + ∂ x φ Θ [ φ Θ ] = a , L = 0 and ∥ x (1) − x N ∥ =
Revised: Indeed, for φ Θ ( x, s ) = as + b for a, b ∈ R d , weget ∂ s φ Θ + ∂ x φ Θ [ φ Θ ] = a , L = 0 and ∥ x (1) − x N ∥ =

Original: ∥ a ∥ 2 N .
Revised: ∥ a ∥ 2 N .

Original: Implication.
Revised: Implication.

Original: The tightness of our bound shows that closeness to the ODE solution is not guaranteed, because we do not know whether C N /N → 0 .
Revised: The tightness of our bound shows that closeness to the ODE solution is not guaranteed, because we do not know whether C N /N → 0 .

Original: Indeed, consider first the residual interpolationφ Θ ( x, s ) = (cid:0) ( n + 1 − Ns ) f
Revised: Indeed, consider first the residual interpolation

Original: ( x, θ Nn )
Revised: [Equation]

Original: + ( Ns − n ) f 
Revised: f [Equation]

Original: ( x, θ Nn +1 ) (cid:1) )
Revised: x, [Equation] Nn +1 ) [Equation]

Original:  1 s ∈ [ nN , n +1 N ] and the simplecase where ∂ x φ Θ [ φ Θ ] = 0 .
Revised: ) 1 s ∈ [ nN , n +1 N ] and the simplecase where ∂ x φ Θ [ φ Θ ] = 0 .

Original: We get ∂
Revised: We get ∂

Original: s φ Θ ( x, s ) =
Revised: s φ Θ ( x, s ) =

Original: N ( f ( x, θ Nn +1 ) − f ( x, θ Nn ))
Revised: N ( f ( x, θ Nn +1 ) − f ( x, θ Nn ))

Original: 1 s ∈ [ nN , n +1 N ] , whichcorresponds to the discrete derivative.
Revised: 1 s ∈ [ nN , n +1 N ] , whichcorresponds to the discrete derivative.

Original: It means that although there is a 1 N factor in our bound, thetime derivative term – without further regularity with depth of the weights, which is at the heart ofsubection 3.2 – usually scales with N : C N N =
Revised: It means that although there is a 1 N factor in our bound, thetime derivative term – without further regularity with depth of the weights, which is at the heart ofsubection 3.2 – usually scales with N : C N N =

Original: O (1) .
Revised: O (1) .

Original: As a first example, consider the simple casewhere f
Revised: As a first example, consider the simple casewhere f

Original: ( x, θ Nn )
Revised: ( x, θ Nn )

Original: = n.
Revised: = n.

Original: This gives x N = x 0 + ( N − 1)while the integration of the Neural ODE (2)leads to x (1) = x 0 + N 2 because φ Θ ( x, s ) =
Revised: This gives x N = x 0 + ( N − 1)while the integration of the Neural ODE (2)leads to x (1) = x 0 + N 2 because φ Θ ( x, s ) =

Original: Ns , so the ∥ x N − x (1) ∥ = 12 is not small.
Revised: Ns , so the ∥ x N − x (1) ∥ = 12 is not small.

Original: Intuitively,this shows that weights cannot scale with depth when using the residual interpolation.
Revised: Intuitively,this shows that weights cannot scale with depth when using the residual interpolation.

Original: Now, consider the weight interpolation, θ Nn = ( − 1) n and suppose f is written as f ( x, θ )
Revised: Now, consider the weight interpolation, θ Nn = ( − 1) n and suppose f is written as f ( x, θ )

Original: =
Revised: =

Original: θ 2 .
Revised: θ 2 .

Original: This givesφ Θ ( ., s )
Revised: This givesφ Θ ( ., s )

Original: = (2 Ns − (2 n + 1)) 2 when s ∈ [ nN , n +1 N ] .
Revised: = (2 Ns − (2 n + 1)) 2 when s ∈ [ nN , n +1 N ] .

Original: Integrating, we get x (1) =
Revised: Integrating, we get x (1) =

Original: while the outputof the ResNet is x N = 1 .
Revised: while the outputof the ResNet is x N = 1 .

Original: Hence ∥ x N − x (1) ∥ = 23 is also not small, even though the weights arebounded.
Revised: Hence ∥ x N − x (1) ∥ = 23 is also not small, even though the weights arebounded.

Original: Thus, one needs additional regularity assumptions on the weights of the ResNet to obtain aNeural
Revised: Thus, one needs additional regularity assumptions on the weights of the ResNet to obtain a

Original:  ODE in the large depth limit.
Revised: Neural ODE in the large depth limit.

Original: Intuitively if the weights are initialized close from one another and theyare updated using gradient descent, they should stay close from one another during training, since thegradients in two consecutive layers will be similar, as highlighted in Eq.
Revised: Intuitively if the weights are initialized close from one anotherand they are updated using gradient descent, they should stay close from one another during training,since the gradients in two consecutive layers will be similar, as highlighted in Eq.

Original: Indeed, we see that ifx n and x n +1 are close, then ∇ x n L and ∇ x n +1 L are close, and then if θ Nn and θ Nn +1 are close, ∇ θ Nn Land ∇ θ Nn +1 L are also close.
Revised: Indeed, wesee that if x n and x n +1 are close, then ∇ x n L and ∇ x n +1 L are close, and then if θ Nn and θ Nn +1 areclose, ∇ θ Nn L and ∇ θ Nn +1 L are also close.

Original: In subsection 3.2, we formalize this intuition and present a positive result for ResNets with linearresidual functions.
Revised: In subsection 3.2, we formalize this intuition and present apositive result for ResNets with linear residual functions.

Original: More precisely, we show that with proper initialization, the difference betweentwo successive parameters is in 1 N during the entire training.
Revised: More precisely, we show that with properinitialization, the difference between two successive parameters is in 1 N during the entire training.

Original: Furthermore, we show that the weightsof the network converge to a smooth function, hence defining a limit Neural ODE.
Revised: Furthermore, we show that the weights of the network converge to a smooth function, hence defininga limit Neural ODE.

Original: 3.2 Linear Case
Revised: 3.2 Linear Case

Original: As a further step towards a theoretical understanding of the connections between ResNets and Neural
Revised: As a further step towards a theoretical understanding of the connections between ResNets and Neural

Original: ODEs we investigate the linear setting, where the residual functions are written f ( x, θ ) =
Revised: ODEs we investigate the linear setting, where the residual functions are written f ( x, θ ) =

Original: θx for anyθ ∈ R d × d .
Revised: θx for anyθ ∈ R d × d .

Original: It corresponds to a deep matrix factorization problem [44, 4, 2, 1].
Revised: It corresponds to a deep matrix factorization problem [50, 4, 2, 1].

Original: As opposed to theseprevious works, we study the infinite depth limit of these linear ResNets with a focus on the learnedweights.
Revised: As opposed to theseprevious works, we study the infinite depth limit of these linear ResNets with a focus on the learnedweights.

Original: We show that, if the weights are initialized close one to another, then at any training time, the weights stay close one to another (Prop. 2) and importantly, they converge to a smooth functionof the continuous depth s as N → ∞ (Th. 1).
Revised: We show that, if the weights are initialized close one to another, then at any training time,the weights stay close one to another (Prop. 2) and importantly, they converge to a smooth functionof the continuous depth s as N → ∞ (Th. 1).

Original: All the proofs are available in appendix A.
Revised: All the proofs are available in appendix A.

Original: Setting.
Revised: Setting.

Original: Given a training set ( x k , y k ) k ∈ [ n ] in R d , we solve the regression problem of mapping x kto y k with a linear ResNet, i.e. f ( x, θ ) = θx , of depth N and parameters ( θ N 1 , . . .
Revised: Given a training set ( x k , y k ) k ∈ [ n ] in R d , we solve the regression problem of mapping x kto y k with a linear ResNet, i.e. f ( x, θ ) = θx , of depth N and parameters ( θ N 1 , . . .

Original: , θ NN ) .
Revised: , θ NN ) .

Original: The ResNettherefore maps x k to Π N x k where Π N :=
Revised: The ResNettherefore maps x k to Π N x k where Π N :=

Original: (cid:81)
Revised: (cid:81)

Original: Nn =

Revised: Nn =

Original: ( I d + θ Nn N ) =
Revised: ( I d + θ Nn N ) =

Original: ( I d + θ NN N ) · · · ( I d + θ N 1 N ) .
Revised: ( I d + θ NN N ) · · · ( I d + θ N 1 N ) .

Original: It is trainedby minimizing the average errors ∥ Π N x k − y k
Revised: It is trainedby minimizing the average errors ∥ Π N x k − y k

Original: ∥
Revised: ∥

Original: 22 , which is equivalent to the deep matrix factorizationproblem:
Revised: 22 , which is equivalent to the deep matrix factorizationproblem:

Original: [Equation]
Revised: [Equation]

Original:  where∥
Revised: (5) where

Original:  A ∥ 2Σ =
Revised: ∥ A ∥ 2Σ =

Original: Tr( A Σ A T ) , Σ is the empirical covariance matrix of the data: Σ := 1 n (cid:80) nk
Revised: Tr( A Σ A T ) , Σ is the empirical covariance matrix of the data: Σ := 1 n (cid:80) nk

Original: =1 x k x ⊤ k ,and B := 1 n (cid:80) nk =1 y k x ⊤ k Σ − 1 .
Revised: =1 x k x ⊤ k ,and B := 1 n (cid:80) nk =1 y k x ⊤ k Σ − 1 .

Original: As is standard, we suppose that Σ is non degenerated.
Revised: As is standard, we suppose that Σ is non degenerated.

Original: We denote by
Revised: We denote by

Original: M > 0 (resp. m > 0 ) its largest (resp.
Revised: M > 0 (resp. m > 0 ) its largest (resp.

Original: smallest) eigenvalue.
Revised: smallest) eigenvalue.

Original: Gradient.
Revised: Gradient.

Original: We denote Π N : n :=
Revised: We denote Π N : n :=

Original: ( I d + θ NN N ) · · · ( I d + θ Nn +1 N ) and Π Nn : := ( I d + θ Nn − 1 N ) · · · ( I d + θ N 1 N ) andwrite the gradient ∇ Nn ( t ) = ∇ θ Nn L  .
Revised: ( I d + θ NN N ) · · · ( I d + θ Nn +1 N ) and Π Nn : := ( I d + θ Nn − 1 N ) · · · ( I d + θ N 1 N )and write the gradient ∇ Nn ( t ) = ∇ θ Nn L ( θ N 1 ( t ) , .. θ Nn ( t ) , .., θ NN ( t )) .

Original: The chain rule gives N ∇ Nn = Π N ⊤ : n (Π N − B )ΣΠ N ⊤ n : .
Revised: The chain rule gives N ∇ Nn =Π N ⊤ : n (Π N − B )ΣΠ N ⊤ n : .

Original: Intuitively,as N goes to + ∞ , the products Π N , Π N : n and Π Nn : should converge to some limit, hence we see that N ∇ Nn scale as 1 .
Revised: Intuitively, as N goes to + ∞ , the products Π N , Π N : n and Π Nn : should convergeto some limit, hence we see that N ∇ Nn scale as 1 .

Original: Therefore, we train θ Nn by the rescaled gradient flow dθ Nn dt ( t ) =
Revised: Therefore, we train θ Nn by the rescaled gradientflow dθ Nn dt ( t ) =

Original: − N ∇ n L ( t ) tominimize L and denote ℓ N ( t ) =
Revised: − N ∇ Nn ( t ) to minimize L and denote ℓ N ( t ) =

Original: L ( θ N 1 ( t ) , . .
Revised: L ( θ N 1 ( t ) , . .

Original: , θ N N ( t )) .
Revised: , θ NN ( t )) .

Original: Two continuous variables involved.
Revised: Two continuous variables involved.

Original: Our results involve two continuous variables: s ∈ [0 , 1] is thedepth of the limit network and corresponds to the time variable in the Neural ODE, whereas t ∈ R
Revised: Our results involve two continuous variables: s ∈ [0 , 1] is thedepth of the limit network and corresponds to the time variable in the Neural ODE, whereas t ∈ R

Original: +is the gradient flow time variable.
Revised: +is the gradient flow time variable.

Original: As is standard in the analysis of convergence of gradient descentfor linear networks, we consider the following assumption:
Revised: As is standard in the analysis of convergence of gradient descentfor linear networks, we consider the following assumption:

Original: Assumption 1.
Revised: Assumption 1.

Original: Suppose that at initialisation one has (cid:112) ℓ N (0) <
Revised: Suppose that at initialisation one has (cid:112) ℓ N (0) <

Original: m 4 √ 2 Me 3 and ∥ θ Nn (0) ∥ ≤ 14 .
Revised: m 4 √ 2 Me 3 and ∥ θ Nn (0) ∥ ≤ 14 .

Original: [Figure] Assumption 1 is the classical assumption in the literature [44, 3] to prove linear convergence of our lossand that the θ Nn ( t ) ’s stay bounded with t .
Revised: Assumption 1 is the classical assumption in the literature [50, 3] to prove linear convergence of ourloss and that the θ Nn ( t ) ’s stay bounded with t .

Original: We nowprove an implicit regularization result showing thatif at initialization, in addition to assumption 1, theweights are close from one another ( O ( 1 N ) ), they willstay at distance O
Revised: Note that this bounded norm assumption implies that1 N θ n (0) = O ( 1 N ) . This is in contrast with classical initialization scales in the feedforward case wherethe initialization only depends on width [19].

Original:  ( 1 N ) : the discrete derivative stayin O ( 1 N ) , which is a central result to consider theinfinite depth limit in ourmain Th. 
Revised: However this initialization scale is coherent with thoseof ResNets for which the scale has to depend on depth [46]. In addition, the experimental findings in[9] suggest that the weights in ResNets scale in 1 N β with β > 0 . We now prove an implicit regularization result showing that if at initialization, in addition to assumption 1, the weights are close from one another ( O ( 1 N ) ), they will stay at distance O ( 1 N ) : the discretederivative stay in O ( 1 N ) , which is a central result to consider the infinite depth limit in our Th. 1.

Original: Proposition 2 (Smoothness in depth of the weights) .
Revised: Proposition 2 (Smoothness in depth of the weights) .

Original: Suppose assumption 1.
Revised: Suppose assumption 1.

Original: Suppose that there exists C 0 > 0 independent of n and N such that [Equation]
Revised: Suppose that thereexists C 0 > 0 independent of n and N such that ∥ θ Nn +1 (0) − θ Nn (0) ∥ ≤ C 0 N .

Original: Then, ∀ t ∈ R + ,∥ θ Nn ( t ) ∥ < 12 , and θ Nn ( t ) admits a limit ψNn ast → + ∞ . Moreover, there exists C > 0 such that [Equation]
Revised: Then, ∀ t ∈ R + ,∥ θ Nn ( t ) ∥ < 12 , and θ Nn ( t ) admits a limit ψ

Original: Sketch of proof. In [44], it is shown that ∥ θ Nn( t ) ∥ ≤ 12 and that the loss satisfies a Polyak-Łojasiewicz(PL) condition. Then we use the PL condition to show that ∇ Nn is integrable so that θ Nn ( t ) admits alimit as t →  ∞ .We then remark that ∇ Nn +1 − ∇ Nn
Revised: Nn as t → + ∞ .

Original: =

Original: ( I + θ N ⊤ n +1 N ) − 1 ( ∇ Nn θ N ⊤ n − θ N ⊤ n +1 ∇ Nn N ) . This impliesthat ∥∇ Nn +1 − ∇ Nn ∥ ≤ 2 N ∥∇ Nn ∥ . The integration of this relation along with the PL conditions leadsthe existence of C > 0independent of n , N and t such that ∥ θ Nn +1 ( t ) − θ Nn ( t ) ∥ ≤ CN .
Revised: Moreover, there exists C > 0 such that∀ t ∈ R + , ∥ θ Nn +1 ( t ) − θ Nn ( t ) ∥ ≤ CN .

Original: For a full proof, see appendix A.2.
Revised: For a full proof, see appendix A.2.

Original: We illustrate Prop. 2 in Figure 2.

Original: The inequality ∥ θ Nn +1 ( t ) −θ N n ( t ) ∥
Revised: The inequality ∥ θ N n +1 ( t ) − θ N n ( t ) ∥

Original: ≤ C N corresponds to a discrete Lipschitz property in depth.
Revised: ≤ C N corresponds to a discrete Lipschitz property in depth.

Original: Indeed, for s ∈ [0 , 1] and t ∈ R + ,let ψ N ( s, t )
Revised: Indeed, for s ∈ [0 , 1] and t ∈ R + , let ψ N ( s, t )

Original: =
Revised: =

Original: θ N ⌊ Ns ⌋ ( t ) .
Revised: θ N ⌊ Ns ⌋ ( t ) .

Original: Then our result gives ∥ ψ N ( n +1 N , t ) − ψ N ( nN , t ) ∥ ≤ CN which implies that∥ ψ N ( s 1 , t ) − ψ N ( s 2 , t ) 
Revised: Thenour result gives ∥ ψ N ( n +1 N , t ) − ψ N ( nN , t ) ∥ ≤ CN which implies that ∥ ψ N ( s 1 , t ) − ψ N ( s 2 , t ) ∥ ≤

Original: ∥ ≤ C | s 1 − s 2 | + CN .
Revised: C | s 1 − s 2 | + CN .

Original: We now turn to the infinite depth limit N → ∞ .
Revised: We now turn to the infinite depth limit N → ∞ .

Original: Th.1 shows that there exists a limit function ψ such that ψ N converges uniformly to ψ in depth s andoptimization time t .
Revised: Th. 1 shows that there exists a limitfunction ψ such that ψ N converges uniformly to ψ in depth s and optimization time t .

Original: Furthermore, this limit is Lipschitz continuous in ( s, t ) .
Revised: Furthermore, this limit is Lipschitz continuous in ( s, t ) .

Original: In addition, we show that the ResNet Π N converges to the limit Neural ODE defined by ψ that is preserved along theoptimization flow, exhibiting an implicit regularization property of deep linear ResNets towards Neural ODEs.
Revised: In addition, we show that the ResNet Π N converges to thelimit Neural ODE defined by ψ that is preserved along the optimization flow, exhibiting an implicitregularization property of deep linear ResNets towards Neural ODEs.

Original: Theorem 1 (Existence of a limit map) .
Revised: Theorem 1 (Existence of a limit map) .

Original: Suppose assumption 1, ∥ θ Nn +1 (0) − θ Nn (0)
Revised: Suppose assumption 1, ∥ θ Nn +1 (0) − θ Nn (0)

Original: ∥ ≤ C 0 N for some
Revised: ∥ ≤ C 0 N for some

Original: C 0 > 0 and that there exists a function ψ init : [0 , 1]
Revised: C 0 > 0 and that there exists a function ψ init : [0 , 1]

Original: → R d × d such that ψ N ( s, 0) → ψ init ( s ) in ∥ .
Revised: → R d × d such that ψ N ( s, 0) → ψ init ( s ) in ∥ .

Original: ∥ ∞uniformly in s as N → ∞ , at speed 1 N .
Revised: ∥ ∞uniformly in s as N → ∞ , at speed 1 N .

Original: Then the sequence ( ψ N ) N ∈ N uniformly converges (in ∥ .
Revised: Then the sequence ( ψ N ) N ∈ N uniformly converges (in ∥ .

Original: ∥ ∞w.r.t ( s, t ) ) to a limit ψ Lipschitz continuous in ( s, t ) and ∥ ψ − ψ N ∥ ∞
Revised: ∥ ∞w.r.t ( s, t ) ) to a limit ψ Lipschitz continuous in ( s, t ) and ∥ ψ − ψ N ∥ ∞

Original: = O ( 1 N ) .
Revised: = O ( 1 N ) .

Original: Furthermore, Π Nuniformly converges as N → ∞ to the mapping x 0 → x 1 where x 1 is the solution at time 1 of the
Revised: Furthermore, Π Nuniformly converges as N → ∞ to the mapping x 0 → x 1 where x 1 is the solution at time 1 of the

Original: Neural ODE d x d s ( s ) =
Revised: Neural ODE d x d s ( s ) =

Original: ψ ( s, t ) x
Revised: ψ ( s, t ) x

Original: ( s ) with initial condition x 0 .
Revised: ( s ) with initial condition x 0 .

Original:  The assumption on the existence of ψ init ensures a convergence at speed 1 N to a Neural ODE atoptimization time 0 .
Revised: [Figure] We illustrate Th. 1 in Figure 2. The assumption on the existence of ψ init ensures a convergence at speed1 N to a Neural ODE at optimization time 0 .

Original: Note that for instance, the constant initialization θ Nn (0)
Revised: Note that for instance, the constant initialization θ Nn (0)

Original: =
Revised: =

Original: θ 0 ∈R
Revised: θ 0 ∈

Original:  d × d satisfiesthis hypothesis.
Revised: R d × d satisfies this hypothesis.

Original: In order to prove Th. 1, for which a full proof is presented in appendix A.6, we firstpresent a useful lemma: the weights of the network have at least one accumulation point.
Revised: In order to prove Th.1, for which a full proof is presented in appendix A.5,we first present a useful lemma: the weights of thenetwork have at least one accumulation point.

Original: Lemma 1 (Existence of limit functions) .
Revised: Lemma 1 (Existence of limit functions) .

Original: For s ∈ [0 , 1] and t ∈ R + , let ψ N ( s, t )
Revised: For s ∈[0 , 1] and t ∈ R + , let ψ N ( s, t )

Original: =
Revised: =

Original: θ N
Revised: θ N

Original: ⌊ Ns ⌋ ( t ) .
Revised: ⌊ Ns ⌋ ( t ) .

Original: Underthe assumptions of Prop. 2, there exists a subsequence ψ σ ( N ) and ψ σ : [0 , 1] × R + → R d × d Lipschitzcontinuous with respect to both parameters s and t such that ψ σ ( N ) → ψ σ uniformly (in ∥ .
Revised: Underthe assumptions of Prop. 2, there exists a subsequence ψ σ ( N ) and ψ σ : [0 , 1] × R + → R d × d Lipschitz continuous with respect to both parameters sand t such that ψ σ ( N ) → ψ σ uniformly (in ∥ .

Original: ∥ ∞ w.r.t( s, t ) ).
Revised: ∥ ∞ w.r.t( s, t ) ).

Original: Sketch of proof.

Original: We first prove that Ψ N satisfies the almost equicontinuity property (i)

Original: ∥

Original: ψ N ( s 1 ) −ψ N ( s 2 )

Original: ∥ ≤ C ′ | t 1 − t 2 |

Original: + C | s 1 − s 2 | + CN and is uniformly bounded (ii) ∀ N ∈ N ,

Original: ∥ ψ N ∥ ∞ ≤ 12 .

Original: Wethen adapt a proof of the Arzelà–Ascoli theorem to prove the existence of a uniformly convergentsubsequence and show its Lipschitz property.

Original: Lemma 1 is proved in appendix A.3, and gives us the existence of a Lipschitz continuous accumulationpoint, but not the uniqueness nor the convergence speed.
Revised: Lemma 1 is proved in appendix A.3, and gives us the existence of a Lipschitz continuous accumulationpoint, but not the uniqueness nor the convergence speed.

Original: For the uniqueness, we show in appendix
Revised: For the uniqueness, we show in appendix

Original: A.6 that, under the assumptions of Th. 1, one has that any accumulation point of ψ σ satisfies the limit
Revised: A.5 that, under the assumptions of Th. 1, one has that any accumulation point of ψ σ satisfies the limit

Original: Neural ODE
Revised: Neural ODE

Original: [Equation]
Revised: [Equation]

Original: and show that F satisfies the hypothesis of the Picard–Lindelöf theorem, hence showing the 249 uniqueness of ψ .
Revised: and show that F satisfies the hypothesis of the Picard–Lindelöf theorem, hence showing the 257 uniqueness of ψ .

Original: We finally show that, as intuitively expected, trajectories of the weights of our linear
Revised: We finally show that, as intuitively expected, trajectories of the weights of our linear

Original: ResNets of depth N and 2 N remain close one to each other.
Revised: ResNets of depth N and 2 N remain close one to each other.

Original: This gives the convergence speed in
Revised: This gives the convergence speed in

Original: Th.
Revised: Th.

Original: See appendix A.4 for a proof.
Revised: See appendix A.4 for a proof.

Original: Lemma 2 (Closeness of trajectories) .
Revised: Lemma 2 (Closeness of trajectories) .

Original: Suppose asumption 1, ∥ θ Nn +1 (0) − θ Nn (0)
Revised: Suppose asumption 1, ∥ θ Nn +1 (0) − θ Nn (0)

Original: ∥ ≤ C 0 N for some
Revised: ∥ ≤ C 0 N for some

Original: C 0 > 0 and that ∥ θ Nn (0) − θ 2 N 2 n (0) ∥
Revised: C 0 > 0 and that ∥ θ Nn (0) − θ 2 N 2 n (0) ∥

Original: =
Revised: =

Original: O ( 1 N ) .
Revised: O ( 1 N ) .

Original: Thenone has that ∀ t ∈ R + , ∥ θ Nn ( t ) − θ 2 N 2 n ( t ) ∥ =
Revised: Then ∀ t ∈ R + , ∥ θ Nn ( t ) − θ 2 N 2 n ( t ) ∥ =

Original: O ( 1 N ) .
Revised: O ( 1 N ) .

Original:  Adjoint Method in Residual Networks
Revised:  Adjoint Method in Residual Networks

Original: In this section, we focus on a particularly useful feature of Neural ODEs and its applicability to
Revised: In this section, we focus on a particularly useful feature of Neural ODEs and its applicability to

Original: ResNets: their memory free backpropagation thanks to the adjoint method.
Revised: ResNets: their memory free backpropagation thanks to the adjoint method.

Original: We consider a ResNet(1) and try to invert it using reverse mode Euler discretization of the Neural ODE (2) when φ Θ isany smooth interpolation of the ResNet.
Revised: We consider a ResNet(1) and try to invert it using reverse mode Euler discretization of the Neural ODE (2) when φ Θ isany smooth interpolation of the ResNet.

Original: This corresponds to defining ˜ x N = x N and iterate forn ∈ { N − 1 , . . . 0 } :
Revised: This corresponds to defining ˜ x N = x N and iterate forn ∈ { N − 1 , . . . 0 } :

Original: [Equation]
Revised: [Equation]

Original: We then use the approximated activations (˜ x n ) n ∈ [ N ] as a proxy for the true activations ( x n ) n ∈ [ N ] tocompute gradients without storing the activations:
Revised: We then use the approximated activations (˜ x n ) n ∈ [ N ] as a proxy for the true activations ( x n ) n ∈ [ N ] tocompute gradients without storing the activations:

Original: [Equation]
Revised: [Equation]

Original: The approximate recovery of the activations in Eq.
Revised: The approximate recovery of the activations in Eq.

Original: (6) is implementable for any ResNet: there is noneed for particular architecture or forward rule modification.
Revised: (6) is implementable for any ResNet: there is noneed for particular architecture or forward rule modification.

Original: The drawback is that the recovery isonly approximate.
Revised: The drawback is that the recovery isonly approximate.

Original: We devote the remainder of the section to the study of the corresponding errors andto error reduction using second order
Revised: We devote the remainder of the section to the study of the corresponding errors andto error reduction using second order

Original: Heun’s method.
Revised: Heun’s method.

Original: We first show that, if f ( ., θ Nn ) and its derivativeare bounded by a constant independent of N , then the error for reconstructing the activations in thebackward scheme (6) is O ( 1 N ) .
Revised: We first show that, if f ( ., θ Nn ) and its derivativeare bounded by a constant independent of N , then the error for reconstructing the activations in thebackward scheme (6) is O ( 1 N ) .

Original: Proofs of the theoretical results are in appendix A.
Revised: Proofs of the theoretical results are in appendix A.

Original: Error for reconstructing activations.
Revised: Error for reconstructing activations.

Original: We consider the following assumption:
Revised: We consider the following assumption:

Original: Assumption 2.
Revised: Assumption 2.

Original: There exists constants C f and C df such that ∀ N ∈ N , ∀ n ∈
Revised: There exists constants C f and L f such that ∀ N ∈ N , ∀ n ∈

Original: [ N − 1] , ∥ f ( ., θ Nn ) ∥ ∞ ≤
Revised: [ N − 1] , ∥ f ( ., θ Nn ) ∥ ∞ ≤

Original: C f and ∥ D x [ f ( ., θ Nn )] ∥ ∞ ≤ L f .
Revised: C f and ∥ ∂ x [ f ( ., θ Nn )] ∥ ∞ ≤ L f .

Original: Then the error made by reconstructing the activations is in O ( 1 N ) .
Revised: Then the error made by reconstructing the activations is in O ( 1 N ) .

Original: Proposition 3 (Reconstruction error) .
Revised: Proposition 3 (Reconstruction error) .

Original: With assumption 2, one has ∥ x n − ˜ x n ∥ ≤ ( e Lf − 1) C f N + O ( 1 N 2 ) .
Revised: With assumption 2, one has ∥ x n − ˜ x n ∥ ≤ ( e Lf − 1) C f N + O ( 1 N 2 ) .

Original: Prop. 3 shows a slow convergence of the error for recovering activations.
Revised: Prop. 3 shows a slow convergence of the error for recovering activations.

Original: This bound does notdepend on the discrete derivative f ( ., θ Nn +1 )
Revised: This bound does notdepend on the discrete derivative f ( ., θ Nn +1 )

Original: − f ( ., θ Nn ) , contrarily to the errors between the ResNetactivations and the trajectory of the interpolating Neural ODE in Prop 1.
Revised: − f ( ., θ Nn ) , contrarily to the errors between the ResNetactivations and the trajectory of the interpolating Neural ODE in Prop 1.

Original: In summary, even thoughregularity in depth is necessary to imply closeness to a Neural ODE, it is not necessary to recoveractivations, and neither gradients, as we now show.
Revised: In summary, even thoughregularity in depth is necessary to imply closeness to a Neural ODE, it is not necessary to recoveractivations, and neither gradients, as we now show.

Original: Error in gradients when using the adjoint method.
Revised: Error in gradients when using the adjoint method.

Original: We use the result obtained in Prop. 3 toderive a bound in O ( 1 N 2 ) on the error made for computing gradients using formulas (7).
Revised: We use the result obtained in Prop. 3 toderive a bound in O ( 1 N 2 ) on the error made for computing gradients using formulas (7).

Original: Proposition 4 (Gradient error) .
Revised: Proposition 4 (Gradient error) .

Original: Suppose assumption 2.
Revised: Suppose assumption 2.

Original: Suppose in addition that ∂ x f ( ., θ ) admits a
Revised: Suppose in addition that ∂ x f ( ., θ ) admits a

Original: Lipschitz constant L df , ∂ θ f ( ., θ ) admits a Lipschitz constant ∆ , and an upper bound Ω , all of whichare independent of θ .
Revised: Lipschitz constant L df , ∂ θ f ( ., θ ) admits a Lipschitz constant ∆ , and an upper bound Ω , all of whichare independent of θ .

Original: Then one has ∥ ˜ ∇ θ Nn L − ∇ θ Nn L ∥ =
Revised: Then one has ∥ ˜ ∇ θ Nn L − ∇ θ Nn L ∥ =

Original: O ( 1 N 2 ) .
Revised: O ( 1 N 2 ) .

Original: For a proof, see appendix A.8, where we give the dependency of our upper bound as a function of∆ , L f , C f , Ω, C df and L df .
Revised: For a proof, see appendix A.7, where we give the dependency of our upper bound as a function of∆ , L f , C f , Ω and L df .

Original: Smoothness-dependent reconstruction with Heun’s method.
Revised: Smoothness-dependent reconstruction with Heun’s method.

Original: The bounds in Prop. 3 and 4 do notdepend on the smoothness with respect to the weights of the f ( ., θ Nn ) .
Revised: The bounds in Prop. 3 and 4 do notdepend on the smoothness with respect to the weights of the f ( ., θ Nn ) .

Original: Only the magnitude of theresiduals plays a role in the correct recovery of the activations and estimation of the gradient.
Revised: Only the magnitude of theresiduals plays a role in the correct recovery of the activations and estimation of the gradient.

Original: Hence,there is no apparent benefit of having such a network behave like a Neural ODE.
Revised: Hence,there is no apparent benefit of having such a network behave like a Neural ODE.

Original: We now turn to
Revised: We now turn to

Original: Heun’s method, a second order integration scheme, and show that in this case smoothness in depth ofthe network improves activation recovery.
Revised: Heun’s method, a second order integration scheme, and show that in this case smoothness in depth ofthe network improves activation recovery.

Original: A HeunNet [26] of depth N with parameters θ N 1 , . .
Revised: A HeunNet [30] of depth N with parameters θ N 1 , . .

Original: , θ NNiterates for n = 0 , . . .
Revised: , θ NNiterates for n = 0 , . . .

Original: , N − 1 :
Revised: , N − 1 :

Original: [Equation]

Original: and

Original: [Equation]
Revised: [Equation]

Original: These forward iterations can once again be approximately reversed by doing for n = N − 1 , . . .
Revised: These forward iterations can once again be approximately reversed by doing for n = N − 1 , . . .

Original: [Equation]
Revised: [Equation]

Original: which also enables approximated backpropagation without storing activations.
Revised: which also enables approximated backpropagation without storing activations.

Original: When discretizing an
Revised: When discretizing an

Original: ODE, Heun’s method has a better O ( 1 N 2 ) error, hence we expect a better recovery than in Prop. 3.
Revised: ODE, Heun’s method has a better O ( 1 N 2 ) error, hence we expect a better recovery than in Prop. 3.

Original: Indeed, we have:
Revised: Indeed, we have:

Original: Proposition 5 (Reconstruction error - Heun’s method) .
Revised: Proposition 5 (Reconstruction error - Heun’s method) .

Original: Assume assumption 2.
Revised: Assume assumption 2.

Original: Denote by L ′ f
Revised: Denote by L ′ f

Original: the
Revised: the

Original: Lipschitz constant of x (cid:55)→ 12 ( f ( x, θ Nn +1 ) + f ( x − 1 N f ( x, θ Nn +1 ) , θ Nn )) , by L θ the Lispchitz constantof
Revised: Lipschitz constant of x (cid:55)→ 12 ( f ( x, θ Nn +1 ) + f ( x − 1 N f ( x, θ Nn +1 ) , θ Nn )) , by L θ the Lispchitz constantof

Original: θ (cid:55)→ f ( · , θ ) and by L ′ θ that of θ (cid:55)→ ∂ x f ( ., θ ) .
Revised: θ (cid:55)→ f ( · , θ ) and by L ′ θ that of θ (cid:55)→ ∂ x f ( ., θ ) .

Original: Let C ′ f
Revised: Let C ′ f

Original: = 14 L ′ θ L θ .
Revised: = 14 L ′ θ L θ .

Original: Finally, define ∆ Nθ :=max n ∥ θ Nn +1 − θ Nn ∥ 2 .
Revised: Finally, define ∆ Nθ :=max n ∥ θ Nn +1 − θ Nn ∥ 2 .

Original: Using Heun’s method, we have: ∥ x n − ˜ x n ∥ ≤ 
Revised: Using Heun’s method, we have: ∥ x n − ˜ x n ∥ ≤ ( e L ′ f − [Equation] ) C ′ f L ′ f N ×

Original: [Equation]
Revised: [Equation]

Original: This bound is very similar to that in proposition 3, with an additional factor ∆ Nθ .
Revised: This bound is very similar to that in proposition 3, with an additional factor ∆ Nθ .

Original: Hence, we see thatunder thesmoothness condition that ∆ Nθ
Revised: Hence, we see thatunder the condition that ∆ Nθ

Original: = O ( 1 N ) , the reconstruction error ∥
Revised: = O ( 1 N ) , the reconstruction error ∥

Original: x n − ˜ x n ∥ is in O ( 1 N 2 ) .
Revised: x n − ˜ x n ∥ is in O ( 1 N 2 ) .

Original: In the linear case, we havepreviously proven under some hypothesis in Prop. 2 that such a conditionon ∆ Nθ holds during training.
Revised: In the linearcase, we have proven under some hypothesis in Prop. 2 that such a condition on ∆ Nθ holds duringtraining.

Original: Consequently, the smoothness of the weights of a HeunNet in turnshelps it recover the activations, while it is not true for a ResNet.
Revised: Consequently, the smoothness of the weights of a HeunNet in turns helps it recover theactivations, while it is not true for a ResNet.

Original: This provides better guarantees onthe error on gradients:
Revised: This provides better guarantees on the error on gradients:

Original: Proposition 6 (Gradient error - Heun’s method) .
Revised: Proposition 6 (Gradient error - Heun’s method) .

Original: Suppose assumption 2.
Revised: Suppose assumption 2.

Original: Suppose in addition that∂ x f ( ., θ ) admits Lipschitz constant L df , ∂ θ f ( ., θ ) admits a Lipschitz constant and an upper bound,all of which are independent of θ .
Revised: Suppose in addition that∂ x f ( ., θ ) admits Lipschitz constant, ∂ θ f ( ., θ ) admits a Lipschitz constant and an upper bound, all ofwhich are independent of θ .

Original: Then one has ∥ ˜ ∇ θ Nn L − ∇ θ Nn L ∥ =
Revised: Then one has ∥ ˜ ∇ θ Nn L − ∇ θ Nn L ∥ =

Original: O ( ∆ Nθ N 2 + 1 N 3 ) .
Revised: O ( ∆ Nθ N 2 + 1 N 3 ) .

Original: Just like with activation, we see that Heun’s method allows for a better gradient estimation whenthe weights are smooth with depth.
Revised: Just like with activation, we see that Heun’s method allows for a better gradient estimation whenthe weights are smooth with depth.

Original: Equivalently, for a fixed depth, this proposition indicates that
Revised: Equivalently, for a fixed depth, this proposition indicates that

Original: HeunNets have a better estimation of the gradient with the adjoint method than ResNets whichultimately leads to better training and overall better performances by such memory-free model.
Revised: HeunNets have a better estimation of the gradient with the adjoint method than ResNets whichultimately leads to better training and overall better performances by such memory-free model.

Original:  Experiments
Revised:  Experiments

Original: We now present experiments to investigate the applicability of the results presented in this paper.
Revised: We now present experiments to investigate the applicability of the results presented in this paper.

Original: Weuse Pytorch [27] and Nvidia Tesla V100 GPUs.
Revised: Weuse Pytorch [31] and Nvidia Tesla V100 GPUs.

Original: Our code will be open-sourced.
Revised: Our code will be open-sourced.

Original: All the experimentaldetails are given in Appendix B, and we provide a recap on ResNet architectures in appendix C.
Revised: All the experimentaldetails are given in appendix B, and we provide a recap on ResNet architectures in appendix C.

Original: [Figure]
Revised: [Figure]

Revised: a i n

Original: [Figure]
Revised: [Figure]

Original: 5.1 Validation of our model with step size 1 N
Revised: 5.1 Validation of our model with step size 1 N

Original: [Figure]

Original: The
Revised: The

Original: ResNet model (1) is different from the classical
Revised: ResNet model (1) is different from the classical

Original: ResNet because ofthe 1 N term.
Revised: ResNet because ofthe 1 N term.

Original: This makes the model depth aware, and we want to studythe impact of this modification on the accuracy on
Revised: This makes the model depth aware, and we want to studythe impact of this modification on the accuracy on

Original: CIFAR and
Revised: CIFAR and

Original: ImageNet.
Revised: ImageNet.

Original: [Table]
Revised: [Table]

Original: We first train a ResNet-101 [18] on CIFAR-10 and ImageNet using the same hyper-parameters.
Revised: We first train a ResNet-101 [20] on CIFAR-10 and ImageNet using the same hyper-parameters.

Original: Experimentaldetails are in appendix B and results are summarized intable 1, showing that the explicit addition of the step size1 N does not affect accuracy.
Revised: Experimentaldetails are in appendix B and results are summarized intable 1, showing that the explicit addition of the step size1 N does not affect accuracy.

Original: In strike contrast,in a weighttied setting, the classical ResNetforward rule is not compatible with the usual training procedureon CIFAR, as opposed to the  1 N scaling, as shown in Figure 3 (a).
Revised: In strike contrast, the classical ResNet rule without the scaling 1 N makes the network behave badly at large depth, while it still workswell with our scaling 1 N , as shown in Figure 3 (a).

Original: On ImageNet, the scaling 1 N alsoleads to similar test accuracy in the weight tied setting: 72 . 5% with 4 blocks per layer, 73 .
Revised: On ImageNet, the scaling 1 N also leads to similartest accuracy in the weight tied setting: 72 . 5% with 4 blocks per layer, 73 .

Original: 2% with 8blocks per layer and 72% with 16 blocks per layer (mean over 2 runs).
Revised: 2% with 8 blocks per layerand 72% with 16 blocks per layer (mean over 2 runs).

Original: 5.2 Adjoint method
Revised: 5.2 Adjoint method

Original: New training strategy.
Revised: New training strategy.

Original: Our results in Prop.
Revised: Our results in Prop.

Original: 3 and 4 assume uniform bounds in N onour residual functions and their derivatives.
Revised: 3 and 4 assume uniform bounds in N onour residual functions and their derivatives.

Original: We also formally proved in the linear settingthat these assumptions hold during the whole learning process if the initial loss is small.
Revised: We also formally proved in the linear settingthat these assumptions hold during the whole learning process if the initial loss is small.

Original: [Table]
Revised: [Table]

Original: A natural idea to start from a small loss is to considera pretrained model.
Revised: A natural idea to start from a small loss is to considera pretrained model.

Original: In addition, we also want our pretrained model to verify assumption 

Revised: In addition, we also want our pretrained model to verify assumption

Original: so we considerthe following setup.
Revised: so we considerthe following setup.

Original: On CIFAR (resp.
Revised: On CIFAR (resp.

Original: ImageNet) wetrain a ResNet with 4 (resp.
Revised: ImageNet) wetrain a ResNet with 4 (resp.

Original: 8) blocks in each layer,where weights are tied within each layer.
Revised: 8) blocks in each layer,where weights are tied within each layer.

Original: A first observation is that one can transfer these weightsto deeper ResNets without significantly affecting the test accuracy of the model: it remains above94 .
Revised: A first observation is that one can transfer these weightsto deeper ResNets without significantly affecting the test accuracy of the model: it remains above94 .

Original: 5% on CIFAR-10 and 72% on ImageNet.
Revised: 5% on CIFAR-10 and 72% on ImageNet.

Original: We then untie the weights of our models and refine them.
Revised: We then untie the weights of our models and refine them.

Original: More precisely, for CIFAR, we then transfer the weights of our model to a ResNet with 4 , 4 , 64 and 4 blocks within each layer and fine-tune it only by refining the third layer, using our adjoint method.
Revised: More precisely, for CIFAR, we then transfer the weights of our model to a ResNet with 4 , 4 , 64 and 4 blocks within each layer and fine-tune it only by refining the third layer, using our adjoint method.

Original: We display in table 2 the median of the new test accuracy, over 5 runs for the initial pretraining of themodel.
Revised: We display in table 2 the median of the new test accuracy, over 5 runs for the initial pretraining of themodel.

Original: For ImageNet, we transfer the weights to a ResNet with 100 blocks per layer and fine-tunethe whole model with our adjoint method for the residual layers.
Revised: For ImageNet, we transfer the weights to a ResNet with 100 blocks per layer and fine-tunethe whole model with our adjoint method for the residual layers.

Original: Results are summarized in table 2.
Revised: Results are summarized in table 2.

Original: To the best of our knowledge, this is the first time a Neural-ODE like ResNet achieves a test-accuracyof 75 .
Revised: To the best of our knowledge, this is the first time a Neural-ODE like ResNet achieves a test-accuracyof 75 .

Original: 1% on ImageNet.
Revised: 1% on ImageNet.

Original: [Figure]
Revised: [Figure]

Original: Failure in usual settings.
Revised: Failure in usual settings.

Original: In Prop. 3 we showed underassumption 2, that is if the residuals are bounded and Lipschitz continuous with constant independent of the depth
Revised: In Prop. 3 we showed underassumption 2, that is if the residuals are bounded and Lipschitz continuous with constant independent of the depth

Original: N , then the error for computing the activations backwardwould scale in 1 N as well as the error for the gradients(Prop. 4).
Revised: N , then the error for computing the activations backwardwould scale in 1 N as well as the error for the gradients(Prop. 4).

Original: First, this results shows that the architectureneeds to be deep enough, because it scales in 1 N : for instance, we fail to train a ResNet-101 [18] on the ImageNetdataset using the adjoint method on its third layer (depth23 ), as shown in Figure 3 (b).
Revised: First, this results shows that the architectureneeds to be deep enough, because it scales in 1 N : for instance, we fail to train a ResNet-101 [20] on the ImageNetdataset using the adjoint method on its third layer (depth23 ), as shown in Figure 3 (b).

Original: Success at large depth.
Revised: Success at large depth.

Original: To further investigate the applicability of the adjoint method for training deeper ResNets,we train a simple ResNet model on the CIFAR data set.
Revised: To further investigate the applicability of the adjoint method for training deeper ResNets,we train a simple ResNet model on the CIFAR data set.

Original: First, the input is processed by a 5 × 5 convolution with16 out channels, and the image is down-sampled to a size10 × 10 .
Revised: First, the input is processed by a 5 × 5 convolution with16 out channels, and the image is down-sampled to a size10 × 10 .

Original: We then apply a batch norm, a ReLU and iteraterelation (1) where f is a pre-activation basic block [19].
Revised: We then apply a batch norm, a ReLU and iteraterelation (1) where f is a pre-activation basic block [21].

Original: We consider the zero residual initialisation: the last batchnorm of each basic block is initialized to zero.
Revised: We consider the zero residual initialisation: the last batchnorm of each basic block is initialized to zero.

Original: We consider different values for the depth N and notice that in this setup, the deeper our model is, the betterit performs in term of test accuracy.
Revised: We consider different values for the depth N and notice that in this setup, the deeper our model is, the betterit performs in term of test accuracy.

Original: We then compare the performance of our model using a ResNet(forward rule (1)) or a HeunNet (forward rule (8)).
Revised: We then compare the performance of our model using a ResNet(forward rule (1)) or a HeunNet (forward rule (8)).

Original: We train our networks using either the classicalbackpropagation or our corresponding proxys using the adjoint method (formulas (6) and (9)).
Revised: We train our networks using either the classicalbackpropagation or our corresponding proxys using the adjoint method (formulas (6) and (9)).

Original: Wedisplay the final test accuracy (median over 5 runs) for different values of the depth N in Figure 4.
Revised: Wedisplay the final test accuracy (median over 5 runs) for different values of the depth N in Figure 4.

Original: The true backpropagation gives the same curves for the ResNet and the HeunNet.
Revised: The true backpropagation gives the same curves for the ResNet and the HeunNet.

Original: Approximatedgradients, however, lead to a large test error at small depth, but give the same performance at largedepth, hence confirming our results in Prop. 4 and 6.
Revised: Approximatedgradients, however, lead to a large test error at small depth, but give the same performance at largedepth, hence confirming our results in Prop. 4 and 6.

Original: In addition,  the  HeunNettrained with the adjointmethod performs better than  the ResNet.
Revised: In addition, at fixed depth, the accuracy whentraining a HeunNet with the adjoint method is better than for the ResNet with the adjoint method.

Original: This is to be linked with the two different bounds in Prop. 4and 6: for the HeunNet, smoothness with depth, which is expected at large depth, according to thetheoretical results for the linear case (Prop. 2), implies a faster convergence to the true gradients forthe HeunNet than for the ResNet.
Revised: This is to be linked with the two different bounds in Prop. 4 and 6: for the HeunNet, smoothness withdepth, which is expected at large depth, according to the theoretical results for the linear case (Prop.2), implies a faster convergence to the true gradients for the HeunNet than for the ResNet.

Original: We finally validate this convergence in Figure 3 (c): the deeper thearchitecture, the better the approximation on the gradients.
Revised: We finallyvalidate this convergence in Figure 3 (c): the deeper the architecture, the better the approximation onthe gradients.

Original: In addition, the HeunNet approximatesthe true gradient better than the ResNet.
Revised: In addition, the HeunNet approximates the true gradient better than the ResNet.

Original: Conclusion, limitations and future works
Revised: Conclusion, limitations and future works

Original: We propose a methodology to analyze how well a ResNet discretizes a Neural ODE.
Revised: We propose a methodology to analyze how well a ResNet discretizes a Neural ODE.

Original: The positiveresults predicted by our theory in the linear case are also observed in practice with real architectures:one can successfully use the adjoint method to train ResNets (or even more effectively HeunNets)using very deep architectures on CIFAR, or fine-tune them on ImageNet, without memory cost in theresidual layers.
Revised: The positiveresults predicted by our theory in the linear case are also observed in practice with real architectures:one can successfully use the adjoint method to train ResNets (or even more effectively HeunNets)using very deep architectures on CIFAR, or fine-tune them on ImageNet, without memory cost in theresidual layers.

Original: However, we also show that for large scale problems such as ImageNet classificationfrom scratch, the adjoint method fails at usual depths.
Revised: However, we also show that for large scale problems such as ImageNet classificationfrom scratch, the adjoint method fails at usual depths.

Original: Our work provides a theoretical guarantee for the convergence to a Neural ODE in the linear settingunder a small loss initialization.
Revised: Our work provides a theoretical guarantee for the convergence to a Neural ODE in the linear settingunder a small loss initialization.

Original: A natural extension would be to study the non-linear case.
Revised: A natural extension would be to study the non-linear case.

Original: Inaddition, the adjoint method is time consuming, and an improvement would be to propose a cheapermethod than a reverse mode traversal of the architecture for approximating the activations.
Revised: Inaddition, the adjoint method is time consuming, and an improvement would be to propose a cheapermethod than a reverse mode traversal of the architecture for approximating the activations.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes](b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [No]
Revised: [No]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] , all the
Revised: [Yes] , all the

Original: assumptions are stated.
Revised: assumptions are stated.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] , the proofs are in the
Revised: [Yes] , the proofs are in the

Original: supplementary material.
Revised: supplementary material.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes] , in appendix
Revised: [Yes] , in appendix

Original: B
Revised: B

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes] , in appendix B
Revised: [Yes] , in appendix B

Original: (c) Did you report error bars (e g., with respect to the random seed after running experi-
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [Yes] , see Figure 4 and table
Revised: [Yes] , see Figure 4 and table

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type

Original: of GPUs, internal cluster, or cloud provider)?
Revised: of GPUs, internal cluster, or cloud provider)?

Original: [Yes]
Revised: [Yes]

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] for Pytorch, CIFAR,
Revised: [Yes] for Pytorch, CIFAR,

Original: ImageNet
Revised: ImageNet

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?

Original: [N/A]
Revised: [N/A]