Original: Characteristic Neural Ordinary Differential Equations
Revised: Characteristic Neural Ordinary Differential Equations

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: We propose Characteristic-Neural Ordinary Differential Equations (C-NODEs), aframework for extending Neural Ordinary Differential Equations (NODEs) beyond
Revised: We propose Characteristic-Neural Ordinary Differential Equations (C-NODEs), aframework for extending Neural Ordinary Differential Equations (NODEs) beyond

Original: ODEs.
Revised: ODEs.

Original: While NODEs model the evolution of  latent variables as the solution to an ODE, C-NODE models the evolution of the latent variables as the solution of afamily of first-order quasi-linear partial differential equations (PDEs) along curveson which the PDEs reduce to ODEs, referred to as characteristic curves.
Revised: While NODEs model the evolution of a latent variables as the solution toan ODE, C-NODE models the evolution of the latent variables as the solution of afamily of first-order quasi-linear partial differential equations (PDEs) along curveson which the PDEs reduce to ODEs, referred to as characteristic curves.

Original: This inturn allows the application of the standard frameworks for solving ODEs, namelythe adjoint method.
Revised: This inturn allows the application of the standard frameworks for solving ODEs, namelythe adjoint method.

Original: Learning optimal characteristic curves for given tasks improvesthe performance and computational efficiency, compared to state of the art NODEmodels.
Revised: Learning optimal characteristic curves for given tasks improvesthe performance and computational efficiency, compared to state of the art NODEmodels.

Original: We prove that the C-NODE framework extends the classical NODE onclassification tasks by demonstrating explicit C-NODE representable functionsnot expressible by NODEs.
Revised: We prove that the C-NODE framework extends the classical NODE onclassification tasks by demonstrating explicit C-NODE representable functionsnot expressible by NODEs.

Original: Additionally, we present C-NODE-based continuousnormalizing flows, which describe the density evolution of latent variables alongmultiple dimensions.
Revised: Additionally, we present C-NODE-based continuousnormalizing flows, which describe the density evolution of latent variables alongmultiple dimensions.

Original: Empirical results demonstrate the improvements providedby the proposed method for classification and density estimation on CIFAR-10,
Revised: Empirical results demonstrate the improvements providedby the proposed method for classification and density estimation on CIFAR-10,

Original: SVHN, and MNIST datasets under a similar computational budget as the existing
Revised: SVHN, and MNIST datasets under a similar computational budget as the existing

Original: NODE methods.
Revised: NODE methods.

Original: The results also provide empirical evidence that the learnedcurves improve the efficiency of the system through a lower number of parametersand function evaluations compared with baselines.
Revised: The results also provide empirical evidence that the learnedcurves improve the efficiency of the system through a lower number of parametersand function evaluations compared with baselines.

Original:  Introduction
Revised:  Introduction

Original: Deep learning and differential equations share many connections, and techniques in the intersectionhave led to insights in both fields.
Revised: Deep learning and differential equations share many connections, and techniques in the intersectionhave led to insights in both fields.

Original: One predominant connection is based on certain neural networkarchitectures resembling numerical integration schemes, leading to the development of Neural
Revised: One predominant connection is based on certain neural networkarchitectures resembling numerical integration schemes, leading to the development of Neural

Original: Ordinary Differential Equations (NODEs) [5].
Revised: Ordinary Differential Equations (NODEs) [5].

Original: NODEs use a neural network parameterization ofan ODE to learn a mapping from observed variables to a latent variable that is the solution to thelearned ODE.
Revised: NODEs use a neural network parameterization of an ODE to learn a mapping from observed variables to a latent variable that is the solution to the learned ODE.

Original: A central benefit of NODEs is the constant memory cost, where backward passes arecomputed using the adjoint sensitivity method rather than backpropagating through individual forwardsolver steps.
Revised: A central benefit of NODEs is the constant memory cost, where backward passes are computedusing the adjoint sensitivity method rather than backpropagating through individual forward solversteps.

Original: Backpropagating through adaptive differential equation solvers to train large NODEswill often result in memory outage, as mentioned in [5].

Original: Moreover, NODEs provide a flexibleprobability density representation often referred to as continuous normalizing flows (CNFs).
Revised: Moreover, NODEs provide a flexible probability density representation often referred to ascontinuous normalizing flows (CNFs).

Original: However,since NODEs can only represent solutions to ODEs, the class of functions is somewhat limitedand may not apply to more general problems that do not have smooth and one-to-one mappings.
Revised: However, since NODEs can only represent solutions to ODEs,the class of functions is somewhat limited and may not apply to more general problems that do nothave smooth and one-to-one mappings.

Original: To address this limitation, a series of analyses based on methods from differential equations havebeen employed to enhance the representation capabilities of NODEs, such as the technique ofcontrolled differential equations [24], learning higher-order ODEs [32], augmenting dynamics [10],and considering dynamics with delay terms [55].
Revised: To address this limitation, a series of analyses based onmethods from differential equations have been employed to enhance the representation capabilities of NODEs, such as the technique of controlled differential equations [23], learning higher-order ODEs[31], augmenting dynamics [10], and considering dynamics with delay terms [53].

Original: Moreover, certain works consider generalizing the ODE case to partial differential equations (PDEs), such as in [40, 44].
Revised: Moreover, certainworks consider generalizing the ODE case to partial differential equations (PDEs), such as in Ruthottoand Haber [39], Sun et al.

Original: However, these methods donot use the adjoint method, removing the primary advantage of constant memory cost.
Revised: However, these methods do not use the adjoint method, removingthe primary advantage of constant memory cost.

Original: This leads usto the central question motivating the work: can we combine the benefits of the rich function class of PDEs with the efficiency of the adjoint method?
Revised: This leads us to the central question motivating thework: can we combine the benefits of the rich function class of PDEs with the efficiency of the adjointmethod?

Original: To do so, we propose a method of continuous-depthneural networks that solves a PDE over parametric curves that reduce the PDE to an ODE.
Revised: To do so, we propose a method of continuous-depth neural networks that solves a PDE over parametric curves that reduce the PDE to an ODE.

Original: Suchcurves are known as characteristics , and they define the solution of the PDE in terms of an ODE[15].
Revised: Such curves are known as characteristics , andthey define the solution of the PDE in terms of an ODE [15].

Original: The proposed Characteristic Neural Ordinary Differential Equations (C-NODE) learn both thecharacteristics and the ODE along the characteristics to solve the PDE over the data space.
Revised: The proposed Characteristic Neural Ordinary Differential Equations (C-NODE) learn both the characteristics and the ODE along thecharacteristics to solve the PDE over the data space.

Original: Thisallows for a richer class of models while still incorporating the same memory efficiency of the adjointmethod.
Revised: This allows for a richer class of models whilestill incorporating the same memory efficiency of the adjoint method.

Original: The proposed C-NODE is also an extension of existing methods, as it improves the empiricalaccuracy of these methods in classification tasks and image quality in generation tasks.
Revised: The proposed C-NODE isalso an extension of existing methods, as it improves the empirical accuracy of these methods inclassification tasks and image quality in generation tasks.

Original:  Related Work
Revised:  Related Work

Original: [Figure]
Revised: [Figure]

Original: We discuss the related work from both  machine learningand numerical analysis perspectives.
Revised: We discuss the related work from both a machine learningand numerical analysis perspective.

Original: 2.1 Machine Learning and ODEs
Revised: 2.1 Machine Learning and ODEs

Original: NODE is often motivated as a continuous form of a Residual Network (ResNet) [17], since the ResNet can be seen as a forward Euler integration scheme on the latent state [48].
Revised: NODE is often motivated as a continuous form of a Residual Network (ResNet) [17], since the ResNet can be seen as a forward Euler integration scheme on the latent state [46].

Original: Specifically, a ResNet is composed of multiple blocks where each block can be represented as:
Revised: Specifically, a ResNet is composed of multiple blocks where each block can be represented as:

Original: [Equation]
Revised: [Equation]

Original: where u t is the evolving hidden state at time t and f ( u t , θ ) represents the gradient at time t , namely dudt ( u t ) .
Revised: where u t is the evolving hidden state at time t and f ( u t , θ ) represents the gradient at time t , namely dudt ( u t ) .

Original: Generalizing the model to a step size given by ∆ t , we have:
Revised: Generalizing the model to a step size given by ∆ t , we have:

Original: [Equation]
Revised: [Equation]

Original: To adapt this model to a continuous setting, we let ∆ t →and obtain:
Revised: To adapt this model to a continuous setting, we let ∆ t →and obtain:

Original: [Equation]
Revised: [Equation]

Original: The model can then be evaluated through existing numerical integration techniques, as proposed by[5]:
Revised: The model can then be evaluated through existing numerical integration techniques, as proposed by[5]:

Original: [Equation]
Revised: [Equation]

Original: Numerical integration can then be treated as a black box, using numerical schemes beyond the forward Euler to achieve higher numerical precision.
Revised: Numerical integration can then be treated as a black box, using numerical schemes beyond the forward Euler to achieve higher numerical precision.

Original: However, since black box integrators can take an arbitrary number of intermediate steps, backpropagating through individual steps would require too much memory since the individual steps must be saved.
Revised: However, since black box integrators can take an arbitrary number of intermediate steps, backpropagating through individual steps would take too much memory since the individual steps must be saved.

Original: Chen et al.
Revised: Chen et al.

Original: [5] addressed this problem by using adjoint backpropagation, which has a constant memory usage.
Revised: [5] solved this problem by using adjoint backpropagation, which has a constant memory usage.

Original: For a given loss function on the terminal state of the hidden state L ( u ( t 1 )) , the adjoint a ( t ) is governed by another ODE:
Revised: For a given loss function on the terminal state of the hidden state L ( u ( t 1 )) , the adjoint a ( t ) is governed by another ODE:

Original: [Equation]
Revised: [Equation]

Original: that dictates the gradient with respect to the parameters.
Revised: that dictates the gradient with respect to the parameters.

Original: The loss L ( u ( t 1 )) can then be calculated bysolving another ODE (the adjoint) rather than backpropagating through the calculations involved inthe numerical integration.
Revised: The loss L ( u ( t 1 )) can then be calculated bysolving another ODE (the adjoint) rather than backpropagating through the calculations involved inthe numerical integration.

Original: However, the hidden state governed by an ODE imposes a limitation on the expressiveness of themapping.
Revised: However, the hidden state governed by an ODE imposes a limitation on the expressiveness of themapping.

Original: For example,  Dupont et al.
Revised: For example, in Dupont et al.

Original: [10] describes a notable limitation of NODEs is in the inabilityto represent dynamical systems with intersecting trajectories.
Revised: [10], the authors describe a notable limitation of NODEsis in the inability to represent dynamical systems with intersecting trajectories.

Original: In response to such limitations, manyworks have tried to increase the expressiveness of the mapping.
Revised: In response to suchlimitations, many works have tried to increase the expressiveness of the mapping.

Original: Dupont et al.
Revised: Dupont et al.

Original: [10] proposed to solvethe intersection trajectories problem by augmenting the vector space, lifting the points into additionaldimensions; Zhu et al.
Revised: [10]proposed to solve the intersection trajectories problem by augmenting the vector space, lifting thepoints into additional dimensions; Zhu et al.

Original: [55] included time delay in the equation to represent dynamical systems of greater complexity; Massaroli et al.
Revised: [53] included time delay in the equation to representdynamical systems of greater complexity; Massaroli et al.

Original: [32] proposed to condition the vector field on the inputs, allowingthe integration limits to be conditioned on the input; Massaroli et al.
Revised: [31] proposed to condition the vector field on the inputs, allowing the integration limits to be conditioned on the input; Massaroli et al.

Original: [32] and Norcliffe et al.
Revised: [31] and Norcliffe et al.

Original: [35]additionally proposed and proved a second-order ODE system can efficiently solve the intersectingtrajectories problem.
Revised: [34] additionally proposed and proved a second-order ODE system can efficientlysolve the intersecting trajectories problem.

Original: Multiple works have attempted to expand NODE systems to other common differential equationformulations.
Revised: Multiple works have attempted to expand NODE systems to other common differential equationformulations.

Original: Sun et al.
Revised: Sun et al.

Original: [44] employed a dictionary method and expanded NODEs to a PDE case,achieving high accuracies both in approximating PDEs and in classifying real-world image datasets.
Revised: [43] employed a dictionary method and expanded NODEs to a PDE case,achieving high accuracies both in approximating PDEs and in classifying real-world image datasets.

Original: However, Sun et al.
Revised: However, Sun et al.

Original: [44] suggested that the method is unstable when training with the adjointmethod and therefore is unable to make use of the benefits that come with training with adjoint.
Revised: [43] suggested that the method is unstable when training with the adjointmethod and therefore is unable to make use of the benefits that come with training with adjoint.

Original: Zhang et al.
Revised: Zhang et al.

Original: [53] proposed a normalizing flow approach based on the Monge-Ampere equation.
Revised: [51] proposed a normalizing flow approach based on the Monge-Ampere equation.

Original: However, Zhang et al.
Revised: However, Zhang et al.

Original: [53] did not consider using adjoint-based training.
Revised: [51] did not consider using adjoint-based training.

Original: Long et al.
Revised: Long et al.

Original: [30, 31], Raissiet al.
Revised: [29, 30], Raissiet al.

Original: [37], Brunton et al.
Revised: [36], Brunton et al.

Original: [3] considered discovering underlying hidden PDEs from data and predictdynamics of complex systems.
Revised: [3] considered discovering underlying hidden PDEs from data and predictdynamics of complex systems.

Original: Kidger et al.
Revised: Kidger et al.

Original: [24], Morrill et al.
Revised: [23], Morrill et al.

Original: [33, 34] used ideas from rough paththeory and controlled differential equations to propose a NODE architecture as a continuous recurrentneural network framework.
Revised: [32, 33] used ideas from rough paththeory and controlled differential equations to propose a NODE architecture as a continuous recurrentneural network framework.

Original: Multiple works have expanded to the stochastic differential equationssetting and developed efficient optimization methods for them [16,  22, 23, 25, 26, 28, 29, 49].
Revised: Multiple works have expanded to the stochastic differential equationssetting and developed efficient optimization methods for them [16, 21, 22, 24, 25, 27, 28, 47].

Original: Salviet al.
Revised: Salviet al.

Original: [41] considered stochastic PDEs for spatio-temporal dynamics prediction.
Revised: [40] considered stochastic PDEs for spatio-temporal dynamics prediction.

Original: Additionally, Chenet al.
Revised: Additionally, Chenet al.

Original: [6] models spatio-temporal data using NODEs, and Rubanova et al.
Revised: [6] models spatio-temporal data using NODEs, and Rubanova et al.

Original: [39], De Brouwer et al.
Revised: [38], De Brouwer et al.

Original: [8]makes predictions on time series data using NODEs.
Revised: [8]makes predictions on time series data using NODEs.

Original: Physical modeling is also a popular applicationof NODEs, as control problems are often governed by latent differential equations that can bediscovered with data driven methods [7, 14, 51, 54].
Revised: Physical modeling is also a popular applicationof NODEs, as control problems are often governed by latent differential equations that can bediscovered with data driven methods [7, 14, 49, 52].

Original: NODE systems have also been used for modeling the flow from a simple probability density toa complicated one [5].
Revised: NODE systems have also been used for modeling the flow from a simple probability density toa complicated one [5].

Original: Specifically, if u ( t ) ∈ R n follows the ODE du ( t ) /dt
Revised: Specifically, if u ( t ) ∈ R n follows the ODE du ( t ) /dt

Original: =
Revised: =

Original: f ( u ( t ))
Revised: f ( u ( t ))

Original: , wheref ( u ( t ))
Revised: , wheref ( u ( t ))

Original: ∈ R n , then its log likelihood from  [5, Appendix A] is given by:
Revised: ∈ R n , then its log likelihood from Chen et al. [5, Appendix A]:

Original: [Equation]
Revised: [Equation]

Original: The trace can be calculated efficiently with a Hutchinson trace estimator [13].
Revised: The trace can be calculated efficiently with a Hutchinson trace estimator [13].

Original: Subsequent workuses invertible ResNet, optimal transport theory, among other techniques to further improve theperformance of CNFs [1, 2, 4, 11, 18, 20, 21, 46, 50, 53].
Revised: Subsequent workuses invertible ResNet, optimal transport theory, among other techniques to further improve theperformance of CNFs [1, 2, 4, 11, 18, 20, 44, 48, 51].

Original: CNF is desirable for having no constraintson the type of neural network used, unlike discrete normalizing flows, which often have constraintson the structure of the latent features [9, 36, 38].
Revised: CNF is desirable for having no constraints onthe type of neural network used, unlike discrete normalizing flows, which often have constraints onthe structure of the latent features [9, 35, 37].

Original: CNFs also inspire development in other generativemodeling methods.
Revised: CNFs also inspire development in other generativemodeling methods.

Original: For instance, a score-based generative model can be seen as a probability flowmodeled with an ODE [42, 47].
Revised: For instance, a score-based generative model can be seen as a probability flowmodeled with an ODE [41, 45].

Original:  Method
Revised:  Method

Original: We describe the proposed C-NODE method in this section by first providing a brief introduction tothe method of characteristics (MoC) for solving PDEs with an illustrative example.
Revised: We describe the proposed C-NODE method in this section by first introducing the method ofcharacteristics (MoC) for solving PDEs with an illustrative example.

Original: We then discusshow we applythe MoC to our C-NODE framework.
Revised: We then discuss how to apply MoC to our C-NODE framework.

Original: We finally discuss the types of PDEs we candescribe using this method.
Revised: We finally discuss the types of PDEs we can describe using thismethod.

Original: 3.1 Method of Characteristics
Revised: 3.1 Method of Characteristics

Original: The MoC provides a procedure for transforming certain PDEs into ODEs along paths known ascharacteristics .
Revised: The MoC provides a procedure for transforming certain PDEs into ODEs along paths known ascharacteristics .

Original: In the most general sense, the method applies to general hyperbolic differentialequations; however, for illustration purposes, we will consider a canonical example using the inviscid
Revised: In the most general sense, the method applies to general hyperbolic differentialequations; however, for illustration purposes, we will consider a canonical example using the inviscid

Original: Burgers equation.
Revised: Burgers equation.

Original: A complete exposition on the topic can be found in  [15, Chapter 9], but weintroduce some basic concepts here for completeness.
Revised: A complete exposition on the topic can be found in Griffiths et al. [15, Chapter 9],but we will introduce some basic concepts here for completeness.

Original: Let u ( x, t ) :
Revised: Let u ( x, t ) :

Original: R × R + → R satisfy thefollowing inviscid Burgers equation
Revised: R × R + → R satisfythe following inviscid Burgers equation

Original: [Equation]
Revised: [Equation]

Original: where we dropped the dependence on x and t for ease of notation.
Revised: where we dropped the dependence on x and t for ease of notation.

Original: We are interested in the solutionof u over some bounded domain Ω ⊂ R × R + .
Revised: We are interested in the solutionof u over some bounded domain Ω ⊂ R × R + .

Original: Consider parametric forms for the spatial componentx ( s ) :
Revised: We will now introduce parametric forms for thespatial component x

Original:  [0 , T ] → R and temporal components t ( s ) :
Revised: ( s ) : [0 , 1] → R and temporal components t ( s ) :

Original: [0 , T ] → R + over the fictitious variable s ∈ [0 , T ] .
Revised: [0 , 1] → R + over the fictitiousvariable s ∈ [0 , 1] .

Original: Intuitively, this allows us to solve an equation on curves x, t as functions ofa variable s which wedenote ( x ( s ) , t ( s )) as the characteristic .
Revised: Intuitively, this allows us to solve an equation on curves x, t as functions of s which we denote ( x ( s ) , t ( s )) as the characteristic .

Original: Expanding, and writing d as the total derivative, we get
Revised: Expanding, and writing d as the total derivative,we get

Original: [Equation]
Revised: [Equation]

Original: Recalling the original PDE in (2) and substituting the proper terms into (3) for dx/ds = u, dt/ds = 1 , d u/ d s = 0 , we then recover (2).
Revised: Recalling the original PDE in (2) and substituting the proper terms into (3) for d x/ d s = u, d t/ d s = 1 , d u/ d s = 0 , we then recover (2).

Original: Note that we now have a system of 3 ODEs, which we cansolve to obtain the characteristics as x ( s ) 
Revised: Solving these equations, we can obtain the characteristics as x ( s ) =

Original: = us + x 0 and t ( s ) =
Revised: us + x 0 and t ( s ) =

Original: s + t 0 as functions of initial conditions x 0 , t 0  .
Revised: s + t 0 as functions of initial conditions x 0 , t 0 , and in the case of time, we let t 0 = 0 .

Original: Finally, by solving over a grid of initial conditions{ x( i ) 0} ∞ i =1 ∈ ∂ Ω , we can obtain the solution of the PDE over Ω .
Revised: Finally, by solving over many initial conditions x 0 ∈ ∂ Ω , we can obtain the solution of the PDE over Ω .

Original: Putting it all together, we have a new ODE that is written as
Revised: Putting it all together, we have a new ODE that is written as

Original: [Equation]
Revised: [Equation]

Original: where we can integrate over s through
Revised: where we can integrate over s through

Original: [Equation]
Revised: [Equation]

Original: using the adjoint method with boundary conditions x 0 , t 0 .
Revised: using the adjoint method with boundary conditions x 0 , t 0 .

Original: This contrasts the usual direct integrationoverthe variable t that is done in NODE; we now jointly couple the integration through the characteristics.
Revised: This contrasts the usual direct integrationover t that is done in NODE; we now jointly couple the integration through the characteristics.

Original: An example of solving this equation over multiple initial conditions is given in Figure 1 withthe contrast to standard NODE integration. To provide some intuition for using MoC, we note that MoC most generally applies to hyperbolic PDEs. The transport equation is an example of this family of PDEs, which roughly describes thepropagation of physical quantities through time.
Revised: Anexample of solving this equation over multiple initial conditions is given in Figure 1.

Original: Such equations are appropriate for deep learningtasks due to their ability to transport data into different regions of the state space.

Original: For instance, in aclassification task, we consider the problem of transporting high-dimensional data points that are notlinearly separable to spaces where they are linearly separable.

Original: Similarly, in generative modeling, wetransport a base distribution to data distribution.

Original: 3.2 Neural Representation of Characteristics
Revised: 3.2 Neural Representation of Characteristics

Original: In the proposed method, we learn the components involved in the MoC, namely the characteristicsand the function coefficients.
Revised: In the proposed method, we learn the components involved in the MoC, namely the characteristicsand the function coefficients.

Original: We now generalize the example given in 3.1, which involved twovariables, to a k -dimensional system.
Revised: We now generalize the example given in 3.1, which involved twovariables, to a k -dimensional system.

Original: Specifically, consider the following nonhomogeneous boundaryvalue problem (BVP)
Revised: Specifically, consider the following nonhomogeneous boundaryvalue problem (BVP)

Original: [Equation]
Revised: [Equation]

Original: Here, u : R k
Revised: Here, u : R k

Original: → R n is a multivariate map, a i : R k + n → R and c : R k + n → R n be functionsdependent on values of u and x ’s. This problem is well-defined and has a solution so long as 129 (cid:80) 
Revised: → R n is a multivariate map, a i : R k + n → R and c : R k + n → R n be functionsdependent on values of u and x ’s. This problem is well-defined and has a solution so long as 121 (cid:80) ki

Original: ki [Equation]
Revised: =1 a i ∂ u ∂x

Original: i is continuous [12].
Revised: i is continuous [12].

Original: MoC has historically been used in a scalar context, but generalization to the vector case is relativelystraightforward.
Revised: MoC has historically been used in a scalar context, but generalization to the vector case is relativelystraightforward.

Original: A proof of the generalization can be found in Appendix B.1.
Revised: A proof of the generalization can be found in Appendix B.1.

Original: We decompose the PDE in (4) into the following system of ODEs
Revised: Following MoC, wedecompose the PDE in (4) into the following system of ODEs

Original: [Equation]
Revised: [Equation]

Original: We represent this ODE system by parameterizing dx i /ds and
Revised: We represent this ODE system by parameterizing dx i /ds and

Original: ∂
Revised: ∂

Original: u
Revised: u

Original: /∂x
Revised: /∂x

Original: i with neural networks.
Revised: i with neural networks.

Original: Consequently, d u /ds is evolving according to (6).
Revised: Consequently, d u /ds is evolving according to (6).

Original: Following this expansion, we arrive at
Revised: Following this expansion, we arrive at

Original: [Equation]
Revised: [Equation]

Original: where we remove u ’s dependency on x ( s ) and x ’s dependency on s for simplicity of notation.
Revised: where we remove u ’s dependency on x ( s ) and x ’s dependency on s for simplicity of notation.

Original: In
Revised: In

Original: Equation (7), the functions J x u and d x /ds are learnable functions which are the outputs of deepneural networks with inputs x , u and parameters Θ 2 .
Revised: Equation (7), the functions J x u and d x /ds are learnable functions which are the outputs of deepneural networks with inputs x , u and parameters Θ 2 .

Original: 3.3 Conditioning on data
Revised: 3.3 Conditioning on data

Original: Previous works primarily modeled the task of classifying a set of data points with a fixed differentialequation, neglecting possible structural variations lying in the data.
Revised: Previous works primarily modeled the task of classifying a set of data points with a fixed differentialequation, neglecting possible structural variations lying in the data.

Original: Here, we condition C-NODEon each data point, thereby solving a PDE with a different initial condition.
Revised: Here, we condition C-NODEon each data point, thereby solving a PDE with a different initial condition.

Original: Specifically, considerthe term given by the integrand in (7).
Revised: Specifically, considerthe term given by the integrand in (7).

Original: The neural network representing the characteristic d x /ds isconditioned on the input data z ∈ R w .
Revised: The neural network representing the characteristic d x /ds isconditioned on the input data z ∈ R w .

Original: Define a feature extractor function g ( · ) : R w → R n and wehave
Revised: Define a feature extractor function g ( · ) : R w → R n and wehave

Original: [Equation]
Revised: [Equation]

Original: By introducing g ( z ) in (8), the equation describing the characteristics changes depending on thecurrent data point.
Revised: By introducing g ( z ) in (8), the equation describing the characteristics changes depending on thecurrent data point.

Original: This leads to the classification task being modeled with a family rather than onesingle differential equation.
Revised: This leads to the classification task being modeled with a family rather than onesingle differential equation.

Original: 3.4 Training C-NODEs
Revised: 3.4 Training C-NODEs

Original: After introducing the main components of C-NODEs, we can integrate them into a unified algorithm.
Revised: After introducing the main components of C-NODEs, we can integrate them into a unified algorithm.

Original: To motivate this section, and to be consistent with part of the empirical evaluation, we will considerclassification tasks with data { ( z j , y j ) } Nj =1 , z j ∈ R w , y j ∈ Z + .
Revised: To motivate this section, and to be consistent with part of the empirical evaluation, we will considerclassification tasks with data { ( z j , y j ) } Nj =1 , z j ∈ R w , y j ∈ Z + .

Original: For instance, z j may be an image,and y j is its class label.
Revised: For instance, z j may be an image,and y j is its class label.

Original: In the approach we pursue here, the image z j is first passed through afeature extractor function g ( · ; Θ 1 ) : R w → R n with parameters Θ 1 .
Revised: In the approach we pursue here, the image z j is first passed through afeature extractor function g ( · ; Θ 1 ) : R w → R n with parameters Θ 1 .

Original: The output of g is the featureu ( j )=
Revised: The output of g is the featureu ( j )=

Original: g ( z j ; Θ 1 ) that provides the boundary condition for the PDE on u ( j ) .
Revised: g ( z j ; Θ 1 ) that provides the boundary condition for the PDE on u ( j ) .

Original: We integrate alongdifferent characteristic curves indexed by s ∈ [0 , T ] with boundary condition u ( j ) ( x (0))
Revised: We integrate alongdifferent characteristic curves indexed by s ∈ [0 , T ] with boundary condition u ( j ) ( x (0))

Original: =
Revised: =

Original: u ( j ) 0 , andcompute the end values as given by (7), where we mentioned in Section 3.2,
Revised: u ( j ) 0 , andcompute the end values as given by (7), where we mentioned in Section 3.2,

Original: [Equation]
Revised: [Equation]

Original: Finally, u ( j ) ( x ( T )) is passed through another neural network, Φ( u ( j ) ( x ( T )); Θ 3 ) with inputu ( j ) ( x ( T )) and parameters Θ 3 whose output are the probabilities of each class labels for image
Revised: Finally, u ( j ) ( x ( T )) is passed through another neural network, Φ( u ( j ) ( x ( T )); Θ 3 ) with inputu ( j ) ( x ( T )) and parameters Θ 3 whose output are the probabilities of each class labels for image

Original: z j .
Revised: z j .

Original: The entire learning is now is reduced to finding optimal weights (Θ 1 , Θ 2 , Θ 3 ) which can be achievedby minimizing the loss
Revised: The entire learning is now is reduced to finding optimal weights (Θ 1 , Θ 2 , Θ 3 ) which can be achievedby minimizing the loss

Original: [Equation]
Revised: [Equation]

Original: where L ( · ) is a loss function of choice.
Revised: where L ( · ) is a loss function of choice.

Original: In Algorithm 1, we illustrate the implementation procedurewith the forward Euler method for simplicity for the framework but note any ODE solver can be used.
Revised: In Algorithm 1, we illustrate the implementation procedurewith the forward Euler method for simplicity for the framework but note any ODE solver can be used.

Original: 3.5 Combining MoC with Existing NODE Modifications
Revised: 3.5 Combining MoC with Existing NODE Modifications

Original: As mentioned in the Section 2, the proposed C-NODEs method can be used as an extension to existing
Revised: As mentioned in the Section 2, the proposed C-NODEs method can be used as an extension to existing

Original: NODE frameworks.
Revised: NODE frameworks.

Original: In all NODE modifications, the underlying expression of (cid:82) b a f ( t, u ; Θ)d t remainsthe same.
Revised: In all NODE modifications, the underlying expression of (cid:82) b a f ( t, u ; Θ)d t remainsthe same.

Original: Modifying this expression to (cid:82) b a J x u ( x , u ; Θ) d x /ds ( x , u ; u 0 ; Θ)d s results in
Revised: Modifying this expression to (cid:82) b a J x u ( x , u ; Θ) d x /ds ( x , u ; u 0 ; Θ)d s results in

Original: the proposed
Revised: the proposed

Original: C-NODE architecture, with the size of x being a hyperparameter.
Revised: C-NODE architecture, with the size of x being a hyperparameter.

Original: for each input data z j do extract image feature u ( s = 0)
Revised: C-NODE has a number of theoretical properties that contribute to its expressiveness. We providesome theoretical results on these properties in the proceeding sections. We also define continuousnormalizing flows (CNFs) with C-NODEs, extending the CNFs originally defined with NODEs.

Original:  =
Revised: As mentioned in [10], one limitation of NODE is that the mappings cannot represent intersectingdynamics. We prove by construction that the C-NODEs can represent some dynamical systems withintersecting trajectories in the following proposition: for each input data z j do extract image feature u ( s = 0) =

Original: g ( z j ; Θ 1 ) with a feature extractor neural network.
Revised: g ( z j ; Θ 1 ) with a feature extractor neural network.

Original: procedure Integration along s = 

Revised: procedure Integration along s =

Original: →for each time step s m do calculate d x ds ( x , u ; g ( z j ; Θ 1 ); Θ 2 ) and J x u ( x , u ; Θ 2 ) .
Revised: →for each time step s m do calculate d x ds ( x , u ; g ( z j ; Θ 1 ); Θ 2 ) and J x u ( x , u ; Θ 2 ) .

Original: calculate d u ds =
Revised: calculate d u ds =

Original: J x u d x ds .
Revised: J x u d x ds .

Original: calculate u ( s m +1 ) =
Revised: calculate u ( s m +1 ) =

Original: u ( s m ) + d u ds ( s m +1 − s m ) .
Revised: u ( s m ) + d u ds ( s m +1 − s m ) .

Original: end for end procedure classify
Revised: end for end procedure classify

Original: u
Revised: u

Original: ( s = 1) with neural network Φ( u ( x ( s = 1)) , Θ 3 ) .
Revised: ( s = 1) with neural network Φ( u ( x ( s = 1)) , Θ 3 ) .

Original: end for
Revised: end for

Original: Properties of C-NODEs

Original: C-NODE has a number of theoretical properties that contribute to its expressiveness.

Original: We providesome theoretical results on these properties in the proceeding sections.

Original: We also define continuousnormalizing flows (CNFs) with C-NODEs, extending the CNFs originally defined with NODEs.

Original: 4.1 Intersecting trajectories

Original: As mentioned in [10], one limitation of NODE is that the mappings cannot represent intersectingdynamics.

Original: We prove by construction that the C-NODEs can represent some dynamical systems withintersecting trajectories in the following proposition:

Original: Proposition 4.1.
Revised: Proposition 4.1.

Original: The C-NODE can represent a dynamical system on u ( s ) , du/ds = G ( s, u ) :
Revised: The C-NODE can represent a dynamical system on u ( s ) , du/ds = G ( s, u ) :

Original: R + × R → R , where when u (0) = 1 , then u (1) = u (0)
Revised: R + × R → R , where when u (0) = 1 , then u (1) = u (0)

Original: + (cid:82) 1 0 G ( s, u ) ds = 0 ; and when u (0) = 0 ,then u (1) =
Revised: + (cid:82) 1 0 G ( s, u ) ds = 0 ; and when u (0) = 0 ,then u (1) =

Original: u (0)
Revised: u (0)

Original: + (cid:82) 1 0 G ( s, u ) ds = 1 .
Revised: + (cid:82) 1 0 G ( s, u ) ds = 1 .

Original: Proof.
Revised: Proof.

Original: See Appendix B.2.
Revised: See Appendix B.2.

Original: 4.2 Density estimation with C-NODEs
Revised: 4.2 Density estimation with C-NODEs

Original: C-NODEs can also be used to define a continuous density flow that models the density of a variableover space subject to the variable satisfying a PDE.
Revised: C-NODEs can also be used to define a continuous density flow that models the density of a variableover space subject to the variable satisfying a PDE.

Original: Similar to the change of log probability of NODEs,as in (1), we provide the following proposition for C-NODEs:
Revised: Similar to the change of log probability of NODEs,as in (1), we provide the following proposition for C-NODEs:

Original: Proposition 4.2.
Revised: Proposition 4.2.

Original: Let u ( s ) be a finite continuous random variable with probability density functionp ( u ( s )) and let u ( s ) satisfy du ( s ) ds = (cid:80) ki
Revised: Let u ( s ) be a finite continuous random variable with probability density functionp ( u ( s )) and let u ( s ) satisfy du ( s ) ds = (cid:80) ki

Original: =1 ∂u∂x i dx i ds .
Revised: =1 ∂u∂x i dx i ds .

Original: Assuming ∂u∂x
Revised: Assuming ∂u∂x

Original: i and dx i ds are uniformly Lipschitzcontinuous in u and continuous in s , then the evolution of the log probability of u follows:
Revised: i and dx i ds are uniformly Lipschitzcontinuous in u and continuous in s , then the evolution of the log probability of u follows:

Original: [Equation]
Revised: [Equation]

Revised: i = ∂u ∂x i

Revised: dx i ds (cid:33)

Original: Proof.
Revised: Proof.

Original: See Appendix B.3.
Revised: See Appendix B.3.

Original: CNFs are continuous and invertible one-to-one mappings onto themselves, i.e., homeomorphisms.
Revised: CNFs are continuous and invertible one-to-one mappings onto themselves, i.e., homeomorphisms.

Original: Zhang et al.
Revised: Zhang et al.

Original: [52] proved that vanilla NODEs are not universal estimators of homeomorphisms, andaugmented neural ODEs (ANODEs) are universal estimators of homeomorphisms.
Revised: [50] proved that vanilla NODEs are not universal estimators of homeomorphisms, andaugmented neural ODEs (ANODEs) are universal estimators of homeomorphisms.

Original: We demonstratethat C-NODEs are pointwise estimators of homeomorphisms, which we formalize in the followingproposition:
Revised: We demonstratethat C-NODEs are universal estimators of homeomorphisms, which we formalize in the followingproposition:

Original: Proposition 4.3.
Revised: Proposition 4.3.

Original: Given any homeomorphism h :
Revised: Given any homeomorphism h :

Original: Υ → Υ , Υ ⊂ R p , initial condition u 0 , and time
Revised: Υ → Υ , Υ ⊂ R p , initial condition u 0 , and time

Original: T > 0 , there exists a flow u ( s, u 0 )
Revised: T > 0 , there exists a flow u ( s, u 0 )

Original: ∈ R n following duds = ∂u∂x dxds
Revised: ∈ R n following duds = ∂u∂x dxds

Original: + ∂u∂t dtds such that u ( T, u 0 )
Revised: + ∂u∂t dtds such that u ( T, u 0 )

Original: =
Revised: =

Original: h ( u 0 ) .
Revised: h ( u 0 ) .

Original: Proof.
Revised: Proof.

Original: See Appendix B.4.
Revised: See Appendix B.4.

Original: [Figure]

Original:  Experiments
Revised:  Experiments

Original: We present experiments on image classification tasks on benchmark datasets, image generation taskson benchmark datasets, PDE modeling [Figure] and time series prediction.
Revised: We present experiments on image classification tasks on benchmark datasets, image generation taskson benchmark datasets, PDE modeling, and time series prediction.

Original: 5.1 Classification Experiments with Image Datasets
Revised: 5.1 Classification Experiments with Image Datasets

Original: We first conduct experiments for classification tasks on high-dimensional image datasets, including
Revised: We first conduct experiments for classification tasks on high-dimensional image datasets, including

Original: MNIST, CIFAR- [Figure] and SVHN.
Revised: MNIST, CIFAR-10, and SVHN.

Original: We provide results for C-NODE and also combine the frameworkwith existing methods, including ANODEs [10], Input Layer NODEs (IL-NODEs) [32], and 2ndOrder NODEs [32].
Revised: We provide results for C-NODE and also combine the frameworkwith existing methods, including ANODEs [10], Input Layer NODEs (IL-NODEs) [31], and 2ndOrder NODEs [31].

Original: For all classification experiments, we set the encoder of input images forconditioning to be identity, i.e., g ( z ) = z , making the input into C-NODE the original image. Thisway, we focus exclusively on the performance of C-NODE.
Revised: [Figure]

Original: The results for the experiments with the adjoint method are reported in Table 1 and in Figure 2a.
Revised: The results for the experiments with the adjoint method are reported in Table 1 and in Figure 2a.

Original: Weinvestigate the performances of the models on classification accuracy and the number of functionevaluations (NFE) taken in the adaptive numerical integration.
Revised: Weinvestigate the performances of the models on classification accuracy and the number of functionevaluations (NFE) taken in the adaptive numerical integration.

Original: NFE is an indicator of the model’scomputational complexity, and can also be interpreted as the network depth for the continuous NODEsystem [5].
Revised: NFE is a indicator of the model’scomputational complexity, and can also be interpreted as the network depth for the continuous NODEsystem [5].

Original: Using a similar number of parameters, combining C-NODEs with different modelsconsistently results in higher accuracies and mostly uses smaller numbers of NFEs, indicating abetter parameter efficiency.
Revised: Using a similar number of parameters, combining C-NODEs with different modelsconsistently results in higher accuracies and mostly uses smaller numbers of NFEs, indicating a betterparameter efficiency.

Original: An ablation study on C-NODEs’ and NODEs’ parameters can be foundin Appendix C.2.The performance improvements can be observed, especially on CIFAR-10 and SVHN, where it seems the dynamics to be learned are too complex for ODE systems, requiring asophisticated model and a large number of NFEs.
Revised: The performance improvements can be observed, especially on CIFAR-10and SVHN, where it seems the dynamics to be learned are too complex for ODE systems, requiringa sophisticated model and a large number of NFEs.

Original: It appears that solving a PDE system along amultidimensional characteristic is beneficial for training more expressive functions with less complexdynamics, as can be seen in Figures 2a, 2b.
Revised: It appears that solving a PDE system along amultidimensional characteristic is beneficial for training more expressive functions with less complexdynamics, as can be seen in Figures 2a, 2b.

Original: We also report training results using a traditional backpropagation through the forward Euler solverin Figure 2b.
Revised: We also report training results using a traditional backpropagation through the forward Euler solverin Figure 2b.

Original: The experiments are performed using the same network architectures as the previousexperiments using the adjoint method.
Revised: The experiments are performed using the same network architectures as the previousexperiments using the adjoint method.

Original: It appears that C-NODEs converge significantly faster thanthe NODEs (usually in one epoch) and generally have a more stable training process with smallervariance.
Revised: It appears that C-NODEs converge significantly faster thanthe NODEs (usually in one epoch) and generally have a more stable training process with smallervariance.

Original: In experiments with MNIST, C-NODEs converge in only one epoch, while NODEs convergein roughly 15 epochs.
Revised: In experiments with MNIST, C-NODEs converge in only one epoch, while NODEs convergein roughly 15 epochs.

Original: This provides additional empirical evidence on the benefits of training using thecharacteristics.
Revised: This provides additional empirical evidence on the benefits of training using thecharacteristics.

Original: As shown in Figures 2a, 2b, compared to training with the adjoint method, trainingwith the forward Euler solver results in less variance, indicating a more stable training process.
Revised: As shown in Figures 2a, 2b, compared to training with the adjoint method, trainingwith the forward Euler solver results in less variance, indicating a more stable training process.

Original: At thesame time, training with the adjoint method results in more accurate models, as the adjoint methoduses a constant amount of memory, and can employ more accurate adaptive ODE solvers.
Revised: At thesame time, training with the adjoint method results in more accurate models, as the adjoint methoduses a constant amount of memory, and can employ more accurate adaptive ODE solvers.

Revised: 5.2 Ablation study on the dimensionality of C-NODE on classification tasks

Revised: We perform an ablation study on the impact of the number of dimensions of the C-NODE weimplement.

Revised: This study allows us to evaluate the relationship between the model performance and themodel’s limit of mathematical approximating power.

Revised: Empirical results show that as we increase thenumber of dimensions used in the C-NODE model, the C-NODE’s performance first improves andthen declines, due to overfitting.

Revised: We have found out that information criteria like AIC and BIC canbe successfully applied for dimensionality selection in this scenario.

Original: 5.2 Continuous normalizing flow with C-NODEs
Revised: Details of the ablation study canbe found in Appendix C. 5.3 Continuous normalizing flow with C-NODEs

Original: We compare the performance of CNFs defined with NODEs to with C-NODEs on MNIST, SVHN,and CIFAR-10.
Revised: We compare the performance of CNFs defined with NODEs to with C-NODEs on MNIST, SVHN,and CIFAR-10.

Original: We use a Hutchinson trace estimator to calculate the trace and use multi-scale
Revised: We use a Hutchinson trace estimator to calculate the trace and use multi-scaleconvolutional architectures as done in Dinh et al.

Original: [Table] convolutional architectures as done in [9, 13] 1 .
Revised: [9], Grathwohl et al.

Original: Differential equations are solved using the RungeKutta method  of order 5 of the Dormand-Prince-Shampine solver and trained with the adjoint method.
Revised: Differential equationsare solved using the adjoint method and a Runge-Kutta of order 5 of the Dormand-Prince-Shampine [Table] solver.

Original: Although the Euler forward method is faster, experimental results show that its fixed step size oftenleads to negative Bits/Dim, indicating the importance of adaptive solvers.
Revised: Although the Euler forward method is faster, experimental results show that its fixed step sizeoften leads to negative Bits/Dim, indicating the importance of adaptive solvers.

Original: As shown in table 2 andfigure 3, using a similar number of parameters, experimental results show that CNFs defined with
Revised: As shown in table 2and figure 3, using a similar number of parameters, experimental results show that CNFs defined with

Original: C-NODEs perform better than CNFs defined with NODEs in terms of Bits/Dim, as well as havinglower variance, and using a lower NFE on all of MNIST, CIFAR-10, and SVHN.
Revised: C-NODEs perform better than CNFs defined with NODEs in terms of Bits/Dim, as well as havinglower variance, and using a lower NFEs on all of MNIST, CIFAR-10, and SVHN.

Original: 5.3 PDE modeling with C-NODEs
Revised: 5.4 PDE modeling with C-NODEs

Revised: [Figure]

Revised: Training dynamics of CNFs on MNIST dataset with adjoint method.

Revised: We present

Revised: Bits/dim of the first 50 training epochs.

Revised: CNODE method achieves higher accuracy, while having a lower variance.

Original: We consider a synthetic regression example for a hyperbolic PDE with a known solution.
Revised: We consider a synthetic regression example for a hyperbolic PDE with a known solution.

Original: Since NODEs assume that the latent state is only dependent on a scalar (namely time), they cannot modeldependencies that vary over multiple spatial variables required by most PDEs.
Revised: Since NODEsassume that the latent state is only dependent on ascalar (namely time), they cannot model dependencies that vary over multiple spatial variables requiredby most PDEs.

Original: We quantifythedifferences in the representation capabilities by examining how well each method can representa linear hyperbolic PDE.
Revised: We quantify the differences in therepresentation capabilities by examining how welleach method can represent a linear hyperbolic PDE.

Original: We also modify the assumptions used in the classification and densityestimation experiments where the boundary conditions were constant as in (4).
Revised: We also modify the assumptions used in the classification and density estimation experiments wherethe boundary conditions were constant as in (4).

Original: We approximate thefollowing BVP:
Revised: Weapproximate the following BVP:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Table]
Revised: [Table]

Original: (10) has an analytical solution given by u ( x, t )
Revised: (10) has an analytical solution given by u ( x, t )

Original: = 2 x exp( t ) 2exp( t )
Revised: = 2 x exp( t ) 2exp( t )

Original: +1 .
Revised: +1 .

Original: We generate a training dataset byrandomly sampling 200 points ( x, t ) , x ∈ [1 , 2] , t ∈ [0 , 1] , as well as values u ( x, t ) at those points.
Revised: We generate a training dataset byrandomly sampling 200 points ( x, t ) , x ∈ [1 , 2] , t ∈ [0 , 1] , as well as values u ( x, t ) at those points.

Original: We test C-NODE and NODE on 200 points randomly sampled as ( x, t ) ∈ [1 , 2] × [0 , 1] .
Revised: We test C-NODE and NODE on 200 points randomly sampled as ( x, t ) ∈ [1 , 2] × [0 , 1] .

Original: For thisexperiment, C-NODE uses 809 parameters while NODE uses 1185 parameters.
Revised: For thisexperiment, C-NODE uses 809 parameters while NODE uses 1185 parameters.

Original: C-NODE deviates8.05% from the test dataset, while NODE deviates 30.52%.
Revised: C-NODE deviates8.05% from the test dataset, while NODE deviates 30.52%.

Original: Further experimental details can be foundin Appendix A.3.5.4 Time series prediction with C-NODEs
Revised: Further experimental details can be foundin Appendix A.3. 5.5 Time series prediction with C-NODEs

Original: [Figure]
Revised: Finally, we test C-NODEs and NODEs on a synthetic time series prediction problem. We definea function by u ( x, t ) = 2 x exp( t ) 2exp( t )+1 , and we sample ˜ u = u ( x, t ) + 0 .

Original: Training dynamics of CNFs on MNIST dataset with adjoint method. We present Bits/dim of thefirst 50 training epochs.
Revised: 1 ϵ t , where ϵ t ∼ N (0 , 1) overx ∈ [1 , 2] , t ∈ [0 , 1] to generate the training dataset.

Original: Finally, we test C-NODEs and NODEs on  thetimeseries prediction problem using the MuJoCo dataset[45]. We follow the experimental settings in [39],where we define an autoregressive model with the encoder being an ODE-RNN model and the decoder being a latent ODE 2 .
Revised: We test the performance on t ∈ [ n, n + 1] withn ∈ { 2 , . . . To make the problem more challenging, x values are omitted, and only t values areprovided during both training and testing.

Original: As shown in Figure 5, C-NODEsachieve lower testing mean squared errors (MSEs). After 100 training epochs, C-NODEs achieve 10.14%lower testing MSEs than NODEs.
Revised: As shown in Table 3, C-NODE produces more profoundimprovements over NODEs as time increases. [Table] [Table]

Original:  Discussion
Revised:  Discussion

Original: We describe an approach for extending NODEs tothe case of PDEs by solving a series of ODEs alongthe characteristics of a PDE.
Revised: We describe an approach for extending NODEs to the case of PDEs by solving a series of ODEs alongthe characteristics of a PDE.

Original: The approach appliesto any black-box ODE solver and can be combined with existing NODE-based frameworks.
Revised: The approach applies to any black-box ODE solver and can combinewith existing NODE-based frameworks.

Original: Weempirically showcase its efficacy on classification tasks while also demonstrating its success inimproving convergence using Euler forward method without the adjoint method.
Revised: We empirically showcase its efficacy on classification taskswhile also demonstrating its success in improving convergence using Euler forward method withoutthe adjoint method.

Original: Additionally, CNODE empirically achieves better performances on density estimation tasks, while being moreefficient with the number of parameters and using lower NFEs.
Revised: Additionally, C-NODE empirically achieves better performances on densityestimation tasks, while being more efficient with the number of parameters and using lower NFEs.

Original: C-NODE’s efficiency over physicalmodeling and time series prediction is also highlighted with additional experiments.
Revised: C-NODE’s efficiency over physical modeling and time series prediction is also highlighted withadditional experiments.

Original: Limitations There are several limitations to the proposed method.
Revised: Limitations There are several limitations to the proposed method.

Original: The MoC only applies tohyperbolic PDEs, and we only consider first-order semi-linear PDEs in this paper.
Revised: The MoC only applies tohyperbolic PDEs, and we only consider first-order semi-linear PDEs in this paper.

Original: This may be alimitation since this is a specific class of PDEs that does not model all data.
Revised: This may be alimitation since this is a specific class of PDEs that does not model all data.

Original: We also did not enforceany particular structure to prevent characteristics from intersecting, which may result in shock wavesand rarefactions.

Original: However, we believe that this is unlikely to happen due to the high dimensionalityof the ambient space.

Original: We additionally note that, compared to ANODE, C-NODE’s training is not asstable.
Revised: We additionally notedthat, compared to ANODE, C-NODE’s training is not as stable.

Original: This can be improved by coupling C-NODEs with ANODEs or other methods.
Revised: This can be improved by coupling C-NODEs with ANODEs or other methods.

Original: A Experimental DetailsA.1 Experimental details of classification tasks
Revised: A Experimental Details

Revised: A.1 Experimental details of classification tasks

Original: We report the average performance over five independent training processes, and the models aretrained for 100 epochs for all three datasets.
Revised: We report the average performance over five independent training processes, and the models aretrained for 100 epochs for all three datasets.

Original: The input for 2nd-Ord, NODE, and C-NODE are the original images.
Revised: The input for 2nd-Ord, NODE, and C-NODE are the original images.

Original: In the IL-NODE, we transformthe input to a latent space before the integration by the integral; that is, we raise the R c × h × wdimensional input image into the R ( c + p ) × h × w dimensional latent feature space 3 .
Revised: In the IL-NODE, we transformthe input to a latent space before the integration by the integral; that is, we raise the R c × h × wdimensional input image into the R ( c + p ) × h × w dimensional latent feature space 1 .

Original: We decode theresult after performing the continuous transformations along characteristics curves, back to the 448 R c × h × w dimensional object space.
Revised: We decode theresult after performing the continuous transformations along characteristics curves, back to the 438 R c × h × w dimensional object space.

Original: Combining this with the C-NODE can be seen as solving a PDEon the latest features of the images rather than on the images directly.
Revised: Combining this with the C-NODE can be seen as solving a PDEon the latest features of the images rather than on the images directly.

Original: We solve first-order PDEs withthree variables in CIFAR-10 and SVHN and solve first-order PDEs with two variables in MNIST.
Revised: We solve first-order PDEs withthree variables in CIFAR-10 and SVHN and solve first-order PDEs with two variables in MNIST.

Original: The number of parameters of the models is similar by adjusting the number of features used in thenetworks.
Revised: The number of parameters of the models is similar by adjusting the number of features used in thenetworks.

Original: We use similar training hyperparameters as [32].
Revised: We use similar training hyperparameters as [31].

Original: Unlike ODEs, we take derivatives with respect to different variables in PDEs.
Revised: Unlike ODEs, we take derivatives with respect to different variables in PDEs.

Original: For a PDE with kvariables, this results in the constraint of the balance equations
Revised: For a PDE with kvariables, this results in the constraint of the balance equations

Original: [Equation]
Revised: [Equation]

Original: This can be satisfied by defining the k -th derivative with a neural network, and integrate k − 1 timesto get the first order derivatives.
Revised: This can be satisfied by defining the k -th derivative with a neural network, and integrate k − 1 timesto get the first order derivatives.

Original: Another way of satisfying the balance equation is to drop thedependency on the variables, i.e., ∀ i ∈ { 1 , 2 , ..., k } ,
Revised: Another way of satisfying the balance equation is to drop thedependency on the variables, i.e., ∀ i ∈ { 1 , 2 , ..., k } ,

Original: [Equation]
Revised: [Equation]

Original: When we drop the dependency, all higher order derivatives are zero, and the balance equations aresatisfied.
Revised: When we drop the dependency, all higher order derivatives are zero, and the balance equations aresatisfied.

Original: All experiments were performed on NVIDIA RTX 3090 GPUs on a cloud cluster.
Revised: All experiments were performed on NVIDIA RTX 3090 GPUs on a cloud cluster.

Original: A.2 Experimental details of continuous normalizing flows
Revised: A.2 Experimental details of continuous normalizing flows

Original: We report the average performance over four independent training processes.
Revised: We report the average performance over four independent training processes.

Original: As shown in Figure 4,compared to NODE, using a C-NODE structure improves the stability of training, as well as having abetter performance.
Revised: As shown in Figure 4,compared to NODE, using a C-NODE structure improves the stability of training, as well as having abetter performance.

Original: Specifically, the standard errors for C-NODEs on MNIST, SVHN, and CIFAR-10are 0.37%, 0.51%, and 0.24% respectively, and for NODEs the standard errors on MNIST, SVHN,and CIFAR-10 are 1.07%, 0.32%, and 0.22% respectively.
Revised: Specifically, the standard errors for C-NODEs on MNIST, SVHN, and CIFAR-10are 0.37%, 0.51%, and 0.24% respectively, and for NODEs the standard errors on MNIST, SVHN,and CIFAR-10 are 1.07%, 0.32%, and 0.22% respectively.

Original: The experiments are developed using code adapted from the code that the authors of [13] provided inhttps://github.com/rtqichen/ffjord .
Revised: The experiments are developed using code adapted from the code that the authors of [13] provided inhttps://github.com/rtqichen/ffjord .

Original: All experiments were performed on NVIDIA RTX 3090 GPUs on a cloud cluster.
Revised: All experiments were performed on NVIDIA RTX 3090 GPUs on a cloud cluster.

Original: A.3 Experimental details of PDE modeling
Revised: A.3 Experimental details of PDE modeling

Original: We want to solve the initial value problem
Revised: We want to solve the initial value problem

Original: [Equation]
Revised: [Equation]

Original: where the exact solution is u ( x, t ) = 2 xe t (2 e t +1) .
Revised: where the exact solution is u ( x, t ) = 2 xe t (2 e t +1) .

Original: Our dataset’s input are 200 randomly sampled points( x, t ) , x ∈ [1 , 2] , t ∈ [0 , 1] , and the dataset’s outputare the exact solutions at those points.
Revised: Our dataset’s input are 200 randomly sampled points( x, t ) , x ∈ [1 , 2] , t ∈ [0 , 1] , and the dataset’s outputare the exact solutions at those points.

Original: For the C-NODE architecture, we define four networks: NN 1 ( x, t ) for ∂u∂x , NN 2 ( x, t ) for ∂u∂t ,
Revised: For the C-NODE architecture, we define four networks: NN 1 ( x, t ) for ∂u ∂x , NN 2 ( x, t ) for ∂u ∂t ,

Original: NN 3 ( t ) for the characteristic path ( x ( s ) , t ( s )) , NN 4 ( x ) for the initial condition.
Revised: NN 3 ( t ) for the characteristic path ( x ( s ) , t ( s )) , NN 4 ( x ) for the initial condition.

Original: The result iscalculated in four steps:
Revised: The result iscalculated in four steps:

Original: Epoch
Revised: Epoch

Original: B i t s / d i m
Revised: B i t s / d i m

Original: C-NODE
Revised: C-NODE

Original: NODE
Revised: NODE

Original: Epoch 2.
Revised: Epoch 2.

Revised: [Figure]

Original: B i t s / d i m
Revised: i t s / d i m

Original: Epoch
Revised: Epoch

Original: B i t s / d i m
Revised: B i t s / d i m

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ds as before.2.
Revised: ds as before.2.

Original: Given x, t , solve equation ι + NN 3 ( NN 4 ( ι ))
Revised: Given x, t , solve equation ι + NN 3 ( NN 4 ( ι ))

Original: [0] ∗ t = x for ι iteratively, with ι n +1 =x − NN 3 ( NN 4 ( ι n ))
Revised: [0] ∗ t = x for ι iteratively, with ι n +1 =x − NN 3 ( NN 4 ( ι n ))

Original: [0] ∗ t .
Revised: [0] ∗ t .

Original: ι 0 is initialized to be x .3.
Revised: ι 0 is initialized to be x .

Original: Calculate initial value u ( x (0) , t (0))
Revised: Calculate initial value u ( x (0) , t (

Original: =
Revised: =

Original: NN 4 ( ι ) .
Revised: NN 4 ( ι ) .

Original: u ( x, t ) =
Revised: u [Equation] x

Original: ∆ u + u ( x (0) , t (0))
Revised: [Equation] t [Equation] [Equation]

Original:  For the NODE architecture, we define one network: NN 1 ( x, t ) for ∂u∂t .
Revised: [Equation] For the NODE architecture, we define one network: NN 1 ( x, t ) for ∂u∂t .

Original: The result is calculated asu ( x, t ) =
Revised: The result is calculated asu ( x, t ) =

Original: (cid:82) t 0 ∂u∂t
Revised: (cid:82) t 0 ∂u∂t

Original: dt = (cid:82) t 

Revised: dt = (cid:82) t

Original: NN 1 dt .
Revised: NN 1 dt .

Original: All experiments were performed on NVIDIA RTX 3080 ti GPUs on a local machine.
Revised: All experiments were performed on NVIDIA RTX 3080 ti GPUs on a local machine.

Original: A.4 Experimentalresults and details of time series predictions
Revised: A.4 Experimental details of time series predictions

Original: A.4.1 Experimental details of time series predictions on MuJoCo dataset

Original: We follow the experimental setup as described in https://github.com/YuliaRubanova/latent_ode .

Original: NODE’s training follows the original setup, with the dimension of the recognitionmodel being 30, the number of units per layer in each of GRU update networks being 100, the numberof units per layer in ODE function being 300, the number of layers in ODE function in generativeand recognition ODE both being 3.

Original: We use a C-NODE with a dimensionality of 128.

Original: The number of units per layer in the networkdescribing d x /d s is 12.

Original: For the network describing ∂ u /∂x i , the dimension of the recognition modelis 30, the number of units per layer in each of GRU update networks is 100, the number of units perlayer in the ODE function is 100, the number of layers in ODE function in generative and recognition

Original: ODE is 1.

Original: [Figure]

Original: with the “Hopper” model from the Deepmind Control Suit [45].

Original: We present testing mean squared error (MSE) of training epochs 10 to 100.

Original: The C-NODE method achieves lower testing MSE while having a lower variance.

Original: A.4.2 Experiment results of time series predictions on synthetic dataset

Original: We test C-NODEs, ANODEs, and NODEs on a synthetic time series prediction problem.

Original: We definea function by u ( x, t )

Original: = 2 x exp( t ) 2exp( t )+1 , and we sample ˜ u

Original: = u ( x, t ) + 0 .

Original: 1 ϵ t , where ϵ t ∼ N (0 , 1) overx ∈ [1 , 2] , t ∈ [0 , 1] to generate the training dataset.

Original: We test the performance on t ∈ [ n, n + 1]with n ∈ { 0 , 1 , . . .

Original: To make the problem more challenging, x values are omitted, and only tvalues are provided during both training and testing.

Original: As shown in Table 3, C-NODE produces moreprofound improvements over NODEs as time increases.

Original: Time

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: We also test C-NODEs, NODEs, and ANODEs on time series prediction with different levels ofnoise.

Original: Specifically, using the same function as above, we form training and testing dataset withϵ t ∼ N (0 , m ) , m ∈ { 0 , 1 , . . .

Original: We test the performance on the time period t ∈ [0 , 1] .

Original: [Table]

Original: Errors are mean errors.

Original: A.4.3 Experimental details of time series predictions on synthetic dataset

Original: We want to predict u ( x, t ) = 2 · x · e t 2 · e t +1 at different time t , with x ∈ [1 , 2] , and x being not accessible tothe network.
Revised: We want to predict u ( x, t ) = 2 · x · e t 2 · e t +1 at different time t , with x ∈ [1 , 2] , and x being not accessible tothe network.

Original: We also provide the network with the value of u (1 , 0) .
Revised: We also provide the network with the value of u (1 , 0) .

Original: We use a 8 dimensional C-NODE network.
Revised: We use a 8 dimensional C-NODE network.

Original: The result is calculated withu ( x, t ) = u (1 , 0) + (cid:90) t 8 (cid:88) i = ∂u
Revised: The result is calculated with

Original: [Equation]

Original: dz

Original: [Equation]
Revised: [Equation]

Original: ds ds.

Original: NODE is calculated with
Revised: NODE is calculated with

Original: [Equation]
Revised: [Equation]

Original: In our experiments, C-NODEs use 1221 parameters, ANODEs use 1270 parameters, NODEs use1290 parameters.
Revised: C-NODE uses 9744 parameters, and NODE uses 9697 parameters.

Original: All experiments were performed on NVIDIA RTX 3080 ti GPUs on a local machine.
Revised: All experiments were performed on NVIDIA RTX 3080 ti GPUs on a local machine.

Original: B Approximation Capabilities of C-NODE
Revised: B Approximation Capabilities of C-NODE

Original: Proposition B.1 (Method of Characteristics for Vector Valued PDEs) .
Revised: Proposition B.1 (Method of Characteristics for Vector Valued PDEs) .

Original: Let u ( x 1 , . . .
Revised: Let u ( x 1 , . . .

Original: , x k )
Revised: , x k )

Original: : R k
Revised: : R k

Original: → R nbe the solution of a first order semilinear PDE on a bounded domain Ω ⊂ R k of the formk 
Revised: → R nbe the solution of a first order semilinear PDE on a bounded domain Ω ⊂ R k of the formk (cid:88) i

Original: [Equation]
Revised: [Equation]

Original: Additionally, let a = ( a 1 , . .
Revised: Additionally, let a = ( a 1 , . .

Original: . , a k ) T : R k + n → R k , c : R k + n → R n be Lipschitz continuousfunctions.
Revised: . , a k ) T : R k + n → R k , c : R k + n → R n be Lipschitz continuousfunctions.

Original: Define a system of ODEs as
Revised: Define a system of ODEs as

Original: [Equation]
Revised: [Equation]

Original: where x 0 and u 0 define the initial condition, ∂ Ω is the boundary of the domain Ω .
Revised: where x 0 and u 0 define the initial condition, ∂ Ω is the boundary of the domain Ω .

Original: Given initialconditions x 0 , u 0 , the solution of this system of ODEs U ( s ) :
Revised: Given initialconditions x 0 , u 0 , the solution of this system of ODEs U ( s ) :

Original: [ a, b ] → R d is equal to the solution ofthe PDE in Equation (11) along the characteristic curve defined by x ( s ) , i.e., u ( x ( s ))
Revised: [ a, b ] → R d is equal to the solution ofthe PDE in Equation (11) along the characteristic curve defined by x ( s ) , i.e., u ( x ( s ))

Original: =
Revised: =

Original: U ( s ) .
Revised: U ( s ) .

Original: Theunion of solutions U ( s ) for all x 0 ∈ ∂ Ω is equal to the solution of the original PDE in Equation(11) for all x ∈ Ω .
Revised: Theunion of solutions U ( s ) for all x 0 ∈ ∂ Ω is equal to the solution of the original PDE in Equation(11) for all x ∈ Ω .

Original: Lemma B.2 (Gronwall’s Lemma [19]) .
Revised: Lemma B.2 (Gronwall’s Lemma [19]) .

Original: Let U ⊂ R n be an open set.
Revised: Let U ⊂ R n be an open set.

Original: Let f : U × [0 , T ] → R n be a continuous function and let h 1 , h 2 : [0 , T ] → U satisfy the initial value problems:
Revised: Let f : U × [0 , T ] → R n be a continuous function and let h 1 , h 2 : [0 , T ] → U satisfy the initial value problems:

Original: [Equation]
Revised: [Equation]

Original: If there exists non-negative constant C such that for all t ∈ [0 , T ]
Revised: If there exists non-negative constant C such that for all t ∈ [0 , T ]

Original: [Equation]
Revised: [Equation]

Original: where ∥ · ∥ is the Euclidean norm.
Revised: where ∥ · ∥ is the Euclidean norm.

Original: Then, for all t ∈ [0 , T ] ,
Revised: Then, for all t ∈ [0 , T ] ,

Original: [Equation]
Revised: [Equation]

Original: B.1 Proof of Proposition B.1 
Revised: B.1 Proof of Proposition B.

Original: This proof is largely based on the proof for the univarate case provided at 4 .
Revised: This proof is largely based on the proof for the univarate case provided at 1 .

Original: We extend for the vectorvalued case.
Revised: We extend for the vectorvalued case.

Original: Proof.
Revised: Proof.

Original: For PDE on u with k input, and an n -dimensional output, we have a i : R k + n → R , ∂ u ∂x i ∈ R n ,and c : R k + n → R n .
Revised: For PDE on u with k input, and an n -dimensional output, we have a i : R k + n → R , ∂ u ∂x i ∈ R n ,and c : R k + n → R n .

Original: In proposition B.1, we look at PDEs in the following form
Revised: In proposition B.1, we look at PDEs in the following form

Original: [Equation]
Revised: [Equation]

Original: Defining and substituting x = ( x 1 , . .
Revised: Defining and substituting x = ( x 1 , . .

Original: , x k ) ⊺ , a = ( a 1 , . .
Revised: , x k ) ⊺ , a = ( a 1 , . .

Original: . , a k ) ⊺ , and Jacobian J ( u ( x )) =(
Revised: . , a k ) ⊺ , and Jacobian J ( u ( x )) =(

Original: ∂
Revised: ∂

Original: u ∂x 1 , ..., ∂ u ∂x k ) ∈ R n × k into Equation (11) result in
Revised: u ∂x 1 , ..., ∂ u ∂x k ) ∈ R n × k into Equation (11) result in

Original: [Equation]
Revised: [Equation]

Original: From proposition B.1, the characteristic curves are given by
Revised: From proposition B.1, the characteristic curves are given by

Original: [Equation]
Revised: [Equation]

Original: , and the ODE system is given by
Revised: and the ODE system is given by

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Define the difference between the solution to (15) and the PDE in (11) as
Revised: Define the difference between the solution to (15) and the PDE in (11) as

Original: [Equation]
Revised: [Equation]

Original: Differentiating ∆( s ) with respect to s and plugging in (14), we get
Revised: Differentiating ∆( s ) with respect to s and plugging in (14), we get

Original: [Equation]
Revised: [Equation]

Original: ) (13) gives us (cid:80) ki
Revised: (13) gives us (cid:80) ki

Original: =1 a i ( x 1 , . . .
Revised: =1 a i ( x 1 , . . .

Original: , x k , u ) ∂ u ∂x i [Equation] c ( x 1 , . . .
Revised: , x k , u ) ∂ u ∂x i − c ( x 1 , . . .

Original: , x k , u ) = 0 .
Revised: , x k , u ) = 0 .

Original: Plugging this equality into (16)and rearrange terms, we have[Equation] U [Equation]
Revised: Plugging this equality into (16)and rearrange terms, we have

Original: −

Original: [Equation]
Revised: [Equation]

Original: Combining terms, we have
Revised: Combining terms, we have

Original: [Equation]
Revised: [Equation]

Original: Applying triangle inequality, we have
Revised: Applying triangle inequality, we have

Original: [Equation]
Revised: [Equation]

Original: By the assumption in proposition B.1, a and c are Lipschitz continuous.
Revised: By the assumption in proposition B.1, a and c are Lipschitz continuous.

Original: By Lipschitz continuity, wehave ∥ a ( U ) − a ( u ))
Revised: By Lipschitz continuity, wehave ∥ a ( U ) − a ( u ))

Original: ∥
Revised: ∥

Original: ≤ A ∥
Revised: ≤ A ∥

Original: u − U ∥ and ∥ c ( U ) − c ( u ))
Revised: u − U ∥ and ∥ c ( U ) − c ( u ))

Original: ∥ ≤ B ∥ u − U ∥ , for some constants A
Revised: ∥ ≤ B ∥ u − U ∥ , for some constants A

Original: and
Revised: and

Original: B in R + .
Revised: B in R + .

Original: Also, for compact set [0 , s 0 ] , s 0 < ∞ , since both u and Jacobian J are continuous mapping,
Revised: Also, for compact set [0 , s 0 ] , s 0 < ∞ , since both u and Jacobian J are continuous mapping,

Original: J ( u ) is also compact.
Revised: J ( u ) is also compact.

Original: Since a subspace of R n is compact if and only it is closed and bounded, J ( u )is bounded [43].
Revised: Since a subspace of R n is compact if and only it is closed and bounded, J ( u )is bounded [42].

Original: Thus, ∥ J ( u ) ∥ ≤ M for some constant M in R + .
Revised: Thus, ∥ J ( u ) ∥ ≤ M for some constant M in R + .

Original: Define C = 2( AM + B ) , wehave
Revised: Define C = 2( AM + B ) , wehave

Original: [Equation]
Revised: [Equation]

Original: From proposition B.1, we have u ( x (0))
Revised: From proposition B.1, we have u ( x (0))

Original: =
Revised: =

Original: U (0) .
Revised: U (0) .

Original: As proved above, we have
Revised: As proved above, we have

Original: [Equation]
Revised: [Equation]

Original: where C < ∞ .
Revised: where C < ∞ .

Original: Thus, by lemma B.2, we have
Revised: Thus, by lemma B.2, we have

Original: [Equation]
Revised: [Equation]

Original: This further implies that U ( s ) =
Revised: This further implies that U ( s ) =

Original: u ( x ( s )) , so long as a and c are Lipschitz continuous.u [Figure] t 0.0 0.
Revised: u ( x ( s )) , so long as a and c are Lipschitz continuous.

Original: x 0.

Original: B.2 Proof of Proposition 4. 
Revised: B.2 Proof of Proposition 4.

Original: Proof.
Revised: Proof.

Original: Suppose have C-NODE given by
Revised: Suppose have C-NODE given by

Original: [Equation]
Revised: [Equation]

Original: Write out specific functions for these terms to match the desired properties of the function.
Revised: Write out specific functions for these terms to match the desired properties of the function.

Original: Defineinitial condition u (0 , 0) = u 0 .
Revised: Defineinitial condition u (0 , 0) = u 0 .

Original: By setting
Revised: By setting

Original: [Equation]
Revised: [Equation]

Original: have the ODE and solution,
Revised: u

Original: [Equation]
Revised: [Figure]

Original: (cid:35)(cid:33)
Revised: t 0.0 0. x 0. have the ODE and solution,

Original: [Equation]
Revised: [Equation]

Original: To be specific, we can represent this system with the following family of PDEs:
Revised: To be specific, we can represent this system with the following family of PDEs:

Original: [Equation]
Revised: [Equation]

Original: We can solve this system to obtain a function that has intersecting trajectories.
Revised: We can solve this system to obtain a function that has intersecting trajectories.

Original: The solution isvisualized in Figure 6, which shows that C-NODE can be used to learn and represent this function G .
Revised: The solution isvisualized in Figure 5, which shows that C-NODE can be used to learn and represent this function G .

Original: It should be noted that this is not the only possible solution to function G , as when ∂t/∂s = 0 ,we fall back to a NODE system with the dynamical system conditioned on the input data.
Revised: It should be noted that this is not the only possible solution to function G , as when ∂t/∂s = 0 , we fallback to a NODE system with the dynamical system conditioned on the input data.

Original: In thisconditioned setting, we can then represent G by stopping the dynamics at different times t as in [32].
Revised: In this conditionedsetting, we can then represent G by stopping the dynamics at different times t as in Massaroli et al.[31].

Original: B.3 Proof of Proposition 4. 
Revised: B.3 Proof of Proposition 4.

Original: The proof uses the change of variables formula for a particle that depends on a vector rather than ascalar
Revised: The proof uses the change of variables formula for a particle that depends on a vector rather than ascalar

Original: and it follows directly from the proof given in  [5, Appendix A].
Revised: and it follows directly from the proof given in Chen et al. [5, Appendix A].

Original: We provide the full proof forcompleteness.
Revised: We provide the fullproof for completeness.

Original: Proof.
Revised: Proof.

Original: Assume (cid:80) ki
Revised: Assume (cid:80) ki

Original: =1 ∂u∂x i dx i ds is Lipschitz continuous in u and continuous in t , so every initial valueproblem has a unique solution [12].
Revised: =1 ∂u∂x i dx i ds is Lipschitz continuous in u and continuous in t , so every initial valueproblem has a unique solution [12].

Original: Also assume u ( s ) is bounded.
Revised: Also assume u ( s ) is bounded.

Original: Want∂p ( u ( s )) ∂s = tr (cid:32) ∂ ∂u k (cid:88) i = ∂u ∂x i
Revised: Want

Original: dx i ds (cid:33) .
Revised: [Equation]

Original: Define T ϵ = u
Revised: Define T ϵ = u

Original: ( s + ϵ ) .
Revised: ( s + ϵ ) .

Original: The discrete change of variables states that u 1 = f ( u 0 ) ⇒ log p ( u 1 )
Revised: The discrete change of variables states that u 1 = f ( u 0 ) ⇒ log p ( u 1 )

Original: =log p ( u 0 ) − log | det ∂f∂u 

Revised: =log p ( u 0 ) − log | det ∂f∂u

Original: | [38].
Revised: | [37].

Original: Take the limit of the time difference between u 0 and u 1 , by definition of derivatives,∂ log p ( u ( s )) ∂t = lim
Revised: Take the limit of the time difference between u 0 and u 1 , by definition of derivatives,

Original: [Equation]
Revised: [Equation]

Original: → 0 + log [Equation] ( u ( [Equation]
Revised: The Jacobi’s formula states that if A is a differentiable map from the real numbers to n × n matrices,then ddt det

Original: [Equation]

Original: u ( s ))

Original: ϵ =

Original: lim

Original: [Equation]

Original: 0 + log

Original: [Equation]

Original: u ( s )

Original: [Equation]

Original: T
Revised: A

Original: [Equation]

Original: ( u ( t ))

Original: [Equation]

Original: log p ( u ( s ))

Original: ϵ =

Original: [Equation]

Original: lim

Original: [Equation]

Original: [Equation]

Original: lim

Original: [Equation]
Revised: [Equation]

Original: |

Original: [Equation]

Original:  =[Equation] | [Equation]
Revised: t ) =

Original: lim

Original: [Equation]

Original: | =

Original: [Equation]

Original: |

Original: The Jacobi’s formula states that if A is a differentiable map from the real numbers to n × n matrices,then ddt det A ( t ) =

Original: tr ( adj ( A ( t )) dA ( t ) dt ) , where adj is the adjugate.
Revised: tr ( adj ( A ( t )) dA ( t ) dt ) , where adj is the adjugate.

Original: Thus, have∂ log p ( u ( t )) ∂t = − lim ϵ → 0 + tr (cid:20) adj (cid:18) ∂ ∂uT ϵ ( u ( s )) (cid:19) ∂
Revised: Thus, have

Original: ∂ϵ ∂ ∂uT ϵ ( u ( s )) (cid:21) = − tr (cid:20)(cid:18) lim ϵ → 0 + adj (cid:18) ∂ ∂uT ϵ ( u ( t ))
Revised: [Equation]

Original: (cid:19)(cid:19) (cid:18) lim
Revised: (cid:18) lim

Original: ϵ → 0 + ∂ ∂ϵ ∂ ∂uT ϵ ( u ( s ))
Revised: ϵ → 0 + ∂ ∂ϵ ∂ ∂uT ϵ ( u ( s ))

Original: (cid:19)(cid:21) =
Revised: (cid:19)(cid:21) =

Original: − tr (cid:20) adj (cid:18)
Revised: − tr (cid:20) adj (cid:18)

Original: ∂
Revised: ∂

Original: ∂uu ( t ) (cid:19) lim
Revised: ∂uu ( t ) (cid:19) lim

Original: ϵ → 0 + ∂ ∂ϵ ∂ ∂uT ϵ ( u ( s ))
Revised: ϵ → 0 + ∂ ∂ϵ ∂ ∂uT ϵ ( u ( s ))

Original: (cid:21) = − tr (cid:20) lim
Revised: (cid:21) = − tr (cid:20) lim

Original: ϵ → 0 + ∂ ∂ϵ ∂ ∂uT ϵ ( u ( s ))
Revised: ϵ → 0 + ∂ ∂ϵ ∂ ∂uT ϵ ( u ( s ))

Original: (cid:21)
Revised: (cid:21)

Original: Substituting T ϵ with its Taylor series expansion and taking the limit, we have
Revised: Substituting T ϵ with its Taylor series expansion and taking the limit, we have∂ log p ( u ( t ))

Revised: ∂t = − tr (cid:18) lim

Revised: ϵ → 0 + ∂ ∂ϵ ∂ ∂u (cid:18) u

Revised: + ϵ du ds

Original: [Equation]
Revised: + O ( ϵ 2 ) + O ( ϵ 3 )

Revised: + ... (cid:19)(cid:19) = − tr (cid:32) lim

Revised: ϵ → 0 + ∂ ∂ϵ ∂ ∂u (cid:32) u

Revised: + ϵ k (cid:88)

Revised: i = ∂u ∂x i

Revised: dx i ds

Revised: + O ( ϵ 2 )

Revised: + O ( ϵ 3 )

Revised: + ...

Revised: (cid:33)(cid:33) = − tr (cid:32) lim

Revised: ϵ → 0 + ∂ ∂ϵ (cid:32)

Revised: I + ∂ ∂uϵ k (cid:88) i = ∂u ∂x i

Revised: dx i ds

Revised: + O ( ϵ 2 )

Revised: + O ( ϵ 3 )

Revised: + ...

Revised: (cid:33)(cid:33) = − tr (cid:32) lim

Revised: ϵ → 0 + (cid:32) ∂

Revised: ∂u k (cid:88) i = ∂u ∂x i

Revised: dx i ds

Revised: + O ( ϵ )

Revised: + O ( ϵ 2 )

Revised: + ...

Revised: (cid:33)(cid:33) = − tr (cid:32) ∂

Revised: ∂u k (cid:88) i = ∂u ∂x i

Revised: dx i ds (cid:33)

Original: B.4 Proof of Proposition 4.
Revised: B.4 Proof of Proposition 4.3

Original: Proof.
Revised: Proof.

Original: To prove proposition 4.3, need to show that for any homeomorphism h ( · ) , there exists au ( s, u 

Revised: To prove proposition 4.3, need to show that for any homeomorphism h ( · ) , there exists au ( s, u

Original: ) ∈
Revised: ) ∈

Original: R n following a C-NODE system such that u ( s = T, u 0 )
Revised: R n following a C-NODE system such that u ( s = T, u 0 )

Original: =
Revised: =

Original: h ( u 0 ) .
Revised: h ( u 0 ) .

Original: Without loss of generality, say T = 1 .
Revised: Without loss of generality, say T = 1 .

Original: Define C-NODEsystem
Revised: Define C-NODE

Original: [Equation]

Original: ( s, u [Equation] = 1 , ∂u∂x ( u ( x, t [Equation]
Revised: system

Original: =

Original: [Equation]

Original: u

Original: [Equation]
Revised: [Equation]

Original: , dtds

Original: [Equation]

Original: , u

Original: [Equation]

Original: ∂u∂t ( u ( x, t ))

Original: = − 1 .

Original: Then, duds = h ( u 0 ) − u 0 .
Revised: Then, duds = h ( u 0 ) − u 0 .

Original: At s = 1 , haveu
Revised: At s = 1 , have

Original: [Equation]

Original: 1 , u

Original: [Equation]
Revised: [Equation]

Original: The inverse map will be defined by integration backwards.
Revised: The inverse map will be defined by integration backwards.

Original: Specifically, have
Revised: Specifically, have

Original: [Equation]
Revised: [Equation]

Original: Thus, for any homeomorphism h ( · ) , there exists a C-NODE system, such that forward integration fortime s = 1 is equivalent as applying h ( · ) , and backward integration for time s = 1 is equivalent toapplying h − 1 ( · ) .
Revised: Thus, for any homeomorphism h ( · ) , there exists a C-NODE system, such that forward integration fortime s = 1 is equivalent as applying h ( · ) , and backward integration for time s = 1 is equivalent toapplying h − 1 ( · ) .

Original: C Ablation Study
Revised: C Ablation Study

Original: C.1 Ablation study on dimension of C-NODE

Original: We perform an ablation study on the impact of the number of dimensions of the C-NODE weimplement.

Original: This study allows us to evaluate the relationship between the model performance and themodel’s limit of mathematical approximating power.

Original: Empirical results show that as we increase thenumber of dimensions used in the C-NODE model, the C-NODE’s performance first improves andthen declines, due to overfitting.

Original: We have found out that information criteria like AIC and BIC canbe successfully applied for dimension selection in this scenario.

Original: In previous experiments, we represent ∂
Revised: In previous experiments, we represent ∂

Original: u /∂x i with separate and independent neural networksc i ( u , θ ) .
Revised: u /∂x i with separate and independent neural networksc i ( u , θ ) .

Original: Here, we represent all k functions as a vector-valued function [ ∂ u /∂x 1 , ..., ∂ u /∂x k ] T .
Revised: Here, we represent all k functions as a vector-valued function [ ∂ u /∂x 1 , ..., ∂ u /∂x k ] T .

Original: Weapproximate this vector-valued function with a neural network c ( u , θ ) .
Revised: Weapproximate this vector-valued function with a neural network c ( u , θ ) .

Original: The model is trained usingthe Euler solver to have better training stability when the neural network has a large number ofparameters.
Revised: The model is trained usingthe Euler solver to have better training stability when the neural network has a large number ofparameters.

Original: Experiment details for the ablation study is as shown in Figures  7, 8, 9.
Revised: Experiment details for the ablation study is as shown in Figures 6, 7, 8.

Original: We show C-NODE’s parameter efficiency over NODE with an ablation study on the image classification task on the CIFAR-10 dataset. Specifically, under a similar training setup, we experimentwith C-NODE with 95071, 55855, and 17379 parameters and experiment with NODE with 96044,56828, and 17444 parameters. As shown in Figure 10, although C-NODE has more variance in itsperformance, it outperforms NODE along the whole training process in all three cases.
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: 2D
Revised: 2D

Original: [Figure]
Revised: [Figure]

Revised: 128D

Original: [Figure]
Revised: [Figure]

Original: 2D
Revised: 2D

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: We additionally provide algorithms for training and sampling CNFs defined with C-NODEs. given probability density function of p ( u ( s = 0)) =
Revised: 128D

Original: p 0 ( · ) for each input data z j do Given (cid:34) u (1) log p ( z j ) − log p ( u (1)) (cid:35) = (cid:34) z j (cid:35) procedure Integrate from 1 → 0 to get (cid:34) u (0) log p ( z j ) − log p ( u (0)) (cid:35) for each time step s m do calculate d x ds ( x , u ; g ( z j ; Θ 1 ); Θ 2 ) and J x u ( x , u ; Θ 2 ) .
Revised: [Figure]

Original: calculate d u ds = J x u d x ds . calculate − tr ( ∂∂ u J x u d x ds ) with Hutchinson trace estimator [13]. calculate (cid:34) u ( s m +1 ) log p ( u ( s m +1 ))
Revised: [Figure]

Original: (cid:35) = (cid:34) u ( s m ) log p ( u ( s m )) (cid:35) + (cid:34) d u dt ∂ log p ( u ( s ))
Revised: 2D

Original: [-∂s (cid:35) ( s m +1 − s m ) . end for evaluate p 0 ( u (0)) calculate log p ( z j ) = (log p ( z j ) − log p ( u (0))) + log p 
-]
Revised: [-∂s (cid:35) ( s m +1 − s m ) . end for evaluate p 0 ( u (0)) calculate log p ( z j ) = (log p ( z j ) − log p ( u (0))) + log p 
-][Figure]

Original: ( u (0)) optimize log p ( z j ) with an optimization algorithm (stochastic gradient descent etc.) end for Algorithm procedure sample u ( s = 0) from base distribution p 0 ( · ) procedure Integrate from 0 → 1 to get u ( s = 1) for each time step s m do calculate d x ds ( x , u ; g ( z j ; Θ 1 ); Θ 2 ) and J x [Equation]
Revised: 32D 64D 128D

Original: [Equation] calculate [Equation] [Equation]
Revised: [Figure]

Original: calculate

Original: [Equation] [Equation] [Equation] end procedure u ( s = 1) is our sample from the CNF
Revised: [Figure]