Original: O SCILLATION
Revised: O SCILLATION

Original: N
Revised: N

Original: EURAL
Revised: EURAL

Original: O
Revised: O

Original: RDINARY D IFFERENTIAL
Revised: RDINARY D IFFERENTIAL

Original: E QUATIONS
Revised: E QUATIONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Neural ordinary differential equations (NODEs) have received a lot of attention in recent years due to their memory efficiency.
Revised: Neural ordinary differential equations (NODEs) have received a lot of attention in recent years due to their memory efficiency.

Original: Different from traditional deep learning, it defines a continuous deep learning architecture based on the theory of ordinary differential equations (ODEs), which also improves the interpretability of deep learning.
Revised: Different from traditional deep learning, it defines a continuous deep learning architecture based on the theory of ordinary differential equations (ODEs), which also improves the interpretability of deep learning.

Original: However, it has several obvious limitations, such as a NODE is not a universal approximator, it requires a large number of function evaluations (NFEs), and it has a slow convergence rate.
Revised: However, it has several obvious limitations, such as a NODE is not a universal approximator, it requires a large number of function evaluations (NFEs), and it has a slow convergence rate.

Original: We address these drawbacks by modeling and adding an oscillator to the framework of the NODEs.
Revised: We address these drawbacks by modeling and adding an oscillator to the framework of the NODEs.

Original: The oscillator enables the trajectories of our model to cross each other.
Revised: The oscillator enables the trajectories of our model to cross each other.

Original: We prove that our model is a universal approximator, even in the original input space.
Revised: We prove that our model is a universal approximator, even in the original input space.

Original: Due to the presence of oscillators, the flows learned by the model will be simpler, thus our model needs fewer NFEs and has a faster convergence speed.
Revised: Due to the presence of oscillators, the flows learned by the model will be simpler, thus our model needs fewer NFEs and has a faster convergence speed.

Original: We apply our model to various tasks including classification and time series extrapolation, then compare several metrics including accuracy, NFEs, and convergence speed.
Revised: We apply our model to various tasks including classification and time series extrapolation, then compare several metrics including accuracy, NFEs, and convergence speed.

Original: The experiments show that our model can achieve better results compared to the existing baselines.
Revised: The experiments show that our model can achieve better results compared to the existing baselines.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Neural Ordinary Differential Equations (NODEs) (
Revised: Neural Ordinary Differential Equations (NODEs) (

Original: Chen et al., 2018) are the latest continuous deep learning architectures that were first developed in the context of continuous recurrent networks (Cohen & Grossberg, 1983).
Revised: Chen et al., 2018) are the latest continuous deep learning architectures that were first developed in the context of continuous recurrent networks (Cohen & Grossberg, 1983).

Original: This continuous deep learning architecture provides a new perspective that theoretically bridges the gap between deep learning and dynamic systems.
Revised: This continuous deep learning architecture provides a new perspective that theoretically bridges the gap between deep learning and dynamic systems.

Original: This deep learning architecture can be efficiently trained with backpropagation and has shown great promise on several tasks including modeling continuous time data, classification, and building normalizing flows.
Revised: This deep learning architecture can be efficiently trained with backpropagation and has shown great promise on several tasks including modeling continuous time data, classification, and building normalizing flows.

Original: The core idea of a NODE is to use a neural network to parameterize the vector field (Chen et al., 2018; Kidger, 2022).
Revised: The core idea of a NODE is to use a neural network to parameterize the vector field (Chen et al., 2018; Kidger, 2022).

Original: Typically, a simple neural network is enough to represent the vector field, which will be optimized during the training process.
Revised: Typically, a simple neural network is enough to represent the vector field, which will be optimized during the training process.

Original: Based on the well-learned vector field, trajectories will be obtained as the estimate functions.
Revised: Based on the well-learned vector field, trajectories will be obtained as the estimate functions.

Original: However, this architecture has several limitations.
Revised: However, this architecture has several limitations.

Original: First, NODEs cannot learn any crossover-mapping functions (Dupont et al., 2019), which results in them not being universal approximators.
Revised: First, NODEs cannot learn any crossover-mapping functions (Dupont et al., 2019), which results in them not being universal approximators.

Original: Second, to optimize the vector field, it will need many function evaluations during both forward evaluation and backpropagation processes of training.
Revised: Second, to optimize the vector field, it will need many function evaluations during both forward evaluation and backpropagation processes of training.

Original: Third, the time and convergence rate of the training process is relatively slow.
Revised: Third, the time and convergence rate of the training process is relatively slow.

Original: The first limitation is caused by the continuity of vector-field-based trajectories because the trajectories in NODEs cannot cross each other at the same time (Massaroli et al., 2020; Norcliffe et al., 2020).
Revised: The first limitation is caused by the continuity of vector-field-based trajectories because the trajectories in NODEs cannot cross each other at the same time (Massaroli et al., 2020; Norcliffe et al., 2020).

Original: This property causes NODEs to be powerless against some special topologies, such as the cases of concentric circles and intersecting lines mentioned by Dupont et al.
Revised: This property causes NODEs to be powerless against some special topologies, such as the cases of concentric circles and intersecting lines mentioned by Dupont et al.

Original: The reason for the second limitation is caused by the straightforward optimization approach of the vector field.
Revised: We conjecture the reason for the second limitation is caused by the straightforward optimization approach of the vector field.

Original: There is no guarantee that learning the vector field is a better choice than learning the estimated functions directly.
Revised: There is no guarantee that learning the vector field is a better choice than learning the estimated functions directly.

Original: Sometimes it will need many function evaluations to optimize the vector field, so the difficulties go beyond learning the estimated functions themselves.
Revised: Sometimes it will need many function evaluations to optimize the vector field, so the difficulties go beyond learning the estimated functions themselves.

Original: The third limitation is caused by the trade-off between accuracy and speed for the ordinary differential equation solver (ODE solver).
Revised: The third limitation is caused by the trade-off between accuracy and speed for the ordinary differential equation solver (ODE solver).

Original: NODEs perform forward evaluation and backpropagation calculations via ODE solvers, which can be treated as black boxes.
Revised: NODEs perform forward evaluation and backpropagation calculations via ODE solvers, which can be treated as black boxes.

Original: If we need to ensure the accuracy of an ODE solver, then we must sacrifice the speed.
Revised: If we need to ensure the accuracy of an ODE solver, then we must sacrifice the speed.

Original: To address the first problem, we intend to add some discrete elements during the process of learning the trajectories.
Revised: To address the first problem, we intend to add some discrete elements during the process of learning the trajectories.

Original: A discrete element can be realized by a “jump” in the trajectory.
Revised: A discrete element can be realized by a “jump” in the trajectory.

Original: This jump allows the trajectories of the NODEs to cross at the same time.
Revised: This jump allows the trajectories of the NODEs to cross at the same time.

Original: It solves the above-mentioned cases.
Revised: It solves the above-mentioned cases.

Original: We also give proof in Section 4.1 that our approach is a universal approximator.
Revised: We also give proof in Section 4.1 that our approach is a universal approximator.

Original: To solve the second and third problems, we propose to join a function g that directly optimizes the trajectories while optimizing the vector field at the same time.
Revised: To solve the second and third problems, we propose to join a function g that directly optimizes the trajectories while optimizing the vector field at the same time.

Original: It shifts part of the burden from optimizing the vector field to optimizing the original estimated function.
Revised: It shifts part of the burden from optimizing the vector field to optimizing the original estimated function.

Original: The addition of g will make the final optimized vector fields less complicated, thus a “simple” vector field can be utilized to estimate the function well.
Revised: The addition of g will make the final optimized vector fields less complicated, thus a “simple” vector field can be utilized to estimate the function well.

Original: This will result in fewer NFEs, less training time, and faster convergence.
Revised: This will result in fewer NFEs, less training time, and faster convergence.

Original: Based on the above two points, we design an oscillator to enhance neural ordinary differential equations, namely Oscillation Neural Ordinary Differential Equations (ONODE).
Revised: Based on the above two points, we design an oscillator to enhance neural ordinary differential equations, namely Oscillation Neural Ordinary Differential Equations (ONODE).

Original: The architecture is shown in Figure 1.
Revised: The architecture is shown in Figure 1.

Original: This oscillator is designed to achieve both “jumping” of trajectories and optimizing the estimation function while simultaneously optimizing the vector field.
Revised: This oscillator is designed to achieve both “jumping” of trajectories and optimizing the estimation function while simultaneously optimizing the vector field.

Original: This design solves the three problems mentioned above to some extent.
Revised: This design solves the three problems mentioned above to some extent.

Original: Our proposed method is not only a universal approximator, but it also reduces the NFEs as well as improves the training convergence speed.
Revised: Our proposed method is not only a universal approximator, but it also reduces the NFEs as well as improves the training convergence speed.

Original: Specifically, the vector field is parameterized by neural network
Revised: Specifically, the vector field is parameterized by neural network

Original: Chen et al can be optimized by ODE solvers which be used such as Runge (1895)
Revised: Chen et al can be optimized by ODE solvers which be used such as Runge (1895)

Original: Kutta (1901)
Revised: Kutta (1901)

Original: Hairer et al is parameterized by shallow neural network structure it has only one hidden layer oscillator be located before vector field modeling, or after the
Revised: Hairer et al is parameterized by shallow neural network structure it has only one hidden layer oscillator be located before vector field modeling, or after the

Original: The oscillator modeling structure may vary slightly in different tasks.
Revised: The oscillator modeling structure may vary slightly in different tasks.

Original: For example, in the usual classification tasks and extrapolating time series tasks, these structures will have the perception of one hidden layer.
Revised: For example, in the usual classification tasks and extrapolating time series tasks, these structures will have the perception of one hidden layer.

Original: In the image classification task, the structure will have two convolutional layers with an activation function.
Revised: In the image classification task, the structure will have two convolutional layers with an activation function.

Original: It is worth noting that since our oscillator is parameterized by a shallow neural network that does not take much space, we can still consider the model to be memory efficient.
Revised: It is worth noting that since our oscillator is parameterized by a shallow neural network that does not take much space, we can still consider the model to be memory efficient.

Original: We will illustrate this in Section 5.
Revised: We will illustrate this in Section 5.

Original: Besides, our oscillator simplifies the learning complexity of the vector field, which leads to a reduction in NFEs and convergence speed.
Revised: Besides, our oscillator simplifies the learning complexity of the vector field, which leads to a reduction in NFEs and convergence speed.

Original: A simple example is shown in Figure 1 right, the final trained vector field will be a very simple one.
Revised: A simple example is shown in Figure 1 right, the final trained vector field will be a very simple one.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: The basic ideas of neural ordinary differential equations were originally considered in Rico-Martinez et al.
Revised: The basic ideas of neural ordinary differential equations were originally considered in Rico-Martinez et al.

Original: (1992), Rico-Martinez & Kevrekidis (1993), and Rico-Martinez et al.
Revised: (1992), Rico-Martinez & Kevrekidis (1993), and Rico-Martinez et al.

Original: For example,
Revised: For example,

Original: Rico-Martinez et al.
Revised: Rico-Martinez et al.

Original: (1992) proposed a vector field that can be trained using a Multilayer Perceptron (MLP).
Revised: (1992) proposed a vector field that can be trained using a Multilayer Perceptron (MLP).

Original: Rico-Martinez & Kevrekidis (1993) used an implicit integrator and recurrent networks for continuous-time modeling of nonlinear systems.
Revised: Rico-Martinez & Kevrekidis (1993) used an implicit integrator and recurrent networks for continuous-time modeling of nonlinear systems.

Original: Chen et al.
Revised: Chen et al.

Original: (2018) specified the architecture of
Revised: (2018) specified the architecture of

Original: NODEs
Revised: NODEs

Original: and applied it to several tasks and achieved good results, including image classification, continuous normalizing flows, and latent ODEs, leading to an explosion of interest in NODEs.
Revised: and applied it to several tasks and achieved good results, including image classification, continuous normalizing flows, and latent ODEs, leading to an explosion of interest in NODEs.

Original: NODEs are a new class of model that transform data continuously through infinite depth architectures (Norcliffe et al., 2020).
Revised: NODEs are a new class of model that transform data continuously through infinite depth architectures (Norcliffe et al., 2020).

Original: In particular, NODEs can be seen as a continual version of Residual
Revised: In particular, NODEs can be seen as a continual version of Residual

Original: Networks (He et al., 2016) by taking the discretization step to infinitely small.
Revised: Networks (He et al., 2016) by taking the discretization step to infinitely small.

Original: This continuous infinite depth architecture will bring several benefits, one of the most important is memory efficiency.
Revised: This continuous infinite depth architecture will bring several benefits, one of the most important is memory efficiency.

Original: According to Chen et al.
Revised: According to Chen et al.

Original: (2018), the scalar-valued loss with respect to all inputs of any ODE solver can be computed directly without backpropagating through the operations of the solver.
Revised: (2018), the scalar-valued loss with respect to all inputs of any ODE solver can be computed directly without backpropagating through the operations of the solver.

Original: The intermediate quantities of the forward pass will not need to be stored.
Revised: The intermediate quantities of the forward pass will not need to be stored.

Original: It causes the NODEs so they can be trained with a constant memory cost.
Revised: It causes the NODEs so they can be trained with a constant memory cost.

Original: However, this continuous infinite depth architecture will bring several drawbacks.
Revised: However, this continuous infinite depth architecture will bring several drawbacks.

Original: Dupont et al.
Revised: Dupont et al.

Original: (2019) showed that NODEs learn representations that preserve the topology of the input space and prove that the existence of functions in NODEs cannot be represented.
Revised: (2019) showed that NODEs learn representations that preserve the topology of the input space and prove that the existence of functions in NODEs cannot be represented.

Original: To address these limitations,
Revised: To address these limitations,

Original: Dupont et al.
Revised: Dupont et al.

Original: introduced Augmented NODEs (ANODEs) which add extra dimensions for the vector field learning.
Revised: introduced Augmented NODEs (ANODEs) which add extra dimensions for the vector field learning.

Original: This approach increases the degrees of freedom of the trajectory by elevating the dimensionality, thus circumventing the problem that trajectories cannot be crossed.
Revised: This approach increases the degrees of freedom of the trajectory by elevating the dimensionality, thus circumventing the problem that trajectories cannot be crossed.

Original: However, it does not inherently overcome the problem that trajectories cannot be traversed and increases the original parameter space.
Revised: However, it does not inherently overcome the problem that trajectories cannot be traversed and increases the original parameter space.

Original: Other limitations of NODEs include the NFEs which can often become prohibitively large (Dupont et al., 2019) and the speed is slow as long as it is needed to obtain good precision (Chen et al., 2018).
Revised: Other limitations of NODEs include the NFEs which can often become prohibitively large (Dupont et al., 2019) and the speed is slow as long as it is needed to obtain good precision (Chen et al., 2018).

Original: These limitations are usually caused by the need for the NODE to learn a complex vector field.
Revised: These limitations are usually caused by the need for the NODE to learn a complex vector field.

Original: Due to the generality of the NODEs’ framework, there is already a lot of related work based on
Revised: Due to the generality of the NODEs’ framework, there is already a lot of related work based on

Original: NODEs.
Revised: NODEs.

Original: Some work focuses on modeling for the time series data.
Revised: Some work focuses on modeling for the time series data.

Original: For example, Norcliffe et al.
Revised: For example, Norcliffe et al.

Original: (2020) focused on the learning dynamics system and proposed the Second Order Neural Ordinary
Revised: (2020) focused on the learning dynamics system and proposed the Second Order Neural Ordinary

Original: Differential Equations (SONODEs).
Revised: Differential Equations (SONODEs).

Original: It can be seen as a system of coupled first-order NODEs.
Revised: It can be seen as a system of coupled first-order NODEs.

Original: Jia & Benson (2019) introduced Neural Jump Stochastic Differential Equations that provided a datadriven approach to learn continuous and discrete dynamic behavior.
Revised: Jia & Benson (2019) introduced Neural Jump Stochastic Differential Equations that provided a datadriven approach to learn continuous and discrete dynamic behavior.

Original: Kidger et al.
Revised: Kidger et al.

Original: (2020) proposed
Revised: (2020) proposed

Original: Neural Controlled Differential Equations to model the irregular time series.
Revised: Neural Controlled Differential Equations to model the irregular time series.

Original: Morrill et al.
Revised: Morrill et al.

Original: (2021) used Neural Rough Differential Equations for modeling long time series.
Revised: (2021) used Neural Rough Differential Equations for modeling long time series.

Original: Other works focused on physical modeling.
Revised: Other works focused on physical modeling.

Original: For example, Greydanus et al.
Revised: For example, Greydanus et al.

Original: (2019) introduced Hamiltonian Neural Networks for learning certain physical laws that follow conservation laws.
Revised: (2019) introduced Hamiltonian Neural Networks for learning certain physical laws that follow conservation laws.

Original: Cranmer et al.
Revised: Cranmer et al.

Original: (2020) proposed
Revised: (2020) proposed

Original: Lagrangian neural networks to learn certain physical laws without requiring canonical coordinates.
Revised: Lagrangian neural networks to learn certain physical laws without requiring canonical coordinates.

Original:  PRELIMINARY
Revised:  PRELIMINARY

Original: NODEs are a family of deep neural network models that can be interpreted as a continuous version of Residual Networks (He et al., 2016).
Revised: NODEs are a family of deep neural network models that can be interpreted as a continuous version of Residual Networks (He et al., 2016).

Original: Recall the formulation of a residual network:
Revised: Recall the formulation of a residual network:

Original: [Equation]
Revised: [Equation]

Original: where the f is the residual block and the θ f represents the parameters of f .
Revised: where the f is the residual block and the θ f represents the parameters of f .

Original: The left side of Equation 1 can be seen as the denominator is 1 , so it can be represented by h t
Revised: The left side of Equation 1 can be seen as the denominator is 1 , so it can be represented by h t

Original: +1 − h t= f ( h t , θ f ) .
Revised: +1 − h t= f ( h t , θ f ) .

Original: When the number of layers becomes infinitely large, and the step becomes infinitely small, Equation 1 will become an ordinary differential equation format, as shown in Equation 2.
Revised: When the number of layers becomes infinitely large, and the step becomes infinitely small, Equation 1 will become an ordinary differential equation format, as shown in Equation 2.

Original: [Equation]
Revised: [Equation]

Original: Thus, the NODE will have the same format as an ODE:
Revised: Thus, the NODE will have the same format as an ODE:

Original: [Equation]
Revised: [Equation]

Original: where x 0 is the input data.
Revised: where x 0 is the input data.

Original: Typically, f will be some standard simple neural architecture, such as a
Revised: Typically, f will be some standard simple neural architecture, such as a

Original: MLP.
Revised: MLP.

Original: The θ f represents trainable parameters in f .
Revised: The θ f represents trainable parameters in f .

Original: To obtain any final state of h ( t ) when t = T , all that is needed is to solve an ordinary differential equation with initial values, which is called an initial value problem (IVP):
Revised: To obtain any final state of h ( t ) when t = T , all that is needed is to solve an ordinary differential equation with initial values, which is called an initial value problem (IVP):

Original: [Equation]
Revised: [Equation]

Original: Thus, a NODE can transform from h (0) to h ( T ) through the solutions to the initial value problem (IVP) of the ODE.
Revised: Thus, a NODE can transform from h (0) to h ( T ) through the solutions to the initial value problem (IVP) of the ODE.

Original: This framework indirectly realizes a functional relationship x → F ( x ) like a general neural network.
Revised: This framework indirectly realizes a functional relationship x → F ( x ) like a general neural network.

Original: By the properties of ODEs, NODEs are always invertible; we can reverse the limits of integration, or alternatively, integrate − f .
Revised: By the properties of ODEs, NODEs are always invertible; we can reverse the limits of integration, or alternatively, integrate − f .

Original: The Adjoint Sensitivity Method (Pontryagin et al., 1961) based on reverse-time integration of an expanded ODE, allows for finding gradients of the initial value problem solutions
Revised: The Adjoint Sensitivity Method (Pontryagin et al., 1961) based on reverse-time integration of an expanded ODE, allows for finding gradients of the initial value problem solutions

Original: h ( T ) with respect to parameters θ f
Revised: h ( T ) with respect to parameters θ f

Original: and the initial values h (0) .
Revised: and the initial values h (0) .

Original: This allows the training
Revised: This allows the training

Original: NODE to use gradient descent, which allows them to combine with other neural network blocks.
Revised: NODE to use gradient descent, which allows them to combine with other neural network blocks.

Original:  T HE P ROPOSED M ETHODS 4. O SCILLATION NODE: A UNIVERSAL APPROXIMATOR
Revised:  T HE P ROPOSED M ETHODS 4. O SCILLATION NODE: A UNIVERSAL APPROXIMATOR

Original: As we mentioned, since the trajectories in NODEs cannot cross each other at the same time, a NODE on its own does not have universal approximation capability.
Revised: As we mentioned, since the trajectories in NODEs cannot cross each other at the same time, a NODE on its own does not have universal approximation capability.

Original: An example from Dupont et al.
Revised: An example from Dupont et al.

Original: (2019) considers a continuous, differentiable, invertible function f ( x ) =
Revised: (2019) considers a continuous, differentiable, invertible function f ( x ) =

Original: − x on X = R .
Revised: − x on X = R .

Original: There is no
Revised: There is no

Original: ODE defined on R that would result in x T = ϕ T ( x 0 ) = − x 0 .
Revised: ODE defined on R that would result in x T = ϕ T ( x 0 ) = − x 0 .

Original: In ODEs, paths ( x t , t ) between the initial value ( x 0 , 0) and final value ( x T , T ) have to be continuous and cannot intersect in X for two different initial values, and the paths corresponding to x → − x and 

Revised: In ODEs, paths ( x t , t ) between the initial value ( x 0 , 0) and final value ( x T , T ) have to be continuous and cannot intersect in X for two different initial values, and the paths corresponding to x → − x and

Original: → 0 would need to intersect.
Revised: → 0 would need to intersect.

Original: Generally, there is a class of functions that a NODE cannot model (Zhang et al., 2020).
Revised: Generally, there is a class of functions that a NODE cannot model (Zhang et al., 2020).

Original: Let X
Revised: Let X

Original: =
Revised: =

Original: R p and
Revised: R p and

Original: Z ⊂ X be a set that partitions X into two or more disjointed, connected subsets C i , for i =
Revised: Z ⊂ X be a set that partitions X into two or more disjointed, connected subsets C i , for i =

Original: [ m ] ; that is, X
Revised: [ m ] ; that is, X

Original: =
Revised: =

Original: Z ∪ ( (cid:83) i C i ) .
Revised: Z ∪ ( (cid:83) i C i ) .

Original: Then, no NODEs can model a mapping h : X → X that satisfy both (1) h is an identity transformation on Z , i.e., ∀ z ∈ Z , h ( z ) =
Revised: Then, no NODEs can model a mapping h : X → X that satisfy both (1) h is an identity transformation on Z , i.e., ∀ z ∈ Z , h ( z ) =

Original: z and (2) h maps some x ∈ C i , for i ̸
Revised: z and (2) h maps some x ∈ C i , for i ̸

Original: = j .
Revised: = j .

Original: Reasoning that the trajectories cannot cross at the same moment, we intend to add the factor of discretization before the continuous trajectory.
Revised: Reasoning that the trajectories cannot cross at the same moment, we intend to add the factor of discretization before the continuous trajectory.

Original: This discretization makes the trajectory no longer subject to the effect of not being able to cross over.
Revised: This discretization makes the trajectory no longer subject to the effect of not being able to cross over.

Original: If the trajectory does cross at some point, then the crossover will be achieved by the oscillator.
Revised: If the trajectory does cross at some point, then the crossover will be achieved by the oscillator.

Original: Based on this idea, we introduce the Oscillation NODEs (ONODEs).
Revised: Based on this idea, we introduce the Oscillation NODEs (ONODEs).

Original: It solves the problem that NODEs’ trajectories cannot be crossed, thus it is a universal approximator and we provide proof in Theorem 4.1.
Revised: It solves the problem that NODEs’ trajectories cannot be crossed, thus it is a universal approximator and we provide proof in Theorem 4.1.

Original: In addition, it has fewer NFEs and faster convergence speed, which we will discuss in Section 4.2.
Revised: In addition, it has fewer NFEs and faster convergence speed, which we will discuss in Section 4.2.

Original: Specifically, an oscillator O : R d → R d approximates function g : X → X ′ .
Revised: Specifically, an oscillator O : R d → R d approximates function g : X → X ′ .

Original: The oscillator O is parameterized by a perceptron with a single hidden layer and acts on the input data x 0 .
Revised: The oscillator O is parameterized by a perceptron with a single hidden layer and acts on the input data x 0 .

Original: The output g ( x 0 ) and input x 0 of the oscillator will have the same dimension R d .
Revised: The output g ( x 0 ) and input x 0 of the oscillator will have the same dimension R d .

Original: A formal description is as follows:
Revised: A formal description is as follows:

Original: [Equation]
Revised: [Equation]

Original: where the x 0 is the input data, h ( t ) is the function we want to estimate.
Revised: where the x 0 is the input data, h ( t ) is the function we want to estimate.

Original: The function f is the first order derivative equation of h with respect to time t , and θ f represents the parameters in f .
Revised: The function f is the first order derivative equation of h with respect to time t , and θ f represents the parameters in f .

Original: The θ g represents the parameters in g .
Revised: The θ g represents the parameters in g .

Original: Figure 2 shows visually how our model works through a toy example.
Revised: Figure 2 shows visually how our model works through a toy example.

Original: We show the proof of our method (Equation 5) is a universal approximator in Theorem 4.1, which is motivated by Kidger (2022).
Revised: We show the proof of our method (Equation 5) is a universal approximator in Theorem 4.1, which is motivated by Kidger (2022).

Original: Before the proof, we give two definitions to represent the space of continuous functions and nonlinear functions, respectively.
Revised: Before the proof, we give two definitions to represent the space of continuous functions and nonlinear functions, respectively.

Original: Definition 4.1 Let cont ( X → Y ) be the space of continuous functions X
Revised: Definition 4.1 Let cont ( X → Y ) be the space of continuous functions X

Original: → Y (with respect to some topologies on X and Y ).
Revised: → Y (with respect to some topologies on X and Y ).

Original: Definition 4.2 Let nonl ( X → Y ) be the space of nonlinear functions X
Revised: Definition 4.2 Let nonl ( X → Y ) be the space of nonlinear functions X

Original: → Y .
Revised: → Y .

Original: Theorem 4.1  Let d ∈ N .
Revised: Theorem 4.1 (ONODEs are universal approximators) Let d ∈ N .

Original: For g ∈ nonl ( R d → R d ) , and f ∈ cont ( R × R d → R d ) which has a unique solution, let ϕ g,f : R d → R d denote the map X→ X ′ with h ( t ) = g ( x t , θ g ) , h ′ ( t ) = f ( h ◦ g ( x t , θ g ) , t, θ ) , h (0) =
Revised: For g ∈ nonl ( R d → R d ) , and f ∈ cont ( R × R d → R d ) which has a unique solution, let ϕ g,f : R d → R d denote the map X

Original: g ( x 0 , θ g ) , for t ∈ [0 , 1] .

Original: Take C ( R d ) to be the family of real functions that one wishes to approximate in R d and set (cid:80) d

Original: = span { ϕ g,f } , then (cid:80) d is dense in C ( R d ) .

Original: Proof : Suppose the input x ∈ R d , the system of ODEs will solve a series of equations y 0 , y 1 , ..., y M as follows: (1) h 0 (0) =

Original: [-g ( x ) ∈ R d and dh 0 dt ( t ) = 0 , (2) h 1 (0) = 
-]
Revised: [-g ( x ) ∈ R d and dh 0 dt ( t ) = 0 , (2) h 1 (0) = 
-]

Original: ∈ R d × d and dh 1 dt = h 0 ( t ) ⊗ h 0 ( t ) , (3) h 2 (0) =

Original: 0 ∈ R d 3 and dh 2 dt ( t ) =

Original:  h1 ( t ) ⊗ h 0 ( t ) ,(4) h 3 (0) = 0 ∈ R d 4 and dh 3 dt ( t ) = h 2 ( t ) ⊗ h 0 ( t ) , ... , (M+1) hM (0)
Revised: → X ′ with h ( t ) , h ′ ( t ) and h (0)

Original: [-= 
-]
Revised: [-= 
-]

Original: ∈ R

Original: d M +1 and dh M dt ( t ) =

Original: h M − 1 ( t ) ⊗ h 0 ( t ) .

Original: The solution will be written immediately: (1) h 0 ( t ) =

Original: g ( x ) , (2) h 1 ( t ) =

Original: (cid:82) t 0 g ( x ) ⊗ g ( x ) ds

Original: = tg ( x ) ⊗ 2 , (3) h 2 ( t )

Original: = (cid:82) t 0 g ( x ) ⊗ 2 t ⊗ g ( x ) ds = 12 t 2 g ( x ) ⊗ 3 , (4) h 3 ( t )

Original: =
Revised: =

Original: (cid:82) t 0 g ( x ) ⊗ 3 t ⊗ g ( x ) ds = 16 t 3 g ( x) ⊗ 4 , ... , (M+1) h M ( t ) = (cid:82) t 0  g( x ) ⊗ ( M +1) t ⊗ g ( x ) ds = 1 M ! t M g ( x ) ⊗ ( M +1) .
Revised: g ( x 0 , θ g ) , for t ∈ [0 , 1] .

Original: Set t = 1 and we will obtain the collection in g (x ) ∈ R d up to degree M + 1 , i.e., { g ( x ) , g ( x ) ⊗ 2 , 12 g ( x ) ⊗ 3 , 16 g ( x ) ⊗ 4 , ..., 1 M ! g ( x ) ⊗ ( M +1) } . The function g ( x ) is realized by a single hidden layer perception utilizing a Relu active function.
Revised: Take C ( R d ) to be the family of real functions that one wishes to approximate in R d and span { ϕ g,f } , then span { ϕ g,f } is dense in C ( R d ) .

Original: Assume µ is a non-negative finite measure on R d with compact support, and continuous with respect to the Lebesgue measure. Then (cid:80) d is dense in L p ( µ ) , 1 ≤ p < ∞ , if and only if, the nonlinearity in g is not a polynomial, where L p ( µ ) is the set of all measurable functions f such that [Equation] Hornik et al., 1989; Hornik, 1991;
Revised: The proof of theorem 4.1 is shown in APPENDIX A.1.

Original: Pinkus, 1999).

Original: Thus, span { ϕ g,f } is dense in C ( R d ) and it is a universal approximator.

Original: 4. ONODE S HAVE A HIGHER COMPUTATIONAL EFFICIENCY
Revised: 4. ONODE S HAVE A HIGHER COMPUTATIONAL EFFICIENCY

Original: Besides our proposed ONODEs, there exists another augmented method that makes a NODE a universal approximator, i.e., ANODEs (Dupont et al., 2019).
Revised: Besides our proposed ONODEs, there exists another augmented method that makes a NODE a universal approximator, i.e., ANODEs (Dupont et al., 2019).

Original: The ANODEs add extra dimensions to the NODEs for the input, thus changing the input from h (0)
Revised: The ANODEs add extra dimensions to the NODEs for the input, thus changing the input from h (0)

Original: ∈ R d to (cid:20) h (0) 

Revised: ∈ R d to (cid:20) h (0)

Original: (cid:21) ∈ R d
Revised: (cid:21) ∈ R d

Original: + a .
Revised: + a .

Original: It bypasses the trajectory crossing problem by adding extra dimensions of R a .
Revised: It bypasses the trajectory crossing problem by adding extra dimensions of R a .

Original: However, this does not essentially change the problem that node trajectories cannot cross.
Revised: However, this does not essentially change the problem that node trajectories cannot cross.

Original: Even in a high-dimensional space, trajectories still cannot cross at the same time.
Revised: Even in a high-dimensional space, trajectories still cannot cross at the same time.

Original: The high-dimensional space, however, causes a rise in degrees of freedom from d to d + a , leading to an increase in the parameter space and theoretical time complexity of estimating the vector field.
Revised: The high-dimensional space, however, causes a rise in degrees of freedom from d to d + a , leading to an increase in the parameter space and theoretical time complexity of estimating the vector field.

Original: Our approach, on the other hand, allows the trajectory to “jump” between adjacent time points by adding the oscillator g .
Revised: Our approach, on the other hand, allows the trajectory to “jump” between adjacent time points by adding the oscillator g .

Original: The degree of its oscillation is controlled by the parameters θ g in g .
Revised: The degree of its oscillation is controlled by the parameters θ g in g .

Original: Thus, when given an input and an output, we do not have to optimize the vector field in the original space
Revised: Thus, when given an input and an output, we do not have to optimize the vector field in the original space

Original: X ∈ R d , because the vector field V in X can be extremely complex.
Revised: X ∈ R d , because the vector field V in X can be extremely complex.

Original: We can optimize the vector field in another space of the same dimension, i.e., X ′ ∈ R d , in which the vector field V ′ may be simple and easy to be optimized.
Revised: We can optimize the vector field in another space of the same dimension, i.e., X ′ ∈ R d , in which the vector field V ′ may be simple and easy to be optimized.

Original: The trajectories based on this vector field V ′ will also become simple.
Revised: The trajectories based on this vector field V ′ will also become simple.

Original: To further illustrate the time efficiency of our method, we consider its training process.
Revised: To further illustrate the time efficiency of our method, we consider its training process.

Original: For a NODE, its forward evaluation and backpropagation can be calculated by using a black-box differential equation solver, called ODE solvers (Kutta, 1901; Hairer, 1987; Runge, 1895).
Revised: For a NODE, its forward evaluation and backpropagation can be calculated by using a black-box differential equation solver, called ODE solvers (Kutta, 1901; Hairer, 1987; Runge, 1895).

Original: The Adjoint Sensitivity
Revised: The Adjoint Sensitivity

Original: Method (Pontryagin et al., 1961) is used for the calculation of backpropagation.
Revised: Method (Pontryagin et al., 1961) is used for the calculation of backpropagation.

Original: It computes gradients by solving a second, augmented ODE backward in time, and it can be applied to all ODE solvers (Chen et al., 2018).
Revised: It computes gradients by solving a second, augmented ODE backward in time, and it can be applied to all ODE solvers (Chen et al., 2018).

Original: For any ODE solvers, both fixed step size and adaptive step size solvers are often reasonable choices for NODEs.
Revised: For any ODE solvers, both fixed step size and adaptive step size solvers are often reasonable choices for NODEs.

Original: Given a final time T , a fixed step size solver will choose the time t i from [0 , T ] where ∆ t =
Revised: Given a final time T , a fixed step size solver will choose the time t i from [0 , T ] where ∆ t =

Original: t i +1 − t i .
Revised: t i +1 − t i .

Original: ∆ t is fixed in advance and independent of i .
Revised: ∆ t is fixed in advance and independent of i .

Original: An adaptive step solver, such as
Revised: An adaptive step solver, such as

Original: Runge-Kutta Method (Runge, 1895) is a relatively modern solver, which can vary the size of the next step so that the error made during the solver is approximately equal to some tolerance.
Revised: Runge-Kutta Method (Runge, 1895) is a relatively modern solver, which can vary the size of the next step so that the error made during the solver is approximately equal to some tolerance.

Original: Typically in practice, a commonly used ODE solver is an adaptive step solver.
Revised: Typically in practice, a commonly used ODE solver is an adaptive step solver.

Original: Although it is not possible to estimate the absolute time theoretically for an adaptive-step-solver-based NODE, we can estimate the computational efficiency by the time complexity as well as the NFEs.
Revised: Although it is not possible to estimate the absolute time theoretically for an adaptive-step-solver-based NODE, we can estimate the computational efficiency by the time complexity as well as the NFEs.

Original: Thus, we present proposition 4.1:
Revised: Thus, we present proposition 4.1:

Original: Proposition 4.1 With the same vector field modeling structure, ONODE has a higher computational efficiency than ANODE.
Revised: Proposition 4.1 With the same vector field modeling structure, ONODE has a higher computational efficiency than ANODE.

Original: Before the proof of the proposition 4.1, first we present lemma 4.1 and its proof.
Revised: Before the proof of the proposition 4.1, first we present lemma 4.1 and its proof.

Original: lemma 4.

Revised: lemma 4.

Original: With the same vector field modeling structure, ONODE has less time complexity as long as N ′ h <
Revised: With the same vector field modeling structure, ONODE has less time complexity as long as N ′ h <

Original: A × N h
Revised: A × N h

Original: N i
Revised: N i

Original: + N
Revised: + N

Original: i + A 2 , where N ′ h is the number of neurons in the hidden layer of g , N h is the number of neurons in the hidden layer of f , A is the augmented dimensions, and N i is the input dimensions.
Revised: i + A 2 , where N ′ h is the number of neurons in the hidden layer of g , N h is the number of neurons in the hidden layer of f , A is the augmented dimensions, and N i is the input dimensions.

Original: Proof: For a fair comparison between ONODE and ANODE, we assume that the input and output dimensions are the same, i.e., N i = N o , where the N o represents the output dimensions.

Original: For the vector field modeling structure, we assume there exist two hidden layers with each hidden layer having N h neurons.

Original: For convenience, we presume that there is only one sample.

Original: For forward evaluation, an ANODE will have the time complexity of O fAnode

Original: =

Original: O (( N i + 1 + A ) N h )

Original: + O ( N h N h )

Original: +

Original: O ( N h ( N i + A ))

Original: + O (( N i + A ) N i )

Original: =

Original: [-O (2 N i N h +
-]
Revised: [-O (2 N i N h +
-]

Original: [-AN h + N 2 h + N h + N 
 i + AN i ) .-]
Revised: [-AN h + N 2 h + N h + N 
 i + AN i ) .-]The proof of lemma 4.1 is shown in APPENDIX A.2.

Original: Our proposed

Original: ONODE will have the time complexity of O fOnode =

Original: O ( N i N ′ h )

Original: + O ( N ′ h

Original: N i )

Original: + O (( N i

Original: + 1) N h )

Original: +

Original: O ( N h N h )

Original: + O ( N h H

Original: i )

Original: =

Original: O (2 N i N h + N 2 h + N h

Original: + 2 N i N ′ h ) .

Original: Since the backpropagation has the same complexity as the forward evaluation, the total time complexities in one iteration for ANODE and ONODE are O Anode =

Original: 2 O fAnode and O Onode

Original: [-= 
-]
Revised: [-= 
-]

Original: O fOnode , respectively.

Original: We can immediately get the solution that O Onode < O Anode as long as N ′ h <

Original: A × N h

Original: N i + N i + A 2 . Since the adaptive step solvers such as Runge-Kutta are used in both ANODEs and ONODEs, we cannot obtain Proposition 4.1 immediately from lemma 4.1.
Revised: Since the adaptive step solvers such as Runge-Kutta are used in both ANODEs and ONODEs, we cannot obtain Proposition 4.1 immediately from lemma 4.1.

Original: However, since the NFEs are equal to the number of iterations of f in one training iteration and we experimentally show that an ONODE requires much fewer NFEs than an ANODE, thus we can obtain Proposition 4.1.
Revised: However, since the NFEs are equal to the number of iterations of f in one training iteration and we experimentally show that an ONODE requires much fewer NFEs than an ANODE, thus we can obtain Proposition 4.1.

Original: It is worth noting that our method can still be augmented by adding additional dimensions.
Revised: It is worth noting that our method can still be augmented by adding additional dimensions.

Original: But the nature of our method differs from ANODE in that the trajectories of our method are crossable in the original input space X ∈ R d .
Revised: But the nature of our method differs from ANODE in that the trajectories of our method are crossable in the original input space X ∈ R d .

Original: However, in the extra space R a , its trajectories remain uncrossable, which is inherited from an ANODE.
Revised: However, in the extra space R a , its trajectories remain uncrossable, which is inherited from an ANODE.

Original: Formally, an ONODE with the extra dimensions will be:
Revised: Formally, an ONODE with the extra dimensions will be:

Original: [Equation]
Revised: [Equation]

Original: where a ( t ) represents the extra dimensions for the original input.
Revised: where a ( t ) represents the extra dimensions for the original input.

Original: The function g represents the oscillator with parameters θ g .
Revised: The function g represents the oscillator with parameters θ g .

Original: The augmented dimension can be seen as an extra hyperparameter to tune.
Revised: The augmented dimension can be seen as an extra hyperparameter to tune.

Original: When the extra dimension is set to 0 , it becomes the ONODEs in Equation 5.
Revised: When the extra dimension is set to 0 , it becomes the ONODEs in Equation 5.

Original: For a more intuitive view, we show how our method works with a toy example and compares it with the NODE and the ANODE in Figure 3.
Revised: For a more intuitive view, we show how our method works with a toy example and compares it with the NODE and the ANODE in Figure 3.

Original: We show that the ONODE with extra dimensions is also a universal approximator in Theorem 4.2.
Revised: We show that the ONODE with extra dimensions is also a universal approximator in Theorem 4.2.

Original: Theorem 4.2  Let d, a ∈ N .
Revised: Theorem 4.2 (ONODEs with extra dimensions are universal approximators) Let d, a ∈ N .

Original: For g ∈ nonl ( R d → R d ) , l ∈ nonl ( R d → R d + a ) , and f ∈ cout ( R × R d + a → R d + a ) which has a unique solution, let ϕ g,l,f  h ( t ) =
Revised: For g ∈ nonl ( R d → R d ) , l ∈ nonl ( R d → R d + a ) , and f ∈ cout ( R × R d + a → R d + a ) which has a unique solution, let ϕ g,l,f : R d → R d + a denote the map X → X ′ with h ( t ) =

Original: (cid:20) g
Revised: (cid:20) g

Original: ( x t , θ g ) a ( t ) (cid:21) , h ′ ( t ) = f
Revised: ( x t , θ g ) a ( t ) (cid:21) , h ′ ( t ) = f

Original: ( (cid:20) h ◦ g 
Revised: ( (cid:20) h ( x t , θ g )

Original: ( x t , θ g ) a( t ) (cid:21) , t, θ f ) , h (0) = (cid:20) g
Revised: a

Original: ( x 0 , θ g ) (cid:21) , for t ∈ [0 , 1] .
Revised: [Equation] (cid:21) , for t ∈ [0 , 1] .

Original: Take C ( R d + a ) to be the family of real functions that one wishes to approximate in R d + a and set (cid:80) d
Revised: Take C ( R d + a ) to be the family of real functions that one wishes to approximate in R d + a and set (cid:80) d

Original: = span { ϕ g,l,f } , then (cid:80) d is dense in C ( R d + a ) .
Revised: = span { ϕ g,l,f } , then (cid:80) d is dense in C ( R d + a ) .

Original: Proof: The proofis similar to that of Theorem 4.1.
Revised: The proof of theorem 4.2 is shown in APPENDIX ?? .

Original: Since the extra dimensions R a will be added into the ONODE through a nonlinear function l , it will not change the result in Theorem 4.1.

Original: Thus, we obtain the result that span { ϕ g,l,f } is dense in C ( R d + a ) .

Original: It is important to note that the extra dimensions change the input space which, depending on the application, may not be desirable.
Revised: It is important to note that the extra dimensions change the input space which, depending on the application, may not be desirable.

Original: We experimentally demonstrate that with or without adding extra dimensions, our model is always better than the original NODEs and ANODEs.
Revised: We experimentally demonstrate that with or without adding extra dimensions, our model is always better than the original NODEs and ANODEs.

Original: In many cases, our model without extra dimensions is better than the ANODEs which have extra dimensions.
Revised: In many cases, our model without extra dimensions is better than the ANODEs which have extra dimensions.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We will demonstrate the superiority of our model in terms of accuracy, the number of function evaluations (NFEs), and convergence speed.
Revised: We will demonstrate the superiority of our model in terms of accuracy, the number of function evaluations (NFEs), and convergence speed.

Original: In Section 5.1, we introduce the datasets and environment settings.
Revised: In Section 5.1, we introduce the datasets and environment settings.

Original: In Section 5.2 we show on two toy datasets and three real-life image datasets that our model has better prediction accuracy, fewer NFEs, and faster convergence speed compared to baselines.
Revised: In Section 5.2 we show on two toy datasets and three real-life image datasets that our model has better prediction accuracy, fewer NFEs, and faster convergence speed compared to baselines.

Original: In
Revised: In

Original: Section 5.3 we apply our model to the extrapolating time series tasks.
Revised: Section 5.3 we apply our model to the extrapolating time series tasks.

Original: We also illustrate that the model has less prediction error and a higher computational efficiency compared to baselines.
Revised: We also illustrate that the model has less prediction error and a higher computational efficiency compared to baselines.

Original: 5. E NVIRONMENT SETUP
Revised: E NVIRONMENT SETUP

Original: Datasets.
Revised: Datasets.

Original: We evaluated our model with two toy datasets and three image datasets in the classification task.
Revised: We evaluated our model with two toy datasets and three image datasets in the classification task.

Original: We evaluated our model using five mathematical functions in the extrapolating time series task.
Revised: We evaluated our model using five mathematical functions in the extrapolating time series task.

Original: Demonstrations of the toy datasets are shown in Figure 2 and Figure 3, respectively.
Revised: Demonstrations of the toy datasets are shown in Figure 2 and Figure 3, respectively.

Original: The first toy dataset consists of four concentric circles, each consisting of 1 , 000 randomly generated data points.
Revised: The first toy dataset consists of four concentric circles, each consisting of 1 , 000 randomly generated data points.

Original: The second toy dataset consists of nine equally spaced data stacks, each consisting of 1 ,randomly generated points based on a Gaussian distribution.
Revised: The second toy dataset consists of nine equally spaced data stacks, each consisting of 1 ,randomly generated points based on a Gaussian distribution.

Original: For the image classification task , we evaluate our model on the MNIST (Deng 2012), CIFAR-10, and CIFAR-100 datasets Krizhevsky
Revised: For the image classification task, we evaluate our model on the MNIST (Deng, 2012), CIFAR-10, and CIFAR-100 datasets (Krizhevsky et al., 2009).

Original: MNIST is a handwritten digit database with a training set of 60 , 000 examples.
Revised: MNIST is a handwritten digit database with a training set of 60 , 000 examples.

Original: The
Revised: The

Original: CIFAR 10 training dataset consists of 60 , [-
-] 32 × 32 color images  ten classes,  CIFARhas 100 classes 
Revised: CIFAR-10 training dataset consists of 60 , [-
-]000 32 × 32 color images in ten classes, and CIFARhas 100 classes containing 600 images each.

Original: Evaluation Metrics and Baselines.
Revised: Evaluation Metrics and Baselines.

Original: For the classification task, we compared our model with
Revised: For the classification task, we compared our model with

Original: NODEs and ANODEs in terms of accuracy, NFEs, and convergence speed.
Revised: NODEs and ANODEs in terms of accuracy, NFEs, and convergence speed.

Original: For the extrapolating time series problem, our baselines include SONODE (Norcliffe et al., 2020) in addition to NODE and the ANODE.
Revised: For the extrapolating time series problem, our baselines include SONODE (Norcliffe et al., 2020) in addition to NODE and the ANODE.

Original: We compare the training and testing mean squared error (MSE), the absolute training time, and the number of parameters.
Revised: We compare the training and testing mean squared error (MSE), the absolute training time, and the number of parameters.

Original: We use the number in parentheses to denote the extra dimension, e.g., ANODE (1) denotes ANODE with the extra dimension of one.
Revised: We use the number in parentheses to denote the extra dimension, e.g., ANODE (1) denotes ANODE with the extra dimension of one.

Original: 5. R ESULTS ON CLASSIFICATION
Revised: 5. R ESULTS ON CLASSIFICATION

Original: Our model performs better than NODEs and ANODEs.
Revised: Our model performs better than NODEs and ANODEs.

Original: We first evaluated our model on two toy datasets.
Revised: We first evaluated our model on two toy datasets.

Original: We model the vector field with a multilayer perceptron containing three hidden layers, each containing 16 neurons, and the activation function is ReLU .
Revised: We model the vector field with a multilayer perceptron containing three hidden layers, each containing 16 neurons, and the activation function is ReLU .

Original: For the oscillator, we model it with a multilayer perceptron containing one hidden layer with 16 neurons.
Revised: For the oscillator, we model it with a multilayer perceptron containing one hidden layer with 16 neurons.

Original: For our models, we use ONODE with and without one extra dimension, respectively.
Revised: For our models, we use ONODE with and without one extra dimension, respectively.

Original: Our models show completely different flow trajectories and feature evolution with NODEs and ANODEs, see Figure 2 and Figure 3.
Revised: Our models show completely different flow trajectories and feature evolution with NODEs and ANODEs, see Figure 2 and Figure 3.

Original: Experiments show that our models have better accuracy and converge faster than NODEs and
Revised: Experiments show that our models have better accuracy and converge faster than NODEs and

Original: ANODEs, regardless of whether they have extra dimensions.
Revised: ANODEs, regardless of whether they have extra dimensions.

Original: For NFEs, our models are also much lower than NODEs and slightly lower than ANODEs.
Revised: For NFEs, our models are also much lower than NODEs and slightly lower than ANODEs.

Original: The results are shown in Figure 4.
Revised: The results are shown in Figure 4.

Original: The rightmost plot indicates the relationship between NFEs and loss.
Revised: The rightmost plot indicates the relationship between NFEs and loss.

Original: Then, we evaluated our model on the image classification task.
Revised: Then, we evaluated our model on the image classification task.

Original: We refer to the NODEs’ modeling method for vector fields, using three convolutional layers, with the number of filters set at 128.
Revised: We refer to the NODEs’ modeling method for vector fields, using three convolutional layers, with the number of filters set at 128.

Original: The activation function is ReLU .
Revised: The activation function is ReLU .

Original: Similarly, our oscillator modeling is also implemented by one convolutional layer.
Revised: Similarly, our oscillator modeling is also implemented by one convolutional layer.

Original: We tested our model on three image datasets, each running for 20 epochs.
Revised: We tested our model on three image datasets, each running for 20 epochs.

Original: We recorded the validation error and average NFEs, and the results are shown in Table 1.
Revised: We recorded the validation error and average NFEs, and the results are shown in Table 1.

Original: The results show that our model is the best in both validation error and average NFEs.
Revised: The results show that our model is the best in both validation error and average NFEs.

Original: [Table]

Original: Our model has higher computational efficiency.
Revised: Our model has higher computational efficiency.

Original: Since our methods only need to learn simpler vector fields and flow trajectories, they require fewer iterations to converge than NODEs and ANODEs.
Revised: Since our methods only need to learn simpler vector fields and flow trajectories, they require fewer iterations to converge than NODEs and ANODEs.

Original: To test this [Table] we measure the NFEs in both the forward evaluation and backpropagation process, then we obtain the total NFEs for each model.
Revised: To test this, we measure the NFEs in both the forward evaluation and backpropagation process, then we obtain the total NFEs for each model.

Original: Take the CIFAR-10 for an example.
Revised: Take the CIFAR-10 for an example.

Original: As seen in Figure 
Revised: As seen in Figure 5, the NFEs required by our models increase slower during both the forward evaluation and backpropagation process while it increases faster for ANODEs and NODEs. This phenomenon

Original: [Table]
Revised: [Table]

Original: the NFEs required by our models increase slower during both the forward evaluation and backpropagation process while it increases faster for ANODEs and NODEs.

Original: This phenomenon is particularly evident for the backpropagation process, where our method requires roughly 23 of the
Revised: is particularly evident for the backpropagation process, where our method requires roughly 23 of the

Original: NFEs of ANODEs and roughly 12 of the NODEs.
Revised: NFEs of ANODEs and roughly 12 of the NODEs.

Original: We obtain similar results for other image datasets such as MNIST and CIFAR-100.
Revised: We obtain similar results for other image datasets such as MNIST and CIFAR-

Original: [Figure]
Revised: [Table]

Revised: 5. R ESULTS ON EXTRAPOLATING TIME SERIES

Original: We investigate the ability of our model to fit and extrapolate time series.
Revised: We investigate the ability of our model to fit and extrapolate time series.

Original: We used five functions, as shown in Figure 6.
Revised: We used five functions, as shown in Figure 6.

Original: Each function has 50 uniformly sampled points in 0 to 10 seconds as the training set, followed by 10 points in 3 seconds as the predicted values.
Revised: Each function has 50 uniformly sampled points in 0 to 10 seconds as the training set, followed by 10 points in 3 seconds as the predicted values.

Original: Our model is set to have one extra dimension.
Revised: Our model is set to have one extra dimension.

Original: The baselines were chosen as NODE, ANODE with one extra dimension, ANODE with three extra dimensions, and SONODE.
Revised: The baselines were chosen as NODE, ANODE with one extra dimension, ANODE with three extra dimensions, and SONODE.

Original: Each model was trained for 500 iterations in the same environment.
Revised: Each model was trained for 500 iterations in the same environment.

Original: First, we compared the absolute training time (Figure 7) for each model.
Revised: First, we compared the absolute training time (Figure 7) for each model.

Original: We found that our model has a faster training speed with SONODE.
Revised: We found that our model has a faster training speed with SONODE.

Original: Then, we compared the training MSE, test MSE, and the number of parameters, as shown in Figure 8.
Revised: Then, we compared the training MSE, test MSE, and the number of parameters, as shown in Figure 8.

Original: For visual comparison, we have ranked the performance of the models according to MSE from smallest to largest.
Revised: For visual comparison, we have ranked the performance of the models according to MSE from smallest to largest.

Original: Our model has the smallest value in most of the training MSE and all the test MSE.
Revised: Our model has the smallest value in most of the training MSE and all the test MSE.

Original: We also found that as the extra dimension increases, there is no guarantee that ANODE becomes better.
Revised: We also found that as the extra dimension increases, there is no guarantee that ANODE becomes better.

Original: Our model has about the same number of parameters as ANODE, but has much smaller than the number of parameters of SONODE.
Revised: Our model has about the same number of parameters as ANODE, but has much smaller than the number of parameters of SONODE.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We propose the ONODE, which is an oscillator-based continuous deep learning model.
Revised: We propose the ONODE, which is an oscillator-based continuous deep learning model.

Original: Due to the presence of an oscillator, the trajectories of our model can “jump”, making it to be a universal approximator.
Revised: Due to the presence of an oscillator, the trajectories of our model can “jump”, making it to be a universal approximator.

Original: We also demonstrate that the ONODE is computationally efficient.
Revised: We also demonstrate that the ONODE is computationally efficient.

Original: We validate our model through various experiments including classification tasks and extrapolation on time series.
Revised: We validate our model through various experiments including classification tasks and extrapolation on time series.

Original: Compared to the baselines, our model achieves the best results on a variety of metrics.
Revised: Compared to the baselines, our model achieves the best results on a variety of metrics.

Revised: A A PPENDIX A. T HE PROOF OF THEOREM 4.

Revised: Proof : Suppose the input x ∈ R d , the system of ODEs will solve a series of equations y 0 , y 1 , ..., y M as follows: (1) h 0 (0) =

Original: {+g ( x ) ∈ R d and dh 0 dt ( t ) = 0 , (2) h 1 (0) = 
+}
Revised: {+g ( x ) ∈ R d and dh 0 dt ( t ) = 0 , (2) h 1 (0) = 
+}

Revised: ∈ R d × d and dh 1 dt = h 0 ( t ) ⊗ h 0 ( t ) , (3) h 2 (0) =

Revised: 0 ∈ R d 3 and dh 2 dt ( t ) =

Revised: h 1 ( t ) ⊗ h 0 ( t ) , (4) h 3 (0) =

Revised: 0 ∈ R

Revised: d 4 and dh 3 dt ( t ) =

Revised: h 2 ( t ) ⊗ h 0 ( t ) , ... , (M+1) h M (0)

Original: {+= 
+}
Revised: {+= 
+}

Revised: ∈ R

Revised: d M +1 and dh M dt ( t ) =

Revised: h M − 1 ( t ) ⊗ h 0 ( t ) .

Revised: The solution will be written immediately: (1) h 0 ( t ) =

Revised: g ( x ) , (2) h 1 ( t ) =

Revised: (cid:82) t 0 g ( x ) ⊗ g ( x ) ds

Revised: = tg ( x ) ⊗ 2 , (3) h 2 ( t )

Revised: = (cid:82) t 0 g ( x ) ⊗ 2 t ⊗ g ( x ) ds = 12 t 2 g ( x ) ⊗ 3 , (4) h 3 ( t )

Revised: =

Revised: (cid:82) t 0 g ( x ) ⊗ 3 t ⊗ g ( x ) ds = 16 t 3 g ( x ) ⊗ 4 , ... , (M+1) h M ( t )

Revised: =

Revised: (cid:82) t 0 g ( x ) ⊗ ( M +1) t ⊗ g ( x ) ds = 1 M !

Revised: t M g ( x ) ⊗ ( M +1) .

Revised: Set t = 1 and we will obtain the collection in g ( x ) ∈ R

Revised: d up to degree M + 1 , i.e., { g ( x ) , g ( x ) ⊗ 2 , 12 g ( x ) ⊗ 3 , 16 g ( x ) ⊗ 4 , ..., 1 M !

Revised: g ( x ) ⊗ ( M +1) } .

Revised: The function g ( x ) is realized by a single hidden layer perception utilizing a Relu active function.

Revised: Assume µ is a non-negative finite measure on R d with compact support, and continuous with respect to the Lebesgue measure.

Revised: Then (cid:80) d is dense in L p ( µ ) , 1 ≤ p < ∞ , if and only if, the nonlinearity in g is not a polynomial, where L p ( µ ) is the set of all measurable functions f such that ∥ f ∥ L p ( µ )

Revised: =

Revised: ( (cid:82) R d | f ( x ) | p dµ ( x )) 1 /p < ∞ (Leshno et al., 1993;

Revised: Hornik et al., 1989; Hornik, 1991;

Revised: Pinkus, 1999).

Revised: Thus, span { ϕ g,f } is dense in C ( R d ) and it is a universal approximator.

Revised: A. T HE PROOF OF LEMMA 4.

Revised: Proof: For a fair comparison between ONODE and ANODE, we assume that the input and output dimensions are the same, i.e., N i = N o , where the N o represents the output dimensions.

Revised: For the vector field modeling structure, we assume there exist two hidden layers with each hidden layer having N h neurons.

Revised: For convenience, we presume that there is only one sample.

Revised: For forward evaluation, an ANODE will have the time complexity of O fAnode

Revised: =

Revised: O (( N i + 1 + A ) N h )

Revised: + O ( N h N h )

Revised: +

Revised: O ( N h ( N i + A ))

Revised: + O (( N i + A ) N i )

Revised: =

Original: {+O (2 N i N h +
+}
Revised: {+O (2 N i N h +
+}

Revised: AN h + N 2 h

Revised: + N h

Original: {++ N 
+}
Revised: {++ N 
+}

Revised: i + AN i ) .

Revised: Our proposed

Revised: ONODE will have the time complexity of O fOnode =

Revised: O ( N i N ′ h )

Revised: + O ( N ′ h

Revised: N i )

Revised: + O (( N i

Revised: + 1) N h )

Revised: +

Revised: O ( N h N h )

Revised: + O ( N h H

Revised: i )

Revised: =

Revised: O (2 N i N h + N 2 h + N h

Revised: + 2 N i N ′ h ) .

Revised: Since the backpropagation has the same complexity as the forward evaluation, the total time complexities in one iteration for ANODE and ONODE are O Anode =

Revised: 2 O fAnode and O Onode

Original: {+= 
+}
Revised: {+= 
+}

Revised: O fOnode , respectively.

Revised: We can immediately get the solution that O Onode < O Anode as long as N ′ h <

Revised: A × N h

Revised: N i

Revised: + N

Revised: i + A 2 .

Revised: A. T HE PROOF OF LEMMA 4.

Revised: Proof: The proof is similar to that of proof A.1.

Revised: Since the extra dimensions R a will be added into the ONODE through a nonlinear function l , it will not change the result in proof A.1.

Revised: Thus, we obtain the result that span { ϕ g,l,f } is dense in C ( R d + a ) .