Original: DISE: D YNAMIC I NTEGRATOR S ELECTION TO M INIMIZE F ORWARD - PASS T IME IN N EURAL ODE S
Revised: DISE: D YNAMIC I NTEGRATOR S ELECTION TO M INIMIZE F ORWARD - PASS T IME IN N EURAL ODE S

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Neural ordinary differential equations (Neural ODEs) are appreciated for their ability to signiﬁcantly reduce the number of parameters when constructing a neural network.
Revised: Neural ordinary differential equations (Neural ODEs) are appreciated for their ability to signiﬁcantly reduce the number of parameters when constructing a neural network.

Original: On the other hand, they are  blamed for their procrastinated forwardpass inference, which is incurred by solving integral problems.
Revised: On the other hand, they are sometimes blamed for their long forwardpass inference time, which is incurred by solving integral problems.

Original: To improve the model accuracy, they rely on advanced solvers, such as the Dormand–Prince (DOPRI) method.
Revised: To improve the model accuracy, they rely on advanced solvers, such as the Dormand–Prince (DOPRI) method.

Original: To solve an integral problem, however, it requires at least tens (or sometimes hundreds) of steps in many Neural ODE experiments.
Revised: To solve an integral problem, however, it requires at least tens (or sometimes thousands) of steps in many Neural ODE experiments.

Original: In this work, we propose to i) directly regularize the step size of DOPRI to make the forward-pass faster and ii) dynamically choose a simpler integrator than DOPRI for a carefully selected subset of input.
Revised: In this work, we propose to i) directly regularize the step size of DOPRI to make the forwardpass faster and ii) dynamically choose a simpler integrator than DOPRI for a carefully selected subset of input.

Original: Because it is not the case that every input requires the advanced integrator, we design an auxiliary neural network to choose an appropriate integrator given input to decrease the overall inference time without signiﬁcantly sacriﬁcing accuracy.
Revised: Because it is not the case that every input requires the advanced integrator, we design an auxiliary neural network to choose an appropriate integrator given input to decrease the overall inference time without signiﬁcantly sacriﬁcing accuracy.

Original: We consider the Euler method, the fourth-order Runge–Kutta (RK4) method, and DOPRI as selection candidates.
Revised: We consider the Euler method, the fourth-order Runge–Kutta (RK4) method, and DOPRI as selection candidates.

Original: We found that a non-trivial percentage of cases can be solved with simple integrators.
Revised: We found that 10-30% of cases can be solved with simple integrators in our experiments.

Original: Therefore, the overall number of functional evaluations (NFEs) decreases up to 78% with improved accuracy.
Revised: Therefore, the overall number of functional evaluations (NFE) decreases up to 78% with improved accuracy.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Neural ordinary differential equations (Neural ODEs) are to learn time-dependent physical dynamics describing hidden vector evolution processes in neural networks (Chen et al., 2018).
Revised: Neural ordinary differential equations (Neural ODEs) are to learn time-dependent physical dynamics describing continuous residual networks (Chen et al., 2018).

Original: It is well known that residual connections are numerically similar to the  Euler method, the simplest integrator to solve ODEs.
Revised: It is well known that residual connections are numerically similar to the explicit Euler method, the simplest integrator to solve ODEs.

Original: In this regard, Neural ODEs are considered as a generalization of residual networks.
Revised: In this regard, Neural ODEs are considered as a generalization of residual networks.

Original: In general, it is agreed by many researchers that Neural ODEs have two advantages and one disadvantage: i) NeuralODEs can reduce the required number of neural network parameters to train, ii)
Revised: In general, it is agreed by many researchers that Neural ODEs have two advantages and one disadvantage: i) Neural

Original:  Neural ODEs can interpret the neural network layer  as a continuous variable and a hidden vector at an arbitrary layer can be calculated, iii) however, Neural ODEs’s forward-pass  can sometimes be numerically unstable  and/or procrastinated to solve an integral problem  (Zhuang et al., 2020b; Finlay et al., 2020; Daulbaev et al., 2020; Quaglino et al., 2020)
Revised: ODEs can sometimes reduce the required number of neural network parameters, e.g., (Pinckaers & Litjens, 2019), ii) Neural ODEs can interpret the neural network layer (or time) as a continuous variable and a hidden vector at an arbitrary layer can be calculated, iii) however, Neural ODEs’s forward-pass inference can sometimes be numerically unstable (i.e., the underﬂow error of DOPRI’s adaptive step size) and/or slow to solve an integral problem (i.e., too many steps in DOPRI) (Zhuang et al., 2020b; Finlay et al., 2020; Daulbaev et al., 2020; Quaglino et al., 2020).

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: Much work has been actively studied to address the numerically unstable nature of solving integral problems.
Revised: Much work has been actively devoted to address the numerically unstable nature of solving integral problems.

Original: In this work, however, we are interested in addressing the problem of procrastinated forward-pass inference.
Revised: In this work, however, we are interested in addressing the problem of long forward-pass inference time.

Original: To overcome the challenge, we i) directly regularize the number of steps of the Dormand– Prince (DOPRI) method (Dormand & Prince, 1980), which means we try to learn an ODE that can be quickly solved by DOPRI, and ii) dynamically select an appropriate integrator for each sample rather than relying on only one integrator.
Revised: To overcome the challenge, we i) directly regularize the numerical errors of the Dormand–Prince (DOPRI) method (Dormand & Prince, 1980), which means we try to learn an ODE that can be quickly solved by DOPRI, and ii) dynamically select an appropriate integrator for each sample rather than relying on only one integrator.

Original: In many cases, Neural ODEs use DOPRI, one of the most advanced adaptive step integrator, for its best accuracy.
Revised: In many cases, Neural ODEs use DOPRI, one of the most advanced adaptive step integrator, for its best accuracy.

Original: However, our method allows that we rely on simpler integrators, such as the Euler method or thefourth-order
Revised: However, our method allows that we rely on simpler integrators, such as the Euler method or the

Original: Runge–Kutta (RK4) method (Ixaru & Vanden Berghe, 2004), for carefully selected inputs.

Original: [Figure]
Revised: [Figure]

Original: Table 1 shows an experimental result that our proposed regularization not only reduces the number of function evaluations (NFEs) — the inference time is linearly proportional to the number of function evaluations in Neural ODEs — but also increases the inference accuracy in the MNIST classiﬁcation task.
Revised: Table 1 shows an experimental result that our proposed regularization not only reduces the number of function evaluations (NFE) — the inference time is linearly proportional to the number of function evaluations in Neural ODEs — but also increases the inference accuracy in the MNIST classiﬁcation task.

Original: We can reduce the inference time by reducing the average number of steps (and thus, the average NFEs) of DOPRI, which can be obtained when the learned ODE is trained to be in a suitable form to solve with DOPRI with a proper regularization.
Revised: We can reduce the inference time by reducing the average number of steps (and thus, the average NFE) of DOPRI, which can be obtained when the learned ODE is trained to be in a suitable form to solve with DOPRI with a proper regularization.

Original: We also found that simpler integrators can be used for carefully selected samples.
Revised: However, the NFE of DOPRI in a step is 6 whereas RK4 has 4 and the Euler method has 1. So, the Euler method is six times faster than DOPRI even when their step sizes are identical. Therefore, the automatic step size adjustment of DOPRI is not enough to minimize the NFE of forward-pass inference (see Section B in Appendix for more detailed descriptions with a concrete example).

Original: To this end, we design an auxiliary network that chooses an appropriate integrator for each sample.
Revised: To this end, we design an auxiliary network that chooses an appropriate integrator for each sample.

Original: The combination of our regularization and the proposed D ynamic I ntegrator SE lection (DISE) shows the best performance in the table.
Revised: The combination of our regularization and the proposed D ynamic I ntegrator SE lection (DISE) shows the best performance in the table.

Original: We conduct experiments for three different tasks and datasets: MNIST image classiﬁcation, PhysioNet mortality prediction, and continuous normalizing ﬂows.
Revised: We conduct experiments for three different tasks and datasets: MNIST image classiﬁcation, PhysioNet mortality prediction, and continuous normalizing ﬂows.

Original: Our method shows the best (or close to the best) accuracy with a much smaller NFEs than state-of-the-art methods.
Revised: Our method shows the best (or close to the best) accuracy with a much smaller NFE than state-of-the-art methods.

Original: Our contributions can be summarized as follows:
Revised: Our contributions can be summarized as follows:

Original: We design an effective regularization to reduce the number of function evaluations (NFEs) of Neural ODEs.
Revised: We design an effective regularization to reduce the number of function evaluations (NFE) of Neural ODEs.

Original: We design a sample-wise dynamic integrator selection (DISE) method to further accelerate Neural ODEs without signiﬁcantly sacriﬁcing model accuracy.
Revised: We design a sample-wise dynamic integrator selection (DISE) method to further accelerate Neural ODEs without signiﬁcantly sacriﬁcing model accuracy.

Original: We conduct in-depth analyses with three popular tasks of Neural ODEs.
Revised: We conduct in-depth analyses with three popular tasks of Neural ODEs.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: In this section, we review the literature on Neural ODEs.
Revised: In this section, we review the literature on Neural ODEs.

Original: In particular, we review recent regularization designs for Neuarl ODEs and numerical methods to solve ODEs
Revised: In particular, we review recent regularization designs for Neuarl ODEs and numerical methods to solve ODEs.

Original: . 2. N EURAL ODE S
Revised: 2. N EURAL ODE S

Original: It had been reported that approximating neural networks with differential equationscan be done by many researchers (Weinan, 2017; Ruthotto & Haber, 2019; Lu et al., 2018; Ciccone et al., 2018; Chen et al., 2018; Gholami et al., 2019).
Revised: It had been attempted by several researchers to model neural networks as differential equations (Weinan, 2017; Ruthotto & Haber, 2019; Lu et al., 2018; Ciccone et al., 2018; Chen et al., 2018; Gholami et al., 2019).

Original: Among them, the seminal neural ordinary differential equations (Neural ODEs), as shown in Fig. 2, consist of three parts in general: a feature extractor, an ODE, and a classiﬁer (Chen et al., 2018; Zhuang et al., 2020a).
Revised: Among them, the seminal neural ordinary differential equations (Neural ODEs), as shown in Fig. 1, consist of three parts in general: a feature extractor, an ODE, and a classiﬁer (Chen et al., 2018; Zhuang et al., 2020a).

Original: Given an input x , the feature extractor produces an input to the ODE, denoted h (0) .
Revised: Given an input x , the feature extractor produces an input to the ODE, denoted h (0) .

Original: Let h ( t ) be a hidden vector at layer (or time) t in the ODE part.
Revised: Let h ( t ) be a hidden vector at layer (or time) t in the ODE part.

Original: In Neural ODEs, a neural network f with a set of parameters, denoted θ , approximates ∂ h ( t ) ∂t and h ( t 1 ) becomes h (0)
Revised: In Neural ODEs, a neural network f with a set of parameters, denoted θ , approximates ∂ h ( t ) ∂t and h ( t 1 ) becomes h (0)

Original: + (cid:82) t 1 t 0 f ( h ( t ) , t ; θ ) dt , where f ( h ( t ) , t ; θ ) =
Revised: + (cid:82) t 1 t 0 f ( h ( t ) , t ; θ ) dt , where f ( h ( t ) , t ; θ ) =

Original: ∂ h ( t ) ∂t .
Revised: ∂ h ( t ) ∂t .

Original: In other words, the internal dynamics of the hidden vector evolution is described by an ODE.
Revised: In other words, the internal dynamics of the hidden vector evolution is described by an ODE.

Original: One key advantage of Neural ODEs is that we can reduce the number of parameters without sacriﬁcing model accuracy.
Revised: One key advantage of Neural ODEs is that we can reduce the number of parameters without sacriﬁcing model accuracy.

Original: For instance, one recent work based on a Neural ODE marked the best accuracy for medical image segmentation with an order of magnitude smaller parameter numbers (Pinckaers & Litjens, 2019).
Revised: For instance, one recent work based on a Neural ODE marked the best accuracy for medical image segmentation with an order of magnitude smaller parameter numbers (Pinckaers & Litjens, 2019).

Original: In general, we calculate h (1) 1 and feed it into the next classiﬁer and its ﬁnal prediction is made.
Revised: In general, we calculate h (1) 1 and feed it into the next classiﬁer and its ﬁnal prediction is made.

Original: One can accordingly modify the architecture in Fig.
Revised: One can accordingly modify the architecture in Fig.

Original: 1 for other types of tasks.
Revised: 1 for other types of tasks.

Original: For simplicity but without loss of generality, we assume the architecture in our discussion.
Revised: For simplicity but without loss of generality, we assume the architecture in our discussion.

Original: Neural ODEs have been used in many tasks, ranging from classiﬁcation and regression to time series forecasting and generative models (Yildiz et al., 2019; Grathwohl et al., 2019; Rubanova et al., 2019).
Revised: Neural ODEs have been used in many tasks, ranging from classiﬁcation and regression to time series forecasting and generative models (Yildiz et al., 2019; Grathwohl et al., 2019; Rubanova et al., 2019).

Original: 2. ODE S OLVERS
Revised: 2. ODE S OLVERS

Original: DOPRI is one of the most powerful integrators  and widely used in Neural ODEs.
Revised: DOPRI is one of the most powerful integrators (Hairer et al., 1993) and widely used in Neural ODEs.

Original: It is a member of the Runge–Kutta family of ODE solvers.
Revised: It is a member of the Runge–Kutta family of ODE solvers.

Original: DOPRI dynamically controls the step size while solving an integral problem.
Revised: DOPRI dynamically controls the step size while solving an integral problem.

Original: It is now the default method for MATLAB, GNU Octave, and Simulink.
Revised: It is now the default method for MATLAB, GNU Octave, and Simulink.

Original: It internally estimates an error by using a heuristic method and the step size is decided by a function inversely proportional to the estimated error — the larger the error, the shorter the step size.
Revised: It internally estimates an error by using a heuristic method and the step size is determined by a function inversely proportional to the estimated error — the larger the error, the shorter the step size.

Original: The error  is estimated by the difference between the fourth-order and the ﬁfth-order Runge–Kutta methods.
Revised: The error at i -th step of DOPRI for an integral problem x , denoted err x,i , is estimated by the difference between the fourth-order and the ﬁfth-order Runge–Kutta methods at the moment.

Original: The intuition behind the heuristic error estimation is simple yet effective.
Revised: The intuition behind the heuristic error estimation is simple yet effective.

Original: One step of DOPRI involves six function evaluations, i.e., six function calls of f .

Original: Among simpler methods, we consider the Euler method, and the fourth-order Runge–Kutta (RK4) method.
Revised: Among simpler methods, we consider the Euler method, and the fourth-order Runge–Kutta (RK4) method.

Original: The Euler method is the simplest method to solve ODEs and both the Euler method and RK4 use a ﬁxed step size.
Revised: The Euler method is the simplest method to solve ODEs and both the Euler method and RK4 use a ﬁxed step size.

Original: Therefore, their solving time is deterministic.
Revised: Therefore, their solving time is deterministic.

Revised: One step of DOPRI involves six function evaluations, i.e., six function calls of f .

Original: The Euler method calls the network f only once in a step and RK4 calls four times.
Revised: The Euler method calls the network f only once in a step and RK4 calls four times.

Revised: Therefore, the Euler method is six times faster than DOPRI for a step.

Original: The number of function evaluations (NFEs) refers to the number of function evaluations to solve an integral problem.
Revised: The term ‘NFE’ refers to the number of function evaluations to solve an integral problem.

Original: For the Euler method and RK4, NFE is deterministic and does not vary.
Revised: For the Euler method and RK4, NFE is deterministic and does not vary.

Original: In DOPRI, however, NFE varies from one sample to another, depending on the estimated error and the number of steps.
Revised: In DOPRI, however, NFE varies from one sample to another, depending on the estimated error and the number of steps.

Revised: We refer readers to Section B in Appendix for more detailed descriptions with a concrete example.

Original: 2. R EGULARIZATIONS IN N EURAL ODE S
Revised: 2. R EGULARIZATIONS IN N EURAL ODE S

Original: To make Neural ODEs faster, one possible way is regularizing the ODE function f .
Revised: To make Neural ODEs faster, one possible way is regularizing the ODE function f .

Original: Two na ¨ ıve methods are regularizing θ with the L 1 or L 2 regularizers (Ng, 2004).
Revised: Two na ¨ ıve methods are regularizing θ with the L 1 or L 2 regularizers (Ng, 2004).

Original: Strictly speaking, these two regularizers are to prevent overﬁtting.
Revised: Strictly speaking, these two regularizers are to prevent overﬁtting.

Original: Therefore, preventing overﬁtting does not necessarily mean quick forward-pass inference.
Revised: Therefore, preventing overﬁtting does not necessarily mean quick forward-pass inference.

Original: To this end, Dupont et al. said that by augmenting h ( t ) with additional zeros, i.e., augmenting the dimensionality of h ( t ) , one can achieve similar effects (Dupont et al., 2019).
Revised: To this end, Dupont et al. showed that by augmenting h ( t ) with additional zeros, i.e., augmenting the dimensionality of h ( t ) , one can achieve similar effects (Dupont et al., 2019).

Original: However, this method is meaningful when we cannot freely control the dimensionality of h ( t ) , which is not our main interest.
Revised: However, this method is meaningful when we cannot freely control the dimensionality of h ( t ) , which is not our setting.

Original: Recently, a kinetic regularization concept has been proposed by Finlay et al.
Revised: Recently, a kinetic regularization concept has been proposed by Finlay et al.

Original: (Finlay et al., 2020), which is written as follows:
Revised: (Finlay et al., 2020), which is written as follows:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Among all regularization terms designed so far, this kinetic regularization’s goal is  closest to ours.
Revised: Among all regularization terms designed so far, this kinetic regularization’s goal is the closest to ours.

Original: It can encourage Neural ODEs to learn straight-line paths from h ( t 0 ) to h ( t 1 ) .
Revised: It can encourage Neural ODEs to learn straight-line paths from h ( t 0 ) to h ( t 1 ) .

Original:  P ROPOSED M ETHOD
Revised:  P ROPOSED M ETHOD

Original: While enabling the design of compact models, Neural ODEs have one critical drawback that it requires solvingan integral problem, for which many approximation methods have been proposed: the Euler method, the RK4 method, the DOPRI method, to name a few.
Revised: While enabling the design of compact models, Neural ODEs have one critical drawback that they require solving integral problems, for which many approximation methods have been proposed: the Euler method, RK4, and DOPRI, to name a few.

Original: Almost all of them are based on discretizing t and converting an integral into a series of additions.
Revised: Almost all of them are based on discretizing t and converting an integral into a series of additions.

Original: In many cases, therefore, it requires a dense discretization, resulting in a long forward-pass.
Revised: In many cases, therefore, it requires a dense discretization, resulting in a long forward-pass inference time.

Original:  In this paper, we tackle the problem of minimizing the number of function evaluations (and thereby, the forward-pass inference time) of Neural ODEs without signiﬁcantly sacriﬁcing model accuracy.
Revised: [Figure] In this paper, we tackle the problem of minimizing the number of function evaluations (and thereby, the forward-pass inference time) of Neural ODEs without signiﬁcantly sacriﬁcing model accuracy.

Original: Our proposed method consists of two parts: i) using the DOPRI’s error estimation as a regularizer and ii) using an auxiliary network to select an appropriate integrator for each input sample.[Figure]
Revised: Our proposed method consists of two parts: i) using the DOPRI’s error estimation as a regularizer and ii) using an auxiliary network to select an appropriate integrator for each input sample.

Original: 3. DOPRI’ S E RROR E STIMATION AS A R EGULARIZER
Revised: 3. DOPRI’ S E RROR E STIMATION AS A R EGULARIZER

Original: We re-implement the DOPRI method in PyTorch and make it return the estimated error terms.
Revised: We re-implement the DOPRI method in PyTorch and make it return the estimated error terms.

Original: Let { err x, 1 , err x, 2 , · · · , err x,N } , where N is the number of function evaluations of DOPRI, be an array of errors estimated by DOPRI while solving an integral problem for an input x .
Revised: Let { err x, 1 , err x, 2 , · · · , err x,N } , where N is the number of steps of DOPRI, be an array of errors estimated by DOPRI while solving an integral problem for an input x .

Original: Note that the adaptive step size isdecided by an inverse function of the error at each step.
Revised: Note that the adaptive step size is an inverse function of the error at each step.

Original: We use the following regularizer while training Neural ODEs:
Revised: We use the following regularizer while training Neural ODEs:

Original: [Equation]
Revised: [Equation]

Original: where x is an input sample for which we have to solve an integral problem, and T is a training set.
Revised: where x is an input sample for which we have to solve an integral problem, and T is a training set.

Original: We note that the maximum value for i varies from one sample to another.

Original: For instance, x can be an image sample to classify.
Revised: For instance, x can be an image sample to classify.

Original: If we train a Neural ODE to classify images with the cross-entropy loss in conjunction with the regularizer, the trained Neural ODE will learn an
Revised: If we train a Neural ODE to classify images with the cross-entropy loss in conjunction with the regularizer, the trained Neural ODE will learn an

Original: ODE that can correctly classify images while reducing the forward-pass time of DOPRI.
Revised: ODE that can correctly classify images while reducing the forward-pass time of DOPRI.

Original: The backward-pass calculation of our proposed regularizer can be done in O ( 1 s avg ) by maintaining the forward-pass computation graph, where s avg is the average step size of DOPRI.
Revised: The backward-pass calculation of our proposed regularizer can be done in O ( 1 s avg ) by maintaining the forward-pass computation graph, where s avg is the average step size of DOPRI.

Original: However, this complexity will decrease as training goes on with our regularizer because the average step size will decrease.
Revised: However, this complexity will decrease as training goes on with our regularizer because the average step size will increase.

Original: 3. A UXILIARY N ETWORK TO S ELECT I NTEGRATOR
Revised: 3. A UXILIARY N ETWORK TO S ELECT I NTEGRATOR

Original: We introduce our auxiliary network v to dynamically select an appropriate integrator given input.
Revised: We introduce our auxiliary network v to dynamically select an appropriate integrator given input.

Original: This network v cooperates with a Neural ODE as shown in Fig.
Revised: This network v cooperates with a Neural ODE as shown in Fig.

Original: Before training the auxiliary network, we ﬁrst train a target Neural ODE.
Revised: Before training the auxiliary network, we ﬁrst train a target Neural ODE.

Original: We use the following loss function to train the target
Revised: We use the following loss function to train the target

Original: Neural ODE:
Revised: Neural ODE:

Original: [Equation]
Revised: [Equation]

Original: where L task is a task-speciﬁc training loss, which can be optimized through the adjoint sensitivity method, and R is an appropriate regularization term.
Revised: where L task is a task-speciﬁc training loss, which can be optimized through the adjoint sensitivity method, and R is an appropriate regularization term.

Original: λ ≥ 0 is a coefﬁcient to emphasize the regularization term.
Revised: λ ≥ 0 is a coefﬁcient to emphasize the regularization term.

Original: We then train the auxiliary network v ( h (0); θ v ) to predict the costs of the Euler method, RK4, and
Revised: We then train the auxiliary network v ( h (0); θ v ) to predict the costs of the Euler method, RK4, and

Original: DOPRI for solving for h (0) .
Revised: DOPRI.

Original: Given a ﬁxed Neural ODE f
Revised: Given a ﬁxed Neural ODE f

Original: and θ , we use the same training data to collect the following data for each integrator:
Revised: and θ , we use the same training data to collect the following data for each integrator:

Original: [Equation]
Revised: [Equation]

Revised: prediction is

Revised: [Equation]

Revised: same as

Revised: [Equation]

Original:  where ∆ > 0 is the number of function evaluations (NFEs) to solve the integral problem of h (0) + (cid:82) tt 

Revised: [Equation] training [Equation] where ∆ > 0 is the number of function evaluations (NFE) to solve the integral problem of h (0) + (cid:82) tt

Original: f ( h ( t ) , t ; θ ) dt , M is a large enough penalty, and α is an exponent.
Revised: f ( h ( t ) , t ; θ ) dt , M is a large enough penalty, and α is an exponent.

Original: We evaluate this cost valuec for each integrator and train the network v . The auxiliary network predicts the costs of the three integrators simultaneously, i.e., its output dimensionality is 3.
Revised: We evaluate this cost value

Original: If the target task is not a classiﬁcation problem, we use the following cost:
Revised: If the target task is not a classiﬁcation problem, we use the following cost:

Original: [Equation]
Revised: [Equation]

Original: where Γ is an error estimation for a certain integrator, such as mean absolute/squared error, KL divergence, and so forth, for which we prefer small values.
Revised: where Γ is an error estimation for a certain integrator, such as mean absolute/squared error, KL divergence, and so forth, for which we prefer small values.

Original: If larger values are preferred, the above deﬁnition should be accordingly modiﬁed.
Revised: If larger values are preferred, the above deﬁnition should be accordingly modiﬁed.

Original: β is a hyperparameter to decide a threshold.
Revised: β is a hyperparameter to decide a threshold.

Original: Note that training v becomes a regression problem with supervision.
Revised: Note that training v becomes a regression problem with supervision.

Original: After many epochs, the auxiliary integrator selection network v is stabilized and we can deploy them.
Revised: After many epochs, the auxiliary integrator selection network v is stabilized and we can deploy it.

Original: The best integrator for h (0) is selected by ﬁnding the smallest predicted cost.
Revised: The best integrator for h (0) is selected by ﬁnding the smallest predicted cost.

Original: We also note that we need to run the network v only once to select an appropriate integrator, which incurs a little overhead.
Revised: We also note that we need to run the network v only once to select an appropriate integrator for a test case, which incurs a little overhead.

Original: The neural network architecture for v should be carefully designed for each application.
Revised: The neural network architecture for v should be carefully designed for each application.

Original: We introduce our general method in this section.
Revised: We introduce our general method in this section.

Original: In the experimental evaluation section, we will introduce our design choice for v in each application.
Revised: In the experimental evaluation section, we will introduce our design choice for v in each application.

Original:  E XPERIMENTAL E VALUATIONS
Revised:  E XPERIMENTAL E VALUATIONS

Original: We describe our detailed experimental environments and results with the following three different tasks: i) MNIST image classiﬁcation, ii)
Revised: We describe our detailed experimental environments and results with the following three different tasks: i) MNIST image classiﬁcation, ii)

Original: PhysioNet mortality prediction, and iii) continuous normalizing ﬂows.
Revised: PhysioNet mortality prediction, and iii) continuous normalizing ﬂows.

Original: We conduct our experiments in the following sequence.
Revised: We conduct our experiments in the following sequence.

Original: We ﬁrst compare the following regularization methods: i) Neural ODEs without any regularizer, ii) with the L 1 regularizer, iii) with the
Revised: We ﬁrst compare the following regularization methods: i) Neural ODEs without any regularizer, ii) with the L 1 regularizer, iii) with the

Original: L 2 regularizer, iv) with the kinetic energy regularizer, and v) with our proposed regularizer.
Revised: L 2 regularizer, iv) with the kinetic energy regularizer, and v) with our proposed regularizer.

Original: At this stage, we do not include the auxiliary network yet.
Revised: At this stage, we do not include the auxiliary network yet.

Original: This stage is to study which regularizer works better than others in each task.
Revised: This stage is to study which regularizer works better than others in each task.

Original: After selecting the best regularizer method for each task, we train all networks including the auxiliary network, which is our main experimental stage.
Revised: After selecting the best regularization method for each task, we train all networks including the auxiliary network.

Original: Throughout these experiments, we show that i) the forward-pass of learned ODEs can be faster by using an appropriate regularizer, and ii) the auxiliary network strategically selects simpler integrators than DOPRI for carefully chosen input samples.
Revised: Throughout these experiments, we show that i) the forward-pass of learned ODEs can be faster by using an appropriate regularizer, and ii) the auxiliary network strategically selects simpler integrators than DOPRI for carefully chosen input samples.

Revised: For each experiment, we show NFE values and unit

Revised: NFE time in seconds and its multiplication will be wall-clock time.

Revised: We repeat training and testing with 10 different seeds and report the mean value in the main paper and the standard deviation value in Appendix.

Revised: Experiments with various integrators are also in Appendix.

Original: All experiments were conducted in the following software and hardware environments: U BUNTU 18.04 LTS, P YTHON 3.6.6, N UMPY 1.18.5, S CIPY 1.5, M ATPLOTLIB 3.3.1, P Y T ORCH 1.2.0,
Revised: All experiments were conducted in the following software and hardware environments: U BUNTU 18.04 LTS, P YTHON 3.6.6, N UMPY 1.18.5, S CIPY 1.5, M ATPLOTLIB 3.3.1, P Y T ORCH 1.2.0,

Original: CUDA 10.0, and NVIDIA Driver 417.22, i9 CPU, and NVIDIA RTX T ITAN .
Revised: CUDA 10.0, and NVIDIA Driver 417.22, i9 CPU, and NVIDIA RTX T ITAN .

Original: 4. MNIST I MAGE C LASSIFICATION
Revised: 4. MNIST I MAGE C LASSIFICATION

Original: We omit the description about MNIST for its popularity.
Revised: We omit the description about MNIST.

Original: We use ODE-Net used in (Chen et al., 2018) to classify MNIST images.
Revised: We use ODE-Net used in (Chen et al., 2018) to classify MNIST images.

Original: Refer to Appendix for detailed descriptions on ODE-Net with its hyperparameter conﬁgurations.
Revised: Refer to Appendix for detailed descriptions on ODE-Net with its hyperparameter conﬁgurations.

Original: [Table]
Revised: [Table]

Original: Auxiliary Network Architecture.
Revised: Auxiliary Network Architecture.

Original: In addition to ODENet, we have one more auxiliary network v whose architecture is summarized in Table 2.
Revised: In addition to ODENet, we have one more auxiliary network v whose architecture is summarized in Table 2.

Original: We use the standard residual block (He et al., 2016) for this network and the proposed network consists of four layers.
Revised: We use the standard residual block (He et al., 2016) for this network and the proposed network consists of four layers.

Original: In comparison with the network f of ODE-Net, its architecture is relatively simpler and it takes 0. NFEs (i.e., 0.0006 seconds) to run once.
Revised: In comparison with the network f of ODE-Net, its architecture is relatively simpler and it takes 0.8 NFEs (i.e., 0.0006 seconds) to run once.

Original: Recall in Table 1 that the network (function) f is evaluated 10-20 times in average to solve integral problems.
Revised: Recall in Table 1 that the network (function) f is evaluated 5-20 times on average to solve integral problems.

Original: Therefore, running the auxiliary network v once to decide the best integrator can be a fruitful investment.
Revised: Therefore, running the auxiliary network v once to decide the best integrator can be a fruitful investment.

Original: Hyperparameters.
Revised: Hyperparameters.

Original: The regularization coefﬁcient λ is set to { 0 . 01 , 0 . 0001 , 0 . 0001 , 0 . 005 } and the starting learning rate is 0 .
Revised: The regularization coefﬁcient λ is set to { 0 . 01 , 0 . 0001 , 0 . 0001 , 0 . 005 } and the starting learning rate is 0 .

Original: 1 with a decay factor { 0 . 1 , 0 . 01 , 0 .
Revised: 1 with a decay factor { 0 . 1 , 0 . 01 , 0 .

Original: 001 } at { 60 , 100 , 140 } epochs.
Revised: 001 } at { 60 , 100 , 140 } epochs.

Original: The exponent α is 0 .
Revised: The exponent α is 0 .

Original: 3 and M is 1,000.
Revised: 3 and M is 1,000.

Original: We use DOPRI as our default solver unless DISE is adopted.
Revised: We use DOPRI as our default solver unless DISE is adopted.

Original: We use the recommended hyperparameters for the base Neural ODE model in the paper or in their github repository, as noted in Appendix.
Revised: We use the recommended hyperparameters of ODE-Net in the paper or in the respective github repository, as noted in Appendix.

Original: Experimental Results.
Revised: Experimental Results.

Original: The results are summarized in Table 1.
Revised: The results are summarized in Table 1.

Original: It needs 26 NFEs when we do not use any regularizer to train the Neural ODE-base classiﬁcation model.
Revised: It needs 26 NFEs when we do not use any regularizer to train ODE-Net.

Original: The kinetic energy-based regularizer improves both the accuracy and the NFE value.
Revised: The kinetic energy-based regularizer improves both the accuracy and the NFE value.

Original: However, its efﬁcacy is comparable to other standard regularizations.
Revised: However, its NFE is comparable to other standard regularizations.

Original: Our regularizer shows the best accuracy  in the table with an NFE of 14, which is much faster than other cases.
Revised: Our regularizer shows an accuracy close to that of the kinetic regularization in the table with an NFE of 14, which is much faster than other cases.

Original: [Table]
Revised: [Table]

Original: We also applied the dynamic integrator selection (DISE) for both of the no-regularizer and our regularizer conﬁgurations.
Revised: We also applied the dynamic integrator selection (DISE) for both of the no-regularizer and our regularizer conﬁgurations.

Original: It is note that the combination of our regularizer and the dynamic selection shows an extraordinary outcome in the table.
Revised: It is worth noting that the combination of our regularizer and the dynamic selection shows an extraordinary outcome in the table.

Original: Even after considering the overheads incurred by the auxiliary network, an NFE of 5.79 is very fast in comparison with other cases.
Revised: Even after considering the overhead incurred by the auxiliary network, an NFE of 5.79 is very fast in comparison with other cases.

Original: At the same time, its accuracy is strictly larger than all other cases except the one with our regularizer only.
Revised: Its accuracy is also larger than many other baselines.

Original: Table 3 shows a percentage of test cases where each integrator is selected by the auxiliary network with our regularizer.
Revised: Table 3 shows a percentage of test cases where each integrator is selected by the auxiliary network with our regularizer.

Original: DOPRI occupies the biggest portion for its higher accuracy than others, i.e., 37%.. RK and the Euler method have more numerical errors than DOPRI and as a result, their estimated costs are larger than that of DOPRI in general (due to the large penalty M ).
Revised: DOPRI occupies the biggest portion for its higher accuracy than others, i.e., 37%. RK4 and the Euler method have more numerical errors than DOPRI and as a result, their estimated costs are larger than that of DOPRI in general (due to the large penalty M ).

Original: We note that theirs rankings in terms of the selection percentage are the same as those in terms of the numerical error.
Revised: We note that their rankings in terms of the selection percentage are the same as those in terms of the numerical error.

Original: One interesting point is that the Euler method also occupies a non-trivial portion in the table.
Revised: One interesting point is that the Euler method also occupies a non-trivial portion in the table.

Original: 4. P HYSIO N ET M ORTALITY C LASSIFICATION
Revised: 4. P HYSIO N ET M ORTALITY C LASSIFICATION

Original: where ψ is Drop Out.

Original: Layer

Original: Design

Original: In Size

Original: Out Size σ ( FC ) 20 ×10 ×

Original: [Table]
Revised: [Table]

Original: σ ( FC ) 10 ×10 × σ ( FC ) 10 ×5 × ψ ( σ ( FC )) 5 ×

Original: [Table]

Original: 5 ×

Original: [Table]

Original: FC 5 ×

Original: [Table]

Original: 3 ×

Original: [Table]

Original: Dataset.
Revised: Dataset.

Original: We use the PhysioNet computing in cardiology challenge dataset released at 2012 (Silva et al., 2010).
Revised: We use the PhysioNet computing in cardiology challenge dataset released at 2012 (Silva et al., 2010).

Original: It is to forecast mortality rates in intensive care unit (ICU) populations.
Revised: It is to forecast mortality rates in intensive care unit (ICU) populations.

Original: The dataset had been collected from 12,000 ICU stays.
Revised: The dataset had been collected from 12,000 ICU stays.

Original: They remove short stays less than 48 hours and recorded up tovariables.
Revised: They remove short stays less than 48 hours and recorded up tovariables.

Original: Each records has a time stamp that indicates an elapsed time after admission to the ICU.
Revised: Each records has a time stamp that indicates an elapsed time after admission to the ICU.

Original: Given a record, we predict whether the patient will die or not.
Revised: Given a record, we predict whether the patient will die or not.

Original: Therefore, the task-speciﬁc loss L task in Eq.
Revised: Therefore, the task-speciﬁc loss L task in Eq.

Original: 3 is a cross-entropy loss.
Revised: 3 is a cross-entropy loss.

Original: They well separated the dataset for training, validating, and testing.
Revised: They well separated the dataset for training, validating, and testing.

Original: We use Latent-ODE (Rubanova et al., 2019) for this task.
Revised: We use Latent-ODE (Rubanova et al., 2019) for this task.

Original: The network (function) architecture of f in Latent-ODE is described in Appendix in conjunction with its hyperparameter conﬁgurations.
Revised: The network (function) architecture of f in Latent-ODE is described in Appendix in conjunction with its hyperparameter conﬁgurations

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Auxiliary Network Architecture.
Revised: Auxiliary Network Architecture.

Original: The auxiliary network for this dataset is shown in Table 4, which consists of four fully connected layers and an output layer.
Revised: The auxiliary network for this dataset is shown in Table 4, which consists of four fully connected layers and an output layer.

Original: There is a dropout at the fourth layer and we found that using a dropout at this position improves the selection quality in some cases.
Revised: There is a dropout at the fourth layer and we found that using a dropout at this position improves the selection quality in some cases.

Original: The time to run the auxiliary network once is comparable to 0.023 NFE (i.e., 0.0003 seconds) in our testing, which is negligible.
Revised: The time to run the auxiliary network once is comparable to 0.023 NFE (i.e., 0.0003 seconds) in our testing, which is negligible.

Original: Hyperparameters.
Revised: Hyperparameters.

Original: The coefﬁcient λ is set to { 0 . 045 , 0 . 0002 , 0 . 0025 , 0 . 015 } respectively, and the starting learning rate is 0 .
Revised: The coefﬁcient λ is set to { 0 . 045 , 0 . 0002 , 0 . 0025 , 0 . 015 } respectively, and the starting learning rate is 0 .

Original: 01 with a decay factor of0 . 999 .
Revised: 01 with a decay factor of

Original: The exponent α is 0 .
Revised: The exponent α is 0 .

Original: The large penalty M is 1,000.
Revised: The large penalty M is 1,000.

Original: The dropout ratio is { 0 , 0 . 25 } .
Revised: The dropout ratio is { 0 , 0 . 25 } .

Original: We use DOPRI as our default solver unless DISE is adopted.
Revised: We use DOPRI as our default solver unless DISE is adopted.

Original: We use the recommended hyperparameters for Latent-ODE in their paper or in their github repository.
Revised: We use the recommended hyperparameters for Latent-ODE in their paper or in their github repository.

Original: Experimental Results.
Revised: Experimental Results.

Original: Table 5 summarizes all the results in terms of the AUC score and NFE.
Revised: Table 5 summarizes all the results in terms of the AUC score and NFE.

Original: Adding an regularizer improves the AUC score and NFE at the same time in all cases.
Revised: Adding an regularizer improves the AUC score and NFE at the same time in all cases.

Original: It is noted that the best AUC score is achieved when we use the kinetic energy-based regularizer for this task among all regularizers.
Revised: It is noted that the best AUC score is achieved when we use the L 1 regularizer for this task among all regularizers.

Original: Our regularizer shows the smallest NFEs among all tested regularizers, i.e., an NFE of 74 with no regularizers vs. an NFE of 38 with our regularizer.
Revised: Our regularizer shows the smallest NFE among all tested regularizers, i.e., an NFE of 74 with no regularizers vs. an NFE of 39.71 with our regularizer.

Original: When being used with DISE, our regularizer marks an NFE of 34.5 and an AUC score of 0.8132, which are the best  in the table.
Revised: When being used with DISE, our regularizer marks an NFE of 34.1 and an AUC score of 0.7604, which is the best AUC to NFE ratio in the table.

Original: [Table]
Revised: [Table]

Original: We summarized the distribution of the integrator selection by DISE with our regularizer in Table 6.
Revised: We summarized the distribution of the integrator selection by DISE with our regularizer in Table 6.

Original: For a majority of cases, it chooses
Revised: For a majority of cases, it chooses

Original: DOPRI due to the difﬁcult nature of the task.
Revised: DOPRI due to the difﬁcult nature of the task.

Original: One interesting point is that RK4 had not been used at all because it does not show any notable difference from the Euler method in this task.
Revised: One interesting point is that RK4 had not been used at all because it does not show any notable difference from the Euler method in this task.

Original: In that regard, choosing the Euler method is a sensible decision.
Revised: In that regard, choosing the Euler method is a sensible decision.

Original: 4. C ONTINUOUS N ORMALIZING F LOWS
Revised: 4. C ONTINUOUS N ORMALIZING F LOWS

Original: Dataset.
Revised: Dataset.

Original: Normalizing ﬂows transform a simple distribution into a richer complex distribution, which can be used for many deep learning applications, such as generative models, reinforcement learning, and variational inference (Rezende & Mohamed, 2015).
Revised: Normalizing ﬂows transform a simple distribution into a richer complex distribution, which can be used for many deep learning applications, such as generative models, reinforcement learning, and variational inference (Rezende & Mohamed, 2015).

Original: It is also known that Neural ODEs can generalize the concept of normalizing ﬂows, which is called continuous normalizing ﬂows.
Revised: It is also known that Neural ODEs can generalize the concept of normalizing ﬂows, which is called continuous normalizing ﬂows.

Original: We use the experimental codes and data released in (Chen et al., 2018).[Table] FC )
Revised: We use the experimental codes and data released in (Chen et al., 2018).

Original: [Table]
Revised: [Table]

Original: Φ( FC )

Original: [Table]

Original: [Table]

Original: Φ( FC ) [Table] 32 × Φ( FC ) 32 ×3 × In this task, we perform a maximum likelihood training between q ( x ) and p ( x ) , where q is a distribution created by a ﬂow model and p is a target distribution.
Revised: In this task, we perform a maximum likelihood training between q ( x ) and p ( x ) , where q is a distribution created by a ﬂow model and p is a target distribution.

Original: Therefore, the task-speciﬁc loss is to maximize E p ( x ) [log q ( x )] , i.e.,  L task = − E p ( x ) [log q ( x )] .
Revised: Therefore, the task-speciﬁc loss is to maximize E p ( x ) [log q ( x )] , i.e., minimize L task = − E p ( x ) [log q ( x )] .

Original: The change of variable theorem is used to measure the probability q ( x ) .
Revised: The change of variable theorem is used to measure the probability q ( x ) .

Original: We set the initial distribution to a Gaussian distribution of
Revised: We set the initial distribution to a Gaussian distribution of

Original: N (0 , 1) and a ﬂow model tries to transform it to the target distribution p .
Revised: N (0 , 1) and a ﬂow model tries to transform it to the target distribution p .

Original: One good characteristic of continuous normalizing ﬂows is that we can reverse the model and extract the reverse-mapping procedure from q to the Gaussian as well.[Table] Our
Revised: One good characteristic of continuous normalizing ﬂows is that we can reverse the model and extract the reverse-mapping procedure from q to the Gaussian as well.

Original: [Table]
Revised: [Table]

Original: Auxiliary Network Architecture.
Revised: Auxiliary Network Architecture.

Original: The auxiliary network for this dataset is shown in Table 7, which consists of three fully connected layers and an output layer.
Revised: The auxiliary network for this dataset is shown in Table 7, which consists of three fully connected layers and an output layer.

Original: The time to run the auxiliary network once is comparable to 0.018 NFE (i.e., 0.000144 seconds) in our testing, which is negligible.
Revised: The time to run the auxiliary network once is comparable to 0.018 NFE (i.e., 0.000144 seconds) in our testing, which is negligible.

Original: Hyperparameters.
Revised: Hyperparameters.

Original: The coefﬁcient λ is set to { 0 . 0005 , 0 . 00001 , 0 . 0003 , 0 . 004 } and the learning rate is 0 . 001 .
Revised: The coefﬁcient λ is set to { 0 . 0005 , 0 . 00001 , 0 . 0003 , 0 . 004 } and the learning rate is 0 . 001 .

Original: The exponent α is 0.03.
Revised: The exponent α is 0.03.

Original: The large penalty M is 1,000.
Revised: The large penalty M is 1,000.

Original: We use DOPRI as our default solver unless DISE is adopted.
Revised: We use DOPRI as our default solver unless DISE is adopted.

Original: We use the recommended hyperparameters for the base continuous normalizing ﬂow model in the original paper or in their github repository.
Revised: We use the recommended hyperparameters for the base continuous normalizing ﬂow model in the original paper or in their github repository.

Original: The threshold β is set to the average negative log-probability loss of the case where we use our regularizer.
Revised: The threshold β is set to the average negative log-probability loss of the case where we use our regularizer.

Original: The rationale behind this threshold is that we select the Euler method only when its loss value is good enough.
Revised: The rationale behind this threshold is that we select the Euler method only when its loss value is good enough.

Original: Experimental Results.
Revised: Experimental Results.

Original: Table 8 shows the performance of various methods.
Revised: Table 8 shows the performance of various methods.

Original: With no regularizers, the base Neural ODE model’s NFEs is very large, i.e., an NFE of 2,318.
Revised: With no regularizers, the base Neural ODE model’s NFE is very large, i.e., an NFE of 2,297.

Original: Our regularizer decreases it to 2,090 and the combination of our regularizer and DISE further decreases to 1,976.
Revised: Our regularizer decreases it to 2,166 and the combination of our regularizer and DISE further decreases to 1,984.

Original: However, the
Revised: However, the

Original: L 2 regularizer rather increases the NFEs and has a negative inﬂuence on it.
Revised: L 2 regularizer rather increases the NFE and has a negative inﬂuence on it.

Original: In  all cases, addingan regularizer decreases the negative log-probability, denoted NLP in the table. At this experiment, NFE takes approximately 0.008 seconds. [Figure]
Revised: In almost all cases, adding

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: t

Original: [Figure]

Original: [Figure]

Original: (e) t = 1 .(f)

Original: Target (g

Original: [Figure]

Original: t

Original: [Figure]
Revised: [Figure]

Original: 0 .(h)

Original: t = 0 .(i) t = 0 .(j) t = 0 .(k) t = 1 .(l)
Revised: an regularizer decreases the negative log-probability, denoted NLP in the table.

Original: Target (m) t = 0 .(n)

Original: t = 0 .(o)

Original: t = 0 .(p) t = 0 .(q)

Original: t = 1 .(r)

Original: Target (s) t = 0 .(t) t = 0 .(u) t = 0 .(v) t = 0 .(w) t = 1 .(x)
Revised: At this experiment, 1 NFE takes approximately 0.008 seconds.

Original: Target Figure 3 shows various transformation processes from the initial Gaussian noise to the target distribution with two circles.
Revised: Figure 3 shows various transformation processes from the initial Gaussian noise to the target distribution with two circles.

Original: All shows reasonable transformations witha little differences.
Revised: All show reasonable transformations with little differences.

Original: Integrator

Original: Percentage

Original: [Table]

Original: [Table]

Original: Euler 12.

Original: [Table]
Revised: [Table]

Original: The selection distribution by the auxiliary network with our regularizer is summarized in Table 9.
Revised: The selection distribution by the auxiliary network with our regularizer is summarized in Table 9.

Original: Being similar to that in PhysioNet,
Revised: Being similar to that in PhysioNet,

Original: DOPRI is selected for many cases, i.e., 87.5%, and the Euler method is used from time to time, i.e., 12.5%.
Revised: DOPRI is selected for many cases, i.e., 87.5%, and the Euler method is used from time to time, i.e., 12.5%.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We tackled one critical problem of Neural ODEs, a delayed process of the forward-pass inference.
Revised: We tackled one critical problem of Neural ODEs, a delayed process of the forward-pass inference.

Revised: Even though DOPRI is able to dynamically adjust its step-sizes, as described earlier, there exists a limitation in saving the forward-pass inference time.

Original: To address the problem, we suggested i) regularizing the DOPRI’s estimated error, which results in  reduced NFEs, and ii) dynamically selecting an appropriate integrator for each input.
Revised: To address the problem, we suggested i) regularizing the DOPRI’s estimated error, which results in a reduced NFE, and ii) dynamically selecting an appropriate integrator for each input.

Original: After introducing a general design, we introduced detailed designs of the auxiliary selection neural network for each dataset and showed that both the model accuracy and the forward-pass inference time can be improved at the same time in all tasks.
Revised: We successfully showed that both the model accuracy and the forward-pass inference time can be improved at the same time in all tasks.

Original: We also showed that a non-trivial percentage of test cases can be solved by the Euler method after adopting our regularizer, i.e., 29% in
Revised: We also showed that nontrivial percentages of test cases can be solved by the Euler method after adopting our regularizer.

Original: MNIST, 12.5% in PhysioNet, and 12.5% in continuous normalizing ﬂows. In particular, for MNIST our model shows more than four times smaller NFEs (i.e., more that four times faster forward-pass inference) in comparison with ODE-Net without any regularizer. 
Revised: In particular, for MNIST our model shows more than four times smaller NFE (i.e., more than four times faster forward-pass inference) in comparison with ODE-Net without any regularizer. One difﬁculty in our work is controlling hyperparameters such as M and α .

Revised: If they are ill-conﬁgured, the auxiliary selection network may choose only one integrator by pursuing either computational correctness (i.e., DOPRI) or computational lightweightness (i.e., the Euler method).

Revised: During our preliminary experiments, we tuned them using training data.