C HARACTERISTIC N EURAL O RDINARY D IFFERENTIAL E QUATIONS
A BSTRACT
We propose Characteristic-Neural Ordinary Differential Equations (C-NODEs), a framework for extending Neural Ordinary Differential Equations (NODEs) beyond
ODEs.
While NODE models the evolution of latent variables as the solution to an
ODE C-NODE models the evolution of the latent variables as the solution of a family of first-order partial differential equations (PDEs) along curves on which the PDEs reduce to ODEs, referred to as characteristic curves.
This reduction along characteristic curves allows for analyzing PDEs through standard techniques used for ODEs, in particular the adjoint sensitivity method.
We also derive CNODE-based continuous normalizing flows which describe the density evolution of latent variables along multiple dimensions.
Empirical results demonstrate the improvements provided by the proposed method for irregularly sampled time series prediction on MuJoCo, PhysioNet and Human Activity datasets and classification and density estimation on CIFAR-10, SVHN and MNIST datasets given a similar computational budget as the existing NODE methods. The results also provide empirical evidence that the learned curves improve the system efficiency using a ∗
Equal contribution † Corresponding Author. lower number of parameters and function evaluations compared with those of the baselines.
I NTRODUCTION
[Figure]
and only represent ( x along , for example, ( x
Deep learning and differential equations share many connections, and techniques in the intersection have led to insights in both fields.
One predominant connection is based on certain neural network architectures resembling numerical integration schemes, leading to the development of Neural Ordinary Differential Equations (NODEs) (Chen et al., 2019b).
NODEs use a neural network parameterization of an ODE to learn a mapping from observed variables to a latent variable that is the solution to the learned ODE.
A central benefit of NODEs is the constant memory cost, when backward passes are computed using the adjoint sensitivity method rather than backpropagating through individual forward solver steps.
Backpropagating through adaptive differential equation solvers to train
NODEs will often result in extensive memory use, as mentioned in Chen et al.
(2019b).
Moreover, NODEs provide a flexible probability density representation often referred to as continuous normalizing flows (CNFs).
However, since
NODEs can only represent solutions to ODEs, the class of functions is somewhat limited and may not apply to more general problems that do not have smooth and one-to-one mappings.
To address this limitation, a series of analyses based on methods from differential equations have been employed to enhance the representation capabilities of NODEs, such as the theory of controlled differential equations (Kidger et al., 2020), learning higher-order ODEs (Massaroli et al., 2021), augmenting dynamics (Dupont et al., 2019), and considering dynamics with delay terms (Zhu et al., 2021).
Additionally, certain works consider generalizing the ODE case to partial differential equations (PDEs), such as in Ruthotto & Haber (2020) and Sun et al.
These PDE-based methods do not use the adjoint method, removing the primary advantage of constant memory cost.
This leads us to the central question motivating the work: can we combine the benefits of the rich function class of PDEs with the efficiency of the adjoint method?
To do so, we propose a method of continuous-depth neural networks that solves a PDE over parametric curves that reduce the PDE to an ODE.
Such curves are known as characteristics , and they define the solution of the PDE in terms of an ODE (Griffiths et al., 2015).
The proposed Characteristic Neural Ordinary Differential
Equations (C-NODE) learn both the characteristics and the ODE along the characteristics to solve the
PDE over the data space.
This allows for a richer class of models while still incorporating the same memory efficiency of the adjoint method.
The proposed C-NODE is also an extension of existing methods, as it improves the empirical accuracy of these methods in classification tasks, time series prediction tasks, and image quality in generation tasks.
R ELATED W ORK
NODE is often motivated as a continuous form of a Residual Network (ResNet)
(He et al., 2015), since the ResNet can be interpreted as a forward Euler integration scheme on the latent state (Weinan, 2017).
Specifically, a ResNet is composed of multiple blocks with each block can be represented as: u t
+1 =
u t + f ( u t , θ ) , where u t is the evolving hidden state at time t and f ( u t , θ ) is interpreted as the gradient at time t , namely du/dt ( u t ) .
Generalizing the model to a step size given by ∆ t results in u t +∆ t = u t + f ( u t , θ )∆ t. To adapt this model to a continuous setting, we let ∆ t → 0 and obtain: lim ∆ t → 0 ( u t +∆ t − u t ) / ∆ t = du ( t ) /dt.
The model can then be evaluated through existing numerical integration techniques, as proposed by Chen et al.
(2019b):
[Equation]
Numerical integration can then be treated as a black box, using numerical schemes beyond the forward Euler to achieve higher numerical precision.
However, since black box integrators can take an arbitrary number of intermediate steps, backpropagating through individual steps would require too much memory since the individual steps must be saved.
Chen et al.
(2019b) addressed this problem by using adjoint backpropagation, which has a constant memory usage.
For a given loss function on the terminal state t = 1 of the hidden state L ( u ( t 1 )) , the adjoint a ( t ) is governed by another ODE:
[Equation]
that dictates the gradient with respect to the parameters.
The loss L ( u ( t 1 )) can then be calculated by solving another ODE (the adjoint) rather than backpropagating through the calculations involved in the numerical integration.
However, assuming that the hidden state is governed by an ODE imposes a limitation on the expressiveness of the mapping.
For example, Dupont et al.
(2019) describes a notable limitation of
NODEs is in the inability to represent dynamical systems with intersecting trajectories.
In response to such limitations, many works have tried to increase the expressiveness of the mapping.
Dupont et al.
(2019) proposed to solve the intersection trajectories problem by augmenting the vector space, lifting the points into additional dimensions; Zhu et al.
(2021) included time delay in the equation to represent dynamical systems of greater complexity; Massaroli et al.
(2021) proposed to condition the vector field on the inputs, allowing the integration limits to be conditioned on the input; Massaroli et al.
(2021) and Norcliffe et al.
(2020) additionally proposed and proved a second-order ODE system can efficiently solve the intersecting trajectories problem.
We note however that the interpretation of NODE as a continuous form of ResNet is also problematic, owing to the fact that the empirical behavior of the ResNet does not match the theoretical properties (Krishnapriyan et al., 2022; Ott et al., 2021).
As such, alternative interpretations of the process represented by ODE have been considered.
In Zhang et al.
(2019), the authors considered an augmentation where the augmented state corresponds to the parameters of the network governing the latent state.
Queiruga et al.
(2021) describes the latent evolution through a series of basis functions thereby allowing important concepts such as BatchNorm to be effectively translated in the continuous setting, achieving state-of-the-art performance on a variety of image classification tasks.
Further improvements to performance have been made by considering different numerical integrators (Matsubara et al., 2021; Zhuang et al., 2020; 2021).
On a related front, multiple works have attempted to expand NODE to other types of differential equation beyond ODEs.
Sun et al.
(2019) employed a dictionary method and expanded NODEs to a
PDE case, achieving high accuracies both in approximating PDEs and in classifying real-world image datasets.
However, Sun et al.
(2019) suggested that the method is unstable when training with the adjoint method and therefore is unable to make use of the benefits that come with training with adjoint.
Zhang et al.
(2018) proposed a density transform approach based on the Monge-Ampere PDE, but did not consider using adjoint-based training.
Multiple works have expanded to the stochastic differential equations setting and developed efficient optimization methods for them including (Güler et al., 2019;
Jia & Benson, 2019; Kidger et al., 2021a;b; Li et al., 2020; Liu et al., 2019; Xu et al., 2022).
Kidger et al.
; Morrill et al.
(2021a;b) used ideas from rough path theory and controlled differential equations to propose a NODE architecture as a continuous recurrent neural network framework.
Salvi et al.
(2022) considered stochastic PDEs for spatio-temporal dynamics prediction.
Additionally, Chen et al.
(2020) models spatio-temporal point-processes using NODEs, and Rubanova et al.
De Brouwer et al.
(2019) makes predictions on time series data using NODEs.
M ETHOD
We describe the proposed C-NODE method in this section by first providing a brief introduction to the method of characteristics (MoC) for solving PDEs with an illustrative example.
We then discuss the types of PDEs we can describe using this method.
We finally discuss how we apply the MoC to our C-NODE framework.
3. M ETHOD OF C HARACTERISTICS
The MoC provides a procedure for transforming certain PDEs into ODEs along paths known as characteristics .
In the most general sense, the method applies to general hyperbolic differential equations.
We will introduce MoC using a canonical example involving the inviscid Burgers equation, and defer to Griffiths et al.
(2015, Chapter 9) for a more complete introduction to the topic.
Let u ( x, t ) : R × R + → R satisfy the following inviscid Burgers equation
[Equation]
where we drop the dependence on x and t for ease of notation.
We are interested in the solution of u over some bounded domain Ω ⊂ R × R + .
Consider parametric forms for the spatial component x ( s ) :
[0 , T ] → R and temporal components t ( s ) :
[0 , T ] → R + over the fictitious variable s ∈ [0 , T ] .
Intuitively, this allows us to solve an equation on curves x, t that are parameterized by a variable s which we denote ( x ( s ) , t ( s )) as the characteristic .
Expanding and writing d as the total derivative, we get
[Equation]
Recall the original PDE in equation 1 and substituting the proper terms into equation 2 for dx/ds = u, dt/ds = 1 , d u/ d s = 0 , we then recover equation 1.
Note that we now have a system of
ODEs which we can solve to obtain the characteristics as x ( s )
= us + x 0 and t ( s )
=
s + t 0 , which are functions of initial conditions x 0 , t 0 .
Finally, by solving over a grid of initial conditions { x ( i ) 0 } ∞ i =1 ∈ ∂ Ω , we can obtain the solution of the PDE over Ω .
Putting it all together, we have a new
ODE that is written as
[Equation]
, where we can integrate over s through
[Equation]
using the adjoint method with boundary conditions x 0 , t 0 .
This contrasts the usual direct integration over the variable t that is done in NODE; we now jointly couple the integration through the characteristics.
An example of solving this equation over multiple initial conditions is given in Figure 1 with the contrast to standard NODE integration.
Hyperbolic PDEs in Machine Learning To motivate using MoC for machine learning problems such as classification or density estimation, we again note that MoC most generally applies to hyperbolic PDEs.
These PDEs roughly describe the propagation of physical quantities through time.
Such equations may be appropriate for deep learning tasks due to their ability to transport data into different regions of the state space.
For instance, in a classification task, we consider the problem of transporting high-dimensional data points that are not linearly separable to spaces where they are linearly separable.
Similarly, in generative modeling, we transport a base distribution to data distribution.
3. N EURAL R EPRESENTATION OF C HARACTERISTICS
In the proposed method, we learn the components involved in the MoC, namely the characteristics and the function coefficients.
We now generalize the example given in 3.1, which involved two variables, to a k -dimensional system.
Specifically, consider the following nonhomogeneous boundary value problem (BVP):
[Equation]
Here, u : R k × R → R n is a multivariate map, a i : R k + n → R and c : R k + n → R n are functions dependent on values of u and x ’s. This problem is well-defined and has a solution as long as (cid:80)
ki
=1 a i ∂ u ∂x i is continuous (Evans, 2010).
MoC is generally used in a scalar context, but the correspondence to the vector case is relatively straightforward.
A proof of this can be found in Appendix B.1.
To begin, we decompose the PDE in equation 3 into the following system of ODEs
[Equation]
We represent this ODE system by parameterizing dx i /ds and
∂
u
/∂x
i with neural networks.
Consequently, d u /ds is evolving according to equation 6.
Following this expansion, we arrive at
[Equation]
where we remove u ’s dependency on x ( s ) and x ’s dependency on s for simplicity of notation.
In equation 7, the functions J x u and d x /ds are represented as neural networks with inputs x , u and parameters Θ 2 .
Conditioning on data Previous works primarily modeled the task of classifying a set of data points with a fixed differential equation, neglecting possible structural variations lying in the data.
Here, we condition C-NODE on each data point, resulting in solving a PDE with a different initial condition and the hyperbolic PDE interpretation of the latent variables.
Consider the term given by the integrand in equation 7.
The neural network representing the characteristic d x /ds is conditioned on the input data z ∈ R w .
Define a mapping g ( · ) : R w → R n
and we have
[Equation]
By introducing g ( z ) in equation 8, the equation describing the characteristics changes depending on the current data point.
This leads to the classification task being modeled with a family rather than one single differential equation and allows the C-NODE system to model dynamical systems with intersecting trajectories.
This property becomes helpful in Proposition 4.1 in proving that C-NODE can represent intersecting dynamics.
3. T RAINING C-NODE S
Having introduced the main components of C-NODEs, we can now integrate them into a unified algorithm.
To motivate this section, and to be consistent with part of the empirical evaluation, we will consider classification tasks with data { ( z j , y j ) } Nj =1 , z j ∈ R w , y j ∈ Z + .
For instance, z j may be an image, and y j is its class label.
In the approach we pursue here, the image z j is first passed through a feature extractor function g ( · ; Θ 1 ) : R w → R n with parameters Θ 1 .
The output of g is the feature u ( j )
=
g ( z j ; Θ 1 ) that provides the boundary condition for the PDE on u ( j ) .
We integrate along different characteristic curves indexed by s ∈ [0 , T ] with boundary condition u ( j ) ( x (0) , 0)
=
u ( j ) 0 , and compute the terminal values as given by equation 7, where we mentioned in Section 3.2,
[Equation]
[Equation] Finally, u ( j ) ( x ( T ) , T ) is passed through another neural network, Φ( u ( j ) ( x ( T )); Θ 3 ) with input u ( j ) ( x ( T ) , T ) and parameters Θ 3 whose output are the probabilities of each class labels for image z j .
The entire learning process is now reduced to finding optimal weights (Θ 1 , Θ 2 , Θ 3 ) which can be achieved by minimizing the loss
[Equation]
where L ( · ) is the corresponding loss function (e.g. cross entropy in classification).
In Algorithm 2, we illustrate the implementation procedure with the forward Euler method for simplicity for the framework but note any ODE solver can be used.
Combining MoC with Existing NODE Modifications As mentioned in Section 2, the proposed
C-NODEs method can be used as an extension to existing NODE frameworks.
In all NODE modifications, the underlying expression of (cid:82) b a f ( t, u ; Θ)d t remains the same.
Modifying this expression to (cid:82) b a J x u ( x , u ; Θ) d x /ds ( x , u ; u 0 ; Θ)d s results in the proposed C-NODE architecture, with the size of x being a hyperparameter.
P ROPERTIES OF C-NODE S
C-NODE has a number of theoretical properties that contribute to its expressiveness.
We provide some theoretical results on these properties in the proceeding sections.
We also define continuous normalizing flows (CNFs) with C-NODEs, extending the CNFs originally defined with NODEs.
Intersecting trajectories As mentioned in Dupont et al.
(2019), one limitation of NODE is that the mappings cannot represent intersecting dynamics.
We prove by construction that conditioning on initial conditions allows C-NODEs to represent some dynamical systems with intersecting trajectories in the following proposition:
Proposition 4.1.
The C-NODE can represent a dynamical system on u ( s ) , du/ds = G ( s, u ) :
R + × R → R , where when u (0) = 1 , then u (1) = u (0)
+ (cid:82) 1 0 G ( s, u ) ds = 0 ; and when u (0) = 0 , then u (1) =
u (0)
+ (cid:82) 1 0 G ( s, u ) ds = 1 .
Proof.
See Appendix B.2.
Density estimation with C-NODEs C-NODEs can also be used to define a continuous density flow that models the density of a variable over space subject to the variable satisfying a PDE, extending the continuous normalizing flows defined with NODEs.
For NODEs, if u ( t ) ∈ R n follows the ODE du ( t ) /dt
=
f ( u ( t )) ,
where f ( u ( t )) ∈ R n , then its log likelihood from Chen et al.
(2019b, Appendix
A) is given by:
[Equation]
Similar to the change of log probability of NODEs, as in equation 10, we provide the following proposition for C-NODEs:
Proposition 4.2.
Let u ( s ) be a finite continuous random variable with probability density function p ( u ( s )) and let u ( s ) satisfy du ( s ) ds = (cid:80) ki
=1 ∂u∂x i dx i ds .
Assuming ∂u∂x
i and dx i ds are uniformly Lipschitz continuous in u and continuous in s , then the evolution of the log probability of u follows:
[Equation]
Proof.
See Appendix B.3.
CNFs are continuous and invertible one-to-one mappings onto themselves, i.e., homeomorphisms.
Zhang et al.
(2020) proved that vanilla NODEs are not universal estimators of homeomorphisms, but augmented neural ODEs (ANODEs) are universal estimators of homeomorphisms.
We demonstrate that C-NODEs are pointwise estimators of homeomorphisms, which we formalize in the following proposition:
Proposition 4.3.
Given any homeomorphism h :
Υ → Υ , Υ ⊂ R p , initial condition u 0 , and time
T > 0 , there exists a flow u ( s, u 0 )
∈ R n following duds = ∂u∂x dxds
+ ∂u∂t dtds such that u ( T, u 0 )
=
h ( u 0 ) .
Proof.
See Appendix B.4.
E XPERIMENTS
We present experiments on image classification tasks, time series prediction tasks, image generation tasks on benchmark datasets, and a synthetic PDE regression task.
5. C LASSIFICATION E XPERIMENTS WITH I MAGE D ATASETS
We first conduct experiments for classification tasks on high-dimensional image datasets, including
MNIST, CIFAR-10, and SVHN.
We provide results for C-NODE and also combine the framework with existing methods, including ANODEs (Dupont et al., 2019), Input Layer NODEs (ILNODEs) (Massaroli et al., 2021), and 2nd-Order NODEs (Massaroli et al., 2021).
For all classification experiments, we set the encoder of input images for conditioning to be identity, i.e., g ( z )
=
z , making the input into C-NODE the original image.
This way we focus exclusively on the performance of
C-NODE.
The results for the experiments using the adjoint method are reported in Table 1.
We investigate the performances of the models on classification accuracy and the number of function evaluations (NFE) taken in the adaptive numerical integration.
NFE is an indicator of the model’s computational complexity and can be interpreted as the network depth for the continuous NODE (Chen et al., 2019b).
Using a similar number of parameters, combining C-NODEs with different models consistently results in higher accuracy and mostly uses a smaller numbers of NFEs, indicating a better parameter efficiency.
An ablation study on C-NODEs’ and NODEs’ parameters can be found in Appendix D.2.
While the average number of function evaluations tends to be lower for C-NODE, we additionally note that, compared to ANODE, training C-NODE with the adjoint method can sometimes have decreased stability.
We define instability as having a NFE > 1000 .
To get a rough idea of the differences in stability, when training NODE, C-NODE, and ANODE for image classification on the SVHN dataset for forty instances, NODE appeared unstable six times, C-NODE was unstable three times, and
ANODE was never unstable.
We note that this was only apparent in the SVHN experiment and when considering C-NODE by itself; the average NFE decreases when adding C-NODE to A-NODE and this was never experienced in the ANODE+C-NODE experiments.
5. T IME SERIES PREDICTION WITH C-NODE S
We test C-NODEs, NODEs, and augmented versions of C-NODEs and NODEs on the time series prediction problem using 100 sequences of each of the following datasets: the PhysioNet dataset, containing measurements from the first 48 hours of patients’ admissions to ICU (Goldberger et al., 2000 (June 13); the Hopper dataset, containing physical simulation results generated with the Hopper model from the Deepmind Control Suite (Rubanova et al., 2019; Tassa et al., 2018); the Human
Activity dataset in the UCI dataset, containing time series data on twelve features from five individuals walking, sitting, and lying (Dua & Graff, 2017); and the synthetic ODE dataset, generated with a first-order ODE whose initial condition follows a Gaussian distribution, and Gaussian noises are added to the observations (Rubanova et al., 2019).
The PhysioNet and Human Activity datasets both contain stochastic processes while the Hopper dataset and the ODE dataset are deterministic.
Since
NODE, ANODE, and C-NODE model deterministic processes, the Hopper dataset and the ODE dataset readily fit the modeling frameworks.
On the other hand, the PhysioNet and Human Activity datasets require augmenting the dynamics of the NODE models with stochasticity to model the arrival of events. [Table]
We follow the experimental setup for interpolation tasks in Rubanova et al.
(2019), where we define an autoregressive model with the encoder being an ODE-RNN model and the decoder being a latent differential equation 2 .
The main purpose of this experiment is to compare the ODE-RNN when using C-NODE versus NODE.
ODE-RNN is a standard method for including ODE modeling in time series tasks as described in Chen et al.
(2019b) and Rubanova et al.
We consider interpolation tasks by first encoding the time series { x i , t i } Ni =0 of length N and computing the approximate posterior q ( z 0 |{ x i , t i } N i =0 ) =
N ( µ z 0 , σ z 0 ) as done in Rubanova et al.
Then, µ ( z 0 ) , , σ ( z 0 ) are computed as ϱ ( ODE-RNN ϕ
( { x i , t i } Ni =0 )) , where ϱ is a function that encodes the terminal hidden states into mean and variance of the latent variable z 0 .
ODE-RNN ( · ) is a model whose states obey an ODE between observations and are updated according to new observations as described in Rubanova et al.
To predict the state at an observation time t i , we sample initial states z 0 from the posterior which are then decoded using another neural network.
We finally average generated observations at each observation time to compute the test data errors.
The results are presented in Table 2 and suggest that C-NODE based models use slightly fewer parameters while achieving lower error rates than NODE models.
C-NODE models the latent dynamics as a first order
PDE, which is a natural extension of the ODE model that NODE uses.
5. C ONTINUOUS NORMALIZING FLOW WITH C-NODE S
We compare the performance of CNFs defined with NODEs to flows defined with C-NODEs on
MNIST and CIFAR-10.
We use the Hutchinson trace estimator to calculate the trace and use multi- [Table] [Table] scale convolutional architectures to model the density transformation as done in (Dinh et al.,
Grathwohl et al., 2019) 1 .
Differential equations are solved using the Runge-Kutta method of orderof the Dormand-Prince-Shampine solver and trained with the adjoint method.
Although the Euler forward method is faster, experimental results show that its fixed step size often leads to negative
Bits/Dim, indicating the importance of adaptive solvers.
As shown in table 3 and figure 7, using a similar number of parameters, experimental results show that CNFs defined with C-NODEs perform better than CNFs defined with NODEs in terms of Bits/Dim, as well as having lower variance
[Table]
and using a lower NFE on both MNIST and CIFAR-10.
5. PDE MODELING WITH C-NODE S
We consider a regression example for a hyperbolic PDE with a known analytical solution.
Since
NODEs assume that the latent state is only dependent on a scalar (namely time), they cannot model dependencies that vary over multiple spatial variables required by most PDEs.
We modify the assumptions used in the classification and density estimation experiments where the boundary conditions were constant as in equation 3.
We approximate the following BVP:
[Equation]
which has an analytical solution given by u ( x, t )
= 2 x exp( t ) 2exp( t )
+1 .
We generate a training dataset by randomly sampling 200 points ( x, t ) , x ∈ [1 , 2] , t ∈ [0 , 1] , as well as values u ( x, t ) at those points.
We test C-NODE and NODE on 200 points randomly sampled as ( x, t ) ∈ [1 , 2] × [0 , 1] .
For this experiment, C-NODE uses 809 parameters while NODE uses 1185 parameters.
We quantify the differences in the representation capabilities by examining how well each method can represent the
PDE.
C-NODE deviates 8.05% from the test set, while NODE deviates 30.52%.
Further experimental details can be found in Appendix A.4.1.
D ISCUSSION
We describe an approach for extending NODEs to the case of PDEs by solving a series of ODEs along the characteristics of a PDE.
The approach applies to any black-box ODE solver and can combine with existing NODE-based frameworks.
We empirically showcase its efficacy on classification tasks while demonstrating its success in improving convergence using Euler forward method without the adjoint method.
C-NODEs also consistently achieve lower testing MSEs over different time series prediction datasets, while having lower standard errors.
Additionally, C-NODEs empirically achieve better performances on density estimation tasks, while being more efficient with the number of parameters and using lower NFEs.
C-NODE’s efficiency over physical modeling is also highlighted with additional experiments.
Limitations There are several limitations to the proposed method.
The MoC only applies to hyperbolic PDEs, and we only consider first-order semi-linear PDEs in this paper.
This may be a limitation since this is a specific class of PDEs that does not model all data.
We also did not enforce any particular structure to prevent characteristics from intersecting, which may result in shock waves and rarefactions.
However, we believe that this is unlikely to happen due to the high dimensionality of the ambient space.
As noted in the experiments section, C-NODE can have decreased stability compared to ANODE when defined by exceeding an extreme threshold of number of function evaluations. A CKNOWLEDGMENTS
This work was supported in part by the Office of Naval Research (ONR) under grant number
N00014-21-1-2590.
AH was supported by NSF-GRFP.
A E XPERIMENTAL D ETAILS
Implementation details of this paper can be found at https://github.com/XingziXu/ NeuralPDE.git . A. E XPERIMENTAL DETAILS OF CLASSIFICATION TASKS
We report the average performance over five independent training processes, and the models are trained for 100 epochs for all three datasets.
We also report the training dynamics of C-NODE and
NODE using the adjoint sensitivity method and the euler backpropagation, as shown in Figure 2.
As shown in Figure 2, using the Euler solver, it appears that C-NODEs converge faster than the vanilla NODEs (usually in one epoch) while generally having a more stable training process with smaller variance.
Additionally, on experiments with MNIST, C-NODEs converge in only one epoch, while NODEs converge in roughly 15 epochs.
This provides additional empirical evidence on the benefits of training using the characteristics.
The input for 2nd-Ord, NODE, and C-NODE are the original images.
In the IL-NODE, we transform the input to a latent space before the integration by the integral; that is, we raise the R c × h × w dimensional input image into the R ( c + p ) × h × w dimensional latent feature space 3 .
For SVHN and CIFAR-10, we assume x ∈
R 3 , i.e., the Jacobian J x u = ( ∂ u /∂x 1 , ∂ u /∂x 2 , ∂ u /∂x 3 ) .
We model each partial derivative ∂
u /∂x i with a separate convolutional network.
The network architecture for the network modeling the partial derivatives is as shown in Table 6.
The network architecture for the network modeling d x /ds is as shown in Table 6.
The architecture in Tables 5, 6 are used for both CIFAR-10 and SVHN.
Note that the network architecture for MNIST differs slightly due to the lower dimensionality of MNIST.
The hyperparameters used are as shown in Table 4.
[Figure] We decode the result after performing the continuous transformations along characteristics curves, back to the R c × h × w dimensional object space.
Combining this with the C-NODE can be seen as solving a PDE on the latest features of the images rather than on the images directly.
Unlike ODEs, we take derivatives with respect to different variables in PDEs.
For a PDE with k variables, this results in the constraint of the balance equations
[Equation]
This can be satisfied by defining the k -th derivative with a neural network, and integrate k − 1 times to get the first order derivatives.
Another way of satisfying the balance equation is to drop the dependency on the variables, i.e., ∀ i ∈ { 1 , 2 , ..., k } ,
[Equation]
When we drop the dependency, all higher order derivatives are zero, and the balance equations are satisfied.
All experiments were performed on NVIDIA RTX 3090 GPUs on a cloud cluster.
[Table]
A. E XPERIMENTAL RESULTS AND DETAILS OF TIME SERIES PREDICTIONS A.2. E XPERIMENTAL DETAILS OF TIME SERIES PREDICTIONS ON REAL - WORLD DATASETS
We provide a more detailed explanation of the time-series experiments which are based on the
ODE-RNN framework described in Rubanova et al.
Our experiments test the effectiveness of
NODE, C-NODE, ANODE, and their combinations under the ODE-RNN framework by computing [Table] [Table] evaluation metrics under the different ODE models.
The ODE-RNN framework involves combining the strengths of both a neural ODE and RNN to initially embed the time series history into a latent distribution parameterized by an RNN and then decode the predicted latent distribution into the original data space.
The main components of the method are illustrated in Algorithm 1.
Forecasts are computed by integrating the latent space in time according to the neural ODE model with initial condition z 0 distributed according to the parameterization by the RNN.
We define the approximate posterior of z 0 as q ( z 0 |{ x i , t i } Ni =0 )
=
N ( µ z 0 , σ z 0 ) , where µ z 0 , σ z 0 = ϱ ( ODE-RNN ϕ ( { x i , t i } Ni =0 )) .
To bypass the RNN requirement of a fixed observation rate, the ODE-RNN model uses states that obey an ODE in between observations and are updated at new observations.
Given a set of time series data { x i , t i } Ni =0 , we embed the time series using the ODE-RNN.
We then pass the output of the
ODE-RNN model through a function ϱ ( · ) to get the mean µ z 0 and the variance σ z 0 of the posterior of z 0 .
To predict the value at timestamp T , we sample K initial conditions z 0 from the posterior q ( z 0 |{ x i , t i } Ni =0 ) , integrate the latent ODE model until timestamp T and use the average of the K integrations as the result.
For all experiments, we follow the experimental setup as described in https://github.com/
YuliaRubanova/latent_ode .
We experiment with the ODE-RNN framework using NODE,
C-NODE, ANODE, and their combinations as the ODE backbone of both the ODE-RNN model and the latent ODE.
Training NODE follows the original setup, with the dimension of the ODE model in the ODE-RNN being 20, the number of units per layer in each of GRU update networks being 100, the number of units per layer in ODE function being around 100, the number of layers in ODE function in generative and the recognition ODE both being 1.
We use a C-NODE with a dimensionality of 8.
The network architecture details are given in Tables 7, 8.
The training hyperparameters are given in Table 9.
The number of units per layer in the network describing d x /d s is 12.
For the network describing ∂ u /∂x i , the dimension of the ODE model in the
ODE-RNN is 20, the number of units per layer in each of GRU update networks is 100, the number of units per layer in the ODE function is tuned to match the number of parameters in the NODE models, the number of layers in ODE function in generative and the recognition ODE is 1.
Input: Data points and their timestamps { ( x i , t i ) } i =1 ,...,N h
=for i in 1 , 2 , ..., N do h ′ i =
ODESolve ( f θ , h
i − 1 ,
( t i − 1 , t i ))
h
i =
RNNCell ( h ′ i , x i ) end for o i =
OutputNN ( h i ) for all i = 1 , ..., N
Return: { o i } i =1 ,...,N ; h N
[Figure]
Augmentation
[Figure] [Figure]
Tanh
N/A N/A
Linear Layer
Tanh
N/A N/A
[Figure] [Figure]
[Figure] [Figure]
[Figure]
[Table]
[Figure]
[Table] Layer
Tanh
N/A N/A
[Table]
[Table]
[Table]
[Table] [Table]
[Table]
We test C-NODEs, ANODEs, and NODEs on a synthetic time series prediction problem. We define a function by u ( x, t ) = 2 x exp( t ) 2exp( t )+1 , and we sample ˜ u
= u ( x, t ) + 0 .
1 ϵ t , where ϵ t ∼ N (0 , 1) over x ∈ [1 , 2] , t ∈ [0 , 1] to generate the training dataset.
We test the performance on t ∈ [ n, n + 1] with n ∈ { 0 , 1 , . . .
[Table]
We also test C-NODEs, NODEs, and ANODEs on time series prediction with different levels of noise.
Specifically, using the same function as above, we form training and testing dataset with ϵ t ∼ N (0 , m ) , m ∈ { 0 , 1 , . . .
We test the performance on the time period t ∈ [0 , 1] .
[Table]
A.2. E XPERIMENTAL DETAILS OF TIME SERIES PREDICTIONS ON SYNTHETIC DATASET
We consider the task of predicting u ( x, t ) = 2 · x · e t 2 · e t +1 at different times t , over x ∈ [1 , 2] .
We specify the initial condition of u (1 , 0) .
We use a 8 dimensional C-NODE network.
The result is calculated with
[Equation]
NODE is calculated with
[Equation]
The experiment results are given in table 11.
In our experiments, C-NODEs use 1221 parameters, ANODEs use 1270 parameters, NODEs use 1290 parameters.
All experiments were performed on NVIDIA RTX 3080 ti GPUs on a local machine.
A. E XPERIMENTAL DETAILS OF CONTINUOUS NORMALIZING FLOWS
We report the average performance over four independent training processes.
As shown in Figure 8, compared to NODE, using a C-NODE structure improves the stability of training, as well as having better performance.
Specifically, the standard errors for C-NODEs on MNIST, SVHN, and CIFARare 0.37%, 0.51%, and 0.24% respectively, and for NODEs the standard errors on MNIST, SVHN, and CIFAR-10 are 1.07%, 0.32%, and 0.22% respectively.
The training dynamics of CNFs on MNIST 50 Epoch 1. B i t s / d i m C-NODE NODE
Epoch
[Figure] C-NODE NODE Epoch
[Figure] Epoch [Figure] The network structures are provided in Tables 12, 13.
The training hyperparameters are provided in Table 14.
The experiments are developed using code adapted from the code that the authors of
Grathwohl et al.
(2019) provided in https://github.com/rtqichen/ffjord .
All experiments were performed on NVIDIA RTX 3090 GPUs on a cloud cluster.
[Table]
[Table]
[Table]
N
[Table]
A
N/A N/A
N/A N/A
[Table] [Table]
A
N/A
N/A
N/A
N/A
[Table]
[Table] [Table]
A
[Table]
A
[Table]
A
[Table]
[Table]
[Table]
/A
[Table] [Table] [Table]
[Table] [Table] [Table]
[Table] [Table]
N/
[Table]
Layer  (3,3) (1,1) (1,1)
[Table]
[Table]
A. E XPERIMENTAL DETAILS OF PDE MODELING A.4. 2- DIMENSIONAL B URGER ’ S E QUATION
We want to solve the initial value problem
[Equation]
where the exact solution is u ( x, t ) = 2 xe t (2 e t +1) .
Our dataset’s input are 200 randomly sampled points ( x, t ) , x ∈ [1 , 2] , t ∈ [0 , 1] , and the dataset’s outputare the exact solutions at those points.
For the C-NODE architecture, we define four networks: NN 1 ( x, t ) for ∂u∂x , NN 2 ( x, t ) for ∂u∂t ,
NN 3 ( t ) for the characteristic path ( x ( s ) , t ( s )) , NN 4 ( x ) for the initial condition.
The result is calculated in four steps:
Integrate ∆ u
= (cid:82) t 0 du ( x ( s ) ,t ( s ))
ds
[Equation]
NN 3 [1] ds as before.
Given x, t , solve equation ι + NN 3 ( NN 4 ( ι ))
[0] ∗ t = x for ι iteratively, with ι n +1 = x − NN 3 ( NN 4 ( ι n ))
[0] ∗ t . ι 0 is initialized to be x . 3. Calculate initial value u ( x (0) , t (0)) =
NN 4 ( ι ) . u ( x, t ) = ∆ u + u ( x (0) , t (0)) . For the NODE architecture, we define one network: NN 1 ( x, t ) for ∂u∂t .
The result is calculated as u ( x, t ) =
(cid:82) t 0 ∂u∂t
dt = (cid:82) t
NN 1 dt .
All experiments were performed on NVIDIA RTX 3080-TI GPUs on a local machine.
A.4. 100- DIMENSIONAL C ONVECTION E QUATION
We additionally experiment on solving a 100-dimensional convection equation given by:
[Equation]
 where x ∈ R 100 , u : R 100 × R + → R , µ : R + → R 100 .
This equation has an analytical solution u ( x , t ) =
exp( − 12 ∥ x − µ ( t ) ∥ 2 ) .
We generate a training dataset with 1000 points, and a testing dataset with 1000 points.
We uniformly sample x , with each x i ∈ [0 , 0 . 5] and t ∈ [0 , 10] .
We calculate the output using the analytical solution.
We define three networks for C-NODE.
The first network models d x /ds : R 101 → R 101 , the second network models J x u : R 101 → R 101 , the third network Γ : R 101 → R models the initial condition u ( x , 0) .
We first integrate ( x , t ) using the first two networks.
The output of these networks is then input to the third network to obtain the value of the solution.
The total number of parameters for the networks is 11611.
We define two networks for NODE.
The first network models d u /ds : R 101 → R 101 , the second network Γ : R 101 → R models the initial condition u ( x , 0) .
We first integrate ( x , t ) using the first network.
The output is R 101 .
We put the output into the second network, and arrive at the output.
The number of parameters used here is 14214.
We test C-NODE and NODE on the dataset with Gaussian noises at different magnitudes.
As shown in Table 15, C-NODE performs 14.2% better than NODE when there is no noise, 44.4% better [Figure] when there’s 10% noise, 39.18% better when there’s 20% noise, and 40.1% better when there’s 30% noise.
The architecture of C-NODE and NODE are as given in Tables 16, 17, 19, 18.
The models are optimized with the AdamW optimizer, with a learning rate of 5 × 10 − 3 and a weight decay of 5 × 10 − 4 .
We also report computation time of NODE and C-NODE using the adjoint method and the Euler forward integration.
As shown in Figure 9, C-NODE uses slightly more computation time for Euler forward integration, and the same amount of time when using the adjoint method.
Although each call of C-NODE uses slightly more time, NODE uses a bigger number of functional evaluations.
This results in roughly a similar computation time for NODE and C-NODE when integrated using the adjoint method.
A note on the difference between physics-informed neural networks (PINNs) PINNs (Raissi et al., 2019) is a method that solves a PDE using a regularization approach by minimizing a regularization term enforcing the PDE.
Comparing this to C-NODE, there are a few apparent differences.
First, PINNs is a very general framework that can accommodate almost any type of PDE.
This contrasts with C-NODE which only computes solutions to hyperbolic PDEs.
On the other hand,
PINNs generally scale poorly with dimension due to the difficulty in computing high dimensional derivatives.
Besides, PINNs require the exact form of the PDEs being solved, whereas C-NODE does not require this information.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
N/A N/A
[Table]
N
[Table]
A
N/A
[Table]
B A PPROXIMATION C APABILITIES OF C-NODE
Proposition B.1 (Method of Characteristics for Vector Valued PDEs) .
Let u ( x 1 , . . .
, x k )
: R k
→ R n be the solution of a first order semilinear PDE on a bounded domain Ω ⊂ R k of the form
[Equation]
Additionally, let a = ( a 1 , . .
. , a k ) T : R k + n → R k , c : R k + n → R n be Lipschitz continuous functions.
Define a system of ODEs as
[Equation]
where x 0 and u 0 define the initial condition, ∂ Ω is the boundary of the domain Ω .
Given initial conditions x 0 , u 0 , the solution of this system of ODEs U ( s ) :
[ a, b ] → R d is equal to the solution of the PDE in Equation equation 12 along the characteristic curve defined by x ( s ) , i.e., u ( x ( s ))
=
U ( s ) .
The union of solutions U ( s ) for all x 0 ∈ ∂ Ω is equal to the solution of the original PDE in Equation equation 12 for all x ∈ Ω .
Lemma B.2 (Gronwall’s Lemma (Howard, 1998)) .
Let U
⊂ R n be an open set.
Let f : U × [0 , T ] → R n be a continuous function and let h 1 , h 2 : [0 , T ] → U satisfy the initial value problems:
[Equation]
If there exists non-negative constant C such that for all t ∈ [0 , T ]
[Equation]
where ∥ · ∥ is the Euclidean norm.
Then, for all t ∈ [0 , T ] ,
[Equation]
B. P ROOF OF P ROPOSITION B.
This proof is largely based on the proof for the univarate case provided at 4 .
We extend for the vector valued case.
Proof.
For PDE on u with k input, and an n -dimensional output, we have a i : R k + n → R , ∂ u ∂x i ∈ R n , and c : R k + n → R n .
In proposition B.1, we look at PDEs in the following form k (cid:88)
[Equation]
Defining and substituting x = ( x 1 , . .
, x k ) ⊺ , a = ( a 1 , . .
. , a k ) ⊺ , and Jacobian J ( u ( x ))
= ( ∂ u ∂x 1 , ..., ∂ u ∂x k ) ∈ R n × k into Equation equation 12 result in
[Equation]
From proposition B.1, the characteristic curves are given by
[Equation]
and the ODE system is given by
[Equation]
Define the difference between the solution to equation 16 and the PDE in equation 12 as
[Equation]
Differentiating ∆( s ) with respect to s and plugging in equation 15, we get
[Equation]
equation 14 gives us (cid:80) ki
=1 a i ( x 1 , . . .
, x k , u ) ∂ u ∂x i − c ( x 1 , . . .
, x k , u ) = 0 .
Plugging this equality into equation 17 and rearrange terms, we have
[Equation]
Combining terms, we have
[Equation]
Applying triangle inequality, we have
[Equation]
By the assumption in proposition B.1, a and c are Lipschitz continuous.
By Lipschitz continuity, we have ∥ a ( U ) − a ( u ))
∥
≤ A ∥
u − U ∥ and ∥ c ( U ) − c ( u ))
∥ ≤ B ∥ u − U ∥ , for some constants A and B in R + .
Also, for compact set [0 , s 0 ] , s 0 < ∞ , since both u and Jacobian J are continuous mapping, J ( u ) is also compact.
Since a subspace of R n is compact if and only it is closed and bounded, J ( u ) is bounded (Strichartz, 2000).
Thus, ∥ J ( u ) ∥ ≤ M for some constant M in R + .
Define C = 2( AM + B ) , we have
[Equation]
From proposition B.1, we have u ( x (0))
=
U (0) .
As proved above, we have
[Equation]
where C < ∞ .
Thus, by lemma B.2, we have
[Equation]
This further implies that U ( s ) =
u ( x ( s )) , so long as a and c are Lipschitz continuous.
B. P ROOF OF P ROPOSITION 4.
Proof.
Suppose have C-NODE given by
[Equation]
Write out specific functions for these terms to match the desired properties of the function.
Define initial condition u (0 , 0) =
u 0 .
By setting
[Equation]
u 0.0 0.2 0.
t 0.0 0.
x 0.
have the ODE and solution,
[Equation]
To be specific, we can represent this system with the following family of PDEs:
[Equation]
We can solve this system to obtain a function that has intersecting trajectories.
The solution is visualized in Figure 10, which shows that C-NODE can be used to learn and represent this function G .
It should be noted that this is not the only possible solution to function G , as when ∂t/∂s = 0 , we fall back to a NODE system with the dynamical system conditioned on the input data.
In this conditioned setting, we can then represent G by stopping the dynamics at different times t as in (Massaroli et al., 2021).
B. P ROOF OF P ROPOSITION 4.
The proof uses the change of variables formula for a particle that depends on a vector rather than a scalar and it follows directly from the proof given in (Chen et al., 2019b, Appendix A).
We provide the full proof for completeness.
Proof.
We initially assume that (cid:80) ki =
1 ∂u∂x i dx i ds is Lipschitz continuous in u and continuous in t , so every initial value problem has a unique solution (Evans, 2010).
We additionally assume u ( s ) is bounded.
We want to show that the probability flow satisfies
[Equation]
Define T ϵ = u
( s + ϵ ) .
The discrete change of variables states that u 1 = f ( u 0 ) ⇒ log p ( u 1 )
= log p ( u 0 ) − log | det ∂f∂u
| (Rezende & Mohamed, 2015).
Taking the limit of the time difference between u 0 and u 1 , and using the definition of derivatives,
[Equation]
The Jacobi’s formula states that if A is a differentiable map from the real numbers to n × n matrices, then ddt det A ( t ) =
tr (cid:16) adj( A ( t )) dA ( t ) dt (cid:17) , where adj denotes the adjugate matrix. Applying this relation, we obtain
[Equation] ϵ ( u ( s )) (cid:19) ∂
∂ϵ ∂ ∂uT ϵ ( u ( s ))
(cid:21) = − tr (cid:20)(cid:18) lim
ϵ → 0 + adj (cid:18) ∂ ∂uT ϵ ( u ( t ))
(cid:19)(cid:19) (cid:18) lim
ϵ
→ 0 + ∂ ∂ϵ ∂ ∂uT ϵ ( u ( s ))
(cid:19)(cid:21) = − tr (cid:20) adj (cid:18) ∂ ∂uu ( t ) (cid:19) lim
ϵ → 0 + ∂ ∂ϵ ∂ ∂uT ϵ ( u ( s ))
(cid:21) = − tr (cid:20) lim
ϵ → 0 + ∂ ∂ϵ ∂ ∂uT ϵ ( u ( s ))
(cid:21)
Substituting T ϵ with its Taylor series expansion and taking the limit, we have the desired result ∂ log p ( u ( t ))
∂t = − tr (cid:18) lim ϵ → 0 + ∂ ∂ϵ ∂ ∂u (cid:18) u
+ ϵduds
+ O ( ϵ 2 )
+ O ( ϵ 3 )
+ ... (cid:19)(cid:19) = − tr (cid:32) lim ϵ → 0 + ∂ ∂ϵ ∂ ∂u (cid:32) u + ϵ k (cid:88) i = ∂u ∂x i
dx i ds
+ O ( ϵ 2 )
+ O ( ϵ 3 )
+ ...
(cid:33)(cid:33) = − tr (cid:32) lim
ϵ → 0 + ∂ ∂ϵ (cid:32)
I + ∂ ∂uϵ k (cid:88) i = ∂u ∂x i
dx i ds
+ O ( ϵ 2 )
+ O ( ϵ 3 )
+ ...
(cid:33)(cid:33) = − tr (cid:32) lim
ϵ → 0 + (cid:32) ∂ ∂u
[Equation]
B. P ROOF OF P ROPOSITION 4.
Proof.
To prove proposition 4.3, we need to show that for any homeomorphism h ( · ) , there exists a u ( s, u 0 )
∈ R n following a C-NODE system such that u ( s = T, u 0 ) =
h ( u 0 ) .
Without loss of generality, we suppose that T = 1 .
Defining a C-NODE system as:
[Equation]
Then, du ds = h ( u 0 ) − u 0 .
At s = 1 , have
[Equation]
The inverse map will be defined by integration backwards.
Specifically, we have
[Equation]
Thus, for any homeomorphism h ( · ) , there exists a C-NODE system, such that forward integration for time s = 1 is equivalent as applying h ( · ) , and backward integration for time s = 1 is equivalent to applying h − 1 ( · ) .
C I NCLUDING I NITIAL S TATE IN NODE’ S I NPUT
Conditioning on the initial condition allows NODE to model intersecting trajectories.
We compare
NODE conditioned on initial values and C-NODE’s performance on image classification tasks on
MNIST, SVHN, and CIFAR-10 datasets, using the forward Euler solver and adjoint solver.
As shown in Figure 11, by conditioning on initial values, NODE’s performances are improved over all datasets when using adjoint integration, and improved on SVHN dataset when using Euler forward integration.
When using Euler forward integration, C-NODE performs better on all three datasets than NODE and conditioned-NODE.
C-NODE higher accuracy within fewer epochs and has lower variance throughout the training process.
All methods use a comparable numbers of parameters [Figure] with C-NODE using the fewest as reported in Table 20.
When using adjoint method, C-NODE performs better on all datasets than NODE, and performs better on SVHN and CIFAR-10 datasets than conditioned-NODE, while achieving similar performance to conditioned-NODE.
It is worth noting that the characteristic curves of the C-NODE also depend on the initial values, which allows C-NODE to model a dynamical system with intersecting trajectories.
C-NODE and conditioned NODE are two different methods of including the initial values.
Empirically, C-NODE achieves better results and is more efficient with parameters, as suggested in Figure 11 and Table 20.
[Table]
All experiments were performed on NVIDIA RTX 3090 GPUs on a cloud cluster.
D A BLATION S TUDY D. A BLATION STUDY ON DIMENSION OF C-NODE
We perform an ablation study on the impact of the number of dimensions of the C-NODE we implement.
This study allows us to evaluate the relationship between the model performance and the model’s limit of mathematical approximating power.
Empirical results show that as we increase the number of dimensions used in the C-NODE model, the C-NODE’s performance first improves and then declines, due to overfitting.
We have found out that information criteria like AIC and BIC can be successfully applied for dimension selection in this scenario.
In previous experiments, we represent ∂
u /∂x i with separate and independent neural networks
c i ( u , θ ) .
Here, we represent all k functions as a vector-valued function [ ∂ u /∂x 1 , ..., ∂ u /∂x k ] T .
We approximate this vector-valued function with a neural network c ( u , θ ) .
The model is trained using the Euler solver to have better training stability when the neural network has a large number of parameters.
Experiment details for the ablation study is as shown in Figures 12, 13, 14.
[Figure]
D. A BLATION STUDY ON NUMBER OF PARAMETERS
We show C-NODE’s parameter efficiency over NODE with an ablation study on the image classification task on the CIFAR-10 dataset.
Specifically, under a similar training setup, we experiment with C-NODE with 95071, 55855, and 17379 parameters and experiment with NODE with 96044, 56828, and 17444 parameters.
As shown in Figure 15, although C-NODE has more variance in its performance, it outperforms NODE along the whole training process in all three cases.
[Figure]
[Figure]
[Figure]
Algorithm 2 C-NODE algorithm using the forward Euler method for each input data
z j do extract image feature u ( s = 0) =
g ( z j ; Θ 1 ) with a feature extractor neural network.
procedure Integration along s =
→for each time step s m do calculate d x ds ( x , u ; g ( z j ; Θ 1 ); Θ 2 ) and J x u ( x , u ; Θ 2 ) .
calculate d u ds =
J x u d x ds .
calculate u ( s m +1 ) =
u ( s m ) + d u ds ( s m +1 − s m ) .
calculate x
( s m +1 )
=
x ( s m ) + d x ds ( s m +1 − s m ) end for end procedure classify
u
( s = 1) with neural network Φ( u ( x ( s = 1)) , Θ 3 ) .
end for
E A LGORITHM FOR IMAGE CLASSIFICAITON WITH C-NODE
We provide algorithm for training C-NODE for the purpose of image classification.
F A LGORITHM FOR CONTINUOUS NORMALIZING FLOWS DEFINED WITH C-NODE
We additionally provide algorithms for training and sampling CNFs defined with C-NODEs.
Algorithm 3 Algorithm for training CNFs defined with C-NODE given probability density function of p ( u ( s = 0))
=
p 0 ( · ) for each input data
z j do
Given (cid:34) u (1) log p ( z j ) − log p ( u (1)) (cid:35) = (cid:34) z j (cid:35) procedure Integrate from 1 → 0 to get (cid:34) u (0) log p ( z j ) − log p ( u (0))
(cid:35) for each time step s m do calculate d x ds ( x , u ; g ( z j ; Θ 1 ); Θ 2 ) and J x u ( x , u ; Θ 2 ) .
calculate d u ds =
J x u d x ds .
calculate − tr ( ∂∂ u J x u d x ds ) with Hutchinson trace estimator (Grathwohl et al., 2019).
calculate (cid:34) u
( s m +1 ) log p ( u ( s m +1 ))
(cid:35) =
(cid:34) u ( s m ) log p ( u ( s m ))
(cid:35) + (cid:34)
d u dt ∂ log p ( u ( s ))
∂s (cid:35) ( s m +1 − s m ) .
end for evaluate p 0 ( u (0)) calculate log p ( z j ) =
(log p ( z j ) − log p ( u (0)))
+ log p
( u (0))
optimize log p ( z j ) with an optimization algorithm (stochastic gradient descent etc.) end for procedure sample u ( s = 0) from base distribution p 0 ( · ) procedure Integrate from 0 → 1 to get u (
1) for each time step s m do calculate d calculate calculate u end for end procedure u