N EURAL D ELAY D IFFERENTIAL E QUATIONS
A BSTRACT
Neural Ordinary Differential Equations (NODEs), a framework of continuousdepth neural networks, have been widely applied, showing exceptional efﬁcacy in coping with some representative datasets.
Recently, an augmented framework has been successfully developed for conquering some limitations emergent in application of the original framework.
Here we propose a new class of continuous-depth neural networks with delay, named as Neural Delay Differential Equations (NDDEs), and, for computing the corresponding gradients, we use the adjoint sensitivity method to obtain the delayed dynamics of the adjoint.
Since the differential equations with delays are usually seen as dynamical systems of inﬁnite dimension possessing more fruitful dynamics, the NDDEs, compared to the NODEs, own a stronger capacity of nonlinear representations.
Indeed, we analytically validate that the NDDEs are of universal approximators, and further articulate an extension of the NDDEs, where the initial function of the NDDEs is supposed to satisfy
ODEs. More importantly, we use several illustrative examples to demonstrate the outstanding capacities of the NDDEs and the NDDEs with ODEs’ initial value.
Speciﬁcally, (1) we successfully model the delayed dynamics where the trajectories in the lower-dimensional phase space could be mutually intersected, while the traditional NODEs without any argumentation are not directly applicable for such modeling, and (2) we achieve lower loss and higher accuracy not only for the data produced synthetically by complex models but also for the real-world image datasets, i.e., CIFAR10, MNIST, and SVHN.
Our results on the NDDEs reveal that appropriately articulating the elements of dynamical systems into the network design is truly beneﬁcial to promoting the network performance.
I NTRODUCTION
A series of recent works have revealed a close connection between neural networks and dynamical systems (E, 2017; Li et al., 2017; Haber & Ruthotto, 2017; Chang et al., 2017; Li & Hao, 2018;
Lu et al., 2018; E et al., 2019; Chang et al., 2019; Ruthotto & Haber, 2019; Zhang et al., 2019a; Pathak et al., 2018; Fang et al., 2018; Zhu et al., 2019; Tang et al., 2020).
On one hand, the deep neural networks can be used to solve the ordinary/partial differential equations that cannot be easily computed using the traditional algorithms.
On the other hand, the elements of the dynamical systems can be useful for establishing novel and efﬁcient frameworks of neural networks.
Typical examples include the Neural Ordinary Differential Equations (NODEs), where the inﬁnitesimal time of ordinary differential equations is regarded as the “depth” of a considered neural network (Chen et al., 2018).
Though the advantages of the NODEs were demonstrated through modeling continuous-time datasets and continuous normalizing ﬂows with constant memory cost (Chen et al., 2018), the limited capability of representation for some functions were also studied (Dupont et al., 2019).
Indeed, the NODEs cannot be directly used to describe the dynamical systems where the trajectories in the lower-dimensional phase space are mutually intersected.
Also, the NODEs cannot model only a few variables from some physical or/and physiological systems where the effect of time delay is inevitably present.
From a view point of dynamical systems theory, all these are attributed to the characteristic of ﬁnite-dimension for the NODEs.
In this article, we propose a novel framework of continuous-depth neural networks with delay, named as Neural Delay Differential Equations (NDDEs).
We apply the adjoint sensitivity method to compute the corresponding gradients, where the obtained adjoint systems are also in a form of delay differential equations.
The main virtues of the NDDEs include:
• feasible and computable algorithms for computing the gradients of the loss function based on the adjoint systems, • representation capability of the vector ﬁelds which allow the intersection of the trajectories in the lower-dimensional phase space, and • accurate reconstruction of the complex dynamical systems with effects of time delays based on the observed time-series data.
R ELATED W ORKS
NODEs Inspired by the residual neural networks (He et al., 2016) and the other analogous frameworks, the NODEs were established, which can be represented by multiple residual blocks as
[Equation]
where h t is the hidden state of the t -th layer, f ( h t ; w t ) is a differential function preserving the dimension of h t , and w t is the parameter pending for learning.
The evolution of h t can be viewed as the special case of the following equation
[Equation]
with ∆ t = 1 .
As suggested in (Chen et al., 2018), all the parameters w t are uniﬁed into w for achieving parameter efﬁciency of the NODEs.
This uniﬁed operation was also employed in the other neural networks, such as the recurrent neural networks (RNNs) (Rumelhart et al., 1986; Elman, 1990) and the ALBERT (Lan et al., 2019).
Letting ∆ t → 0 and using the uniﬁed parameter w instead of w t , we obtain the continuous evolution of the hidden state h t as
[Equation]
w ) , which is in the form of ordinary differential equations.
Actually, the NODEs can act as a feature extraction, mapping an input to a point in the feature space by computing the forward path of a
NODE as:
[Equation]
where h (0)
= input is the original data point or its transformation, and T is the integration time (assuming that the system starts at t = 0 ).
Under a predeﬁned loss function L ( h ( T )) , (Chen et al., 2018) employed the adjoint sensitivity method to compute the memory-efﬁcient gradients of the parameters along with the ODE solvers.
More precisely, they deﬁned the adjoint variable, λ ( t ) =
∂L ( h ( T ))
∂ h ( t ) , whose dynamics is another
ODE, i.e.,
[Equation]
The gradients are computed by an integral as:
[Equation]
(Chen et al., 2018) calculated the gradients by calling an ODE solver with extended ODEs (i.e., concatenating the original state, the adjoint, and the other partial derivatives for the parameters at each time point into a single vector).
Notably, for the regression task of the time series, the loss function probably depend on the state at multiple observational times, such as the form of
L ( h ( t 0 ) , h ( t 1 ) , ..., h ( t n )) .
Under such a case, we must update the adjoint state instantly by adding the partial derivative of the loss at each observational time point.
As emphasized in (Dupont et al., 2019), the ﬂow of the NODEs cannot represent some functions omnipresently emergent in applications.
Typical examples include the following two-valued function with one argument: g 1-D (1) = − 1 and g 1-D ( − 1)
= 1 .
Our framework desires to conquer the representation limitation observed in applying the NODEs.
Optimal control As mentioned above, a closed connection between deep neural networks and dynamical systems have been emphasized in the literature and, correspondingly, theories, methods and tools of dynamical systems have been employed, e.g. the theory of optimal control (E, 2017; Li et al., 2017; Haber & Ruthotto, 2017;
Chang et al., 2017; Li & Hao, 2018; E et al., 2019; Chang et al., 2019; Ruthotto & Haber, 2019; Zhang et al., 2019a).
Generally, we model a typical task using a deep neural network and then train the network parameters such that the given loss function can be reduced by some learning algorithm.
In fact, training a network can be seen as solving an optimal control problem on difference or differential equations (E et al., 2019).
The parameters act as a controller with a goal of ﬁnding an optimal control to minimize/maximize some objective function.
Clearly, the framework of the NODEs can be formulated as a typical problem of optimal control on
ODEs.
Additionally, the framework of NODEs has been generalized to the other dynamical systems, such as the Partial Differential Equations (PDEs) (Han et al., 2018;
Long et al., 2018; 2019;
Ruthotto & Haber, 2019; Sun et al., 2020) and the Stochastic Differential Equations (SDEs) (Lu et al., 2018; Sun et al., 2018; Liu et al., 2019), where the theory of optimal control has been completely established.
It is worthwhile to mention that the optimal control theory is tightly connected with and beneﬁts from the method of the classical calculus of variations (Liberzon, 2011).
We also will transform our framework into an optimal control problem, and ﬁnally solve it using the method of the calculus of variations.
T HE FRAMEWORK OF NDDE S 3. F ORMULATION OF NDDE S
1 t   x ( t ) 1 t
In this section, we establish a framework of continuous-depth neural networks.
To this end, we ﬁrst introduce the concept of delay deferential equations (DDEs).
The DDEs are always written in a form where the derivative of a given variable at time t is affected not only by the current state of this variable but also the states at some previous time instants or time durations (Erneux, 2009).
Such kind of delayed dynamics play an important role in description of the complex phenomena emergent in many real-world systems, such as physical, chemical, ecological, and [Equation]
%( ) (cid:1872)(cid:3)(cid:3404)(cid:3)(cid:882) =
$ (cid:215) (cid:3) [(cid:127)(cid:129)
[Equation]
0] R d R d R d R d ODE (cid:24) DE f
[Equation]
h
[Equation]
w ) f
( h
[Equation]
(cid:1) w )
R R (cid:282)(cid:286)(cid:367)(cid:258)(cid:455)(cid:3)(cid:286)(cid:296)(cid:296)(cid:286)(cid:272)(cid:410) h ( (cid:17) )
[Figure]
physiological systems.
In this article, we consider a system of DDE with a single time delay:
[Equation]
where the positive constant τ is the time delay and h ( t )
=
φ ( t ) is the initial function before the time t = 0 .
Clearly, in the initial problem of ODEs, we only need to initialize the state of the variable at t = 0 while we initialize the DDEs using a continuous function.
Here, to highlight the difference between ODEs and DDEs, we provide a simple example:
[Equation]
where τ = 1 (with time delay) or τ = 0 (without time delay).
As shown in Figure 2 , the DDE ﬂow can map -1 to 1 and 1 to -1; nevertheless, this cannot be made for the ODE whose trajectories are not intersected with each other in the t - x space in Figure 2.
3. A DJOINT M ETHOD FOR NDDE S
Assume that the forward pass of DDEs is complete.
Then, we need to compute the gradients in a reverse-mode differentiation by using the adjoint sensitivity method (Chen et al., 2018;
Pontryagin et al., 1962).
We consider an augmented variable, named as adjoint and deﬁned as
[Equation]
where L ( · ) is the loss function pending for optimization.
Notably, the resulted system for the adjoint is in a form of DDE as well.
Theorem 1 (Adjoint method for NDDEs).
Consider the loss function L ( · ) .
Then, the dynamics of adjoint can be written as
[Equation]
where χ [0 ,T − τ ] ( · ) is a typical characteristic function.
We provide two ways to prove Theorem 1, which are, respectively, shown in Appendix.
Using h ( t ) and λ ( t ) , we compute the gradients with respect to the parameters w as:
[Equation]
Clearly, when the delay τ approaches zero, the adjoint dynamics degenerate as the conventional case of the NODEs (Chen et al., 2018).
We solve the forward pass of h and backward pass for h , λ and dLd w by a piece-wise ODE solver, which is shown in Algorithm 1.
For simplicity, we denote by f ( t ) and g ( t ) the vector ﬁled of h and λ , respectively.
Moreover, in this paper, we only consider the initial function φ ( t ) as a constant function, i.e., φ ( t ) =
h 0 .
Assume that T = n · τ and denote f k ( t ) =
f ( k · τ + t ) , g k ( t ) =
g ( k · τ + t ) and λ k ( t ) =[-λ ( k · τ + t ) . Algorithm Input: dynamics parameters w , time delay τ , start time 0 , stop time T = n · τ , ﬁnal state h ( T ) , loss gradient ∂L/∂ h ( T ) ∂L∂ w = 0 | w | for i in range( n − 1 , − 1 , − 1 ) : s 
-]
λ ( k · τ + t ) .
In the traditional framework of the NODEs, we can calculate the gradients of the loss function and recompute the hidden states by solving another augmented ODEs in a reversal time duration.
However, to achieve the reverse-mode of the NDDEs in Algorithm 1, we need to store the checkpoints of the forward hidden states h
( i · τ ) for i = 0 , 1 , ..., n , which, together with the adjoint λ ( t ) , can help us to recompute h ( t ) backwards in every time periods.
The main idea of the Algorithm 1 is to convert the DDEs as a piece-wise ODEs such that one can naturally employ the framework of the
NODEs to solve it.
The complexity of Algorithm 1 is analyzed in Appendix. Algorithm
{+Input: dynamics parameters w , time delay τ , start time 0 , stop time T = n · τ , ﬁnal state h ( T ) , loss gradient ∂L/∂ h ( T ) ∂L∂ w = 0 | w | for i in range( n − 1 , − 1 , − 1 ) : s 
+}
=
[ h ( T ) , h ( T − τ ) , ..., h ( τ ) , ∂L ∂ h ( T ) , ..., ∂L ∂ h (( i +1) · τ ) , ∂L∂ w ] def aug dynamics([ h n − 1 ( t ) ,..., h 0 ( t ) , λ n − 1 ( t ) ,.., λ i ( t ) , .], t , w ): return [ f n − 1 ( t ) , f n − 2 ( t ) , ..., f 0 ( t ) , g n − 1 ( t ) , ..., g i ( t ) , − λ i ( t ) (cid:62) ∂f i ( t ) ∂ w ] [ ∂L ∂ h ( i · τ ) , ∂L∂ w ] =ODESolve( s 0 , aug dynamics, τ , 0 , w ) return ∂L∂ h (0) , ∂L∂ w  I LLUSTRATIVE E XPERIMENTS 4. E XPERIMENTS ON SYNTHETIC DATASETS
Here, we use some synthetic datasets produced by typical examples to compare the performance of the NODES and the NDDEs.
In (Dupont et al., 2019), it is proved that the NODEs cannot represent the function g : R d → R , deﬁned by
[Equation]
where 0 < r 1 < r 2 < r 3 and (cid:107) · (cid:107) is the Euclidean norm.
The following proposition show that the
NDDEs have stronger capability of representation.
Proposition 1 The NDDEs can represent the function g ( x ) speciﬁed above.
To validate this proposition, we construct a special form of the NDDEs by   d h
i ( t ) dt
= {+(cid:107) h t − τ (cid:107) − r, t > = 
+}
and i = 1 , d h i ( t ) dt
{+= 0 , t > = 
 and i = 2 , ..., d, h ( t ) = x , t < = 0 .+}
(6) where r :=
( r 1 + r 2 ) / 2 is a constant and the ﬁnal time point T is supposed to be equal to the time delay τ with some sufﬁcient large value.
Under such conﬁgurations, we can linearly separate the two clusters by some hyper plane.
Here trans formed data separable.
In Fig.
3 , we, respectively, show the original data of g ( x ) for the dimension d = 2 and the transformed data by the DDEs (6) with the parameters r 1 = 1 , r 2 = 2 , r 3 = 3 , T = 10 , and τ = 10 .
Clearly, the transformed data by the DDEs are linearly separable.
We also train the NDDEs to represent the g ( x ) for d = 2 , whose evolutions during the training procedure are, respectively, shown in Fig.
This ﬁgure also includes the evolution of the NODEs.
Notably, while the NODEs is struggled to break apart the annulus, the NDDEs easily separate them.
The training losses and the ﬂows of the NODEs and the NDDEs are depicted, respectively, in Fig.
Particularly, the NDDEs achieve lower losses with the faster speed and directly separate the two clusters in the original 2 -D space; however, the NODEs achieve it only by increasing the dimension of the data and separating them in a higher-dimensional space.
In general, we have the following theoretical result for the NDDEs, whose proof is provided in
Appendix.
[Figure]
T r a i n i n g l o ss the population, r is the growth rate.
The Mackey-Glass system is written as ˙ x = β x ( t − τ ) 1+ x n ( t − τ ) − γx ( t ) , where x ( t ) is the number of the blood cells, β, n, τ, γ are the parameters of biological signiﬁcance.
The NODEs and the NDDEs are tested on these two dynamics.
As shown in Figure 7, a very low training loss is achieved for the NDDEs while the the loss of the NODEs does not go down afterwards, always sustaining at some larger value.
As for the predicting durations, the NDDEs thereby achieve a better performance than the NODEs. 4. E XPERIMENTS ON IMAGE DATASETS Here, we apply the NODEs and the NDDEs to the image datasets. For the NODEs, we model the vector ﬁled f ( h ( t )) as the convolutional architectures together with the same hyperparameter setups in (Dupont et al., 2019).
The initial hidden state h (0) ∈ R c × h × w with respect to an image, where c, h, w are, respectively, the number of channels, height, and width of the image.
For the NDDEs, we design the vector ﬁled as f (concat( h ( t ) , h ( t − τ ))) , mapping the space from R 2 c × h × w to R c × h × w , where concat( · , · ) executes the concatenation of two tensors on the channel dimension.
Its initial function is designed as a constant function, i.e., h ( t ) is the input/image for t < 0 .
Theorem 2 (Universal approximating capability of the NDDEs).
For any given continuous function
F : R n → R n , if one can construct a neural network for approximating the map G ( x ) =
1 T [ F ( x ) − x ] , then there exists an NDDE of n -dimension that can model the map x
(cid:55)→ F ( x ) , that is, h ( T ) ≈
F ( x ) with the initial function φ ( t ) =
x for t ≤ 0 .
Additionally, NDDEs are suitable for ﬁtting the time series with the delay effect in the original systems, which cannot be easily achieved by using the NODEs.
To illustrate this, we use a model of 2 -D DDEs, written as ˙ x
= A tanh( x ( t ) + x ( t − τ )) with x ( t ) =
x 0 for t < 0 .
Given the time series generated by the system, we use the NDDEs and the NODEs to ﬁt it, respectively.
Figureshows that the NNDEs approach a much lower loss, compared to the NODEs.
More interestingly, the NODEs prefer to ﬁt the dimension of x 2 and its loss always sustains at a larger value, e.g. 0 .in
Figure 6 .
The main reason is that two different trajectories generated by the autonomous ODEs cannot intersect with each other in the phase space, due to the uniqueness of the solution of ODEs.
We perform experiments on another two classical DDEs, i.e., the population dynamics and the
Mackey-Glass system (Erneux, 2009).
Speciﬁcally, the equation of the dimensionless population dynamics is ˙ x = rx ( t )(1 − x ( t − τ )) , where x ( t ) is the ratio of the population to the carrying capacity of the population, r is the growth rate and τ is the time delay.
The Mackey-Glass system is written as ˙ x = β x ( t − τ ) 1+ x n ( t − τ ) − γx ( t ) , where x ( t ) is the number of the blood cells, β, n, τ, γ are the parameters of biological signiﬁcance.
The NODEs and the NDDEs are tested on these two dynamics.
As shown in Figure 7, a very low training loss is achieved for the NDDEs while the the
[Figure]
loss of the NODEs does not go down afterwards, always sustaining at some larger value.
As for the predicting durations, the NDDEs thereby achieve a better performance than the NODEs.
The details of the training could be found in Appendix
[Figure]
4. E XPERIMENTS ON IMAGE DATASETS
For the image datasets, we not only use the NODEs and the NDDEs but also an extension of the
NDDEs, called the NODE+NDDE, which treats the initial function as an ODE.
In our experiments, such a model exhibits the best performance, revealing strong capabilities in modelling and feature representations.
Moreover, inspired by the idea of the augmented NODEs (Dupont et al., 2019), we extend the NDDEs and the NODE+NDDE to the A+NDDE and the A+NODE+NDDE, respectively.
Precisely, for the augmented models, we augment the original image space to a higher-dimensional space, i.e., R c × h × w → R ( c + p ) × h × w , where c , h , w , and p are, respectively, the number of channels, height, width of the image, and the augmented dimension.
With such conﬁgurations of the same augmented dimension and approximately the same number of the model parameters, comparison studies on the image datasets using different models are reasonable.
For the NODEs, we model the vector ﬁled f ( h ( t )) as the convolutional architectures together with a slightly different hyperparameter setups in (Dupont et al., 2019).
The initial hidden state is set as h (0)
∈ R c × h × w with respect to an image.
For the NDDEs, we design the vector ﬁled as f (concat( h ( t ) , h ( t − τ ))) , mapping the space from R 2 c × h × w to R c × h × w , where concat( · , · ) executes the concatenation of two tensors on the channel dimension.
Its initial function is designed as a constant function, i.e., h ( t ) is identical to the input/image for t < 0 .
For the NODE+NDDE, we model the initial function as an ODE, which follows the same model structure of the NODEs.
For the augmented models, the augmented dimension is chosen from the set { 1, 2, 4 } .
Moreover, the training details could be found in Appendix, including the training setting and the number of function evaluations for each model on the image datasets.
The training processes on MNIST, CIFAR10, and SVHN are shown in Fig.
Overall, the NDDEs and its extensions have their training losses decreasing faster than the NODEs/ANODEs which achieve lower training and test loss.
Also the test accuracies are much higher than the that of the
NODEs/ANODEs (refer to Tab. 1).
Naturally, the better performance of the NDDEs is attributed to the integration of the information not only on the hidden states at the current time t but at the previous time t − τ .
This kind of framework is akin to the key idea proposed in (Huang et al., 2017), where the information is processed on many hidden states.
Here, we run all the experiments fortimes independently.
[Figure]
D ISCUSSION
In this section
[Figure]
we present the limitations of the NDDEs and further suggest several potential directions for future works.
We add the delay effect to the NDDEs, which renders the model absolutely irreversible.
Algorithm 1 thus requires a storage of the checkpoints of the hidden state h ( t ) at every instant of the multiple of τ .
Actually, solving the DDEs is transformed to solving the ODEs of an increasingly high dimension with respect to the ratio of the ﬁnal time T and the time delay τ .
This deﬁnitely indicates a high computational cost.
To further apply and improve the framework of the
NDDEs, a few potential directions for future works are suggested, including:
[Table]
each model, the initial (resp.
ﬁnal) time is set as 0 (resp. 1), and the delays of the NDDEs and its extensions are all set as 1, simply equal to the ﬁnal time.
Applications to more real-world datasets.
In the real-world systems such as physical, chemical, biological, and ecological systems, the delay effects are inevitably omnipresent, truly affecting the dynamics of the produced time-series (Bocharov & Rihan, 2000; Kajiwara et al., 2012).
The NDDEs are undoubtedly suitable for realizing model-free and accurate prediction (Quaglino et al., 2019).
Additionally, since the NODEs have been generalized to the areas of Computer Vision and Natural
Language Processing He et al.
; Yang et al.
Liu et al.
(2020), the framework of the
NDDEs probably can be applied to the analogous areas, where the delay effects should be ubiquitous in those streaming data.
Extension of the NDDEs.
A single constant time-delay in the NDDEs can be further generalized to the case of multiple or/and distributed time delays (Shampine & Thompson, 2001).
As such, the model is likely to have much stronger capability to extract the feature, because the model leverages the information at different time points to make the decision in time.
All these extensions could be potentially suitable for some complex tasks.
However, such complex model may require tremendously huge computational cost.
Time-dependent controllers.
From a viewpoint of control theory, the parameters in the
NODEs/NDDEs could be regarded as time-independent controllers, viz.
constant controllers.
A natural generalization way is to model the parameters as time-dependent controllers.
In fact, such controllers were proposed in (Zhang et al., 2019b), where the parameters w ( t ) were treated as another learnable ODE ˙ w = q ( w ( t ) , p ) , q ( · , · ) is a different neural network, and the parameters p and the initial state w (0) are pending for optimization.
Also, the idea of using a neural network to generate the other one was initially conceived in some earlier works including the study of the hypernetworks (Ha et al., 2016).
C ONCLUSION
In this article, we establish the framework of NDDEs, whose vector ﬁelds are determined mainly by the states at the previous time.
We employ the adjoint sensitivity method to compute the gradients of the loss function.
The obtained adjoint dynamics backward follow another DDEs coupled with the forward hidden states.
We show that the NDDEs can represent some typical functions that cannot be represented by the original framework of NODEs.
Moreover, we have validated analytically that the
NDDEs possess universal approximating capability.
We also demonstrate the exceptional efﬁcacy of the proposed framework by using the synthetic data or real-world datasets.
All these reveal that integrating the elements of dynamical systems to the architecture of neural networks could be potentially beneﬁcial to the promotion of network performance.
A UTHOR C ONTRIBUTIONS
WL conceived the idea, QZ, YG & WL performed the research, QZ & YG performed the mathematical arguments, QZ analyzed the data, and QZ & WL wrote the paper.
A CKNOWLEDGMENTS
WL was supported by the National Key RD Program of China (Grant no. 2018YFC0116600), the National Natural Science Foundation of China (Grant no. 11925103) and by the STCSM (Grant No. 18DZ1201000, 19511101404, 19511132000)
A A PPENDIX A. T HE F IRST P ROOF OF T HEOREM
Here we present a direct proof of the adjoint method for the NDDEs.
For the neat of the proof the following notations are slightly different from those in the main text.
Let x ( t ) obey the DDE written as
[Equation]
whose adjoint state is deﬁned as
[Equation]
where L is the loss function that is, L :
= L ( X ( T )) .
After discretizing the above DDE, we have
[Equation]
[Equation]
According to the deﬁnition of λ ( t ) and applying the chain rule
[Equation]
we have
[Equation]
[Equation]
which implies,
[Equation]
[Equation]
where χ [0 ,T − τ ] ( · ) is a characteristic function.
For the parameter θ ( t ) , the result can be analogously derived as
[Equation]
In this article, θ ( t ) is considered to be a constant variable, i.e., θ ( t ) ≡ θ , which yields:
[Equation]
To summarize, we get the gradients with respect to x (0) and θ in a form of augmented DDEs.
These
DDEs are backward in time and written in an integral form of
[Equation]
A. T HE S ECOND P ROOF OF T HEOREM
Here, we mainly employ the Lagrangian multiplier method and the calculus of variation to derive the adjoint method for the NDDEs.
First, we deﬁne the Lagrangian by
[Equation]
We thereby need to ﬁnd the so-called Karush-Kuhn-Tucker (KKT) conditions for the Lagrangian, which is necessary for ﬁnding the optimal solution of θ .
To obtain the KKT conditions, the calculus of variation is applied by taking variations with respect to λ ( t ) and x ( t ) .
For λ ( t ) , let ˆ λ ( t ) be a continuous and differentiable function with a scalar (cid:15) .
We add the perturbation ˆ λ ( t ) to the Lagrangian L , which results in a new Lagrangian, denoted by
[Equation]
In order to obey the optimal condition for λ ( t ) , we require the following equation
[Equation]
to be zero.
Due to the arbitrariness of ˆ λ ( t ) , we obtain
[Equation]
[Equation]
which is exactly the DDE forward in time.
Analogously, we can take variation with respect to x ( t ) and let ˆ x ( t ) be a continuous and differentiable function with a scalar (cid:15) .
Here, ˆ x ( t ) =
for t ∈ [ − τ, 0] .
We also denote by ˆ y ( t ) :
=
ˆ x ( t − τ ) for t ∈ [0 , T ] .
The new Lagrangian under the perturbation of ˆ x ( t ) becomes
[Equation]
[Equation]
We then compute the d L ( (cid:15) ) d(cid:15)
[Equation]
which gives
[Equation]
integration by parts
[Equation]
no variation on the interval [
[Equation]
τ, 0] [Equation] dt [Equation]
[Equation]
Notice that d L ( (cid:15) ) d(cid:15) | (cid:15) =
= 0 is satisﬁed for all continuous differentiable ˆ x ( t ) at the optimal x ( t ) .
Thus, we have
[Equation]
Therefore, the adjoint state follows a DDE as well.
A. T HE P ROOF OF T HEOREM
The validation for Theorem 2 is straightforward.
Consider the NDDEs in the following form:
[Equation]
where τ equals to the ﬁnal time T .
Due to h ( t ) = x for t ≤ 0 , then the vector ﬁeld of the
NDDEs in the interval [0 , T ] is constant.
This implies h ( T ) =
x + T · f ( x ; w ) .
Assume that the neural network f ( x ; w ) is able to approximate the map G ( x ) =
1 T [ F ( x ) − x ] .
Then, we have h ( T ) =
x + T · 1 T [ F ( x ) − x ] =
F ( x ) .
The proof is completed.
A. C OMPLEXITY A NALYSIS OF THE A LGORITHM
As shown in (Chen et al., 2018), the memory and the time costs for solving the NODEs are O (1) and
O ( L ) , where L is the number of the function evaluations in the time interval [0 , T ] .
More precisely, solving the NODEs is memory efﬁcient without storing any intermediate states of the evaluated time points in solving the NODEs.
Here, we intend to analyze the complexity of Algorithm 1.
It should be noting that the state-of-the-art for the DDE software is not as advanced as that for the ODE software.
Hence, solving a DDE is much difﬁcult compared with solving the ODE.
There exists several DDE solvers, such as the popular solver, the dde23 provided by MATLAB.
However, these DDE solvers usually need to store the history states to help the solvers access the past time state h
( t − τ ) .
Hence, the memory cost of DDE solvers is O ( H ) , where H is the number of the history states.
This is the major difference between the DDEs and the ODEs, as solving the ODEs is memory efﬁcient, i.e.,
O (1) .
In Algorithm 1, we propose a piece-wise ODE solver to solve the DDEs.
The underlying idea is to transform the DDEs into the piece-wise ODEs for every τ time periods such that one can naturally employ the framework of the NODEs.
More precisely, we compute the state at time kτ by solving an augmented ODE with the augmented initial state, i.e., concatenating the states at time − τ, 0 , τ, ..., ( k − 1) τ into a single vector.
Such a method has several strengths and has weaknesses as well.
The strengths include:
• One can easily implement the algorithm using the framework of the NODEs, and • Algorithm 1 becomes quite memory efﬁcient
O ( n ) , where we only need to store a small number of the forward states, h (0) , ..., h ( nτ ) , to help the algorithm compute the adjoint and the gradients in a reverse mode.
Here, the ﬁnal T is assumed to be not very large compared with the time delay τ , for example, T = nτ with a small integer n .
Notably, we chosen n = 1 (i.e., T = τ = 1 . 0 ) of the NDDEs in the experiments on the image datasets, resulting in a quite memory efﬁcient computation.
Additionally, the weaknesses involve:
• Algorithm 1 may suffer from a high memory cost, if the ﬁnal T is extremely larger than the τ (i.e., many forward states, h (0) , ..., h ( nτ ) , are required to be stored), and • the increasing augmented dimension of the ODEs may lead to the heavy time cost for each evaluation in the ODE solver.
B E XPERIMENTAL D ETAILS B. S YNTHETIC DATASETS
For the classiﬁcation of the dataset of concentric spheres, we almost follow the experimental conﬁgurations used in (Dupont et al., 2019).
The dataset contain 2000 data points in the outer annulus and 1000 data points in the inner sphere.
We choose r 1 = 0 .
5 , r 2 = 1 . 0 , and r 3 = 1 . 5 .
We solve the classiﬁcation problem using the NODE and the NDDE, whose structures are described, respectively, as:
Structure of the NODEs: the vector ﬁeld is modeled as
[Equation]
where W in ∈ R 32 × 2 , W ∈ R 32 × 32 , and W out ∈ R 2 × 32 ; 2.
Structure of the NDDEs: the vector ﬁeld is modeled as
[Equation]
where W in ∈ R 32 × 2 , W ∈ R 32 × 32 , and W out ∈ R 2 × 32 .
For each model, we choose the optimizer Adam with 1e-3 as the learning rate, 64 as the batch size, and 5 as the number of the training epochs.
We also utilize the synthetic datasets to address the regression problem of the time series.
The optimizer for these datasets is chosen as the Adam with 0 .
as the learning rate and the mean absolute error (MAE) as the loss function.
In this article, we choose the true time delays of the underlying systems for the NDDEs.
We describe the structures of the models for the synthetic datasets as follows.
For the DDE ˙ x
= A tanh( x ( t ) + x ( t − τ )) , we generate a trajectory by choosing its parameters as τ = 0 .
5 , A =
[[ − 1 , 1] , [ − 1 , − 1]] , x ( t ) =
[0 , 1] for t ≤ 0 , the ﬁnal time T = 2 . 5 , and the sampling time period equal to 0 .
We model it using the NODE and the NDDE, whose structures are illustrated, respectively, as:
Structure of the NODE: the vector ﬁeld is modeled as
[Equation]
where W in ∈ R 10 × 2 and W out ∈ R 2 × 10 ; 2.
Structure of the NDDE: the vector ﬁeld is modeled as
[Equation]
where W in ∈ R 10 × 2 and W out ∈ R 2 × 10 .
We use the whole trajectory to train the models with the total iterations equal to 5000 .
For the population dynamics ˙ x
= rx ( t )(1 − x ( t − τ 1 )) with x ( t ) = x 0 for t ≤ 0 and the MackeyGlass systems ˙ x
= β x ( t − τ ) 1+ x n ( t − τ 2 ) − γx ( t ) with x ( t ) =
x 0 for t ≤ 0 , we choose the parameters as τ 1 = 1 , τ 2 = 1 , r = 1 . 8 , β = 4 , n = 9 . 65 , and γ = 2 .
We then generate 100 different trajectories within the time interval [0 , 8] with 0 .
as the sampling time period for both the population dynamics and the Mackey-Glass systems.
We split the trajectories into two parts.
The ﬁrst part within the time interval [0 , 3] is used as the training data.
The other part is used for testing.
We model both systems by applying the NODE, the NDDE, and the ANODE, whose structures are introduced, respectively, as:
Structure of the NODE: the vector ﬁeld is modeled as
[Equation]
where W in ∈ R 10 × 1 , W ∈ R 10 × 10 , and W out ∈ R 1 × 10 ; 2.
Structure of the NDDE: the vector ﬁeld is modeled as
[Equation]
where W in ∈ R 10 × 2 , W ∈ R 10 × 10 , and W out ∈ R 1 ×
Structure of the ANODE: the vector ﬁeld is modeled as
[Equation]
where W in ∈ R 10 × 2 , W ∈ R 10 × 10 , W out ∈ R 2 × 10 , and the augmented dimension is equal to 1 .
Notably, we need to align the augmented trajectories with the target trajectories to be regressed.
To do it, we choose the data in the ﬁrst component and exclude the data in the augmented component, i e., simply projecting the augmented data into the space of the target data.
We train each model for the total 3000 iterations.
B. I MAGE DATASETS
The image experiments shown in this article are mainly depend on the code provided in (Dupont et al., 2019).
Also, the structures of each model almost follow from the structures proposed in (Dupont et al., 2019).
More precisely, we apply the convolutional block with the following strutures and dimensions
• 1 × 1 conv, k ﬁlters, 0 paddings, • ReLU activation function, • 3 × 3 conv, k ﬁlters, 1 paddings, • ReLU activation function, • 1 × 1 conv, c ﬁlters, 0 paddings,
where k is different for each model and c is the number of the channels of the input images.
In
Tab. 2, we specify the information for each model.
As can be seen, we ﬁx approximately the same number of the parameters for each model.
We select the hyperparamters for the optimizer Adam through 1e-3 as the learning rate, 256 as the batch size, and 30 as the number of the training epochs.
C A DDITIONAL RESULTS FOR THE EXPERIMENTS
[Table]
Under review as a conference paper at ICLR
[Figure]
[Figure]
) x
[Figure]
T e s t
T r a i n i ng
NODE
ANODE
NDDE
[Figure]
Under review as a conference paper at ICLR
[Figure]
[Figure] [Figure]