LORD: L OWER -D IMENSIONAL E MBEDDING OF L OGS IGNATURE IN N EURAL R OUGH D IFFERENTIAL E QUATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
The problem of processing very long time-series data (e.g., a length of more than 10,000) is a long-standing research problem in machine learning.
Recently, one breakthrough, called neural rough differential equations (NRDEs), has been proposed and has shown that it is able to process such data.
Their main concept is to use the log-signature transform, which is known to be more efﬁcient than the Fourier transform for irregular long time-series, to convert a very long timeseries sample into a relatively shorter series of feature vectors.
However, the logsignature transform causes non-trivial spatial overheads.
To this end, we present the method of LO we R - D imensional embedding of log-signature (LORD), where we deﬁne an NRDE-based autoencoder to implant the higher-depth log-signature knowledge into the lower-depth log-signature.
We show that the encoder successfully combines the higher-depth and the lower-depth log-signature knowledge, which greatly stabilizes the training process and increases the model accuracy.
In our experiments with benchmark datasets, the improvement ratio by our method is up to 75% in terms of various classiﬁcation and forecasting evaluation metrics.
I NTRODUCTION
Time-series data occurs frequently in real-world applications, e.g., stock price forecasting (Ariyo et al., 2014; Siami-Namini et al., 2018;
Jhin et al., 2021), trafﬁc forecasting (Reinsel, 2003; Fu, 2011; Bai et al., 2020; Fang et al., 2021), weather forecasting (Shi et al., 2015; Seo et al., 2018;
Brouwer et al., 2019; Ren et al., 2021), and so on.
However, it is known that very long timeseries data (e.g., a time-series length of more than 10,000) is not straightforward to process with deep learning despite various techniques ranging from recurrent neural networks (RNNs) to neural ordinary/controlled differential equations (NODEs and NCDEs).
RNNs are known to be unstable when training with such very long sequences and the maximum length that can be processed by NODEs and NCDEs is more or less the same as that by RNNs (Morrill et al., 2021;
Aicher et al., 2020; Trinh et al., 2018; Stoller et al., 2019; Bai et al., 2018).
However, one breakthrough has been recently proposed, namely neural rough differential equations (NRDEs).
NRDEs are based on the rough path theory which was established to make sense of the controlled differential equation:
[Equation]
where X is a continuous control path, and z ( t ) is a hidden vector at time t .
A prevalent example of X is a (semimartingale)
Wiener process coupled with time (i.e. dX t = d ( t, W t )) , in which case the equation reduces to a stochastic differential equation.
In this sense, the rough path theory extends stochastic differential equations beyond the semimartingale environments (Lyons et al., 2004).
One key concept in the rough path theory is the log-signature of a path.
It had been proved that the log-signature of a path with bounded variations is unique under mild conditions (Lyons & Xu, 2018; Geng, 2017) and most time-series data that happens in the ﬁeld of deep learning has bounded variations.
Therefore, one can interpret that the log-signature is a unique feature of the path.
Given a N -length time-series sample { x i } Ni =0 annotated with its observation time-points { t i } Ni =0 , where t 0 = 0 , t N = T , and t i <
t i +1 , NRDEs construct a continuous path X ( t ) , where t ∈ t 0 t 1 tt N ... x 1 xx N x
Path X
Neural RDE
Response
Dimensionality Reduction
NRDE
Dim.
Reduction
Higher-depth z (0) z ( T )
P (a) Directly reducing the dimensionality Path X Response Encoding
NRDE Recovery t 0 t
[Figure]
[Figure]
Neural RDE
Lightweight
[Figure]
Heavyweight Decoder [Figure] Higher
[Figure]
z (0) z ( T )
[Figure]
The dark gray means a processing in a higher-dimensional space and the light gray means that in a lower-dimensional space.
Our embedding method in (b) is better than the baseline in (a) in that i) the higher-dimensional processing is deferred to the decoder which will be discarded after pre-training the autoencoder, and ii) our method does not involve any higher-dimensional processing during the main training and inference processes.
We pre-train the autoencoder and in the main training step, we discard the decoder, ﬁx the encoder, and train only the main NRDE for a downstream task (cf.
Table 1).
In this way, we can exclude the higher-dimensional processing as early as possible.
[0 , T ] , with an interpolation algorithm, where X ( t i ) =
( x i , t i ) for t i ∈ { t i } Ni =0 .
In other words, the path has the same value as the observation ( x i , t i ) , when t i is one of the observation timepoints and otherwise, interpolated values.
As shown in Fig.
1, a log-signature (each dotted yellow box in the ﬁgure) is calculated every P -length sub-path, and another time-series of log-signatures, denoted { LogSig Dr i ,r i +1 ( X ) } (cid:98) TP (cid:99)− 1 i = , is created.
The log-signature calculation has one important hyperparameter D called depth — the higher the depth is, the more accurately represented each sub-path is (cf.
Eq. 4).
For instance, the best accuracy score is 0.81 for D = 3 vs. 0.78 for D = 2 in
EigenWorms . The sub-path length P and the depth D decides the number and the dimensionality of log-signatures, respectively.
However, one downside is dim( LogSig Dr i ,r i +1 ( X )) >
dim( X ( t )) , where dim( x ) means the dimensionality of x .
As a matter of fact, dim( LogSig Dr i ,r i +1 ( X )) grows rapidly w.r.t.
dim( X ( t ))
(Morrill et al., 2021). Since the dimensionality of the input data is, given a dataset, ﬁxed, we need to decrease D to reduce overheads.
In general, NRDEs require more parameters to process higher-dimensional log-signatures (as shown in Table 4 where the original NRDE design always requires more parameters for D = 3 in comparison with D = 2 .)
[Figure]
To this end, we propose to embed the higher-depth signatures onto a lower-dimensional space to i) decrease the complexity of the main
NRDE (the solid blue box in Fig. 1), and ii) increase the easiness of training (cf.
Fig. 2) and as a result, its model accuracy as well.
Fig.
1 shows two possible approaches where we directly reduce the dimensionality of the higher-depth log-signature in Fig.
1 (a) or we adopt the autoencoder architecture to combine both the higherdepth and the lower-depth signatures in Fig.
1 (b) — the ﬁrst approach in Fig.
1 (a) is a baseline model, and our proposed model is the second approach in Fig.
1 (b).
Autoencoders are frequently used for (unsupervised) dimensionality reduction although there also exist other approaches.
Since the higher-depth log-signature should be reconstructed from the embedded vector, one can consider that the higher-depth log-signature is indirectly embedded into the vector.
Moreover, the decoder is discarded after its pre-training so our method does not involve any high-cost computation with the higher-depth log-signature in the main training and inference steps as clariﬁed in Table 1.
[Table]
Our proposed method, LO we R - D imensional embedding of log-signature (LORD), adopts an
NRDE-based autoencoder to combine the higherdepth and the lower-depth log-signature knowledge, and utilizes the embedded knowledge by the encoder for a downstream machine learning task as shown in Fig.
1 (b).
The encoder embeds the lower-depth log-signature from the continuous path X , and the decoder reconstructs the higherdepth log-signature.
This speciﬁc design is where our key idea lies in.
The encoded lower-depth log-signature will contain both the lower-depth and the higher-depth log-signature knowledge because i) it is an encoding of the lower-depth log-signature and ii) the decoder should recover the higher-depth log-signature from it.
After pre-training the autoencoder, we discard the decoder to exclude the higher-dimensional processing as early as possible from our framework.
After that, we ﬁx the encoder and train only the main NRDE.
We conduct experiments with six very long time-series benchmark datasets and ﬁve baselines including NODE, NCDE, and NRDE-based models.
RNN-based models cannot process the very long time-series datasets and we exclude them.
Our proposed method signiﬁcantly outperforms all existing baselines.
Our contributions can be summarized as follows:
We design an NRDE-based continuous autoencoder to combine the higher-depth and the lower-depth log-signature information.
Since the decoder is discarded after being trained, we adopt the asymmetric architecture that the encoder is lightweight and the decoder is heavyweight in terms of the number of parameters.
Our proposed method outperforms all baselines by large margins (up to 75% for various evaluation metrics, e.g., R 2 ) in standard benchmark datasets with much smaller model sizes in comparison with the original NRDE design.
R ELATED W ORK AND P RELIMINARIES
NODEs NODEs (Chen et al., 2018) use the following equation:
[Equation]
which means z ( T ) is solely deﬁned by the initial value z (0) — the entire evolving path from z (0) to z ( T ) is deﬁned by the initial value in ODEs.
NODEs are not considered as a continuous analogue to RNNs but to residual networks (Chen et al., 2018).
NCDEs Let { x i } Ni =0 be an N -length time-series sample and { t i } Ni =0 be its observation timepoints. NCDEs are different from NODEs in that they use the following equation: [Equation] where X ( t ) is a continuous interpolated path from { ( x i , t i ) }
Ni =0 , where t N = T .
In this regards, NCDEs can also be considered as a continuous analogue to RNNs. The adjoint sensitivity method can be used to decrease the space complexity of gradient calculation instead of backpropagation (Kidger et al., 2020). However, NCDEs has an inherent disadvantage that they are weak at processing very long timeseries samples, because NCDEs directly process the very long time-series samples. Signature Transform Let [ r i , r i +1 ] be a time duration, where r i < r
i +1 .
The signature of the time-series sample is deﬁned as follows:
[Equation]
dt
[Equation]
dt
[Equation]
[Table]
Dr i [Table] However, the signature has redundancy, e.g., S 1 , 2 a,b ( X ) + S 2 , 1 a,b ( X ) =
S 1 a,b ( X ) S 2 a,b ( X ) where we can know a value if knowing three others.
The log-signature LogSig Dr i ,r i +1 ( X ) is then created after dropping the redundancy from the signature.
In this work, we use this log-signature deﬁnition in default to design our method.
By creating a log-signature for every P -length sub-path, the entire sequence length can be reduced — moreover, the log-signature is a unique feature of the subpath (Lyons & Xu, 2018;
Geng, 2017).
For complexity reasons, we typically use the log-signature of depth D ≤ 3 (Morrill et al., 2021).
NRDEs Owing to the log-signature transform of time-series, NRDEs (Morrill et al., 2021) are deﬁned as follows:
[Equation]
where LogSig D r i ,r i +1 ( X ) means the log-signature created from the path X within the interval [ r i , r i +1 ) .
LogSig Dri,ri[-+
-]
[Equation]
− r i is a piecewise approximation of the time-derivative of the log-signature in the short interval [ r i , r i +1 ) .
D means the depth of the log-signature.
Once we deﬁne the subpath length P , the intervals { [ r i , r i +1 ) } (cid:98) TP (cid:99)− 1 i = are decided, i.e., P = r
i +1 − r i for all i .
Then, the time-series of LogSig Dr i ,r i +
( X ) constitutes { LogSig Dr i ,r i +1 ( X ) } (cid:98) TP (cid:99)− 1 i = in our notation.
NRDEs use Eq.
5 to derive z ( T ) from z (0) , which can be considered as a continuous analogue to
RNNs since it continuously reads the time-derivative of the log-signature.
Therefore, z ( T ) is deﬁned by the initial value z (0) and the sequence of the time-derivative of the log-signature.
We can also use the adjoint sensitivity method to train NRDEs.
Long Sequence Time-series Input (LSTI)
The problem of processing very long time-series data is a long standing research problem.
There are three ways to solve the LSTI problem.
First, it reduces the sequence through truncating/summarizing/sampling from a very long input sequence.
However, this method may lose information affecting the prediction accuracy.
The second is to give the gradient transformation.
As the sequence becomes longer, the gradient vanishing problem occurs.
Therefore, in (Aicher et al., 2020), the model is trained using only the gradient of the last step.
It also solves the problem using auxiliary losses (Trinh et al., 2018).
Finally, CNNs (Stoller et al., 2019; Bai et al., 2018) were used to solve the LSTI problem.
The convolutional ﬁlter captures long term dependencies, but the receptive ﬁelds increase exponentially, breaking the sequence.
P ROPOSED M ETHOD
We describe our proposed method, LORD-NRDE.
We ﬁrst clarify the motivation of our work and then describe our proposed model design.
Motivation It is known that NRDEs are a generalization of NCDEs (Morrill et al., 2021,
Section 3.2).
However, one problem in utilizing NRDEs in real-world environments is that dim( LogSig Dr i ,r i +1 ( X )) is a rapidly growing function of dim( X ( t ))
(Morrill et al., 2021, Section A).
This rapid blow-up of dimensionality causes two problems : i) it hinders us from applying
NRDEs to high-dimensional time-series data, and ii) it makes the training process complicated since the hidden representation of the large input should be made for a downstream task.
Therefore, we propose to embed the log-signature onto a lower dimensional space so that the training process becomes more tractable and enhance the applicability of NRDEs to high-dimensional data.
After the embedding, in other words, the dimensionality of embedded vector is typically smaller than that of the log-signature with depth D [-
-]2 in our setting but it has knowledge of the log-signature with depth D 2 > D 1 .
[Figure]
Input: Training data D train , Validating data D val , Maximum iteration numbers max iter AE and max iter TASK 1 Initialize the parameters of the encoder (i.e., θ f , θ φ e ), the decoder (i.e., θ o , θ φ s ), and the main NRDE (i.e., θ g , θ φ z , θ φ output ). * Pre-training of the autoencoder */ 2 for max iter AE iterations do Train the encoder and the decoder using [Table]
[Table]
-training of
[Table]
-NRDE */ 4 for max iter [Table] iterations do Train the
[Table]
NRDE using
[Table]
Validate the best main NRDE parameters with D
[Table]
; 7 return the encoder and the main NRDE parameters;
Overall Architecture We describe our proposed method, LORD-NRDE, which consists of three
NRDEs: i) an encoder NRDE, ii) a decoder NRDE and iii) a main NRDE to derive z ( T ) from z (0) .
The role of each NRDE is as follows:
The encoder NRDE continuously embeds the log-signature with depth D 1 onto another space.
We use e ( t ) to denote the vector embedded from the log-signature at time t . 2.
The decoder NRDE reconstructs the log-signature with depth D 2 > D 1 from e ( t ) .
The main NRDE reads e ( t ) to evolve z ( t ) and derive z ( T ) .
There is an output layer which reads z ( T ) and makes inference.
We note that dim( LogSig D 2 r i ,r i +1 ( X )) >
dim( e ( t )) , where t ∈ [0 , T ] .
We require that the encoder produces embedding vectors that contain both the log-signature knowledge with D 1 and D 2 depth.
LORD-NRDE
To this end, we propose the following method:
[Equation]
where dim( e ( t )) < dim( s ( t ))
=
dim( LogSig D 2 r
i ,r i +1 ( X )) , where t ∈ [ r i , r i +1 ) , with D 2 > D 1 .
The term d e ( t ) can be considered as an embedding of the log-signature with depth D 1 at time t , and e ( t ) as an embedding of X ( t ) constructed by the log-signature with depth D 1 .
Similarly, d s ( t ) is a reconstructed log-signature with depth D 2 from the embedding, and s ( t ) as X ( t ) reconstructed by the log-signature with depth D 2 .
Since e (0) and s (0) mean the initial values, we set φ e ( X (0); θ φ e ) , φ s ( e (0); θ φ s ) , where φ means a transformation function.
Therefore, e ( t ) and s ( t ) constitute an autoencoder based on NDREs as shown in Fig.
In our design, we use the asymmetrical setting for them, where the encoder RDE is lightweight and the decoder RDE is heavyweight in terms of the number of parameters, considering the applicability of our design to real-world environments.
Since the decoder NRDE is discarded after its pre-training, the heavyweight setting causes no problems afterwards — our method requires more resources for the pre-training process though.
There are three functions g , f , and o in Eqs.
6 to 8 and their deﬁnitions are as follows:
[Equation]
where FC h is a fully-connected (FC) layer whose output size is h .
N g , N f , N o means the number of
FC layers.
ψ and η are the hyperbolic tangent and rectiﬁed linear unit activations, respectively.
Training Method To train the NRDE-based autoencoder, we use the following loss:
[Equation]
where M = (cid:98) TP (cid:99) − 1 , and c AE is a coefﬁcient of the L 2 regularization terms.
c e also regularizes the scale of the learned embedding.
In addition to L AE , we also have a task loss L TASK which is the standard cross-entropy (CE) loss or the mean squared loss (MSE) loss.
From z ( T ) , we have an output layer, which is typically a fully-connected layer with an appropriate ﬁnal activation function such as softmax, to produce a prediction ˆ y .
θ φ output denotes the parameters of the output layer.
Using the ground-truth information y , we deﬁne the task loss L TASK .
Therefore, the ﬁnal loss can be written as follows:
[Equation]
[Equation] where c TASK is a coefﬁcient of the L 2 regularization terms, and CE ( y, ˆ y ) means a cross-entropy loss — we assume classiﬁcation but it can be accordingly changed for other tasks.
To implement, we deﬁne the following augmented ODE:
[Equation]
where we replace d e ( t ) dt with f ( e ( t ); θ f ) LogSig D 1 {+ri,ri +
+}
( X ) r
i +1 − r i (as deﬁned in Eq. 7), and the initial values are deﬁned as follows:
[Equation]
where φ z and φ e are transformation functions to produce the initial values from the initial observation X (0) , and φ s is a transformation function to produce the initial reconstructed value from the initial embedded vector e (0) .
We use a fully-connected layer for each of them.
Alg.
1 shows the training algorithm.
In Line 3, we pre-train the autoencoder for max iter AE iterations.
After discarding the decoder and ﬁxing the encoder, we then train the main NRDE in Linefor max iter TASK iterations.
Using D val , we validate and choose the best parameters.
E XPERIMENTAL E VALUATIONS
We describe our experimental environments and results.
The mean and variance of 5 different seeds are reported for model evaluation.
We refer to Appendix A for reproducibility.
4. E XPERIMENTAL E NVIRONMENTS
Datasets We use six real-word dataset which all contain very long time-series samples.
There are 3 classiﬁcation datasets in the University of East Anglia ( UEA ) repository (Tan & Webb):
EigenWorms , CounterMovementJump , and SelfRegulationSCP2 .
EigenWorms has a time-series length of 17,984 and a channel of 7 which contains the movement data of roundworms.
The goal is to classify each worm among 5 worm types.
CounterMovementJump has 4,250 length and 4 channels which means accelerations data of each 3D-axis.
Using accelerations data, the type of jump is predicted among 3 types.
The object of SelfRegulationSCP2 is to classify whether the subject moves the computer’s cursor up or down, using 8 channels of the EEG data.
Its time-series length is 1,153.
For forecasting, three Beth Israel Deaconess Medical Centre ( BIDMC ) datasets are used, which come from the TSR archive (Tan & Webb).
Using the PPG and ECG information, each task is to predict a person’s heart rate ( HR ), respiratory rate ( RR ), or oxygen saturation ( SpO2 ), respectively.
The time-series length is 4,000.
Baselines There are three types of baselines in our Experiments, ranging from NODE to NCDE and NRDE-based baseline models.
ODE-RNN is one of the state-of-the-art NODE models in processing time-series.
Following the suggestion in (Morrill et al., 2021), we merge P observations into one merged observation and feed it into ODE-RNN , in which case ODE-RNN is able to process much longer time-series. The dimensionality of the merged observation is P times larger than that of the original observation or equivalently, the length of the merged time-series is P times shorter than that of the original one.
NCDE is the original NCDE design in (Kidger et al., 2020).
Attentive NCDE ( ANCDE ) is an extension of NCDE by adding an attention mechanism into it, which signiﬁcantly outperforms NCDE
(Jhin et al., 2021).
For the NRDE-type baselines, we consider i) the original NRDE design (Morrill et al., 2021), and ii) the one in Fig.
1 (a) which directly embeds the higherdepth log-signature into a lower-dimensional space, each of which is called NRDE and DE-NRDE , respectively.
For those NRDE-based models, we clarify its one important hyperparameter, depth , as part of its name, e.g., NRDE 2 means NRDE with D = 2 .
One important point is that NRDE is a generalization of NCDE .
Therefore, NRDE 1 is theoretically identical to NCDE , if using the linear interpolation method, and for this reason, we set D > 1 for NRDE .
Hyperparameters When using NRDEs, there is one crucial hyperparameter, the log-signature depth D .
We use two depth settings which were used in (Morrill et al., 2021).
For our LORD-NRDE, there are two depths, D 1 and D 2 , where D 1 < D 2 .
LORD D 1 → D 2 means that LORD-NRDE’s decoder recovers the D 2 -depth log-signature from the D 1 -depth log-signature.
The number of layers in the encoder, decoder and main NRDE, N g , N f , and N o of Eqs.
9 to 11, are in { 2, 3 } .
The hidden sizes, h g , h f , and h o of Eqs.
9 to 11, are in { 32, 64, 128, 192 } .
The coefﬁcients of the L 2 regularizers in Eqs.
13 and 14 are in { 1 × 10 − 5 , 1 × 10 − 6 } .
The coefﬁcient of the embedding regularizer, c e in Eq.
13 is in { 0, 1, 10 } .
The max iteration numbers, max iter AE and max iter TASK in Alg. 1, are in { 400, 500, 1000, 1500, 2000 } .
The learning rate of the pre-training and main-training is 1 × 10 − 3 .
We also set dim( e ( t )) = dim( LogSig D 1 r i ,r i +1 ( X )) , where t ∈ [ r i , r i +1 ) . We also conduct experiments by setting the sub-path length P to 4, 8, 32, 64, 128, 256, or 512 observations.
In other words, we create one log-signature for every P input observation for NCDE and
NRDE-based models.
For ODE-RNN , as described earlier, we simply concatenate P observations into one observation.
The ﬁnal time T is large in our experiments, e.g., T > 10 , 000 , and the number of log-signatures is (cid:98) TP (cid:99) .
We test both the adjoint sensitivity method and the backpropagation through the solver.
Evaluation Methods We reuse the very long time-series classiﬁcation and forecasting evaluation methods of (Morrill et al., 2021) and extend the methods by adding more datasets and more evaluation metrics.
We use accuracy, macro F1, and ROCAUC for binary classiﬁcation; accuracy, macro/weighted F1, and ROCAUC for multi-class classiﬁcation; and R 2 , explained variance, mean squared error (MSE), and mean absolute error (MAE) for forecasting — we list the complete results in Appendix C after introducing key results in the main manuscript.
We also show the number of parameters for each model — for our LORD , we exclude the parameter numbers of the decoder since it is discarded after the pre-training.
We train and test each model 5 times with different seeds.
If the mean of score is the same, the smaller standard deviation is better.
4. E XPERIMENTAL R ESULTS
Summary of Experimental Results Since our main result tables have many items, we quickly summarize their highlights in Tables 2 and 3.
To calculate the improvement in evaluation metrics, we use the ratio of improvement over NRDE 2 for each metric averaged over all the sub-path lengths and all datasets, e.g., we calculate
[Equation]
and
[Equation]
for all classiﬁcation cases and average them.
The ratio of the number of parameters is calculated similarly.
Overall, LORD outperforms other baselines with smaller numbers of parameters.
LORD D 1 → D 2 has a larger model size, excluding its decoder, than NRDE D 1 whereas it has a much smaller model [Table] [Table] size than NRDE D 2 .
However [Table] the performance of LORD D 1 → D 2 is similar to or better than that of NRDE
D [Table] which proves the efﬁcacy of our method.
Using the encoder-decoder structure [Table] the high complexity of processing log-signatures can be reduced and it makes our model smaller and more amenable to train [Table]
In many cases for LORD [Table] the adjoint method and the backpropagation method are comparable in terms of model accuracy.
Interestingly [Table] we found that the backpropagation through the Euler method is fast enough with a neglectable sacriﬁce of accuracy for several cases. Detailed Experimental Results Table 4 show detailed results for some selected evaluation metrics — full tables with all metrics are in Appendix C.
One point is that many best outcomes are made with moderate P settings.
For EigenWorms , ODE-RNN can’t achieve good scores for all P settings.
Our proposed model, LORD , achieves the best scores.
In particular, LORD 1 → 3 with P = 32 has a much smaller number of parameters, compared with other baselines that have similar performance.
LORD 2 → 3 also signiﬁcantly outperforms NRDE 3 for both accuracy and mode size.
For CounterMovementJump and SelfRegulationSCP2 , ODE-RNN is better than other NRDE and NCDE -based baselines.
However, LORD marks the best scores in all cases.
In all BIDMC experiments, LORD shows outstanding performance.
LORD 2 → 3 shows the best performance in almost all cases.
Compared with NCDE and NRDE , DE-NRDE has 40 ∼ 60% improvements and LORD has 50 ∼ 70% improvements.
However, LORD ’s model size is reduced by 60% whereas
DE-NRDE ’s model size is reduced by 20%.
Therefore, our proposed method is a better embedding method for log-signatures.
Unlike EigenWorms , the time-series length in this dataset is rather short.
For that reason, ODE-RNN performs better.
Overall, DE-NRDE 3 with P = 128 has the best performance, among the baseline models.
This means that the training difﬁculty by the large logsignature size can be alleviated by the direct embedding.
However, LORD outperforms DE-NRDE .
[Figure]
Additional Results and Visualization Fig.
4 visually compares the three log-signatures and we note that the embedded signature of d e ( t ) has the characteristics of both D 1 = 1 and D 2 = 3 .
We refer to our supplementary material for other visualization, detailed results, sensitivity analyses, and reproducibility information.
L IMITATIONS AND C ONCLUSIONS
The log-signature transform of NRDEs is suitable to irregular and long time-series.
However, the log-signature transform requires larger memory footprints and to this end, we presented an autoencoder-based method to embed their higher-dimensional log-signature into a lowerdimensional space, called LORD .
Our method is carefully devised to eradicate the higherdimensional computation as early as right after the pre-training of the autoencoder.
In the standard benchmark experiments of very long time-series, our proposed method signiﬁcantly outperforms existing methods and shows relatively smaller model sizes in terms of the number of parameters.
Nonetheless, it is still in its early phase to enhance NRDEs and there exist a couple of items to be studied further, e.g., processing long time-series with many channels. [Table]
E THICS S TATEMENT
Time-series classiﬁcation and forecasting is closely related to our daily life, ranging from stock price forecasting, weather forecasting to patient mortality classiﬁcation and so on.
Therefore, our enhanced technology has many beneﬁts to our society.
It is hard to say that deep neural networks processing time-series have ethical problems.
R EPRODUCIBILITY S TATEMENT
We include detailed reproducibility information in Appendix A.
In particular, we clarify the neural network architectures we used and their best hyperparameter sets for all experiments in our paper.
We also submit our source codes and data with appropriate shell scripts to easily reproduce some selected results due to the 100MB limitation of the supplementary zip ﬁle.
A B EST H YPER P ARAMETERS
Our software and hardware environments are as follows: U BUNTU 18.04 LTS, P YTHON 3.7.10,
P YTORCH 1.8.1, CUDA 11.4, and NVIDIA Driver 470.42.01, i9 CPU, and NVIDIA RTX A6000.
A. B ASELINE
There are 5 baselines in our experiments.
Each model has its own hyerparameters.
ODE-RNN method has a hyperparameter of hidden path size.
NCDE and NRDE have also the same hyperparameter, and in addition, the size of hidden dimension and the number of hidden layers in their CDE and
RDE functions.
The Attentive NCDE ( ANCDE ) has all of those hyperparameters and in addition, the attention channel size.
In DE-NRDE , there are addition hyperparameters related to embedding, which are the embedding compression ratio, the size of hidden dimension, and the number of hidden layers in the embedding layer.
In (Morrill et al., 2021), there are three models , ODE-RNN , NCDE and NRDE , and four datasets,
EigenWorms , BIDMC32HR , BIDMC32RR , and BIDMC32SpO2 .
For these combinations, we set the hyperparameters as reported in (Morrill et al., 2021).
In other cases, we ﬁnd the best hyperparameters in the following range.
All kinds of hidden size in ODE-RNN , NCDE , ANCDE , and NRDE are in { 32, 64, 128, 256 } .
The number of hidden layers in those is in { 2,3 } .
In DE-NRDE , the embedding compression ratio, the hidden size, and the number of hidden layers in the embedding layer are in { 0.3, 0.5, 0.7 } , { 64, 128 } , and { 1,2 } , respectively.
Tables 5 to 8 show the best hyperparameters of the baselines.
A. LORD-NRDE
Table 9 shows the best hyperparameter conﬁguration of our method in each dataset.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
B V ISUALIZATION
We show other PCA based visualizations of the log-signatures.
Using PCA, we extract the distributions of the most important principle components of { LogSig D 1 r i ,r i +1 ( X ) } (cid:98) TP (cid:99)− 1 i = , { LogSig D 2 r i ,r i +1 ( X ) } (cid:98) TP (cid:99)− 1 i = , and d e ( t ) .
If two paths have similar distributions on those components, the information contained by the two paths is similar.
In Fig.
5, the distribution of d e ( t ) is somewhere in between the distributions of { LogSig D 2 r i ,r i +1 ( X ) } (cid:98) TP (cid:99)− 1 i = and { LogSig D 1 r
i ,r i +1 ( X ) } (cid:98) TP (cid:99)− 1 i = .
From the encoder-decoder learning, d e ( t ) can learn the information of { LogSig D 2 r
i ,r i +1 ( X ) } (cid:98) TP (cid:99)− 1 i = .
[Figure]
[Equation]
[Figure]
[Table]
[Figure]
[Table]
(e)
[Figure]
[Table] [Figure]
D
[Figure]
r i ,r i +1 ( X ) } (cid:98) TP (cid:99)− 1 i = , and d e ( t ) in
EigenWorms and BIDMC .
C F ULL E XPERIMENTAL RESULTS
In Tables 10 to 15, all experimental results are shown.
#Params(D) and #Params(R) denote the number of parameters in the decoder part and in the rest parts except the decoder, respectively.
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Figure]
D S ENSITIVITY E XPERIMENTS
Fig.
6 and Table 16 show that in general, the performance of LORD is improved as max iter AE gets larger in EigenWorms ( P = 128 ).
Our encoder-decoder structure can successfully integrate the lower-depth log-signature and the higher-depth log-signature information into the embedding vector e
[Figure]
and the well-integrated information helps the model perform better.
In general, the lower-depth and the higher-depth log-signature information are blended better as the encoder-decoder structure is more trained
[Figure]
E T RAIN L OSS
Fig.
7 visualizes several training cases for our method and the original NRDE design.
In general, our method shows much better stability across the entire training period.
F E ND - TO -E ND T RAINING
LORD has two distinguished characteristics in its training process.
First, LORD uses a two-phase training strategy, a pre-training and a main training processes.
Second, the training process of LORD is not end-to-end.
In other words, the decoder is not used and the encoder is ﬁxed in the main training phase.
In this section, we test with various end-to-end-training settings for LORD . There are three possible conﬁgurations for the end-to-end-training. The ﬁrst conﬁguration is training the encoder (rather than ﬁxing it) with L TASK in the main training phase, which is denoted as FineTuning .
The second one is Co-Train where both L AE and L TASK are used for training the encoder, the decoder and the main NRDE — recall that in FineTuning , we use only L TASK
[Figure]
[Table]
to train the encoder.
The last method is Co-Train(w
[Table]
o
[Table]
pre) which is the same as Co-Train without any pre-training process.
The same hyperparameters are used for all end-to-end models.
We experiment with EigenWorms ( P = 128 ).
Table 16 shows the results.
Among the three endto-end models, Co-Train shows the best performance.
Co-Train(w
[Table]
o. pre) has the worst performance in general.
These results prove that the pre-training makes the main training easier.
[Table]
[Table]
[Table]
[Table]
[Table]
G A DDITIONAL E XPERIMENTS WITH SHORT TIME - SERIES
In this section
[Table]
we experiment with short time-series datasets.
We use CharacterTrajectory and LiveFuelMoistureContent from (Tan & Webb)
[Table]
The object of CharacterTrajectory is to classify 20 characters using the trajectory information of writing on a tablet.
Its length is 182.
LiveFuelMoistureContent consists of daily reﬂectance data at 7 spectral bands for predicting moisture rate in vegetation.
Its length is 365.
In Tables 18 to
[Table]
we summarize hyperparameters.
Because of the short time-series lengths, we test P = 1 for all except NRDE -based models.
In Table
[Table]
NRDE achieves the best scores among baseline models and LORD achieves the best scores among all methods.
In Table 24, DE-NRDE 2 is the best.
Except DE-NRDE 2 , ODE-RNN is the best.
However, LORD also has comparable scores.
From these results, NRDE and NRDE -based models are empirically useful for short-time series as well.
[Table]
[Table]
[Table]