GT-GAN: General Purpose Time Series Synthesis with Generative Adversarial Networks
Abstract
Time series synthesis is an important research topic in the ﬁeld of deep learning,which can be used for data augmentation.
Time series data types can be broadlyclassiﬁed into regular or irregular.
However, there are no existing generativemodels that show good performance for both types without any model changes.
Therefore, we present a general purpose model capable of synthesizing regularand irregular time series data.
To our knowledge, we are the ﬁrst designing ageneral purpose time series synthesis model, which is one of the most challengingsettings for time series synthesis.
To this end, we design a generative adversarialnetwork-based method, where many related techniques are carefully integrated intoa single framework, ranging from neural ordinary/controlled differential equationsto continuous time-ﬂow processes.
Our method outperforms all existing methods
Introduction
Time series data occurs frequently in real-world applications
[Reinsel, 2003, Fu, 2011, Li et al., 2018,
Yu et al., 2018, Wu et al., 2019, Guo et al., 2019, Bai et al., 2019, Song et al., 2020, Huang et al., 2020a,
Ren et al., 2021, Tekin et al., 2021].
Among many tasks related to time series, synthesizing time seriesdata is one of the most important tasks because real-world time series data is frequently imbalancedand/or insufﬁcient.
Since regular and irregular time series data have different characteristics, however,different model designs had been adopted for them.
Therefore, existing time series synthesis workfocuses on either regular or irregular time series synthesis [Yoon et al., 2019, Alaa et al., 2021].
Toour knowledge, there are no existing methods that work well for both types.
Regular time series means regularly sampled observations without any missing ones, and irregulartimes series means that some observations are missing from time to time.
Irregular time series is muchharder to process than regular time series.
For instance, it is known that neural networks performbetter after transforming time series data into its frequency domain, i.e., the Fourier transform, andsome time series generative models use this approach [Alaa et al., 2021].
However, it is not easy toobserve pre-determined frequencies from highly irregular time series [Kidger et al., 2019]. However,continuous-time models [Chen et al., 2018, Kidger et al., 2020, Brouwer et al., 2019] show goodperformance in processing both regular and irregular time series.
By resorting to them, we propose ageneral purpose model that can synthesize both time series types without any model changes.
To achieve the goal, we design a sophisticated model which utilizes a diverse set of technologies,ranging from generative adversarial networks (GANs [Goodfellow et al., 2014]), and autoencoders(AEs) to neural ordinary differential equations (NODEs [Chen et al., 2018]), neural controlled differential equations (NCDEs [Kidger et al., 2020]), and continuous time-ﬂow processes (CTFPs [Denget al., 2020]), which reﬂects the difﬁculty of the problem.
Fig.
1 shows the overall design of our proposed method.
One of the key points in our model designis that we combine the adversarial training of GANs and the exact maximum likelihood training of
Encoder realorfake
[Figure]
AE
GAN
Log-density
Synthesis
[Figure]
CTFPs into a single framework.
However, the exact maximum likelihood training is applicable onlyto invertible mapping functions whose input and output sizes are the same.
Therefore, we designan invertible generator, and adopt an autoencoder, on whose hidden space our GAN performs theadversarial training.
[Figure]
In other words, i) the hidden vectorsize of the encoder is the same as thenoisy vector of the generator, ii) thegenerator produces a set of fake hidden vectors, iii) the decoder convertsthe set into a fake continuous path(cf.
Fig. 2), and iv) the discriminatorprovides feedback after reading thesampled fake sample.
We note thatin the third step, a fake continuouspath is created by the decoder.
Therefore, we can sample any arbitrary regular/irregular time series sample from the fake path, which shows the ﬂexibility in our method.
We conduct experiments with 4 datasets and 7 baselines.
Since our method is able to support both theregular and irregular time series synthesis, we test for both of them.
Our method outperforms otherbaselines in both environments.
Our contributions can be summarized as follows:
We design a model based on various state-of-the-art deep learning technologies.
Our method
is able to process any types of time series data, ranging from regular to irregular, without
any model changes.
Our experimental results and visualization prove the efﬁcacy of the proposed model.
Since our task is one of the most challenging tasks for time
series synthesis, the proposedmodel architecture is carefully designed.
Our ablation studies show that our proposed modeldoes not work well if any part is missing.
Related work and preliminaries
GANs are one of the most representative generative technology.
Ever since the ﬁrst introduction in itsseminal research paper, GANs have been adopted to main different domains.
Recently, researchersfocused on their synthesis for time series data.
Therefore, there have been proposed several GANs fortime series synthesis.
C-RNN-GAN [Mogren, 2016] has a regular GAN framework that can be appliedto sequential data by using LSTM in its generator and discriminator.
Recurrent Conditional GAN(RCGAN
[Esteban et al., 2017]) took a similar approach except that its generator and discriminatortake conditional input for better synthesis.
WaveNet [van den Oord et al., 2016] also generates timeseries data from the conditional probability of previous data by using the dilated casual convolution.
WaveGAN [Donahue et al., 2019] has a similar approach with DCGAN
[Radford et al., 2016], whereits generator is based on WaveNet.
We can modify the teacher-forcing (T-Forcing [Graves, 2014])and professor-forcing (P-Forcing [Lamb et al., 2016]) models to generate time series data from noisevectors, although they are not GAN models, by using the forecasting characteristic of those models.
[Figure]
Proposed method
In this section, we describe our design.
Since our general purpose time series synthesis is a challengingtask, the proposed design is much more complicated than other baselines.
3.1 Overall workﬂow
We ﬁrst describe the overall workﬂow in our model design, which consists of several different datapaths (and several different training methods based on the data paths) as follows:
Autoencoder path:
Given an time series sample { ( t i , x reali ) } Ni =0 , the encoder producesa set of hidden vectors { h reali } Ni =0 .
The decoder recovers a continuous path ˆ X real ,which enhances the ﬂexibility of our proposed method.
From the path ˆ X real , we sample { ( t i , ˆ x reali ) } Ni =0 .
We train the encoder and the decoder using the standard autoencoder(AE) loss to match x reali and ˆ x reali for all i .
Adversarial path:
Given a set of noisy vectors { z i } Ni =0 , our generator produces a set offake hidden vectors { h fakei } Ni =0 .
The decoder recovers a fake continuous path ˆ X fake from{ h fakei } N i =0 .
We sample { ( t j , x fakej ) } M j =0 from ˆ X fake and feed it into the discriminator.
For irregular time series synthesis, we sample t j in [0 , T ] .
We train the generator, thedecoder, and the discriminator using the standard adversarial loss.
Log-density path:
Given a set of hidden vectors { h reali } Ni =0 for an time series sample
{ ( t i , x reali ) } Ni =0 , the inverse path of the generator reproduces a set of noisy vectors { ˆ z i } Ni =
.We feed { ˆ z i } Ni =0 into its forward path again to reproduce { ˆ h reali } Ni =0 , where ˆ h reali = h reali
for all i .
During the forward pass, we calculate the negative log probability of − log p (ˆ h reali )for all i with the change of variable theorem and minimize it for training, being inspired
by Grover et al.
[2018] and Deng et al.
In particular, we note that the dimensionality of the hidden space in the autoencoder is the same asthat of the latent input space of the generator, i.e., dim( h )
= dim( z ) .
This is needed for the exactlikelihood training in the generator — the change of variable theorem requires that the input andoutput sizes are the same to estimate the exact likelihood.
In addition to it, we let the autoencoderand the generator share the workload to synthesize fake time series by combining them into asingle framework, i.e., the generator synthesizes fake hidden vectors and the decoder reproduceshuman-readable fake time series from them.
3.2 Autoencoder
Encoder General NCDEs, which are considered as a continuous analogue to recurrent neuralnetworks (RNNs), are deﬁned as follows:
[Equation]
where X ( t ) is a continuous path created from a raw discrete time series sample { ( t i , x reali ) } Ni =0 by aninterpolation
algorithm — we note that X ( t i ) =
( t i , x reali ) for all i , and for other non-observed timepoints the interpolation algorithm ﬁlls out values.
Note that NCDEs keep reading the time-derivativeof X ( t ) , denoted ˙
X ( t ) def = dX ( t ) dt .
In our case, we collect { h reali } Ni =0 as follows:
[Equation]
where h real= FC dim( x ) → dim( h ) ( x real) and FC input _ size → output _ size is a fully-connected layer with 117 speciﬁc input and output sizes.
We refer to Appendix B for the ODE function f deﬁnition.
Therefore, the input time series { ( t i , x reali ) } Ni =0 is represented by a set of hidden vectors{ ( t i , h reali ) } Ni =0 .
Because NCDEs are a continuous analogue to RNNs, it shows the best ﬁt toprocessing irregular time series [Kidger et al., 2020].
Decoder Our decoder, which reproduces a time series from its hidden representations, is based on
GRU-ODEs [Brouwer et al., 2019] and is deﬁned as follows:
[Equation]
=
d ( t i ) + (cid:90) t i + t
i g ( d ( t )
[Equation]
t
[Equation]
θ g ) dt, (3) d ( t i +1 ) =
GRU ( h i +1 , ¯ d ( t i +1 )) , (4) ( t i +1 , ˆ
[Equation]
i
[Equation]
) =
( t i +
[Equation]
FC dim( d ) → dim( x ) ( d ( t i +1 ))) , (5) where d ( t 0 ) =
FC dim( h ) → dim( d ) ( h 0 ) and h
i means either the i -th real or fake hidden vector, i.e.,h reali or h fakei — recall that in Fig.
3, the decoder is involved in both the autoencoder and thesynthesis processes.
ˆ x means a reproduced copy of x .
GRU-ODEs uses the technology called neuralordinary differential equations (NODEs) to continuously interpret GRUs and we refer to Appendix Bfor the ODE function g deﬁnition.
In particular, the gated recurrent unit (GRU) at Eq.
(4) is called as jump which is known to be effectivein processing time series with NODEs [Brouwer et al., 2019, Jia and Benson, 2019].
We train theencoder-decoder using the standard reconstruction loss between x reali and ˆ x reali for all i in all trainingtime series samples.
3.3 Generative adversarial network
[Figure]
Generator Whereas generators typically read a noisy vector togenerate a fake sample in standard GANs, our generator reads acontinuous path (or time series) sampled from a Wiener processto generate a fake time series sample — this generation conceptis known as continuous time ﬂow processes (CTFPs [Deng et al.,2020]).
Fig.
4 shows an example of our generation process.
Theinput to our generation process is a random path sampled froma Wiener process, which is represented by a time series of latentvectors { ( t i , z i ) } N i =0 in the path, and the output is a path of hiddenvectors which is also represented by a time series of hidden vectors { ( t i , h fakei ) } Ni =0 .
Therefore, ourgenerator can be written as follows:
[Equation]
where w i (0)
=
z i , a i (0) =
t i .
Here, τ means a virtual time variable of the integral problem, and t iis a real physical time contained in a time series sample { ( t i , x reali ) } Mi =0 .
We note that this designcorresponds to a NODE model augmented with a i ( t ) .
We refer to Appendix B for the ODE functionr deﬁnition.
Owing to the invertible nature of NODEs, we can calculate the exact log-density of h reali , i.e., theprobability that h reali is generated by the generator, using the change of variable theorem and the
Hutchinson’s stochastic trace estimator as follows [Grathwohl et al., 2019, Deng et al., 2020]:
[Equation]
where ˆ w (0) means ˆ z reali in Fig.
ˆ h reali means a reproduced copy of h reali by our generator.
Eq.
(7)corresponds to “CTFP − 1 ”, and Eqs.
(6) and (8) to “CTFP” in Fig.
We note that in Eq.
(7), theintegral time is reversed to solve the reverse-mode integral problem.
Therefore, we minimize the negative log-density, denoted − log Pr(ˆ h reali ) , for each t i , and ourgenerator is trained by the two different training paradigms: i) the adversarial training against thediscriminator, and ii) the maximum likelihood estimator (MLE) training with the log-density.
Discriminator We design our discriminator based on the GRU-ODE technology as follows:
[Figure]
[Equation]
+1 )
=
c ( t i ) + (cid:90) t i
+ t i q ( c ( t ) , t
[Equation]
θ q ) dt, (9) c ( t
[Equation]
) =
GRU (
[Equation] [Equation]
c ( t
[Equation]
+1 ))
, (10) where c ( t 0 ) =
FC dim( x ) → dim( c ) ( x 0 ) , and x i means the i -th time series value, i.e., x reali or x fakei .
The ODE function q has the same architecture as g but with its own parameters θ q .
After that, wecalculate the real or fake classiﬁcation y =
σ ( FC dim( c ) → 2 ( c ( t N ))) , where σ is a softmax activation.
3.4 Training method
We use the mean squared reconstruction loss, i.e., the mean of (cid:107) x reali − ˆ x reali (cid:107) 22 for all i , to train theencoder-decoder architecture.
Then, we use the standard GAN loss to train the generator and thediscriminator.
In our preliminary experiments, we found that the original GAN loss is suitable for ourtask.
Instead of other variations, such as WGAN-GP [Gulrajani et al., 2017], therefore, we use thestandard GAN loss.
We train our model in the following sequence:
We pre-train the encoder-decoder networks the reconstruction loss for K AE iterations.
After the above pre-training step, we start to jointly train all networks in the following
sequence for K JOINT iterations: i) training the encoder-decoder networks with the re-
construction loss, ii) training the discriminator-generator networks with the GAN loss, iii)
training the decoder to improve the discriminator’s classiﬁcation output with the discrimina-
tor loss and iv) the generator with the MLE loss every P MLE iteration.
We found that too
frequent MLE training incurs mode-collapse so we use it every P MLE iteration.
In particular, the 2-ii step to train the decoder to help the discriminator out is one additional pointwhere the autoencoder and the GAN are integrated into a single framework.
In other words, thegenerator should deceive both the decoder and the discriminator.
The well-posedness 1 of NCDEs and GRU-ODEs was already proved in Lyons et al.
[2007, Theorem1.3] and Brouwer et al.
[2019] under the mild condition of the Lipschitz continuity.
We show thatour NCDE layers are also well-posed problems.
Almost all activations, such as ReLU, Leaky ReLU,
SoftPlus, Tanh, Sigmoid, ArcTan, and Softsign, have a Lipschitz constant of 1.
Other commonneural network layers, such as dropout, batch normalization and other pooling methods, have explicit
Lipschitz constant values.
Therefore, the Lipschitz continuity of ODE/CDE functions can be fulﬁlledin our case.
In other words, it is a well-posed training problem.
As a result, our training algorithmsolves a well-posed problem so its training process is stable in practice.
Experimental evaluations
Our software and hardware environments are as follows: U BUNTU 18.04 LTS, P YTHON 3.8.10,
P YTORCH 1.8.1, T ENSORFLOW 2.5.0, CUDA 11.2, and NVIDIA Driver 417.22, i9 CPU, and
NVIDIA RTX 3090.
The mean and variance of 10 runs are reported for model evaluation.
4.1 Experimental environments
Datasets We conduct experiments with 2 simulated and 2 real-world datasets.
Sines has 5 featureswhere each feature is created with different frequencies and phases independently.
For each feature,i ∈ { 1 , ..., 5 } , x i ( t ) =
sin (2 πf i t
+ θ i ) , where f i ∼ U [0 , 1] and θ i ∼ U [ − π, π ] .
MuJoCo ismultivariate physics simulation time series data with 14 features.
Stocks is the Google stock pricedata from 2004 to 2019.
Each observation represents one day and has 6 features.
Energy is a UCIappliance energy prediction dataset with 28 values.
To create the challenging irregular environments,30, 50, 70% of observations from each time series sample { ( t i , x reali ) } Ni =0 is randomly dropped —in other words, N decreases to 0 .
7 N, 0 .
5 N, 0 .
3 N .
Dropping random values has been mainly used tocreate irregular time series environments in the literature [Kidger et al., 2019, Xu and Xie, 2020,
Huang et al., 2020b, Tang et al., 2020, Zhang et al., 2021, Jhin et al., 2021, Deng et al., 2021].
Therefore, we conduct experiments with both the regular and the irregular environments.
Baselines We consider the following baselines for the regular time series experiments: TimeGAN,
RCGAN, C-RNN-GAN, WaveGAN, WaveNet, T-Forcing, and P-Forcing.
For the irregular experiments, we exclude WaveGAN and WaveNet, which cannot handle irregular time series, and redesignother baselines by replacing their GRU with GRU- (cid:52) t and GRU-Decay (GRU-D)
[Che et al., 2018].
GRU- (cid:52) t and GRU-D are effective models for processing irregular time series data.
GRU- (cid:52) t additionally uses the time difference between observations as input.
GRU-D is a modiﬁcation of GRU- (cid:52) tto learnt exponential decays between observations.
TimeGAN- (cid:52) t , RCGAN- (cid:52) t , C-RNN-GAN- (cid:52) t ,
T-Forcing- (cid:52) t , and P-Forcing- (cid:52) t (resp.
TimeGAN-D, RCGAN-D, C-RNN-GAN-D, T-Forcing-D,and P-Forcing-Decay) are modiﬁed with GRU- (cid:52) t (resp.
GRU-D) and can handle irregular data.
Our ablation studies also involve many advanced methods, based on NODEs, VAEs, ﬂow models, andso forth.
We intentionally leave these advanced methods for our ablation studies since our proposedmethod internally has them as sub-parts.
Evaluation metrics For quantitative evaluation of synthesized data, it is evaluated with the discriminative score and the predictive score used in TimeGAN [Yoon et al., 2019].
The discriminative scoremeasures the similarity between the original data and the synthesized data.
After learning a modelthat classiﬁes the original data and the synthesized data using a neural network, it is tested whether theoriginal data and the synthesized data are classiﬁed well.
The discriminative score is | Accuracy-0.5 | ,and if the score is low, classiﬁcation is difﬁcult, so the original data and the synthesized data aredecided to be similar.
The predictive score measures the effectiveness of the synthesized data usingthe train-synthesis-and-test-real (TSTR) method.
After training a model that predicts the next stepusing the synthesized data, the mean absolute error (MAE) is calculated between the predicted valuesand the ground-truth values in test data.
If the MAE is small, the model trained using the synthesizeddata is decoded to be similar to the original data.
For qualitative evaluation, the synthetic data isvisualized with the original data.
There are two methods for visualization.
One is to project originaland synthetic data in a two dimensional space using t-SNE [Van der Maaten and Hinton, 2008].
Theother one is the kernel density estimation to draw data distributions.
4.2 Experimental results
Regular time series synthesis
In Table 1, we list the results of the regular time series synthesis.
GT-GAN shows better performance on most cases than TimeGAN, the previous state-of-the-art
[Table]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
model.
As shown in the 1 st row in Fig.
5, GT-GAN covers original data areas better than TimeGAN.
In addition, the 2 nd row in Fig.
5 is the distributions of the fake data generated by GT-GAN and
TimeGAN.
The synthesized data’s distributions from GT-GAN are more similar to those of theoriginal data than TimeGAN, which shows the efﬁcacy of the explicit likelihood training of GT-GANagainst the implicit likelihood training of TimeGAN.
Irregular time series synthesis In Tables 2, 3, and 4, we list the results of the irregular timeseries synthesis.
GT-GAN shows better discriminative and predictive scores than other baselinesin all cases.
In Table 2, where we drop random 30% of observations from each time series sample,
GT-GAN shows the best outcomes, outperforming TimeGAN by large margins.
Baselines modiﬁedwith GRU- (cid:52) t and those with GRU-Decay show comparable results and it is hard to say one is betterthan the other in this table.
In Table 3 (50% dropped), many baselines do not show reasonable synthesis quality, e.g., TimeGAND, TimeGAN- (cid:52) t , RCGAN-D, C-RNN-GAN-D, and C-RNN-GAN- (cid:52) t have a discriminative scoreof 0.5.
Surprisingly, T-Forcing-D, T-Forcing- (cid:52) t , P-Forcing-D, and P-Forcing- (cid:52) t work well in thiscase.
However, our model clearly shows the best performance in all datasets.
Baselines modiﬁedwith GRU- (cid:52) t show slightly better than them modiﬁed with GRU-Decay in this case.
[Table] [Table]
[Figure]
Finally, Table 4 (70% dropped) shows the results of the most challenging experiments in our paper.
All baselines do not work well because of the high dropping rate.
T-Forcing-D, T-Forcing- (cid:52) t ,
P-Forcing-D, and P-Forcing- (cid:52) t , which showed reasonable performance with a dropping rate nolarger than 50%, do not work well in this case.
This shows that they are vulnerable to highly irregulartime series data.
Other GAN-based baselines are vulnerable as well.
Our method greatly outperformsall existing methods, e.g., a discriminative score of 0.278 by GT-GAN vs. 0.436 by T-Forcing-Dvs.
0.288 by P-Forcing- (cid:52) t for Sines, and a predictive score of 0.051 by GT-GAN vs. 0.114 by
T-Forcing-D vs. 0.053 by T-Forcing- (cid:52) t for MuJoCo.
Fig 6 visually compare our method and thebest performing baseline for the dropping rate of 70% — ﬁgures for other dropping rates and data arein Appendix I and similar patterns are observed in them.
[Figure]
[Table]
Ablation & sensitivity analyses
[Table]
GT-GAN is characterized by the
MLE training with the negative logdensity in Eq.
(8), and the pretraining step of the encoder and decoder.
Table 5 shows the results ofvarious GT-GAN modiﬁcations withsome training mechanism removed.
The model using the negative log-
[Table]
density training shows better performance than the model not using it.
That is, the MLE trainingmakes the synthetic data more like the real data.
When the pre-trained autoencoder is not used, thepredictive score is better than GT-GAN.
However, the discriminative score is the worst.
In Table 6, we alter the architecture for our model.
We modify our proposed GT-GAN model by removing its sub-parts to create simpler ablationmodels: i)
In the ﬁrst ablation model, we removethe autoencoder and perform the adversarial training only with our generator and discriminator,denoted “GT-GAN (w.o. AE)”.
In other words,our generation directly outputs raw observations(instead of hidden vectors), which will be fed intoour GRU-ODE-based discriminator.
ii)
The second ablation model, denoted “GT-GAN (Flow only)”,has only our CTFP-based generator and we train it with the maximum likelihood training — wenote that this construction is the same as training ﬂow-based models.
This model is equivalent tothe original CTFP model
[Deng et al., 2020].
iii)
The third ablation model has only the autoencoder,denoted “GT-GAN (AE only)”.
However, we convert it to a variational autoencoder (VAE) model.
In the full GT-GAN model, the encoder produces a set of hidden vectors { ( t i , h reali ) } Ni =0 .
In thisablation model, however, this is changed to { ( t i , N ( h reali , 1 )) }
Ni =0 , where N ( h reali , 1 ) means the unit
Gaussian centered at h reali .
The decoder is the same as its full model.
We use the variational trainingfor this model.
Among the ablation models, GT-GAN (Flow only) outperforms the discriminatorscore in most cases.
However, our full model is clearly the best in all cases.
Our study shows thatthe ablation models of GT-GAN do not perform as well as its full model if any parts are missing, asshown in Fig 21.
Refer to Appendix E for the ablation studies with other datasets.
The hyperparameters that signiﬁcantly affect model performance are the absolute tolerance (atol),the relative tolerance (rtol), and the period of the MLE training ( P MLE ) for the generator.
The atoland rtol determine the error control performed by the ODE solvers in CTFPs.
We test with variousoptions of the hyperparameters in Appendix F.
We found that there is an appropriate error tolerance(atol, rtol) depending on the data input size.
For example, the datasets with small input sizes (i.e.,
Sines, Stocks) have good discriminator scores with (1e-2, 1e-3), and the datasets with large inputsizes (i.e., Energy, MuJoCo) show good results with (1e-3, 1e-2).
Conclusions and discussions
Time series synthesis is an important research topic in deep learning and had been separately studiedfor regular or irregular time series synthesis.
However, there are still no existing generative modelsthat can handle both regular and irregular time series without model changes.
Our proposed method,
GT-GAN, is based on various advanced deep learning technologies, ranging from GANs to NODEs,
NCDEs, and so forth, and is able to process all possible types of time series without any changes inits model architecture and parameters.
Our experiments, which incorporate various synthetic andreal-world datasets, prove the efﬁcacy of the proposed method.
In our ablation studies, only our fullmethod without any missing parts shows reasonable synthesis capabilities.
Limitations Our model shows the best performance in both regular and irregular time series synthesis.
However, since our model has a complicated architecture, many hyperparameters exist.
Sometimes itis hard to train such large models, which involves a large scale hyperparameter search.
Societal impacts Time series data is one of the most widely used data in the ﬁeld of machine learning.
In many cases, time series data carries sensitive personal information, in which case one can use ourmethod to synthesize fake time series and protect privacy.
Likewise, we believe that our method hasmuch more positive impacts on our society than negative ones.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes] We did.
(b) Did you describe the limitations of your work?
[Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeSection 5(d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See oursupplemental material.(b)
Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] In Appendix G, we specify all the training details.(c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] In Section 4, We report the results of 10 experiments bycalculating the mean and variance.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section 44.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Appendix C
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]We release our model.
See our supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A Datasets
We use 2 simulated (Sines, MuJoCo) and 2 real-world (Stocks, Energy) datasets.
Table.
7 shows thestatistics of the datasets.
All datasets are available online via the link.
We note that in some of ourdatasets, the time series length N can be varied from one time series sample to another.
However, ourframework has no problems in dealing with those varying lengths.
[Table]
B ODE/CDE functions in GT-GANB.1 Encoder
Our encoder based on NCDEs has the following CDE function f .
[Table]
B.2 Decoder, Discriminator
Our decoder and discriminator based on GRU-ODEs have the following ODE functions.
3 They havethe same architecture but their parameters are separated.
[Table]
r t = Sigmoid(Linear)
N × dim( h ) N × dim( h ) z t = Sigmoid(Linear)
N × dim( h ) N × dim( h ) u t = Tanh(Linear)
N × dim( h ) N × dim( h )
[Table]
[Equation]
[Table]
[Equation]
[Table] [Table]
B.3 Generator
Our generator has the following ODE function f in Table 11.
[Table]
C Baselines
For the regular time series baseline models, i.e., TimeGAN, RCGAN, C-RNN-GAN, T-forcing, and
P-forcing, we use the 3-layer GRU-based neural network architecture with a hidden size that is 4times larger than the input size.
We use or modify the following accessible source codes to run.
• TimeGAN : https://github.com/jsyoon0823/TimeGAN
[Table]
• RCGAN :
https://github.com/3778/Ward2ICU
[Table]
• C-RNN-GAN : https://github.com/olofmogren/c-rnn-gan
[Table]
• T-forcing, P-forcing : https://github.com/mojesty/professor_forcing
[Table]
• GRU-D : https://github.com/zhiyongc/GRU-D
[Table]
Because ordinary GRUs can not be applied to irregular time series, we replace the ﬁrst layer GRU to
GRU- (cid:52) t and GRU-D in all those baselines so that the redesigned baseline models, i.e., TimeGAN- (cid:52) t ,
RCGAN- (cid:52) t , C-RNN-GAN- (cid:52) t , T-forcing- (cid:52) t ,
P-forcing- (cid:52) t , TimeGAN-D, RCGAN-D, C-RNNGAN-D, T-forcing-D and P-forcing-D, can process irregular time series data.
D Evaluation metrics
[Figure]
For fair comparison, we reuse the experimental environments of TimeGAN for the discriminativescore.
However, we found that TimeGAN’s predictive task is rather straightforward as shown in
Fig.
8 (a).
It predicts only one element in yellow from other four past elements in blue.
Sinceonly one element is used for evaluation, we found that the original predictive score of TimeGANcan be biased.
Instead, our predictive task predicts the entire vector, as shown in Fig.
8 (b), andtherefore, our predictive score is measured under much more challenging environments.
We usethis more challenging predictive score deﬁnition for our irregular time series synthesis.
We stickto the TimeGAN’s deﬁnition for the regular time series experiment for fair comparison but use ourchallenging predictive score metric for all other experiments.
E Additional ablation studies
In Tables 12 to 17, we report the missing ablation study tables in the main paper.
[Table] [Table]
[Table] [Table]
[Table] [Table]
F Sensitivity analyses
We provide performance (discriminative score and predictive score) depending on hyperparameters(i.e.
atol (absolute tolerance) =
{ 1 e − 1 , 1 e − 2 , 1 e − 3 } , rtol (relative tolerance) =
{ 1 e − 1 , 1 e −2 , 1 e − 3 } and P MLE = { 1 , 2 , 3 } ) for each different datasets.
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
G
The best hyperparamter set for GT-GAN
[Table]
• ‘atol’ means absolute tolerance for the generator.
[Table]
• ‘rtol’ means relative tolerance for the generator.
[Table]
• ‘ P MLE ’means the period of the negative log-density training for the generator.
[Table]
• ‘ K AE ’ means the autoencoder’s pre-training iteration numbers.
[Table]
• ‘d-layer’ means the number of discriminator’s GRU layers.
[Table]
• ‘r-acti’ means the last activation function of the decoder.
[Table]
• ‘reg-recon’ means the reconstruction regularization for the generator.
[Table]
• ‘reg-kinetic’ means the kinetic-energy regularization for the generator.
[Table]
• ‘reg-jacobian’ means the Jacobian-norm2 regularization for the generator.
[Table]
• ‘reg-direct’ means the directional-penalty regularization for the generator.
[Table] [Table]
H Model size & training time comparison
In Table
[Table]
we report the model size and training time of our method and TimeGAN, one of the bestperforming baseline.
As shown, our model has much smaller numbers of parameters than TimeGAN.
However, it take much longer time to train our model than TimeGAN.
This is mainly because we needto solve various differential equations, which is not needed for TimeGAN.
The memory requirementsare more or less the same in both models.
Therefore, these exist pros and cons for our method incomparison with the state-of-the-art baseline.
[Table]
I Visualizations with t-SNE and data distribution
[Table]
We introduce additional visualization outcomes in Figs.
13 to
[Table]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
J
AE number K train period
We describe the training method in Alg.
We ﬁrst pre-train the autoencoder in the ﬁrst while loop,followed by the second while loop for the main training step.
The main training step consists of i)ﬁne-tuning the autoencoder, ii) training the generation with the log-density loss, iii) training the GANpart with the adversarial loss.
In order to see the efﬁcacy of the log-density training, we conduct two more studies. The ﬁrst model GT-GAN (w/o Eq.
(8)) is a model in which the generator is trained only with adversarial loss. Thesecond model GT-GAN (supervised loss) replacing the log-density loss to a supervised loss.
Toobtain the supervised loss, like TimeGAN, we added a supervisor network between the encoder anddecoder. [Table] According to the above results, it was conﬁrmed that even if TimeGAN’s supervised loss is used, nobetter results than those of our original design are obtained (the predictive score is slightly improvedthough). In other words, this experiment conﬁrms the importance of the log-density path in ourmodel.
[Table]
[Table]
L Efﬁcacy of the NCDE-based encoder
We execute two experiments to justify using an NCDE-based encoder.
First, we experiment by
[Table]
replacing the encoder of TimeGAN with our NCDE-based encoder.
Second, the NCDE-basedencoder of GT-GAN is changed to GRU- (cid:52) t .
The results are in Tables 21 and 22.
Our model showsthe best outcomes when we use the NCDE-based encoder.
M Role of each network
Although our model looks complicated, we use an appropriate network for each part to ﬁt its role.
The role of each part is as follows:
Encoder The neural CDE-based encoder is able to encode a regular/irregular time series sample intoa regular/irregular hidden vector sequence.
Neural CDEs are sometimes called continuous RNNs andare specially designed for the representation learning of irregular time series.
As reported in our ﬁrstemail, generation quality is severely degraded when this network is substituted with GRU- (cid:52) t .
Decoder The GRU-ODE-based decoder is able to decode a regular/irregular hidden vector sequenceinto a regular/irregular time series sample.
One beauty of this decoder is, as shown in Fig.
2 in ourmain paper,
that the sampling time point and the sample length can be freely determined by users.
Generator
The CTFP-based invertible generator was intentionally selected by us since we canperform both the log-likelihood and the adversarial training together.
Since this network is a key partof our model, we wanted to use the two different training paradigms.
Our ablation studies about thelog-likelihood and supervised-learning training in Table.
20 justify our design selection.
Discriminator The GRU-ODE-based discriminator is able to process regular/irregular time series.
Unlike the encoding task of the neural CDE-based encoder, we observed faster and better results withthe GRU-ODE-based discriminator.
Moreover, neural CDEs require interpolation of input as a preprocessing.
We can do this for real data before training.
However, it is hard to perform dynamically forthe fake hidden vector sequence due to its excessive computation amount.
In particular, it signiﬁcantlydelays the overall training process if we use a neural CDE-based discriminator.
In general, our key design points lie in utilizing i) the continuous-time method-based autoencoder, andii) the CTFP-based generator.
Therefore, we can stabilize the generating performance for complicatedirregular time series as well.
N Discriminative vs. predictive score
[Figure]
In Table 5, GT-GAN without pre-training performs better than GT-GAN in terms of the predictive score.
In Fig.
21, however, GT-GAN generates samples a little out of the original datadistribution whereas GT-GAN
w/o pre-traininghas a severe mode-collapse problem (i.e., generating in a narrow region).
We conjecture that thosesamples a little outside the original data distribution make the prediction tasks’ scores a little low.
However, note that GT-GAN can successfully recall almost the entire data region.