GT-GAN: General Purpose Time Series Synthesis with Generative Adversarial Networks
Abstract
Time series synthesis is an important research topic in the ﬁeld of deep learning, which can be used for data augmentation.
Time series data types can be broadly classiﬁed into regular or irregular.
However, there are no existing generative models that show good performance for both types without any model changes.
Therefore, we present a general purpose model capable of synthesizing regular and irregular time series data.
To our knowledge, we are the ﬁrst designing a general purpose time series synthesis model, which is one of the most challenging settings for time series synthesis.
To this end, we design a generative adversarial network-based method, where many related techniques are carefully integrated into a single framework, ranging from neural ordinary/controlled differential equations to continuous time-ﬂow processes.
Our method outperforms all existing methods.
Introduction
Time series data occurs frequently in real-world applications
[Reinsel, 2003, Fu, 2011, Li et al., 2018,
Yu et al., 2018, Wu et al., 2019, Guo et al., 2019, Bai et al., 2019, Song et al., 2020, Huang et al., 2020a,
Ren et al., 2021, Tekin et al., 2021].
Among many tasks related to time series, synthesizing time series data is one of the most important tasks because real-world time series data is frequently imbalanced and/or insufﬁcient.
Since regular and irregular time series data have different characteristics, however, different model designs had been adopted for them.
Therefore, existing time series synthesis work focuses on either regular or irregular time series synthesis [Yoon et al., 2019, Alaa et al., 2021].
To our knowledge, there are no existing methods that work well for both types.
Regular time series means regularly sampled observations without any missing ones, and irregular times series means that some observations are missing from time to time.
Irregular time series is much harder to process than regular time series.
For instance, it is known that neural networks perform better after transforming time series data into its frequency domain, i.e., the Fourier transform, and some time series generative models use this approach [Alaa et al., 2021].
However, it is not easy to observe pre-determined frequencies from highly irregular time series [Kidger et al., 2019].
However, continuous-time models [Chen et al., 2018, Kidger et al., 2020, Brouwer et al., 2019] show good performance in processing both regular and irregular time series.
By resorting to them, we propose a general purpose model that can synthesize both time series types without any model changes.
To achieve the goal, we design a sophisticated model which utilizes a diverse set of technologies, ranging from generative adversarial networks (GANs [Goodfellow et al., 2014]), and autoencoders (AEs) to neural ordinary differential equations (NODEs [Chen et al., 2018]), neural controlled differ-
[Figure]
GAN
Log-density
Synthesis
[Figure]
ential equations (NCDEs [Kidger et al., 2020]), and continuous time-ﬂow processes (CTFPs [Deng et al., 2020]), which reﬂects the difﬁculty of the problem.
Fig.
1 shows the overall design of our proposed method.
One of the key points in our model design is that we combine the adversarial training of GANs and the exact maximum likelihood training of
CTFPs into a single framework.
However, the exact maximum likelihood training is applicable only to invertible mapping functions whose input and output sizes are the same.
Therefore, we design an invertible generator, and adopt an autoencoder, on whose hidden space our GAN performs the adversarial training.
[Figure]
In other words, i) the hidden vector size of the encoder is the same as the noisy vector of the generator, ii) the generator produces a set of fake hidden vectors, iii) the decoder converts the set into a fake continuous path (cf.
Fig. 2), and iv) the discriminator provides feedback after reading the sampled fake sample.
We note that in the third step, a fake continuous path is created by the decoder.
Therefore, we can sample any arbitrary regular/irregular time series sample from the fake path, which shows the ﬂexibility in our method.
We conduct experiments with 4 datasets and 7 baselines.
Since our method is able to support both the regular and irregular time series synthesis, we test for both of them.
Our method outperforms other baselines in both environments.
Our contributions can be summarized as follows:
We design a model based on various state-of-the-art deep learning technologies.
Our method is able to process any types of time series data, ranging from regular to irregular, without any model changes.
Our experimental results and visualization prove the efﬁcacy of the proposed model.
Since our task is one of the most challenging tasks for time series synthesis, the proposed model architecture is carefully designed.
Our ablation studies show that our proposed model does not work well if any part is missing.
Related work and preliminaries
GANs are one of the most representative generative technology.
Ever since the ﬁrst introduction in its seminal research paper, GANs have been adopted to main different domains.
Recently, researchers focused on their synthesis for time series data.
Therefore, there have been proposed several GANs for time series synthesis.
C-RNN-GAN [Mogren, 2016] has a regular GAN framework that can be applied to sequential data by using LSTM in its generator and discriminator.
Recurrent Conditional GAN (RCGAN [Esteban et al., 2017]) took a similar approach except that its generator and discriminator take conditional input for better synthesis.
WaveNet [van den Oord et al., 2016] also generates time series data from the conditional probability of previous data by using the dilated casual convolution. [Figure]
WaveGAN [Donahue et al., 2019] has a similar approach with DCGAN
[Radford et al., 2016], where its generator is based on WaveNet.
We can modify the teacher-forcing (T-Forcing [Graves, 2014]) and professor-forcing (P-Forcing [Lamb et al., 2016]) models to generate time series data from noise vectors, although they are not GAN models, by using the forecasting characteristic of those models.
TimeGAN [Yoon et al., 2019] is yet another model for time series synthesis. This model aims mainly at synthesizing fake regular time series samples. They proposed a framework where the adversarial training of GANs and the supervised training of predicting x i +1 from x i , where x i and x i +1 mean two multivariate time series values at time t i and t i +1 , respectively.
Proposed method
In this section, we describe our design.
Since our general purpose time series synthesis is a challenging task, the proposed design is much more complicated than other baselines.
3.1 Overall workﬂow
We ﬁrst describe the overall workﬂow in our model design, which consists of several different data paths (and several different training methods based on the data paths) as follows:
Autoencoder path:
Given an time series sample { ( t i , x reali ) } Ni =0 , the encoder produces a set of hidden vectors { h reali } Ni =0 .
The decoder recovers a continuous path ˆ X real , which enhances the ﬂexibility of our proposed method.
From the path ˆ X real , we sample { ( t i , ˆ x reali ) } Ni =0 .
We train the encoder and the decoder using the standard autoencoder (AE) loss to match x reali and ˆ x reali for all i . 2.
Adversarial path:
Given a set of noisy vectors { z i } Ni =0 , our generator produces a set of fake hidden vectors { h fakei } Ni =0 .
The decoder recovers a fake continuous path ˆ X fake from { h fakei } Ni =0 .
We sample { ( t j , x fakej ) } Mj =0 from ˆ X fake and feed it into the discriminator.
For irregular time series synthesis, we sample t j in [0 , T ] .
We train the generator, the decoder, and the discriminator using the standard adversarial loss.
Log-density path:
Given a set of hidden vectors { h reali } Ni =0 for an time series sample { ( t i , x reali ) } Ni =0 , the inverse path of the generator reproduces a set of noisy vectors { ˆ z i } Ni [-=
-]=0 .
We feed { ˆ z i } Ni =0 into its forward path again to reproduce { ˆ h reali } Ni =0 ,
where ˆ h reali = h reali for all i .
During the forward pass, we calculate the negative log probability of − log p (ˆ h reali ) for all i with the change of variable theorem and minimize it for training, being inspired by Grover et al.
[2018] and Deng et al.
In particular, we note that the dimensionality of the hidden space in the autoencoder is the same as that of the latent input space of the generator, i.e., dim( h )
= dim( z ) .
This is needed for the exact likelihood training in the generator — the change of variable theorem requires that the input and output sizes are the same to estimate the exact likelihood.
In addition to it, we let the autoencoder and the generator share the workload to synthesize fake time series by combining them into a single framework, i.e., the generator synthesizes fake hidden vectors and the decoder reproduces human-readable fake time series from them.
3.2 Autoencoder
Encoder General NCDEs, which are considered as a continuous analogue to recurrent neural networks (RNNs), are deﬁned as follows:
[Equation]
where X ( t ) is a continuous path created from a raw discrete time series sample { ( t i , x reali ) } Ni =0 by an interpolation algorithm — we note that X ( t i ) =
( t i , x reali ) for all i , and for other non-observed timepoints the interpolation algorithm ﬁlls out values.
Note that NCDEs keep reading the time-derivative of X ( t ) , denoted ˙
X ( t ) def = dX ( t ) dt .
In our case, we collect { h reali } Ni =0 as follows:
[Equation]
where h real= FC
[Equation]
dim
[Equation] ) ( x [Equation] ) and FC input _ size → output _ size is a fully-connected layer with speciﬁc input and output sizes.
We refer to Appendix B for the ODE function f deﬁnition.
Therefore, the input time series { ( t i , x reali ) } Ni =0 is represented by a set of hidden vectors { ( t i , h reali ) } Ni =0 .
Because NCDEs are a continuous analogue to RNNs, it shows the best ﬁt to processing irregular time series [Kidger et al., 2020].
Decoder Our decoder, which reproduces a time series from its hidden representations, is based on
GRU-ODEs [Brouwer et al., 2019] and is deﬁned as follows:
[Equation]
where d ( t 0 )
=
FC dim( h ) → dim( d ) ( h 0 ) and h
i means either the i -th real or fake hidden vector, i.e., h reali or h fakei — recall that in Fig.
3, the decoder is involved in both the autoencoder and the synthesis processes.
ˆ x means a reproduced copy of x .
GRU-ODEs uses the technology called neural ordinary differential equations (NODEs) to continuously interpret GRUs and we refer to Appendix B for the ODE function g deﬁnition.
In particular, the gated recurrent unit (GRU) at Eq.
(4) is called as jump which is known to be effective in processing time series with NODEs [Brouwer et al., 2019, Jia and Benson, 2019].
We train the encoder-decoder using the standard reconstruction loss between x reali and ˆ x reali for all i in all training time series samples.
3.3 Generative adversarial network
Generator Whereas generators typically read a noisy vector to generate a fake sample in standard GANs, our generator reads a continuous path (or time series) sampled from a Wiener process to generate a fake time series sample — this generation concept is known as continuous time ﬂow processes (CTFPs [Deng et al., 2020]).
Appendix.
B.3 shows an example of our generation process.
The input to our generation process is a random path sampled from a Wiener process, which is represented by a time series of latent vectors { ( t i , z i ) } Ni =0 in the path, and the output is a path of hidden vectors which is also represented by a time series of hidden vectors { ( t i , h fakei ) } Ni =0 .
Therefore, our generator can be written as follows:
[Equation]
where w i (0)
=
z i , a i (0) =
t i .
Here, τ means a virtual time variable of the integral problem, and t i is a real physical time contained in a time series sample { ( t i , x reali ) } Mi =0 .
We note that this design corresponds to a NODE model augmented with a i ( t ) .
We refer to Appendix B for the ODE function r deﬁnition.
Owing to the invertible nature of NODEs, we can calculate the exact log-density of h reali , i.e., the probability that h reali is generated by the generator, using the change of variable theorem and the
Hutchinson’s stochastic trace estimator as follows [Grathwohl et al., 2019, Deng et al., 2020]:
[Equation]
where ˆ w (0) means ˆ z reali in Fig.
ˆ h reali means a reproduced copy of h reali by our generator.
Eq.
(7) corresponds to “CTFP − 1 ”, and Eqs.
(6) and (8) to “CTFP” in Fig.
We note that in Eq.
(7), the integral time is reversed to solve the reverse-mode integral problem.
Therefore, we minimize the negative log-density, denoted − log Pr(ˆ h reali ) , for each t i , and our generator is trained by the two different training paradigms: i) the adversarial training against the discriminator, and ii) the maximum likelihood estimator (MLE) training with the log-density.
Discriminator We design our discriminator based on the GRU-ODE technology as follows:
[Equation]
where c ( t 0 ) =
FC dim( x ) → dim( c ) ( x 0 ) , and x i means the i -th time series value, i.e., x reali or x fakei .
The ODE function q has the same architecture as g but with its own parameters θ q .
After that, we calculate the real or fake classiﬁcation y =
σ ( FC dim( c ) → 2 ( c ( t N ))) , where σ is a softmax activation.
The role of each part of our proposed model is in Appendix M.
3.4 Training method
We use the mean squared reconstruction loss, i.e., the mean of (cid:107) x real i − ˆ x real i (cid:107) 2 2 for all i , to train the encoder-decoder architecture.
Then, we use the standard GAN loss to train the generator and the discriminator.
In our preliminary experiments, we found that the original GAN loss is suitable for our task.
Instead of other variations, such as WGAN-GP [Gulrajani et al., 2017], therefore, we use the standard GAN loss.
We train our model in the following sequence:
We pre-train the encoder-decoder networks the reconstruction loss for K AE iterations.
After the above pre-training step, we start to jointly train all networks in the following sequence for K JOINT iterations: i) training the encoder-decoder networks with the reconstruction loss, ii) training the discriminator-generator networks with the GAN loss, iii) training the decoder to improve the discriminator’s classiﬁcation output with the discriminator loss and iv) the generator with the MLE
loss every P MLE iteration.
We found that too frequent MLE training incurs mode-collapse so we use it every P MLE iteration.
In particular, the 2-ii step to train the decoder to help the discriminator out is one additional point where the autoencoder and the GAN are integrated into a single framework.
In other words, the generator should deceive both the decoder and the discriminator.
Our training algorithm refer to
Appendix
The well-posedness 1 of NCDEs and GRU-ODEs was already proved in Lyons et al.
[2007, Theorem 1.3] and Brouwer et al.
[2019] under the mild condition of the Lipschitz continuity.
We show that our NCDE layers are also well-posed problems.
Almost all activations, such as ReLU, Leaky ReLU,
SoftPlus, Tanh, Sigmoid, ArcTan, and Softsign, have a Lipschitz constant of 1.
Other common neural network layers, such as dropout, batch normalization and other pooling methods, have explicit
Lipschitz constant values.
Therefore, the Lipschitz continuity of ODE/CDE functions can be fulﬁlled in our case.
In other words, it is a well-posed training problem.
As a result, our training algorithm solves a well-posed problem so its training process is stable in practice.
Experimental evaluations
Our software and hardware environments are as follows: U BUNTU 18.04 LTS, P YTHON 3.8.10,
P YTORCH 1.8.1, T ENSORFLOW 2.5.0, CUDA 11.2, and NVIDIA Driver 417.22, i9 CPU, and
NVIDIA RTX 3090.
The mean and variance of 10 runs are reported for model evaluation.
4.1 Experimental environments
Datasets We conduct experiments with 2 simulated and 2 real-world datasets.
Sines has 5 features where each feature is created with different frequencies and phases independently.
For each feature, i ∈ { 1 , ..., 5 } , x i ( t ) =
sin (2 πf i t
+ θ i ) , where f i ∼ U [0 , 1] and θ i ∼ U [ − π, π ] .
MuJoCo is multivariate physics simulation time series data with 14 features.
Stocks is the Google stock price data from 2004 to 2019.
Each observation represents one day and has 6 features.
Energy is a UCI appliance energy prediction dataset with 28 values.
To create the challenging irregular environments, 30, 50, 70% of observations from each time series sample { ( t i , x reali ) } Ni =0 is randomly dropped — in other words, N decreases to 0 .
7 N, 0 .
5 N, 0 .
3 N .
Dropping random values has been mainly used to create irregular time series environments in the literature [Kidger et al., 2019, Xu and Xie, 2020,
Huang et al., 2020b, Tang et al., 2020, Zhang et al., 2021, Jhin et al., 2021, Deng et al., 2021].
Therefore, we conduct experiments with both the regular and the irregular environments.
Baselines We consider the following baselines for the regular time series experiments: TimeGAN,
RCGAN, C-RNN-GAN, WaveGAN, WaveNet, T-Forcing, and P-Forcing.
For the irregular experiments, we exclude WaveGAN and WaveNet, which cannot handle irregular time series, and redesign other baselines by replacing their GRU with GRU- (cid:52) t and GRU-Decay (GRU-D)
[Che et al., 2018].
GRU- (cid:52) t and GRU-D are effective models for processing irregular time series data.
GRU- (cid:52) t additionally uses the time difference between observations as input.
GRU-D is a modiﬁcation of GRU- (cid:52) t to learnt exponential decays between observations.
TimeGAN- (cid:52) t , RCGAN- (cid:52) t , C-RNN-GAN- (cid:52) t ,
T-Forcing- (cid:52) t , and P-Forcing- (cid:52) t (resp.
TimeGAN-D, RCGAN-D, C-RNN-GAN-D, T-Forcing-D, and P-Forcing-Decay) are modiﬁed with GRU- (cid:52) t (resp.
GRU-D) and can handle irregular data.
Our ablation studies also involve many advanced methods, based on NODEs, VAEs, ﬂow models, and so forth.
We intentionally leave these advanced methods for our ablation studies since our proposed method internally has them as sub-parts.
Evaluation metrics For quantitative evaluation of synthesized data, it is evaluated with the discriminative score and the predictive score used in TimeGAN [Yoon et al., 2019].
The discriminative score measures the similarity between the original data and the synthesized data.
After learning a model that classiﬁes the original data and the synthesized data using a neural network, it is tested whether the original data and the synthesized data are classiﬁed well.
The discriminative score is | Accuracy-0.5 | , and if the score is low, classiﬁcation is difﬁcult, so the original data and the synthesized data are decided to be similar.
The predictive score measures the effectiveness of the synthesized data using the train-synthesis-and-test-real (TSTR) method.
After training a model that predicts the next step using the synthesized data, the mean absolute error (MAE) is calculated between the predicted values and the ground-truth values in test data.
If the MAE is small, the model trained using the synthesized data is decoded to be similar to the original data.
For qualitative evaluation, the synthetic data is visualized with the original data.
There are two methods for visualization.
One is to project original
[Table] [Table] [Figure] and synthetic data in a two dimensional space using t-SNE [Van der Maaten and Hinton, 2008].
The other one is the kernel density estimation to draw data distributions.
4.2 Experimental results
Regular time series synthesis
In Table 1, we list the results of the regular time series synthesis.
GT-GAN shows better performance on most cases than TimeGAN, the previous state-of-the-art model.
As shown in the 1 st row in Fig.
4, GT-GAN covers original data areas better than TimeGAN.
In addition, the 2 nd row in Fig.
4 is the distributions of the fake data generated by GT-GAN and
TimeGAN.
The synthesized data’s distributions from GT-GAN are more similar to those of the original data than TimeGAN, which shows the efﬁcacy of the explicit likelihood training of GT-GAN against the implicit likelihood training of TimeGAN.
Irregular time series synthesis In Tables 2, 3, and 4, we list the results of the irregular time series synthesis.
GT-GAN shows better discriminative and predictive scores than other baselines in all cases.
In Table 2, where we drop random 30% of observations from each time series sample,
GT-GAN shows the best outcomes, outperforming TimeGAN by large margins.
Baselines modiﬁed with GRU- (cid:52) t and those with GRU-Decay show comparable results and it is hard to say one is better than the other in this table. [Table] [Table] [Figure]
In Table 3 (50% dropped)
[Figure] many baselines do not show reasonable synthesis quality, e [Figure] g., TimeGAND, TimeGAN- (cid:52) t , RCGAN-D, C-RNN-GAN-D, and C-RNN-GAN- (cid:52) t have a discriminative score of 0.5.
Surprisingly, T-Forcing-D, T-Forcing- (cid:52) t , P-Forcing-D [Figure] and P-Forcing- (cid:52) t work well in this case.
However, our model clearly shows the best performance in all datasets.
Baselines modiﬁed with GRU- (cid:52) t show slightly better than them modiﬁed with GRU-Decay in this case.
[Table]
[Figure] [Table] [Figure] [Table]
Finally
[Figure]
Table 4 (70% dropped) shows the results of the most challenging experiments in our paper.
All baselines do not work well because of the high dropping rate.
T-ForcingD, T-Forcing- (cid:52) t , P-Forcing-D, and
P-Forcing- (cid:52) t , which showed reasonable performance with a dropping rate no larger than 50%, do not work well in this case.
This shows that they are vulnerable to highly irregular time series data.
Other GAN-based baselines are vulnerable as well.
Our method greatly outperforms all existing methods, e.g., a discriminative score of 0.278 by GT-GAN vs. 0.436 by T-Forcing-D vs. 0.288 by P-Forcing- (cid:52) t for Sines, and a predictive score of 0.051 by GT-GAN vs. 0.114 by T-Forcing-D vs. 0.053 by
T-Forcing- (cid:52) t for MuJoCo.
Fig 5 visually compare our method and the best performing baseline for
[Table]
the dropping rate of 70% — ﬁgures for other dropping rates and data are in Appendix I and similar patterns are observed in them.
Ablation & sensitivity analyses GT-GAN is characterized by the MLE training with the negative log-density in Eq.
(8), and the pre-training step of the encoder and decoder.
Table 5 shows the results of various GT-GAN modiﬁcations with some training mechanism removed.
The model using the negative log-density training shows better performance than the model not using it.
That is, the MLE training makes the synthetic data more like the real data.
When the pre-trained autoencoder is not used, the predictive score is better than GT-GAN.
However, the discriminative score is the worst.
In Table 6, we alter the architecture for our model.
We modify our proposed GT-GAN model by removing its sub-parts to create simpler ablation models: i)
In the ﬁrst ablation model, we remove the autoencoder and perform the adversarial training only with our generator and discriminator, denoted “GT-GAN (w.o. AE)”.
In other words, our generation directly outputs raw observations (instead of hidden vectors), which will be fed into our GRU-ODE-based discriminator.
ii)
The second ablation model, denoted “GT-GAN (Flow only)”, has only our CTFP-based generator and we train it with the maximum likelihood training — we note that this construction is the same as training ﬂow-based models.
This model is equivalent to the original CTFP model [Deng et al., 2020].
iii)
The third ablation model has only the autoencoder, denoted “GT-GAN (AE only)”.
However, we convert it to a variational autoencoder (VAE) model.
In the full GT-GAN model, the encoder produces a set of hidden vectors { ( t i , h reali ) } Ni =0 .
In this ablation model, however, this is changed to { ( t i , N ( h reali , 1 )) }
Ni =0 , where N ( h reali , 1 ) means the unit
Gaussian centered at h reali .
The decoder is the same as its full model.
We use the variational training for this model.
Among the ablation models, GT-GAN (Flow only) outperforms the discriminator score in most cases.
However, our full model is clearly the best in all cases.
Our study shows that the ablation models of GT-GAN do not perform as well as its full model if any parts are missing, as shown in Fig 6.
Refer to Appendix E for the ablation studies with other datasets.
The hyperparameters that signiﬁcantly affect model performance are the absolute tolerance (atol), the relative tolerance (rtol), and the period of the MLE training ( P MLE ) for the generator.
The atol and rtol determine the error control performed by the ODE solvers in CTFPs.
We test with various options of the hyperparameters in Appendix F.
We found that there is an appropriate error tolerance (atol, rtol) depending on the data input size.
For example, the datasets with small input sizes (i.e.,
Sines, Stocks) have good discriminator scores with (1e-2, 1e-3), and the datasets with large input sizes (i.e., Energy, MuJoCo) show good results with (1e-3, 1e-2).
Conclusions
Time series synthesis is an important research topic in deep learning and had been separately studied for regular or irregular time series synthesis.
However, there are still no existing generative models that can handle both regular and irregular time series without model changes.
Our proposed method,
GT-GAN, is based on various advanced deep learning technologies, ranging from GANs to NODEs, and NCDEs, and is able to process all possible types of time series without any changes in its model architecture and parameters.
Our experiments, which incorporate various synthetic and real-world datasets, prove the efﬁcacy of the proposed method.
In our ablation studies, only our full method without any missing parts shows reasonable synthesis capabilities.
The limitations and societal impacts of our proposed model are in Appendix O. Acknowledgments and Disclosure of Funding
Noseong Park is the corresponding author.
This work was supported by the Yonsei University
Research Fund of 2021, the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korean government (MSIT)
(No. 2020-0-01361, Artiﬁcial
Intelligence Graduate School Program at Yonsei University), and the LG Display research fund (C2022000673).
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] We did.
(b) Did you describe the limitations of your work?
[Yes] See Section(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] See Section(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See our supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] In Appendix G, we specify all the training details.
(c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[Yes] In Section 4, We report the results of 10 experiments by calculating the mean and variance.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See Appendix C (b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] We release our model.
See our supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Datasets
We use 2 simulated (Sines, MuJoCo) and 2 real-world (Stocks, Energy) datasets.
Table.
7 shows the statistics of the datasets.
All datasets are available online via the link.
We note that in some of our datasets, the time series length N can be varied from one time series sample to another.
However, our framework has no problems in dealing with those varying lengths.
[Table]
B ODE/CDE functions in GT-GAN B.1 Encoder
Our encoder based on NCDEs has the following CDE function f .
[Table]
B.2 Decoder, Discriminator
Our decoder and discriminator based on GRU-ODEs have the following ODE functions.
They have the same architecture but their parameters are separated.
[Table]
r t = Sigmoid(Linear)
N × dim( h ) N × dim( h ) z t = Sigmoid(Linear)
N × dim( h ) N × dim( h ) u t = Tanh(Linear)
N × dim( h ) N × dim( h ) dh
= (1 − z t ) ∗ ( u t − h t ) N × dim(
[Table]
[Table]
[Equation]
B.3 Generator
Our generator has the following ODE function f in Table 11.
[Table]
C Baselines
For the regular time series baseline models, i.e., TimeGAN, RCGAN, C-RNN-GAN, T-forcing, and
P-forcing, we use the 3-layer GRU-based neural network architecture with a hidden size that istimes larger than the input size.
We use or modify the following accessible source codes to run.
• TimeGAN : https://github.com/jsyoon0823/TimeGAN • RCGAN :
https://github.com/3778/Ward2ICU • C-RNN-GAN : https://github.com/olofmogren/c-rnn-gan • T-forcing, P-forcing : https://github.com/mojesty/professor_forcing • GRU-D : https://github.com/zhiyongc/GRU-D
Because ordinary GRUs can not be applied to irregular time series, we replace the ﬁrst layer GRU to
GRU- (cid:52) t and GRU-D in all those baselines so that the redesigned baseline models, i.e., TimeGAN- (cid:52) t ,
RCGAN- (cid:52) t , C-RNN-GAN- (cid:52) t , T-forcing- (cid:52) t ,
P-forcing- (cid:52) t , TimeGAN-D, RCGAN-D, C-RNNGAN-D, T-forcing-D and P-forcing-D, can process irregular time series data.
D Evaluation metrics
[Figure]
For fair comparison, we reuse the experimental environments of TimeGAN for the discriminative score.
However, we found that TimeGAN’s predictive task is rather straightforward as shown in
Fig.
8 (a).
It predicts only one element in yellow from other four past elements in blue.
Since only one element is used for evaluation, we found that the original predictive score of TimeGAN can be biased.
Instead, our predictive task predicts the entire vector, as shown in Fig.
8 (b), and therefore, our predictive score is measured under much more challenging environments.
We use this more challenging predictive score deﬁnition for our irregular time series synthesis.
We stick to the TimeGAN’s deﬁnition for the regular time series experiment for fair comparison but use our challenging predictive score metric for all other experiments.
E Additional ablation studies
In Tables 12 to 17, we report the missing ablation study tables in the main paper.
[Table] [Table]
[Table] [Table]
[Table] [Table]
F Sensitivity analyses
We provide performance (discriminative score and predictive score) depending on hyperparameters (i [Figure]
e. atol (absolute tolerance) = { 1 e − 1 , 1 e − 2 , 1 e − 3 } , rtol (relative tolerance) = { 1 e − 1 , 1 e − 2 , 1 e − 3 } and P MLE = { 1 , 2 , 3 } ) for each different datasets
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
G The best hyperparamter set for GT-GAN
• ‘atol’ means absolute tolerance for the generator.
• ‘rtol’ means relative tolerance for the generator.
• ‘ P MLE ’means the period of the negative log-density training for the generator.
• ‘ K AE ’ means the autoencoder’s pre-training iteration numbers.
• ‘d-layer’ means the number of discriminator’s GRU layers.
• ‘r-acti’ means the last activation function of the decoder.
• ‘reg-recon’ means the reconstruction regularization for the generator.
• ‘reg-kinetic’ means the kinetic-energy regularization for the generator.
• ‘reg-jacobian’ means the Jacobian-norm2 regularization for the generator.
• ‘reg-direct’ means the directional-penalty regularization for the generator.
[Table]
H Model size & training time comparison
In Table
[Table]
we report the model size and training time of our method and TimeGAN, one of the best performing baseline.
As shown, our model has much smaller numbers of parameters than TimeGAN.
However [Table] it take much longer time to train our model than TimeGAN.
This is mainly because we need to solve various differential equations, which is not needed for TimeGAN.
The memory requirements are more or less the same in both models.
Therefore, these exist pros and cons for our method in comparison with the state-of-the-art baseline.
[Table]
I Visualizations with t-SNE and data distribution
We introduce additional visualization outcomes in Figs.
13 to 20.
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
J
Algorithm number train period
We describe the training method in Alg.
We ﬁrst pre-train the autoencoder in the ﬁrst while loop, followed by the second while loop for the main training step.
The main training step consists of i) ﬁne-tuning the autoencoder, ii) training the generation with the log-density loss, iii) training the GAN part with the adversarial loss.
K Efﬁcacy of the log-density training
In order to see the efﬁcacy of the log-density training, we conduct two more studies.
The ﬁrst model
GT-GAN (w/o Eq.
(8)) is a model in which the generator is trained only with adversarial loss.
The second model GT-GAN (supervised loss) replacing the log-density loss to a supervised loss.
To obtain the supervised loss, like TimeGAN, we added a supervisor network between the encoder and decoder.
[Table]
According to the above results, it was conﬁrmed that even if TimeGAN’s supervised loss is used, no better results than those of our original design are obtained (the predictive score is slightly improved though).
In other words, this experiment conﬁrms the importance of the log-density path in our model.
[Table] [Table]
L Efﬁcacy of the NCDE-based encoder
We execute two experiments to justify using an NCDE-based encoder.
First, we experiment by replacing the encoder of TimeGAN with our NCDE-based encoder.
Second, the NCDE-based encoder of GT-GAN is changed to GRU- (cid:52) t .
The results are in Tables 21 and 22.
Our model shows the best outcomes when we use the NCDE-based encoder.
M Role of each network
Although our model looks complicated, we use an appropriate network for each part to ﬁt its role.
The role of each part is as follows:
Encoder The neural CDE- based encoder is able to encode a regular/irregular time series sample into a regular/irregular hidden vector sequence.
Neural CDEs are sometimes called continuous RNNs and are specially designed for the representation learning of irregular time series.
As reported in our ﬁrst email, generation quality is severely degraded when this network is substituted with GRU- (cid:52) t .
Decoder The GRU-ODE-based decoder is able to decode a regular/irregular hidden vector sequence into a regular/irregular time series sample.
One beauty of this decoder is, as shown in Fig.
2 in our main paper,
that the sampling time point and the sample length can be freely determined by users.
Generator
The CTFP-based invertible generator was intentionally selected by us since we can perform both the log-likelihood and the adversarial training together.
Since this network is a key part of our model, we wanted to use the two different training paradigms.
Our ablation studies about the log-likelihood and supervised-learning training in Table.
20 justify our design selection.
Discriminator The GRU-ODE-based discriminator is able to process regular/irregular time series.
Unlike the encoding task of the neural CDE-based encoder, we observed faster and better results with the GRU-ODE-based discriminator.
Moreover, neural CDEs require interpolation of input as a preprocessing.
We can do this for real data before training.
However, it is hard to perform dynamically for the fake hidden vector sequence due to its excessive computation amount.
In particular, it signiﬁcantly delays the overall training process if we use a neural CDE-based discriminator.
In general, our key design points lie in utilizing i) the continuous-time method-based autoencoder, and ii) the CTFP-based generator.
Therefore, we can stabilize the generating performance for complicated irregular time series as well.
N Discriminative vs. predictive score
In Table 5, GT-GAN without pre-training performs better than GT-GAN in terms of the predictive score.
In Fig.
21, however, GT-GAN generates samples a little out of the original data distribution whereas GT-GAN
w/o pre-training has a severe mode-collapse problem (i.e., generating in a narrow region).
We conjecture that those samples a little outside the original data distribution make the prediction tasks’ scores a little low.
However, note that GT-GAN can successfully recall almost the entire data region.
O Discussions
Limitations Our model shows the best performance in both regular and irregular time series synthesis.
However, since our model has a complicated architecture, many hyperparameters exist.
Sometimes it is hard to train such large models, which involves a large scale hyperparameter search.
Societal impacts Time series data is one of the most widely used data in the ﬁeld of machine learning.
In many cases, time series data carries sensitive personal information, in which case one can use our method to synthesize fake time series and protect privacy.
Likewise, we believe that our method has much more positive impacts on our society than negative ones.