PSA-GAN: P ROGRESSIVE S ELF A TTENTION GAN S FOR S YNTHETIC T IME S ERIES
Flunkert flunkert@
Jan Gasthaus
AI Labs gasthaus@amazon com
SE tim januschowski
A BSTRACT
Realistic synthetic time series data of sufﬁcient length enables practical applications in time series modeling tasks, such as forecasting, but remains a challenge.
In this paper, we present PSA-GAN , a generative adversarial network (GAN) that generates long time series samples of high quality using progressive growing of
GANs and self-attention.
We show that PSA-GAN can be used to reduce the error in several downstream forecasting tasks over baselines that only use real data.
We also introduce a Frechet Inception distance-like score for time series, Context-FID, assessing the quality of synthetic time series samples.
We ﬁnd that Context-FID is indicative for downstream performance.
Therefore, Context-FID could be a useful tool to develop time series GAN models.
I NTRODUCTION
In the past years, methods such as (Salinas et al., 2020; Franceschi et al., 2019; Kurle et al., 2020; de B´ezenac et al., 2020; Oreshkin et al., 2020a; Rasul et al., 2021; Cui et al., 2016; Wang et al., 2017) have consistently showcased the effectiveness of deep learning in time series analysis tasks.
Although these deep learning based methods are effective when sufﬁcient and clean data is available this assumption is not always met in practice.
For example, sensor outages can cause gaps in IoT data, which might render the data unusable for machine learning applications (Zhang et al., 2019b).
An additional problem is that time series panels often have insufﬁcient size for forecasting tasks, leading to research in meta-learning for forecasting (Oreshkin et al., 2020b).
Cold starts are another common problem in time series forecasting where some time series have little and no data (like a new product in a demand forecasting use case).
Thus, designing ﬂexible and task-independent models that generate synthetic, but realistic time series for arbitrary tasks are an important challenge.
Generative adversarial networks (GAN) are a ﬂexible model family that has had success in other domains.
However, for their success to carry over to time series, synthetic time series data must be of realistic length, which current state-of-the-art synthetic time series models struggle to generate because they often rely on recurrent networks to capture temporal dynamics (Esteban et al., 2017;
Yoon et al., 2019).
In this work, we make three contributions: i)
We propose PSA-GAN a progressively growing, convolutional time series GAN model augmented with self-attention (Karras et al., 2017; Vaswani et al., 2017).
PSA-GAN scales to long time series because the progressive growing architecture starts modeling the coarse-grained time series features and moves towards modeling ﬁne-grained details during training.
The self-attention mechanism captures long-range dependencies in the data (Zhang et al., 2019a).
ii)
We show empirically that PSA-GAN samples are of sufﬁcient quality and length to boost several downstream forecasting tasks: far-forecasting and data imputation during inference, data imputation of missing value stretches during training, forecasting under cold start conditions, and data augmentation.
Furthermore, we show that PSA-GAN can be used as a forecasting model and has competitive performance when using the same context information as an established baseline.
iii)
Finally, we propose a Frechet Inception distance (FID)-like score (Salimans et al., 2016), ContextFID, leveraging unsupervised time series embeddings (Franceschi et al., 2019).
We show that the lowest scoring models correspond to the best-performing models in our downstream tasks and that the Context-FID score correlates with the downstream forecasting performance of the GAN model (measured by normalized root mean squared error).
Therefore, the Context-FID could be a useful general-purpose tool to select GAN models for downstream applications.
We structure this work as follows:
We discuss the related work in Section 2 and introduce the model in Section 3.
In Section 4, we evaluate our proposed GAN model using the proposed Context-FID score and through several downstream forecasting tasks.
We also directly evaluate our model as a forecasting algorithm and perform an ablation study.
Section 5 concludes this manuscript.
R ELATED W ORK
GANs (Goodfellow et al., 2014) are an active area of research (Karras et al., 2019; Yoon et al., 2019;
Engel et al., 2019; Lin et al., 2017; Esteban et al., 2017; Brock et al., 2018) that have recently been applied to the time series domain (Esteban et al., 2017; Yoon et al., 2019) to synthesize data (Takahashi et al., 2019; Esteban et al., 2017), and to forecasting tasks (Wu et al., 2020).
Many time series GAN architectures use recurrent networks to model temporal dynamics (Mogren, 2016; Esteban et al., 2017; Yoon et al., 2019).
Modeling long-range dependencies and scaling recurrent networks to long sequences is inherently difﬁcult and limits the application of time series GANs to short sequence lengths (less than 100 time steps) (Yoon et al., 2019; Esteban et al., 2017).
One way to achieve longer realistic synthetic time series is by employing convolutional (van den Oord et al., 2016;
Bai et al., 2018; Franceschi et al., 2019) and self-attention architectures (Vaswani et al., 2017).
Convolutional architectures are able to learn relevant features from the raw time series data (van den Oord et al., 2016; Bai et al., 2018;
Franceschi et al., 2019), but are ultimately limited to local receptive ﬁelds and can only capture long-range dependencies via many stacks of convolutional layers.
Self-attention can bridge this gap and allow for modeling long-range dependencies from convolutional feature maps, which has been a successful approach in the image (Zhang et al., 2019a) and time series forecasting domain (Li et al., 2019; Wu et al., 2020).
Another technique to achieve long sample sizes is progressive growing, which successively increases the resolution by adding layers to generator and discriminator during training (Karras et al., 2017).
Our proposal, PSA-GAN , synthesizes progressive growing with convolution and self-attention into a novel architecture particularly geared towards time series.
Another line of work in the time series ﬁeld is focused on developing suitable loss functions for modeling ﬁnancial time series with GANs where speciﬁc challenges include heavy tailed distributions, volatility clustering, absence of autocorrelations, among others (Cont, 2001; Eckerli & Osterrieder, 2021).
To this end, several models like QuantGAN (Wiese et al., 2020), (Conditional) SigWGAN
(Ni et al., 2020; 2021), and DAT-GAN (Sun et al., 2020) have been proposed (see (Eckerli & Osterrieder, 2021) for review in this ﬁeld).
This line of work targets its own challenges by developing new loss functions for ﬁnancial time series, which is orthogonal to our work, i.e. we focus on neural network architectures for time series GANs and show its usefulness in the context of time series forecasting.
Another challenge is the evaluation of synthetic data.
While the computer vision domain uses standard scores like the Inception Score and the Frechet Inception distance (FID) (Salimans et al., 2016; Heusel et al., 2017), such universally accepted scores do not exist in the time series ﬁeld.
Thus, researchers rely on a Train on Synthetic–Test on Real setup and assess the quality of the synthetic time series in a downstream classiﬁcation and/or prediction task (Esteban et al., 2017; Yoon et al., 2019).
In this work, we build on this idea and assess the GAN models through downstream forecasting tasks.
Additionally, we suggest a Frechet Inception distance-like score that is based on unsupervised time series embeddings (Franceschi et al., 2019).
Critically, we want to be able to score the ﬁt of our ﬁxed length synthetic samples into their context of (often much longer) true time series, which is taken into account by the contrastive training procedure in Franceschi et al.
As we will later show, the lowest scoring models correspond to the best performing models in downstream tasks.
M ODEL
Problem formulation We denote the values of a time series dataset by z i,t ∈ R , where i ∈ { 1 , 2 , . . .
, N } is the index of the individual time series and t ∈ { 1 , 2 , . .
, T } is the time index.
Additionally, we consider an associated matrix of time feature vectors X 1: T = ( x 1 , . . .
, x T ) in
R D × T .
Our goal is to model a time series of ﬁxed length τ , ˆ Z i,t,τ =
(ˆ z i,t , . . .
, ˆ z i,t + τ − 1 ) , from this dataset using a conditional generator function G and a ﬁxed time point t .
Thus, we aim to model ˆ
Z i,t,τ = G ( n , φ ( i ) , X t : t + τ − 1 ) , where n ∈ R τ is a noise vector drawn from a Gaussian distribution of mean zero and variance one; φ is an embedding function that maps the index of a time series to a vector representation, that is concatenated to each time step of X t : t + τ − 1 .
An overview of the model architecture is shown in Figure 1 and details about the time features are presented in Appendix A.
Spectral Normalised Residual
The generator and use a main function m that is a composition of convolution, self-attention and spectral normalisation
[Equation]
where f ( x ) =
LR(SN( c ( x ))) and m ( y ) =
γ SA( y )
+ y , c is a one dimensional convolution operator,
LR the LeakyReLU operator (Xu et al., 2015), SN the spectral normalisation operator (Miyato et al., 2018) and SA the self-attention module.
The variable n f is the number of in and out-channels of c , l is the length of the sequence.
Following the work of (Zhang et al., 2019a), the parameter γ is learnable.
It is initialized to zero to allow the network to learn local features directly from the building block f and is later enriched with distant features as the absolute value of gamma increases, hereby more heavily factoring the self-attention term SA .
The module m is referenced as residual self-attention in Figure 1 ( right ).
Downscaling and Upscaling The following sections mention upscaling ( UP ) and downscaling ( DOWN ) operators that double and halve the length of the time series, respectively.
In this work, the upscaling operator is a linear interpolation and the downscaling operator is the average pooling.
PSA-GAN PSA-GAN is a progressively growing GAN (Karras et al., 2017); thus, trainable modules are added during training.
Hereby, we model the generator and discriminator as a composition of functions: G =
g L +1 ◦ ... ◦
g 1 and D = d 1 ◦ ... ◦
d L +1 where each function g i and d i for i ∈ [1 , L +1] corresponds to a module of the generator and discriminator.
G ENERATOR
As a preprocessing step, we ﬁrst map the concatenated input [ n , φ ( i ) , X t : t + τ − 1 ] from a sequence of length τ to a sequence of length 8, denoted by ˜ Z 0 , using average pooling.
Then, the ﬁrst layer of the generator g 1 applies the main function m :
[Equation]
For i ∈ [2 , L ] , g i maps an input sequence ˜
Z i − 1 to an output sequence ˜
Z i by applying an upscaling of the input sequence and the function m ◦ f :
[Equation]
The output of g i is concatenated back to the time features X t : t + τ − 1 and forwarded to the next block.
Lastly, the ﬁnal layer of the generator g L +1 reshapes the multivariate sequence ˜ Z L to a univariate time series ˆ Z i,t,τ of length τ = 2 L +3 using a one dimensional convolution and spectral normalisation.
D ISCRIMINATOR
The architecture of the discriminator mirrors the architecture of the generator.
It maps the generator’s output ˆ Z i,t,τ and the time features X t : t + τ − 1 to a score d .
The ﬁrst module of the discriminator d L +1 uses a one dimensional convolution c 1 and a LeakyReLU activation function:
[Equation]
For i ∈
[ L + 1 , 2] , the module d i applies a downscale operator and the main function m :
[Equation]
The last module d 1 turns its input sequence into a score:
[Equation]
where FC is a fully connected layer.
PSA-GAN-C
We introduce another instantiation of PSA-GAN in which we forward to each generator block g
i knowledge about the past.
The knowledge here is a sub-series ˆ Z i,t − L C ,L C in the range [ t − L C , t − 1] , with L C being the context length.
The context ˆ Z i,t − L C ,L C is concatenated along the feature dimension, i.e. at each time step, to the output sequence of g i .
It is then passed through a two layers perceptron to reshape the feature dimension and then added back to the output of g i .
3. L OSS FUNCTIONS
PSA-GAN is trained via the LSGAN loss (Mao et al., 2017), since it has been shown to address mode collapse (Mao et al., 2017).
Furthermore, least-squares type losses in embedding space have been shown to be effective in the time series domain (Mogren, 2016; Yoon et al., 2019).
Additionally, we use an auxiliary moment loss to match the ﬁrst and second order moments between the batch of synthetic samples and a batch of real samples:
[Equation]
where µ is the mean operator and σ is the standard deviation operator.
The real and synthetic batches have their time index and time series index aligned.
We found this combination to work well for
PSA-GAN
empirically.
Note that the choice of loss function was not the main focus of this study and we think that our choice can be improved in future research.
Training procedures GANs are notoriously difﬁcult to train, have hard-to-interpret learning curves, and are susceptible to mode collapse.
The training procedure to address those issues together with other training and tuning details is presented in Appendix B–E.
E XPERIMENTS
The evaluation of synthetic time series data from GAN models is challenging and there is no widely accepted evaluation scheme in the time series community.
We evaluate the GAN models by two guiding principles: i)
Measuring to what degree the time series recover the statistics of the training dataset. ii)
Measuring the performances of the GAN models in challenging, downstream forecasting scenarios.
For i), we introduce the Context-FID (Context-Frechet Inception distance) score to measure whether the GAN models are able to recover the training set statistics.
The FID score is widely used for evaluating synthetic data in computer vision (Heusel et al., 2017) and uses features from an inception network (Szegedy et al., 2016) to compute the difference between the real and synthetic sample statistics in this feature space.
In our case, we are interested in how well the the synthetic time series windows ”ﬁt” into the local context of the time series.
Therefore, we use the time series embeddings from Franceschi et al.
(2019) to learn embeddings of time series that ﬁt into the local context.
Note that we train the embedding network for each dataset separately.
This allows us to directly quantify the quality of the synthetic time series samples (see Appendix D for details).
For ii), we set out to mimic several challenging time series forecasting tasks that are common for time series forecasting practitioners.
These tasks often have in common that practitioners face missing or corrupted data during training or inference.
Here, we set out to use the synthetic samples to complement an established baseline model, DeepAR , in these forecasting tasks.
These tasks are: far-forecasting and missing values during inference, missing values during training, cold starts, and data augmentation.
We evaluate these tasks by the normalized root mean squared distance (NRMSE).
Additionally, we evaluate PSA-GAN model when used as a forecasting model.
Note that, where applicable, we re-train our GAN models on the modiﬁed datasets to ensure that they have the same data available as the baseline model during the downstream tasks.
We also considered NBEATS (Oreshkin et al., 2020a) and Temporal Fusion Transformer ( TFT ) (Lim et al., 2021) as alternative forecasting models.
However, we found that DeepAR performed best in our experiments and therefore we report these results in the main text (see Appendix F for experiment details).
Please refer to Appendix G for the evaluation of NBEATS and TFT (Tables S2–S6 and
Figures S3–S6).
In addition, we perform an ablation study of our model and discuss whether the Context-FID scores are indicative for downstream forecasting tasks.
4. D ATASETS AND B ASELINES
We use the following public, standard benchmark datasets in the time series domain: M4, hourly time series competition data (414 time series) (Makridakis et al., 2020);
Solar, hourly solar energy collection data in Alabama State (137 stations) (Lai et al., 2018); Electricity, hourly electricity consumption data (370 customers) (Dheeru & Karra Taniskidou, 2017); Trafﬁc: hourly occupancy rate of lanes in San Francisco (963 lanes) (Dheeru & Karra Taniskidou, 2017).
Unless stated otherwise, we split all data into a training/test set with a ﬁxed date and use all data before that date for training.
For testing, we use a rolling window evaluation with a window size of 32 and seven windows.
We minmax scale each dataset to be within [0 , 1] for all experiments in this paper (we scale the data back before evaluating the forecasting experiments).
In lieu of ﬁnding public datasets that represent the downstream forecasting tasks, we modify each the datasets above to mimic each tasks for the respective experiment (see later sections for more details).
We compare PSA-GAN with different GAN models from the literature ( TIMEGAN (Yoon et al., 2019) and EBGAN (Zhao et al., 2017)).
In what follows PSA-GAN-C and PSA-GAN denote our proposed model with and without context, respectively.
In the forecasting experiments, we use the GluonTS (Alexandrov et al., 2020) implementation of DeepAR which is a well-performing forecasting model and established baseline (Salinas et al., 2020).
[Table]
the mean and standard deviation.
4. D IRECT E VALUATION WITH C ONTEXT -FID S CORES
Table 1 shows the Context-FID scores for PSA-GAN , PSA-GAN-C and baselines.
For all sequence lengths, we ﬁnd that PSA-GAN or PSA-GAN-C consistently produce the lowest Context-FID scores.
For a sequence length of 256 time steps, TIMEGAN is the second best performing model for all datasets.
Note that even though using a context in the PSA-GAN model results in the best overall performance, we are interested to use the GAN in downstream tasks where the context is not available.
Thus, the next section will use PSA-GAN without context, unless otherwise stated.
4. E VALUATION ON F ORECASTING T ASKS
In this section, we present the results of the forecasting tasks.
We ﬁnd that synthetic samples do not improve over baselines in all cases.
However, we view these results as a ﬁrst attempt to use GAN models in these forecasting tasks and believe that future research could improve over our results.
Far-forecasting: In this experiment, we forecast far into the future by assuming that the data points between the training end time and the rolling evaluation window are not observed.
For example, the last evaluation window would have 32 ∗ 6 unobserved values between the training end time and the forecast start date.
This setup reﬂects two possible use cases: Forecasting far into the future (where no context data is available) and imputing missing data during inference because of a data outage just before forecasting.
Neural network-based forecasting models such as DeepAR struggle under these conditions because they rely on the recent context and need to impute these values during inference.
Furthermore, DeepAR only imputes values in the immediate context and not for the lagged values.
Here, we use the GAN models during inference to ﬁll the missing observations with synthetic data.
As a baseline, we use DeepAR and impute the missing observations of lagged values with a moving average (window size 10) during inference.
Here, we ﬁnd that using the synthetic data from GAN models drastically improve over the DeepAR baseline and using samples from PSA-GAN results into the lowest NRMSE for three out of four datasets (see left Table in Figure 2).
Figure 2 also shows the NRMSE as a function of the forecast window.
Figure 3 shows an example of a imputed time series and the resulting forecast from the Electricity dataset when using PSA-GAN .
Missing Value Stretches:
Missing values are present in many real world time series applications and are often caused by outages of a sensor or service (Zhang et al., 2019b).
Therefore, missing values in real-world scenarios are often not evenly distributed over the time series but instead form missing value ”stretches”.
In this experiment, we simulate missing value stretches and remove time series values of length 50 and 110 from the datasets.
This results into 5.4-7.7% missing values for a stretch length of 50 and and 9.9-16.9% missing values for a stretch length of 110 (depending on the dataset.)
Here, we split the training dataset into two parts along the time axis and only introduce missing values in the second part.
We use the ﬁrst (unchanged) part of the training dataset to train the GAN models and both parts of the training set to train DeepAR .
We then use the GAN models
[Figure] [Figure]
The orange line between the dashed lines corresponds to the PSA-GAN imputed sequence and is used by
DeepAR for forecasting.
Note that this part is unobserved in this experiment.
The imputed sequence allows to generate a reasonable forecast, even if many data points before the forecast start date are unobserved.
to impute missing values during training and inference of DeepAR .
Figure 4 shows that using the
GAN models to impute missing values during training DeepAR on data with missing value stretches reduces its forecasting error.
While all GAN models reduce the NRMSE in this setup, PSA-GAN is most effective in reducing the error in this experiment.
See Figure S1 for a detailed split by dataset.
[Figure]
Cold Starts:
In this experiment, we explore whether the GAN models can support DeepAR in a cold start forecasting setting.
Cold starts refer to new time series (like a new product in demand forecasting) for which little or no data is available.
Here, we randomly truncate 10%, 20%, and 30% of the time series from our datasets such that only the last 24 (representing 24 hours of data) values before inference are present.
We only consider a single forecasting window in this experiment.
We then again use the GAN models to impute values for the lags and context that DeepAR conditions for
[Table]
Mean and 95% conﬁdence intervals obtained by re-running each method ﬁve times.
DeepAR is only run on the real time series data.
The other models correspond to DeepAR and one of the
GAN models for augmentation forecasting the cold start time series.
Figure 5 shows the forecasting error of the different models for the cold start time series only.
In this experiment
PSA-GAN and TIMEGAN improve the NRMSE over DeepAR and are on-par overall (mean NMRSE 0 .
70 and 0 .
71 for PSA-GAN and TIMEGAN , respectively).
See Figure S2 for a detailed split by dataset
[Figure]
ten runs.
Note that this ﬁgure only shows the error of the cold start time series.
Overall,
TIMEGAN and PSA-GAN improve the NRMSE over DeepAR in this setup, while PSA-GAN is the best method when 30% of time series are cold starts.
Data Augmentation:
In this experiment, we average the real data and GAN samples to augment the data during training.
During inference, DeepAR is conditioned on real data only to generate forecasts.
In Table
we we can see that none of the GAN models for data augmentation consistently improves over DeepAR .
Overall, TIMEGAN is the best-performing GAN model but plain DeepAR still performs better.
This ﬁnding is aligned with recent work in the image domain where data augmentation with GAN samples does not improve a downstream task (Ravuri & Vinyals, 2019).
We hypothesize that the GAN models cannot improve in the data augmentation setup because they are trained to generate realistic samples and not necessarily to produce relevant invariances.
Furthermore, the dataset sizes might be sufﬁcient for DeepAR to train a well-performing model and therefore the augmentation might not be able to reduce the error further.
More research is required to understand whether synthetic data can improve forecasting models via data augmentation.
Forecasting Experiments: In this experiment, we use the GAN models directly for forecasting (Table 3, example samples in Appendix H).
We can see that DeepAR consistently performs best.
This is expected as DeepAR takes into account context information and lagged values.
This kind of information is not available to the GAN models.
To test this we further consider PSA-GAN-C , i.e.
PSA-GAN with 64 previous time series values as context, and further evaluate DeepAR with drawing lags only from the last 64 values ( DeepAR-C ).
We can see that in this case PSA-GAN-C outperforms
DeepAR-C in 3 out of 4 datasets and PSA-GAN performs on par with DeepAR-C .
Moreover, both
PSA-GAN and PSA-GAN-C are the best performing GAN models.
Adding lagged values to the GAN models as context could further improve their performance and adversarial/attention architectures have been
[Table]
4. A BLATION S TUDY
Figure 6 shows the results of our ablation study where we disable important components of our model: moment loss, self-attention, and fading in of new layers.
We measure the performance of the ablation models by Context-FID score.
Overall, our propost logog ed PSA-GAN model performs better than the ablations which conﬁrms that these components contribute to the performance of the model.
over three on four datasets and the 68% conﬁdence interval.
PSA
GAN the lowest
Context-FID score which con-
ﬁrms that the proposed model requires these components for good performance.
SA : Selfattention, ML : Moment loss, fade in : Progressive fade
in
of new layers
4. L OW C ONTEXT -FID S CORE M ODELS C ORRESPOND TO B EST - PERFORMING F ORECASTING M ODELS
One other observation is that the lowest Context-FID score models correspond to the best models in the data augmentation and far-forecasting experiments.
PSA-GAN and TIMEGAN produce the lowest
Context-FID samples and both models also improve over the baseline in most downstream tasks.
Overall, PSA-GAN has lowest Context-FID and also outperforms the other models in the downstream forecasting tasks, except for the cold start task.
Additionally, we calculated the Context-FID scores for the ablation models mentioned in Figure 6 (but with a target length of 32 ) and the NRMSE of these models in the forecasting experiment (as in Table 3).
We ﬁnd that the Pearson correlation coefﬁcient between the Context-FID and forecasting NRMSE is 0.71 and the Spearman’s rank correlation coefﬁcient is 0.67, averaged over all datasets.
All datasets (except Traffic ) have a correlation coefﬁcient of at least 0.73 in either measure (see Table S1 in the Appendix).
C ONCLUSION
We have presented PSA-GAN , a progressive growing time series GAN augmented with self-attention, that produces long realistic time series and improves downstream forecasting tasks that are challenging for deep learning-based time series models.
Furthermore, we introduced the Context-FID score to assess the quality of synthetic time series samples produced by GAN models.
We found that the lowest Context-FID scoring models correspond to the best-performing models in downstream tasks.
We believe that time series GANs that scale to long sequences combined with a reliable metric to assess their performance might lead to their routine use in time series modeling.
R EPRODUCIBILITY STATEMENT
Details necessary for reproducing our experiments are given in the Appendix.
In particular, details on training are provided in Sections B and C, together with hyperparameter tuning in Section E and further experimental settings in Section F.
The code we used in the paper is available under: https://github.com/mbohlkeschneider/psa-gan and we will additionally disseminate PSA-GAN via GluonTS: https://github.com/awslabs/gluon-ts .
A PPENDIX A T IME S ERIES F EATURES
We use time-varying features to represent the time dimension for the GAN models to enable training on time series windows sampled from the full-length time series .
We use HourOfDay , DayOfWeek ,
DayOfMonth and DayOfYear features.
Each feature is encoded using a single index number (for example, between [0 , 365[ for DayOfYear ) and normalized to be within [-0.5, 0.5].
We also use an age feature to represent the age of the time series to be log (2 . 0 + t ) where t is the time index.
Additionally, we embed the index of the individual time series into a 10 dimensional vector with an embedding network to and use it as a non time-varying feature.
DeepAR uses the same features in forecasting.
B T RAINING P ROCEDURE
PSA-GAN uses spectral normalization and progressive fade-in of new layers to stabilize training.
Progressive fade in of new layers As training progresses, new layers are added to double the length of time series.
However, simply adding new untrained layers drastically changes the number of parameters and the loss landscape, which destabilizes training.
Progressive fade in of new layers is a technique introduce by Karras et al., 2017 to mitigate this effect where each layer g i and d i is smoothly introduced.
For i ∈ [2 , L ] the fade in of new layers is written as:
[Equation]
where α is a scalar initialized to be zero and grows linearly to one over 500 epochs.
Spectral Normalisation Both the generator and the discriminator beneﬁt from using spectral normalisation layers.
In the discriminator, Spectral Normalisation stabilizes training (Miyato et al., 2018).
It constrains the Lipschitz constant of the discriminator to be bounded by one.
In the generator the spectral normalisation stabilises the training and avoids escalation of the gradient magnitude (Zhang et al., 2019a).
C T RAINING D ETAILS
All GAN models in this paper have been implemented using PyTorch from Paszke et al.
PSA-GAN has been trained over 6500 epochs, where a new block is added every 1000 epochs and is faded over 500 epochs.
At each epoch, PSA-GAN trains on 100 batches of size 512.
It optimises its parameters using the Adam (Kingma & Ba, 2014) with a learning rate of 0 . 0005 and betas of (0 . 9 , 0 . 999) for both the generator and the discriminator.
PSA-GAN (and its variants) has been trained on ml.p2.xlarge Amazon instances forhours.
The number of in-channels and out-channels n f in the Main Block (Figure 1, Right ) equals 32 .
The number of out-channels in the convolution layer c equals 1 (Eq.6 ).
The number of time features D equals 15 .
For the case of PSA-GAN-C, the MLP block is a two layers perceptron with LeakyReLU.
It incorporates knowledge of the past by selecting the L C = 64 time points and concatenates them with its input.
D C ONTEXT -FID S CORE ON T IME S ERIES E MBEDDINGS
To compute the Context-FID, we replace InceptionV3 (Szegedy et al., 2016) with the encoder E of Franceschi et al.
(2019), which we train separately for each dataset.
Our proposed Context-FID score is computed as follows: we ﬁrst select a time range [ t, t + τ ] .
We then sample a batch of synthetic time series ˆZ t,τ =
[ ˆ Z 1 ,t,τ , ..., ˆ Z N,t,τ ] and a batch of real time series Z t,τ =
[ Z 1 ,t,τ , ..., Z N,t,τ ] that we encode with E into ˆZ et,τ and Z e t,τ respectively.
Finally, we compute the FID score of the embeddings.
E H YPERPARAMETER TUNING
We did limited hyperparameter tuning in this study to ﬁnd default hyperparemters that perform well across datasets.
We performed the following grid-search: the batch size , the number of batches used per epoch , the presence of Self-attention and the presence of the
Progressive fade in of new layers mechanism explained in section B, paragraph B.
The range considered for each hyper-parameter is: batch size : [64 , 128 , 256 , 512] , the number of batches used per epoch :
[50 , 100 , 200] , the presence of Self-attention :
[ True , False ] and the presence of the Progressive fade in of new layers : [ True , False ] .
The set of hyperparameters is shared across the four datasets we train PSA-GAN on (which means we do not tune hyperparameters per datasets).
We select the ﬁnal hyperparameter set by lowest
Context-FID score on the training set, averaged across all datasets.
F F ORECASTING E XPERIMENT D ETAILS
We use the DeepAR (Salinas et al., 2020) implementation in GluonTS (Alexandrov et al., 2020) for the forecasting experiments.
We use the default hyperparameters of DeepAR with the following exceptions: epochs=100 , num batches per epoch=100 , dropout rate=0.01 , scaling=
False , prediction length=32 , context length=64 , use feat static cat=
True .
We use the Adam optimizer with a learning rate of 1e − 3 and weight decay of 1e − 8 .
Additionally, we clip the gradient to 10 . 0 and reduce the learning rate by a factor of 2 for every 10 consecutive updates without improvement, up to a minimum learning rate of 5e − 5 .
DeepAR also uses lagged values from previous time steps that are used as input to the LSTM at each time step t .
We set the time series window size that DeepAR samples to 256 and truncate the default hourly lags to ﬁt the context window, the prediction window, and the longest lag into the window size 256.
We use the following lags: [1, 2, 3, 4, 5, 6, 7, 23, 24, 25, 47, 48, 49, 71, 72, 73, 95, 96, 97, 119, 120, 121, 143, 144, 145].
For NBEATS (Oreshkin et al., 2020a), we use the default parameters and set the trainer settings to epochs=100 , num batches per epoch=50 .
Note that in the original paper, NBEATS is an ensemble of 180 networks.
Due to the high computational cost, we only ensemble three networks with the settings context length=64 , context length=96 , context length= and mean absolute percentage error as the loss function.
For TFT (Lim et al., 2021), we use the default hyperparameters except: epochs=100 , num batches per epoch=50 , context length=128 .
For all methods, we used the implementations available in GluonTS (Alexandrov et al., 2020).
G A DDITIONAL E XPERIMENTS
[Table]
[Figure] [Figure]
TIMEGAN and PSA
GAN improve over DeepAR in
[Table]
N B E A T S E B G
A N
T I M E G A
N
P S A - G A N
[Figure] [Figure]
[Figure] [Figure]
N
[Figure]
M
[Figure]
E
Stretch length =
N B E A T S E B G
A N
T I M E G A
N
P S A - G A N
Stretch length =
[Figure]
NRMSE
Dataset
[Table]
Solar
Trafﬁc
[Table]
PSA
[Table] [Table]
[Figure]
N
[Figure]
G
[Figure]
T F T E B G A N
T I M E G A
N
P S A - G A N
[Figure] [Figure]
NRMSE
[Table]
Solar
[Table]
TIMEGAN
[Table]
PSA-GAN 0 .
[Table] [Table]
[Table]
H S AMPLE FORECASTS
We present more sample forecasts under the framework of the forecasting experiment.
In Figures S7–
S10 we present sample forecast for Electricity, M4, Solar Energy, and Trafﬁc datasets, respectively.
We can see that the synthetic time series generated by PSA-GAN and PSA-GAN-C consistently convey a better approximation to the ground truth values.
One can further observe that this task is particularly challenging for GANs as data generation is located in a time-frame that is not covered while training.
[Figure] [Figure]
[Figure] [Figure]