Forecasting Market Prices using DL with Data Augmentation and Meta-learning: ARIMA still wins!
Abstract
Deep-learning techniques have been successfully used for time-series forecasting and have often shown superior performance on many standard benchmark datasets as compared to traditional techniques.
Here we present a comprehensive and comparative study of performance of deep-learning techniques for forecasting prices in ﬁnancial markets.
We benchmark state-of-the-art deep-learning baselines, such as NBeats, etc., on data from currency as well as stock markets.
We also generate synthetic data using a fuzzy-logic based model of demand driven by technical rules such as moving averages, which are often used by traders.
We benchmark the baseline techniques on this synthetic data as well as use it for data augmentation.
We also apply gradient-based meta-learning to account for nonstationarity of ﬁnancial time-series.
Our extensive experiments notwithstanding, the surprising result is that the standard ARIMA models outperforms deep-learning even using data augmentation or meta-learning.
We conclude by speculating as to why this might be the case.
Introduction
Forecasting prices in ﬁnancial markets has long been attempted using a variety of statistical as well as modern machine-learning techniques; in theory markets are ‘efﬁcient’, and should not be predictable, making this task potentially ill-posed to start with.
Nevertheless, ﬁnancial markets are rife with algorithmic trading engines that aim at precisely this, attempting to discover any lingering inefﬁciencies that could yield a usable trading signal.
The behaviour of stock markets is a result of complex combination of several trading heuristics followed by a large number of groups of traders ranging from big ﬁnancial ﬁrms to individual investors, each with varying impact.
Previous works (Wang (2015b,a)) have shown that these trading strategies and the behaviour arising from their combination can be modeled as a set of mathematical equations for the excess demand function using fuzzy logic.
These set of equations are capable of modelling the real-world ﬁnancial markets to a good extent as shown in Wang (2015c).
These models can then be used to generate data of desired amount which will follow the behaviour arising from the combination of all the trading heuristics considered while building the model.
Present day deep learning models require large amounts of data to train.
The foremost problem in using deep-learning for market data forecasting is the lack of availability of large datasets; after all, there is only so much historical data available.
We hypothesize that synthetic data can be used to improve the performance of existing Deep Learning models on forecasting real-world market data.
This is based on the premise that training on even very dissimilar data can help in improving the performance of deep learning models as shown in Malhotra et al.
[Table]
Meta-learning is a machine learning paradigm wherein a model is trained on multiple similar tasks; for example, tasks having slightly different data distributions.
Gradient-based meta-learning using
MAML introduced in Finn et al.
(2017) is a model-agnostic procedure wherein a base model is adapted using the training data of each task via a few gradient steps.
During meta-training, the based model is trained in an outer-loop, again using gradient descent, so that its post-adaptation performance on the test data of the meta-training tasks is optimized.
The goal is to initialize the model with a set of parameters which can easily be adapted to different downstream tasks, potentially helping the model to generalize to out of distribution data.
Financial data can be thought of as undergoing frequent changes of distribution as economic conditions and investor sentiments ﬂuctuate due to global and national events
We present a comprehensive study in which we attempt using meta-learning and synthetically generated data for data augmentation, to improve performance of deep-learning models for forecasting ﬁnancial market data.
We report results across various deep learning models: MLPs, LSTMs,
Transformers (Vaswani et al.
and two state of the art time-series forecasting models: NBEATS (Oreshkin et al.
and Temporal Fusion Transformers (TFT) (Lim et al.
Surprisingly, we ﬁnd that none of these models is better than the traditional ARIMA model, even using data augmentation or applying meta-learning.
Related work
Makridakis and Hibon (2000) and Makridakis et al.
(2018) provide an in depth review of the performance of various statistical and Machine Learning algorithms for time-series forecasting in the
M3 competition.
Recently, Oreshkin et al.
(2020) proposed N-BEATS, a deep learning univariate time-series forecasting model which is capable of beating statistical approaches on the M4 benchmark.
Lim et al.
(2019) introduced Temporal Fusion Transformer, a state of the art multivariate time-series forecasting model which shows improved performance as compared to ARIMA and ETS on a number of real-world tasks.
DeepAR proposed in Flunkert et al.
(2017) uses auto-regressive RNN for probabilistic forecasting.
Oreshkin et al.
(2021) studies zero-shot learning for univariate time series forecasting using Model Agnostic Meta Learning as proposed in Finn et al.
(2017) on N-BEATS.
Another very recent work, Elsayed et al.
(2021) shows that traditional forecasting approaches such as
Gradient Tree Boosting Models are still capable of beating a number of state of the art deep learning approached on an array of different datasets.
Methodology
Wang (2015b) uses fuzzy logic to develop a mathematical model of the excess demand of the stock market by taking different trading heuristics into account.
This mathematical is then used to generate synthetic data using:
[Equation]
where p t denotes the price of a share at a give time-step, a i ( t ) denotes the inﬂuence of a particular heuristic, and ed i ( x t ) denotes the excess-demand function for the heuristic with x t being a time dependent variable.
We consider synthetic data generated from Heuristic 1 discussed in Wang (2015b) for all purposes.
Using fuzzy logic, excess demand can be modeled into the expression given below; twelve such heuristics are discussed in Wang (2015b).
[Equation]
where x ( m,n ) 1 ,t is the ratio of the natural logarithm of the two moving averages of length m and n of the stock price.
The price of a stock at time step t is denoted by p t .
µ A i are a fuzzy membership functions for each of the 7 rules as in Wang (2015b), describing when each rule applies.
The moving average of length n and the ratio of the natural logarithms of two moving averages of length m and n can then be calculated as: ¯
[Equation]
[Equation]
We experiment with two different techniques in an attempt to improve market data forecasting using synthetic data:
Data Augmentation: We augment the real world data with an equal amount of synthetic data during training the models.
For each epoch, batches of real-world data and synthetic data are fed randomly to the deep learning model until all the batches of both synthetic and real world data are exhausted.
We use a lookback length of 20 for the real-world data and a lookback length of 5 for the synthetic data.
For models with ﬁxed input length such as MLP and NBEATS, we interpolate the lookback windows of the synthetic data to make their length equal to those of the real-world data.
1: Initialize the synthetic dataloader S , the real-world dataloader R with appropriate batch sizes and the number of epochs e 2: n ← len ( S ) ; m ← len ( R ) 3: for i in e do 4: s ← 0 ; r ←5: while s < n or r < m do 6: a ∼ U (0 , 1) 7: if a > 0 .
5 then 8: if s < n then 9: batch ∼ S 10:
Input the lookback window into the model 11:
Calculate the loss on the batch 12:
Backpropagate on the loss 13: s ← s +14: end if 15: else 16: if r < m then 17: batch ∼ R 18: Input the lookback window into the model 19: Calculate the loss on the batch 20: Backpropagate on the loss 21: r ← r +22: end if 23: end if 24: end while 25: end for
Meta Learning: We create the tasks for meta-training from the training part of a dataset and those for meta-testing from testing part of the dataset.
For synthetic data, these divisions are made within each of the 40 time-series.
For creating each task for meta-training, a time-step d is chosen randomly from the training data.
Let l be the total length of the window ( lookback + prediction ).
For each such time-step d , k such windows, [ d − l − j, d − j ] are sampled randomly from before d , where j ∼ U [1 , a ] .
This forms the training set of the task.
Similarly, r windows are sampled randomly after d which forms the test set of the task.
The same procedure is used on the testing dataset to create meta-testing tasks.
For synthetic data, meta-training and meta-testing tasks are created from each series and then merged.
All the deep learning models are then trained using meta-learning on both real-world and synthetic data individually using First Order Model Agnostic Meta Learning algorithm
(Nichol et al.
Empirical evaluation 4.1 Materials
We use run experiments on three datasets: one synthetic and two real world datasets.
Synthetic Data: We use equations (1) and (2) with ( m, n ) =
(1 , 5) to generate 40 synthetic time series each containing 500 time-steps.
Each of these time-series is initialized with different seeds and a random-walk for 100 time-steps.
For normal training and testing, 36 time-series are used as [Table] training data and 2 time-series are used for validation and testing data each.
For meta learning, the ﬁrst 400 time-steps of each time series are used as the meta-training data and the last 100 time-steps are used as the meta-testing data.
Banknifty Data: Banknifty data is a time-series of closing values of the banknifty index, recorded every week day from 1st January, 2016 to 31st May, 2021.
After cleaning the data we split the data into a training time series containing 1050 time-steps and testing and validation time series, each containing 144 time-steps.
For meta learning the the time series consisting of 1050 time-steps is used to create tasks for meta-training and the rest of the time series consisting of 288 time-steps (144 + 144) is used for creating meta-testing tests.
Forex Data: Forex Data is a time-series of spot prices of 1 INR in USD, recorded every weekday from 1st January, 2010 to 14th August, 2020.
The total length of the time series is 2562 time-steps of which the the ﬁrst 2050 time-steps are used as the training data, time steps 2045 to 2306 are used as the validation data and time-steps 2301 to 2562 are used as testing data.
For meta learning, the ﬁrst 2050 time-steps are used for creating meta-training tasks and the remaining data is used to create meta-testing tasks.
4.2 Results
Table 1 compares 1 the performance of deep learning models trained using different training techniques with ARIMA.
We use a multitude of deep learning models ranging from simple models to current state of the art models for time-series forecasting.
Results were obtained for: MLP, LSTM, Transformers , NBEATS and TFT.
All the experiments were performed using a 12GB NVIDIA Tesla K80 GPU, 12GB RAM and an Intel(R) Xeon( R)
CPU with cores.
The results have been reported for two well known metrics: Root Mean Squared Error (RMSE) and Mean Absolute Percentage Error (MAPE)
The lookback window length for the synthetic dataset has been taken as 5 since each time-step depends on the last 5 time-steps and the lookback window length for both the real-world datasets
[Table]
has been taken as 20 which was arrived at by experimenting with different lookback lengths.
The lookback windows are locally normalized before passing through the model which allows the realworld and the synthetic data to be combined during data augmentation.
We evaluate each model for a prediction horizon of 10 time-steps and the results are calculated after de-normalizing the predictions.
We further decompose the results obtained to get results for one-step ahead prediction, ﬁve step ahead prediction and 6th to 10th step prediction.
Results for one step (OS) and ten step (TS) ahead predictions have been reported
4.2.1 Rollout Testing
We further experiment with try rollout testing for models trained with and without data augmentation on non-autoregressive models (MLP, NBEATS and TFT).
In rollout testing, we take into account the previous n points to predict the next time-step where n is the length of the lookback window.
The lookback window is of ﬁxed length and keeps sliding one step at a time as and when new predictions are made.
We take n = 5 for synthetic data in an attempt to make it easier for the predictor to capture the actual relationship between the time-steps (remember that we use n = 5 in equation 2 for generating the data).
n = 20 for Banknifty and Forex datasets.
The results are presented in Table 2.
Clearly, state of the art deep learning models for time-series forecasting fail to beat classical approaches such as ARIMA on the real-world data, even after data augmentation and meta learning.
Deep Learning models successfully beat ARIMA on synthetic data which is deterministic.
Real-world ﬁnancial data is very noisy and chaotic.
Machine Learning models rely on a global feedback signal or a recent signal for optimization.
Due to the (ﬁnancial) data being noisy and chaotic and the fact that deep learning models are often trained on multiple time-series, the feedback signals are also very noisy leading to poor optimization.
The synthetic data on the other hand is deterministic, hence leading to noiseless signals due to which deep learning models perform better on synthetic data.
ARIMA on the other hand relies on a purely local signal computed every time it’s applied.
A local signal is much stronger when there is more noise in the longer signals, e.g., because the inefﬁciencies that lead to potential trading signals are not similar over time and change continuously.
Conclusion
We present a comprehensive and comparative study on performance of different Deep Learning techniques and ARIMA, a classical approach on market data forecasting.
While current state of the art deep models are often capable of outperforming classical forecasting techniques, they fail to do so on ﬁnancial data - an initally surprising negative result.
We attribute the reason to the fact that ﬁnancial data is inherently chaotic.
Autoregressive models such as ARIMA in general perform better for prediction when there is less reason to believe that the regularities enabling prediction are themselves non-stationary and cannot therefore be exploited by machine-learning models.
We ﬁnd that this is indeed true for the ﬁnancial data that we experimented on.
A Appendix A.1 Model Conﬁgurations and Hyperparameters
We use or MLP, LSTM, NBEATS and TFT and data loading and pre-processing utilities.
For Transformer, we use the PyTorch’s nn.
Transformer.
Any parameters not included here have been used with their default values as deﬁned in the respective libraries.
We use 5 fast adaptation steps for MAML throughout all the experiments.
A.1.1 Transformer
learning rate 1ebatch size weight decay 1ed model attn heads encoders decoders feedforward dim
MAML lr 1elearner lr (SN, BN, FR) 1e-5, 1e-3, 1e-
learning rate 1ebatch size weight decay # stacks stack type generic backcast loss ratio 0.
layer width
MAML lr 1elearner lr (SN, BN, FR) 1e-3, 1e-3, 1eA
.1.3 Temporal Fusion Transformer
learning rate 1ebatch size weight decay hidden size # lstm layers # attn heads hidden continuous size # output quantiles 3 (0.1, 0.5, 0.9)
MAML lr 1elearner lr (SN, BN, FR) 1e-4, 1e-3, 1e-  learning rate 1ebatch size weight decay 1e- # layers hidden state size
MAML lr 1elearner lr (SN, BN, FR) 1e-3, 1e-3, 1e-
A.1.5 MLP
We use hidden layers of uniform size and ReLU activation learning rate 1ebatch size weight decay # hidden layers hidden size
MAML lr 1elearner lr (SN, BN, FR) 1e-2, 1e-4, 1e-