Spatio-temporal Stacked LSTM for Temperature Prediction in Weather Forecasting
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Long Short-Term Memory (LSTM) is a well-known method used widely on sequence learning and time series prediction.
In this paper we deployed stacked
LSTM model in an application of weather forecasting.
We propose a 2-layer spatiotemporal stacked LSTM model which consists of independent LSTM models perlocation in the ﬁrst LSTM layer.
Subsequently, the input of the second LSTMlayer is formed based on the combination of the hidden states of the ﬁrst layer
LSTM models.
The experiments show that by utilizing the spatial information theprediction performance of the stacked LSTM model improves in most of the cases.
Introduction
The weather system is a challenging complex system and state-of-the-art methods utilize Numerical
Weather Prediction (NWP) which is a computationally intense method [2].
Recently, data-drivenapproaches for weather forecasting have become a major interest of researchers as they are computationally simpler and more straightforward [3, 9, 10, 11].
Long Short-Term Memory (LSTM), proposed by Hochreiter & Schmidhuber [8], is a popular type ofrecurrent neural network which is able to capture long-term dependencies.
LSTMs have been widelyused and shown signiﬁcant performance on different sequence learning problems and time seriesprediction [14, 17, 7, 4, 12, 18].
LSTMs have been also used in analyzing spatio-temporal datasets[15, 13].
Stacked LSTM is a deep architecture which consists of more than one layer of LSTM andthe input of each LSTM layer is the hidden states of the previous LSTM layer [6, 17].
In this paper a spatio-temporal stacked LSTM model is proposed and its performance is evaluatedon the application of temperature prediction.
In the proposed model independent LSTM modelsper location are trained and afterward, the input of the second layer LSTM is formed based on thecombination of the hidden states of the LSTM models in the ﬁrst layer.
It is worth mentioning thatthe general structure of the spatio-temporal stacked LSTM is similar to the approach proposed by
Su et al.
in the framework of Convolutional Neural Networks for 3D shape recognition [16].
Notethat both stacked LSTM and spatio-temporal stacked LSTM methodologies can be explained as amulti-view approach as, similar to [9], they are fusing the information of different cities.
Stacked
LSTM and spatio-temporal stacked LSTM beneﬁt from early fusion and intermediate fusion of theinformation from different views respectively.
Spatio-temporal Stacked LSTM
Assuming i [ l ] t , f
[ l ] t , o [ l ] t , c [ l ] t and h [ l ] t to be the values of the input gate, forget gate, output gate, memorycell and hidden state at time t in the sequence and layer l respectively, and x t,k be the input of thesystem at time t at location k , the stacked LSTM model based on the architecture of the LSTM cell deﬁned in [5] is shown in Table 1.
Note that x t as an input of the model is a concatenation of thevariables from all locations; i.e. x t =
[ x t, 1 , x t, 2 , · · · , x t,c ] T .
In this study we focus on a 2-layerstacked LSTM; however, the methodology can be extended to a larger number of layers.
The fullweight matrices W xj for j ∈ { i, f, o, c } are the weights that connect the input to the correspondinggates and the memory cell.
The weight matrices W cj for j ∈ { i, f, o } are diagonal matrices thatconnect the cell memory to different gates.
Note that the number of neurons for all gates is a predeﬁnedparameter and the equations are applied for each neuron.
For simplicity, we use column vectors w [ l ] lstmand b [ l ] lstm to indicate all the elements in { W [ l ] xi , W
[ l ] ci , W
[ l ] xf , W
[ l ] hf , W [ l ] cf , W
[ l ] xc , W
[ l ] hc , W [ l ] xo , W
[ l ] ho , W
[ l ] co }and { b [ l ] i , b
[ l ] f , b [ l ] c , b [ l ] o } respectively.
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
− 1 ,c [2] t − 1 ,h [1] t ; w [2]lstm ,b [2]lstm )
In Table 2 the equations for the proposed spatio-temporal stacked LSTM model are shown.
Asis explained, instead of one LSTM model in the ﬁrst layer, there are independent LSTM models per location.
Hence, having 5 locations, 5 LSTM models are created.
The full weight matrices W xj,k for j ∈ { i, f, o, c } refer to the connection of the data of location k to differentgates in the corresponding LSTM model.
Similarly, other weight matrices, biases, gate values,memory cell and hidden state used subscript k to indicate that they correspond to the LSTMrelated to the location k .
For simplicity, we use column vectors w [ l ] lstm ,k and b [ l ] lstm ,k which include all the elements in { W [ l ] xi,k , W
[ l ] ci,k , W
[ l ] xf,k , W
[ l ] hf,k , W [ l ] cf,k , W
[ l ] xc,k , W
[ l ] hc,k , W
[ l ] xo,k , W
[ l ] ho,k , W [ l ] co,k }and { b [ l ] i,k , b [ l ] f,k , b [ l ] c,k , b
[ l ] o,k } to refer to the parameters for the LSTM part related to location k fork ∈ { 1 , 2 , · · · c } in the ﬁrst layer.
The information from different locations are then combined bymerging the hidden states of the ﬁrst layer and passing it as input to the second layer.
For the second
LSTM layer the deﬁnition of w [ l ] lstm and b [ l ] lstm remains the same.
Figure 1 depicts the stacked LSTMand spatio-temporal stacked model when the number of layers is equal to two.
As is shown, for thestacked LSTM model, the hidden states of the ﬁrst layer are used as the input of the second layer.
For the second LSTM layer the deﬁnition of w [ l ] lstm and b [ l ] lstm remains the same.
However, in caseof spatio-temporal stacked LSTM, there are independent LSTM models per location, and afterwardthe input of the second LSTM layer is deﬁned based on the combination of the hidden states of the
LSTM models of the ﬁrst layer.
Note that if the number of layers is more that two, the mergingof the hidden states is possible at any layer before the last LSTM layer; e.g. if we have a 3-layerspatio-temporal stacked LSTM model, the combination of the hidden states can happen after theﬁrst LSTM layer or the second one.
Note that in both stacked LSTM and spatio-temporal stacked
LSTM models, after the second LSTM layer, a dense layer is used.
The ﬁnal prediction can be doneby using ˆ y t + T + q = w T dense h [2] t + T + b [2]dense where q is the number of days ahead to predict, T is theinput sequence length, and w dense and b dense are the weights and bias term in the dense layer.
For theexperiments, we use a quadratic loss function to train the network and utilize L 2 -norm regularizationto avoid overﬁtting.
One of the advantages of the proposed spatio-temporal stacked LSTM method is a smaller numberof parameters in comparison with stacked LSTM.
Assume the total number of neurons in the ﬁrstlayer and second layer is similar in both cases; in other words, if the number of neurons in stacked
LSTM model in the ﬁrst layer is n 1 , then in the spatio-temporal stacked LSTM, each LSTM model inthe ﬁrst layer has n 1 c neurons where c is the number of locations.
In this case, the spatio-temporalstacked LSTM is similar to the case that the full weight matrices in stacked LSTM are considered tobe block diagonal at which point each block is related to a location.
Hence, the number of parametersto be optimized is smaller in the proposed method.
This makes the spatio-temporal stacked LSTM a
[Table] [Table]
k h [1] t − 1 ,k + W [1] co,k c [1] t,k + b [1] o,k ) o
[2] t = σ ( W [2] xo h [2] t + W [2] ho h
[2] t − 1 + W [2] co c [2] t + b [2] o ) h
[1] t,k
= o [1] t,k (cid:12) tanh( c [1] t,k ) h [2] t
= o
[2] t (cid:12) tanh( c [2] t )
Summary c [1] t,k = f ( c [1] t − 1 ,k ,h [1] t − 1 ,k ,x t,k ; w [1]lstm ,k ,b [1]lstm ,k ) c [2] t = f
( c [2] t − 1 ,h [2] t − 1 ,h [1] t ; w [2]lstm ,b [2]lstm ) h [1] t,k
= g ( h [1] t − 1 ,k ,c [1] t − 1 ,k ,x t,k ; w [1]lstm ,k ,b [1]lstm ,k ) h [2] t = g ( h [2] t − 1 ,c [2] t − 1 ,h [1] t ; w [2]lstm ,b [2]lstm )
LSTM LSTM
LSTM …
D e n s e
L a y e r ℎ 𝑡[2]
𝑥 𝑡+1 𝑥 𝑡 𝑦 𝑡 𝑦 𝑡+ 𝑥 𝑡+𝑇 𝑦 𝑡+𝑇 𝑦 𝑡+𝑇+𝑠
LSTM
LSTM
LSTM … ℎ 𝑡+1[2] ℎ 𝑡+𝑇 [2] ℎ 𝑡 [1] ℎ 𝑡[1] ℎ 𝑡+1[1] ℎ 𝑡+𝑇 [1] ℎ 𝑡+1[1] (
[Figure]
LSTM LSTM
LSTM … ℎ 𝑡[2] 𝑦 𝑡 𝑦 𝑡+ 𝑦 𝑡+𝑇 𝑦 𝑡+𝑇+𝑠 ℎ 𝑡+1[2] ℎ 𝑡+𝑇[2]
D e n s e L a y e r 𝑥
𝑡+1,1 𝑥 𝑡, 𝑥 𝑡+𝑇,
LSTM
LSTM
LSTM … ℎ 𝑡,1[1] ℎ 𝑡+𝑇,1 [1] ℎ 𝑡+1,1[1] 𝑥 𝑡+1,𝑐 𝑥
𝑡,𝑐
𝑥
𝑡+𝑇,𝑐
LSTM
LSTM
LSTM … ℎ 𝑡,𝑐[1] ℎ 𝑡+𝑇,𝑐[1] ℎ 𝑡+1,𝑐[1] . . .
Information from location Information from location
[Figure]
better choice when the number of samples in the training set is relatively small.
On the other hand,in spatio-temporal stacked LSTM, the relation between the locations are taken into account in thesecond LSTM layer by combining the hidden states from the ﬁrst layer.
Experiments
In this paper, the data have been collected fromthe Weather Underground company website [1]and cover a time period from the beginning of2007 to mid-2014 for 5 cities including Brussels,
Antwerp, Liege, Amsterdam and Eindhoven.
Toevaluate the performance of the proposed methods in various weather conditions, two test setsare deﬁned: (i) from mid-November 2013 to midDecember 2013 (Nov/Dec) and (ii) from midApril 2014 to mid-May
(Apr/May).
Thedata contain 18 measured weather variables, suchas temperature and humidity, for each day percity.
In order to beneﬁt from all available data,the training data that is used for each test set includes the data from the beginning of 2007 untilthe day before the corresponding test set.
[Figure]
In this study, the LSTM cell architecture described by Zaremba & Vinyals [19] implemented in
TensorFlow has been used for the experiments.
The considered range for the tuning parameters were selected empirically.
For the number of neurons, in the stacked LSTM we examined{ 20 , 40 , 80 , 160 , 320 , 640 } in the ﬁrst layer and { 32 , 64 , 128 , 256 } for the second layer.
In case ofthe spatio-temporal stacked LSTM, the number of neurons in LSTM per location in the ﬁrst layer
[Table]
is considered to be in the set of { 4 , 8 , 16 , 32 , 64 , 128 } .
Note that as there are 5 locations in the ﬁrstlayer, the total number of neurons in the ﬁrst layer are similar in both models.
For the inner state, wedeployed both tanh and sigmoid as the activation function.
In the experiments, the sequence length isconsidered to be 10.
The experiments were conducted for the prediction of the minimum and maximum temperature in
Brussels for 1 to 6 days ahead.
To avoid local minima problems in neural networks, the experimentsare repeated 5 times and the median Mean Absolute Error (MAE) and the Mean Squared Error (MSE)on both test sets are presented in Table 3.
As is shown, using sigmoid as the inner activation functioncan result in better performance.
Moreover, it can be seen that in most of the cases taking the spatialinformation into account can improve the performance.
This is more evident in case the activationfunction in the inner state is tanh.
In addition, the comparison between the performance of the stacked LSTM, the proposed methodand Weather Underground over the two test sets together is depicted in Figure 3.
As is shown, although very few locations aretaken into account, the LSTM models (as datadriven approaches) outperform the state-of-theart method used for minimum temperature prediction.
Also, for maximum temperature predictionthe performance of the LSTM models are competitive with the performance of the state-of-the-artmethods.
Conclusion
In this study, we proposed a spatio-temporalstacked LSTM model at which point in the ﬁrstlayer different LSTM models are considered perlocation, and then the corresponding hidden statesare merged and given as input to the next layer.
The proposed method was deployed in an application of weather forecasting.
The experimental results suggest that consideringspatio-temporal property of the data in the LSTMmodel can improve the performance of the prediction.
Moreover, it is shown that the proposedmethod is competitive with the state-of-the-artmethod in weather forecasting.
6 Days ahead0.51.52.
M AE f o r M i n .
T e m p .
[Table]
-Stacked
[Table]
[Figure]
temperature prediction for Weather Underground, stacked LSTM, and Spatio-Temporal stacked