Spatio-Temporal Feature Learning
Anonymous Author(s)
AfﬁliationAddress email
Abstract
In modern medicine, patient vital sign information is often collected ashigh-dimensional multi-channel time series data, which contains both spatialand temporal information.
In this paper, we propose a hybrid featurelearning model containing both spatial and temporal autoencoders to learndeep feature representations of time series data.
We use a publicly availableelectroencephalograph (EEG) dataset to evaluate our model’s classiﬁcationperformance and compare the results to: (i) using raw data as features, and (ii)features learned from various combinations of spatial and temporal autoencoders.
Our ﬁndings highlight that the way in which we exploit spatial and temporalcorrelations makes a signiﬁcant difference and demonstrate the effectiveness of ourmodel in processing multivariate time series patient data.
Introduction
In modern medicine, patient data is often collected as high-dimensional multi-channel time seriesdata, which contain both spatial and temporal information.
An important example of such dataare electroencephalograph (EEG) signals, which measure the electrical activity of the brain fromelectrodes placed along the scalp (see Fig. 1).
The records are usually time series data withlarge dimensionality and high sampling frequency.
This makes the use of EEG signals for diseaseclassiﬁcation and predictive diagnosis challenging as large-scale data processing suffers from thecurse of dimensionality [1].
Therefore, extracting a compact feature representation from the raw datafacilitates practical and (hopefully) better clinical decision making.
Interestingly, EEG signals typically havespatial and temporal correlations due to theplacement of the electrodes on the scalp (asshown in Figure 1).
However, extractingspatio-temporal correlations and useful featuresfrom EEG signals requires expert knowledgein conventional feature extraction approaches.
These supervised approaches are domain-drivenand sometimes do not generalize well.
Toovercome the pitfalls of supervised methods,the autoencoder as a data-driven unsupervisedapproach has been proposed to directly learnfeature representations from large-scale data[2].
However, recent work in [3], [4] and [5]does not speciﬁcally cater for learning compactrepresentations and dependencies from both thespatial and temporal dimensions of the data.
Inspired by [6] which uses the idea of spatio-temporal autoencoders to predict the next video frame, inthis paper, we propose a hybrid feature learning model containing spatial and temporal autoencodersto learn deep feature representations from both the spatial and temporal domains.
We evaluate itsperformance and compare the results to: (i) using raw data as features, and (ii) features learned fromvarious combinations of spatial and temporal autoencoders.
Our ﬁndings highlight that the way inwhich we exploit spatial and temporal correlations makes a signiﬁcant difference and demonstratethe effectiveness of our model in processing multivariate time series patient data.
Exploiting Spatial and Temporal Information
It is well known that compact representations can be derived from raw data in an unsupervisedmanner using autoencoders [2].
In this paper, we explore an autoencoder type structure to captureboth spatial and temporal dependencies in raw time series data in an unsupervised manner.
We usea standard neural network based autoencoder [2] to extract the spatial information, and we choosean autoencoder constructed with long short term memory (LSTM) cells [7] to learn the temporalinformation.
These two autoencoders, called the spatial autoencoder (SAE) and temporal autoencoder(TAE) respectively, are shown in Fig.
Not surprisingly, the SAE and TAE can be stacked in variousways to yield hybrid combinations.
What is surprising is that the way in which we stack them leads towidely varying performance and suggests that, at least for the datasets we studied, stacking an SAEfollowed by a TAE is better for classiﬁcation (as compared to stacking a TAE followed by an SAE).
2.1 Capturing Spatial Features
The SAE is a standard autoencoder constructed with multi-layer neural networks, consisting of aninput layer, an encoder and a decoder.
We use X to represent the raw patient data, and x to representthe reconstructed patient data from an autoencoder.
X is the multivariate time series data, which hasthe dimension of S × T , where S is the number of channels, and T is the length of the time series.
For example, in EEG data, S is the number of electrodes placed on the scalp; in ICU data, S could bethe number of vital signs.
Note that the reconstructed patient data from an autoencoder, x has thesame dimension as X .
The encoder in SAE contains only one layer with S (cid:48) number of neurons that compress the number ofchannels in the input data X from S to S (cid:48) while learning a more compact representation Y ∈ R S (cid:48)
× Tin the spatial domain.
This encoding process does not touch the information carried in the temporaldomain.
The decoder in SAE reconstructs the input data x based on the encoded features Y .
SAEcan be optimized by stochastic gradient descent and trained by minimizing the reconstruction errorbetween X and x through the mean squared error loss function
L ( X, x ) .
2.2 Capturing Temporal Features
Similar to the SAE, the TAE also has an input layer, an encoder and a decoder.
In this paper, we use
LSTM cells with standard structure to construct the encoder and the decoder of the TAE.
A total of T (cid:48)
LSTM cells in the encoder are used to learn and extract temporal dependencies from the multivariatetime series inputs X , which have dimension of S × T .
We take all the outputs from every LSTMunit in the TAE encoder, and consider them as the temporal feature representations h ∈ R S × T (cid:48) of theinput data.
The decoder contains another set of T LSTM units which are attached after the encoder tomap the encoded temporal feature representations h back to the input of TAE, X .
The TAE can beoptimized by gradient descent and trained by minimizing the reconstruction error between X and xthrough the mean squared error loss function
L ( X, x ) .
2.3 Hybrid Stacked Combinations
As motivated by [8], the two types of autoencoders described above can be stacked together to learndeeper feature representations from the spatial and temporal domains.
For example, we use SSAE todenote the autoencoder which stacks two SAEs together and TTAE to denote the autoencoder whichstacks two TAE in a chain.
In these stacked autoencoders models, raw data is passed into the ﬁrststage to learn one set of feature representations and then passed into the next stage to continue tolearn another set of feature representations.
The autoencoder at each stage is trained and optimizedindividually.
Unlike SAEs and TAEs, where information in only one domain is captured, it is natural to stackan SAE and TAE together (ﬁrst explored in [6] for video frame prediction).
We explore variouscombinations of the SAE and TAE to learn compact spatio-temporal feature representations frommultivariate time series medical patient data.
We denote the combination of one SAE followedby one TAE as STAE, and one TAE followed by one SAE as TSAE.
Taking STAE as an example,multivariate time series data is passed into the SAE ﬁrst
so as the spatial information in the raw datais extracted without touching the time domain.
The outputs from the SAE, which contains compactspatial feature representations are then fed into a TAE to further learn representations capturing thetemporal characteristics.
The resultant feature set learned from STAE is the deep spatio-temporalfeature representation of the raw multivariate time series patient data.
We note that [9] also considered a STAE-type structure for multivariate time series data and include adiscussion and comparison in Sec. 4.
Classiﬁcation Performance and Results
In this section, we present the classiﬁcation problem and dataset considered in this paper, and evaluatethe classiﬁcation performance of various feature representations learned from the dataset with threedifferent classiﬁers.
The EEG dataset extracted from the UCI Machine Learning Repository [10] contains EEG time seriesdata from 1200 different patients.
Each patient’s EEG time series data contains measurements from64 channels for 256 time stamps.
Therefore, one piece of EEG data has a dimension of 64 × 256 .
Each patient is labeled as either alcoholic (Class 0) or non-alcoholic (Class 1).
In the EEG datasetused in our study, both Class 0 and Class 1 contain 600 patients equally.
The classiﬁcation problemconsidered in this study is to classify whether a patient is alcoholic (Class 0) or non-alcoholic (Class1), based on her EEG measurements.
The results presented here are averaged based on 5-fold crossvalidation repeated 10 times.
In Table 1, we compare the classiﬁcation performance using the raw EEG data and the differentfeature sets learned by various autoencoder combinations, for three different classiﬁers: supportvector machine (SVM), gradient boosting (GB) and random forest (RF) [1].
With the SVM classiﬁer,it can be seen that the feature set learned by STAE achieves the highest classiﬁcation accuracyamong all different feature learning methods, and also outperforms the raw data.
Using the STAEfeature set, we can classify a patient to be alcoholic with 82.1% accuracy, which is about 2.4%, 4.9%and 5.3% higher than using the feature sets extracted from SAE, TAE, and TSAE for classiﬁcationrespectively.
Similar observations in classiﬁcation accuracy can also been seen in Table 1 with the GBand RF classiﬁers.
These observations imply that feature representations extracted from the spatial ortemporal domain solely do not well represent the multivariate time series data.
Furthermore, Table 1 shows that feature representations learned from STAE achieve the highestscores in the AUC-ROC and F1-score with the SVM classiﬁer.
Similar patterns can be seen from the
AUC-ROC and F1-score with the GB and RF classiﬁers.
[Table]
We also notice that with all three different classiﬁers, classiﬁcation with feature sets extracted by
SAE and TAE purely do not always achieve better accuracies and F1-scores as compared to theclassiﬁcation with the raw EEG data.
This may be because the information carried by one domain(i.e.
spatial or temporal) is not a good representative for the entire dataset distribution.
In other words,the dependencies between the spatial and temporal measurements are not fully captured.
Another interesting comparison is between the STAE and TSAE models.
Since both feature learningmodels use the combination of SAE and TAE (albeit in different order), one would expect them tohave similar performance.
Surprisingly, the classiﬁcation results of the two differ signiﬁcantly.
Theresults presented in Table 1 show that, STAE-SVM outperforms TSAE-SVM by 5.3% in accuracyand F1-score, and 4.2% in AUC-ROC score.
Similar results are obtained with the GB classiﬁer,where STAE-GB performs better than TSAE-GB by around 9.1% in both accuracy and F1-score, and10.2% in AUC-ROC score.
The same pattern is also found with the RF classiﬁer, where STAE-RFperforms better than TSAE-RF by over 13.0% in accuracy, AUC-ROC score and F1-score.
We alsonote that TSAE performs poorly as compared to the raw EEG data with all three classiﬁers.
Our ﬁnding that STAE consistently beats the SAE/TAE/TSAE combinations is robust in the sensethat we have tested it on strong classiﬁers such as SVM, weak classiﬁers such as decision trees (DT),and on a variety of other time-series datasets.
We plan to present these results at the workshop itself.
Reﬂections
In this paper, we propose to learn deep feature representations from multivariate time series patientdata via spatial and temporal autoencoders.
When stacking a single type of autoencoder in asequence, feature representations from one domain can be learned.
Feature representations thatcontain information in multiple domains can be obtained from a hybrid model which contains astacked combination of different types of autoencoders.
Our key ﬁnding is that the order of theautoencoders in the stacking sequence seems to make a signiﬁcant difference in the performance overa wide range of classiﬁers.
We contend that a more careful study and explanation of this phenomenonare excellent open research questions relevant to the community.
We note that [9] carries out a related study, in which the combination of a classic neural networkbased autoencoder and an LSTM autoencoder is used to learn features in patient data with missingobservations.
The main difference between this previous study and our work in this paper has to dowith the feature representation output from the temporal autoencoder.
The autoencoder in [9] uses theinformation at the last time stamp as the feature representation, which is similar to what is done forvideo prediction [6].
However, in our paper, information at each time stamp are used as the featurerepresentations for the entire time series.
We ﬁnd that, for multivariate time series data, keeping onlythe information at the last time stamp is inferior to keeping information at every time stamp.
For comparison, we conduct a similar experiment to one done in [9] where 30% of the raw datais dropped to emulate missing observations.
We apply linear imputation before using our hybridcombination of SAE and TAE.
With our model, STAE-SVM with 30% missing observations canachieve 73.5% classiﬁcation accuracy and an AUC-ROC score of 0.807.
These numbers are slightlyhigher than the 72.3% accuracy and 0.790 AUC-ROC reported in [9].
The improvement in the resultsdemonstrates the effectiveness of our model in learning spatio-temporal features from multivariatetime series data.