From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper21/Authors
title: Response to comments
comment: 1. The comment about why stacking order affects performance is one we are thinking deeply about. One possible reason might be that when data is passed into the LSTM unit, the multiple features  are summarized, possibly reducing the original spatial dependencies. When you capture the spatial dependencies first, you don’t necessarily break the temporal dependencies in the process.  This could explain why STAE is better for classification. 

2. Regarding the comment about compression: The hidden states of the LSTM unit have one output at each time step and there are multiple LSTM units. The amount of compression depends upon the number of LSTM units, which is typically less than the number of features. 

3. Regarding further experiments, we observed similar results for the PTB Diagnostic ECG database on Physionet. We will discuss these results and other issues related to performance at the workshop.  

4. Thanks for the suggestion on the paper title. We plan to change the title to: Deep Spatio-Temporal Feature Learning using Autoencoders 

5. Thanks for the pointers to relevant literature. We plan to take a closer look at these works.

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper21/Authors
title: Response to comments
comment: 1. Thanks for the suggestion on the paper title. We plan to change the title to: Deep Spatio-Temporal Feature Learning using Autoencoders 

2. The comment about why stacking order affects performance is one we are thinking deeply about. One possible reason might be that when data is passed into the LSTM unit, the multiple features  are summarized, possibly reducing the original spatial dependencies. When you capture the spatial dependencies first, you don’t necessarily break the temporal dependencies in the process.  This could explain why STAE is better for classification. 

3. Regarding further experiments, we observed similar results for the PTB Diagnostic ECG database on Physionet. We will discuss these results and other issues related to performance at the workshop.  

4. Regarding the comment about just using an LSTM, we agree that the LSTM can capture both spatial and temporal dependencies to a certain degree. In fact, the TAE architecture is just an LSTM. However, the STAE stacked architecture consistently outperforms TAE, suggesting it captures both types of dependencies better. 

5. Thanks for the comment regarding baselines. In the workshop paper, we were focusing on the feature learning and hence included only simple classifiers. We will explore and discuss deep classifiers as a baseline.

From: ~Ransalu_Senanayake1
comment: Dear authors,

You should now be able to see the reviews. We encourage you to engage in the discussion by commenting on the reviews. Please attempt to clarify the issues highlighted by the reviewers.  

Regards,
Ransalu, Neal, Fabio, and Girish
title: Rebuttal

From: ~Ransalu_Senanayake1
comment: Brief Summary:
-	The paper presents a method that suggests that the order that one stacks multiple autoencoders learned on both spatial and temporal data has a measurably significant effect on performance

General Comments:
-	The title is all broadly encompassing while the method and application is very specific. Possible suggestions would be “Spatio-temporal Feature Leaning with Autoencoders” or “Deep Spatio-temporal Feature Learning” or “Deep Spatio-temporal Feature Learning for electroencephalography” 

Strengths:
-	There are good comparisons against standard baseline methods
-	The application domain is interesting and clearly relevant to the workshop focus
-	The work appears to extend similar work used in video frame prediction into a more general framework

Weaknesses:
-	It is not clear to what extent different architecture orderings affect performance. While the ordering has been highlighted briefly, it is not clear what parameterizations have what effect
-	The method is only evaluated on a single dataset and it is not clear how the developed methodology generalizes to different situations (i.e. different datasets)
-	It is not clear why one could not just use an LSTM which should be capable, to some degree, of capturing both spatial and temporal relations in its latent layers. A comparison would be useful
-	The autoencoders specifically appear to be used as a dimensionality reduction and there are no baselines against default network types (e.g. just an LSTM, or any standard NN for classification)

Summary of Recommendation:
-	Weak Accept
title: From: NIPS 2018 Workshop Spatiotemporal Paper21 AnonReviewer2 (appeared as an official comment)

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper21/AnonReviewer2
relevance: 5: very high
impact: 3: borderline
novelty: 3: borderline

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper21/AnonReviewer1
evaluation: This paper presents a technique to compress EEG signals while capturing spatiotemporal relationships. To this end, spatial data are encoded using an autoencoder, and temporal data are compressed using an LSTM. Experiments indicate that the proposed hybrid approach has superior performance.

As a general concept, I liked the idea of spatiotemporal data compression. However, the paper should be well-motivated why this is an important problem. Capturing space-time dependencies is well-corroborated by experiments. It's a major strength of the paper to have SAE and TAE alone in addition to testing TSAE and STAE. However, the architecture setup in section 2.3 and the claim "the way in which we stack them leads to 53 widely varying performance and suggests that, at least for the datasets we studied, stacking an SAE 54 followed by a TAE is better for classification (as compared to stacking a TAE followed by an SAE)" should be discussed in depth. For instance, why do you think this is the case? What is the intuition? 

Further details about the architecture and experiments should be provided. I did not understand how the LSTM architecture used here is capable of compressing data. Experiments with more datasets are expected. It is extremely important to evaluate the quality of compression and how sensitive is the compression to the parameters such as the size of the autoencoder. 

Importantly, the title of the paper is an overly generic. I would suggest a title along the lines of "Spatiotemporal data compression for electroencephalography using deep neural networks."

    1. What did you learn from reading this paper that you did not know before?
Importance of capturing spatiotemporal relationships in EEG data.
    2. What was the most exciting part of the work?
Spatiotemporal data compression.
    3. What in your opinion makes this work stand apart from other papers in this area?
Testing spatial, temporal, and spatiotemporal domains separately.
    4. Does the paper fit into the scope of the workshop?
Yes
    5. Are you aware of relevant literature that you may want to make the authors aware of?
* Spatiotemporal Sparse Bayesian Learning with Applications to Compressed Sensing of Multichannel Physiological Signals
* Compression of Spatio-temporal Data

Recommendation: poster presentation

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper21/AnonReviewer1
relevance: 5: very high
impact: 3: borderline
novelty: 3: borderline

From: ~Oliver_Paul_Richter1
comment: Thanks for the write up! Some comments/questions from my side:
Throughout the paper it is unclear to me, how exactly you set up your architecture. Since the architecture is the main focus of your paper, I would explain it in more detail.
 - You say that you "stack" the autoencoders. Given two autoencoders, with encoders e_1 and e_2 and decoders d_1 and d_2, do you mean the stacked autoencoder is e_1>e_2>d_2>d_1 (hopefully) or e_1>d_1>e_2>d_2. From the text one might guess that it is the first arrangement, but a more precise formulation would help.
 - You use an LSTM for the temporal encoding. However, from the paper it is unclear to me, whether you understood what an LSTM actually does.  In the reflection (line 156-157) you state: "[...] information at each time stamp are used as the feature representations for the entire time series." You also refer to the "output" of multiple LSTM as the encoding and you say, that the time dimension T gets T' (line 70-79). My questions here:
 a) Are you using a single LSTM cell with T' hidden units? Or S hidden units and T=T'? Or what are the hidden state sizes of your LSTMs?
 b) Do you take the hidden state of the LSTM as encoding or the output (after the output gate)? The former gives an unconstrained latent space while the later is constraint by the activation function of the output (normally a tanh)
 c) Are you aware that an LSTM projects for each time step the input on to the number of hidden units and adjusts the hidden state according to last hidden state, input and forget estimates? I.e., are you aware that a LSTM also learns spatial relations?
 d) Are you aware that if you take every output at every time step of the LSTM as latent code of the autoencoder, you might as well just learn a spatial feature extraction and no time dependencies what so ever?
 - You never mention, how big your latent encodings actually are. (What is S'? What is T'?) Since an autoencoder is mainly about compression, it would be interesting how much you actually compress the features.
 - On your evaluation: Cross-validation is normally used for model selection (e.g. whether to use an SVM or a Random Forest). Model accuracy on the other hand should be reported on a held out test set which was not used in any way during training. Or is that what you report?
 - And last but not least: Why don't you train a neural network classification model directly? (Or at least as comparison?)
title: Question on model architecture and evaluation method