Paper:
Many state-of-the-art methods of recognizing human action are based on attention mechanism, which shows the importance of attention mechanism in action recognition. With the rapid development of neural networks, human action recognition has been achieved great improvement by using convolutional neural networks (C-NN) or recurrent neural networks (RNN). In this paper, we propose a model based on spatial-temporal attention weighted LSTM. This model pays attention to the key part in each video frame, and also focuses on the important frames in each video sequence, thus the most important theme for our model is how to find out the key point spatially and the key frames temporally. We show a feasible architecture which can solve those two problems effectively and achieve a satisfactory result. Our model is trained and tested on three datasets including UCF-11, UCF-101, and HMDB51. Those results demonstrate a high performance of our model in human action recognition.

Review:
# 1. Summary
This paper presents a spatio-temporal attention LSTM for action recognition, where attention decides which pixels and frames are more important for classification. ConvNet features are extracted, a first layer of attention looks at the pixel level, then a second layer is applied at the temporal level. An LSTM is used to connect frame representation through time.
Weaknesses:
* The paper do not present substantial novelty compared to previous work. In fact, it has a strong overlap with (Song et al., 2017) (see #3)
* Some modeling choices are not well motivated (see #2)
* Ablation study showing that each modeling decision are motivated from a practical perspective is missing (see #4)
* The paper fails in comparing with relevant papers (see #4)
# 2. Clarity and Motivation
* Page 1 “many new deep learning methods of action recognition would use iDT as one part of their networks to optimize their models ”: this statement is not clear, please provide references of methods that do this. To my knowledge iDTs are part of the input of a ConvNet or used as complementary feature to other networks (e.g., C3D, I3D, …).
* Page 1, “Two-stream CNN (Feichtenhofer et al., 2016b)”: The reference might not really accurate. The citation should be: Two-Stream Convolutional Networks for Action Recognition in Videos Karen Simonyan, Andrew Zisserman.
*Page 1, “The pure RNN-based models are usually used on skeleton data“: it is not clear right away why the authors discuss some paper about skeleton data since it is not an application studied in this paper.
* Page 3, Sec. 2.2, “\alpha_t is a matrix”: why is it a matrix? It seems that it is a vector of length 196.
* By reshaping the features as Fig. 2, you loose the spatial consistency between neighbour pixels. How does Eq 2 deals with this? It seems that a better approach will be to have convolutions instead of fully-connected layers in Eq. 2. Have the authors considered this option?
* In neural machine translation models, usually the weights are normalized with a softmax before the weighted average of Eq. 3 and 6. It seems that here the alphas are normalized but the betas are not before Eq. 6. Any explanation about this?
* My comment above seems that is also related with the need of the regularisation term in Eq. 8. Probably it is not really needed in case that the betas are also normalized.
* A discussion is missing why the two attention models (spatial and temporal) are different. In principle, one could adopt the same kind of attention model for both the spatial and temporal component. One reason to have different models would be to consider the spatial relations between neighbour features in a frame (which is not the case of this model, as I highlighted above)
* Page 6, “The second term is applied to force the model to pay more spatial attention on more relevant parts in the frame automatically […]. The third term is used to restrict the unlimited increasing of temporal attention“: this sentence is a bit unclear. More details and intuition about how the 2nd and 3rd term work would be appreciated.
# 3. Novelty
From a methodological point of view the paper is not novel enough. It seems that the model is a combination between of the soft attention mechanism (Xu et al., 2015) (Sharma et al., 2016) and the temporal attention mechanism (Song et al., 2017). The overlap with the latter paper is substantial. From the application point of view, there is also little novelty given that the paper is tested on action recognition using relatively-small datasets.
# 4. Experimentation
Since this paper is an application paper, and there is not much novelty about the model, one would expect a comprehensive set of experiments. 
* Ablation study is missing. Looking at the model in Fig. 3, it seems that not all the components are required. Some questions should be: 
** Since attention is already working on the temporal domain, why do we need an LSTM model which seems redundant? 
** What is the impact of removing the first layer of attention (\alpha)? And the temporal one (\beta)?
* The selected datasets are bit small scale. It would have been nice to see some results with bigger and more challenging datasets, such as Kinetics or similar.
* The paper fails in comparing with relevant papers (table 1). The topic of action recognition is widely explored, specifically for the datasets used in this paper. References and comparison numbers can be found here, for example: http://www.actionrecognition.net/files/dsetdetail.php?did=6;  and   http://www.actionrecognition.net/files/dsetdetail.php?did=5;