Original: W HERE AND W HEN TO L OOK ? S PATIO - TEMPORAL A TTENTION FOR A CTION R ECOGNITION IN V IDEOS
Revised: W HERE AND W HEN TO L OOK ? S PATIO - TEMPORAL A TTENTION FOR A CTION R ECOGNITION IN V IDEOS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Inspired by the observation that humans are able to process videos efﬁciently by only paying attentionwhen and where  it is needed, we propose a novel spatialtemporal attention mechanism for video-based action recognition.
Revised: Inspired by the observation that humans are able to process videos efﬁciently by only paying attention where and when it is needed, we propose a novel spatialtemporal attention mechanism for video-based action recognition.

Original: For spatial attention, we learn a saliency mask  to allow the model to focus on the most salient parts of the feature maps.
Revised: For spatial attention we learn a saliency mask only using convolutional layers to allow the model to focus on the most salient parts of the feature maps.

Original: For temporal attention, we employ a soft temporal attention mechanism to identify the most relevant frames from an input video.
Revised: For temporal attention we employ a convolutional LSTM based attention mechanism to identify the most relevant frames from an input video.

Original: Further, we propose a set of regularizers that ensure that our attention mechanism attends to coherent regions in space and time.
Revised: Further, we propose a set of regularizers that ensure that our attention mechanism attends to coherent regions in space and time.

Original: Our model is efﬁcient, as it proposes a separable spatio-temporal mechanism for video attention, while being able to identify important parts of the video  both spatially and temporally.
Revised: Our model can not only effectively improve video action recognition accuracy but also can localize discriminative regions both spatially and temporally, despite only trained in a weakly-supervised manner with only classiﬁcation labels (no bounding box spatial labels and time frame temporal labels).

Original: We demonstrate the efﬁcacy of our  approach on three public video action recognition datasets.
Revised: We evaluate our proposed approach on several public video action recognition datasets with ablation studies.

Original: The proposed approach leads to state-of-the-art performance on all of them, including the new large-scale Moments in Time dataset.

Original: Furthermore, we quantitatively and qualitatively evaluate our model’s ability toaccurately localize discriminative regions spatially and critical frames temporally.
Revised: Furthermore, we quantitatively and qualitatively evaluate our model’s ability to localize discriminative regions spatially and critical frames temporally.

Original: This is despite our model only being trained with per video classiﬁcation labels.
Revised: Experimental results demonstrate the efﬁcacy of our approach, showing superior or comparable accuracy with the state-of-the-art methods with the same input

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Machine learning models been widely adopted for solving various real world tasks, ranging from visual recognition (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014; He et al., 2016; Huang et al., 2017) to natural language processing (Iyyer et al., 2015; Goldberg, 2016). Although recordbreaking accuracy has been reported in many of the problems, an open question remains: “why does my model make that prediction?”
Revised: An important property of human perception is that one does not need to process a whole scene in its entirety at once. Instead, humans focus attention selectively on parts of the visual space to acquire information where and when it is needed, and combine information from different ﬁxations over time to build up an internal representation of the scene (Rensink, 2000), which can then be used for interpretation or decision making.

Original: (Koh & Liang, 2017). By understanding the reason or logic behind the decision process of a learning algorithm, one can further improve the model, discover new science (Shrikumar et al., 2017), provide end-users with explanations and guide them on how to effectively employ the results (Ribeiro et al., 2016).
Revised: In computer vision and natural language processing, over the last couple of years, attention models have proved similarly important.

Original: In particular, forvideo action recognition, a proper attention model can help answer the question of where and when it needs to look at the image evidence to draw a classiﬁcation decision.
Revised: Particularly for the tasks where interpretation or explanation requires only a small portion of the image or video.

Original: The attention mechanism in the human visual system is perhaps one of the most fascinating facets of intelligence. Instead of compressing an entire image into a static representation, attention allows for salient features to dynamically come to the forefront as needed. One important property of human perception is that one does not tend to process a whole scene in its entirety at once.
Revised: Examples include visual question answering (Lu et al., 2016; Xu & Saenko, 2016;

Original: Instead, humans focus attention selectively on parts of the visual space to acquire information when and where it is needed, and combine information from different ﬁxations over time to build up an internal representation of the scene (Rensink, 2000). The attention mechanism (Xu et al., 2015; Bahdanau et al., 2015; Cho et al., 2015; Vaswani et al., 2017) provides a way to bridge the gap between the black-box decision process and human interpretability.
Revised: Xiong et al., 2016), activity recognition (Sharma et al., 2015; Girdhar & Ramanan, 2017; Li et al., 2018b), and natural machine translation (Bahdanau et al., 2015). These models have also provided a level of interpretability, by visualizing regions selected or attended over for a particular task or decision. In particular, for video action classiﬁcation, a proper attention model can help answer the question of where and when it needs to look at the image evidence to draw a classiﬁcation decision.

Original: It intuitively explains which part the model attends to when making a particular decision, which is very helpful in real applications, e.g., medical AI systems or self-driving cars.
Revised: It intuitively explains which part the model attends to when making a particular decision, which is very helpful in real applications, e.g., medical AI systems or self-driving cars.

Original: In this paper, we propose a novel spatio-temporal attention mechanism that is designed to address these challenges.
Revised: In this paper, we propose a novel spatio-temporal attention mechanism that is designed to address these challenges.

Original: Our attention mechanism is efﬁcient, due to its space- and time- separability, and yet ﬂexible enough to enable encoding of effective regularizers (or priors).
Revised: Our attention mechanism is efﬁcient, due to its space- and time- separability, and yet ﬂexible enough to enable encoding of effective regularizers (or priors).

Original: As such, our attention mechanism consists of spatial and temporal components shown in Fig.
Revised: As such, our attention mechanism consists of spatial and temporal components shown in Fig.

Original: The spatial attention component, that attenuates frame-wise CNN image features, consists of the saliency mask; regularized 
Revised: The spatial attention component, that attenuates frame-wise CNN image features, consists of the saliency mask; regularized to be discriminative and spatially smooth.

Revised: The temporal component consists of a uni-modal soft attention mechanism that aggregates information over the near-by attenuated frame features before passing it into Convolutional LSTM for class prediction.

Revised: Contributions:

Revised: In summary, the main contributions of this work are: (1) We introduce a simple yet effective spatial-temporal attention for video action recognition, which consists of the saliency mask

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: -temporal attention for video action recognition.
Revised: -temporal attention for video action recognition.

Original: The convolutional features are attended over both spatially, in each frame, and subsequently temporally.
Revised: The convolutional features are attended over both spatially, in each frame, and subsequently temporally.

Original: Both attentions are soft, meaning that the effective ﬁnal representation at time t of an RNN, used to make the prediction, is a spatio-temporally weighted aggregation of convolutional features across the video along with the past hidden state from t − 1 .
Revised: Both attentions are soft, meaning that the effective ﬁnal representation at time t of an RNN, used to make the prediction, is a spatio-temporally weighted aggregation of convolutional features across the video along with the past hidden state from t − 1 .

Original: For details please refer to Sec. 3.
Revised: For details please refer to Sec. 3.

Original: to be discriminative andspatially smooth. The temporalcomponent consists of a uni-modal soft attention mechanism that aggregates information over the near-by attenuated frame features before passing it into Convolutional LSTM for class prediction. Contributions:
Revised: for spatial attention learned by ConvNets and temporal attention learned by convolutional LSTM.

Original: In summary, the main contributions of this work are: (1) We introduce a simple yet effective spatial-temporal attention mechanism for video action recognition; (2) We introduce three different regularizers, two for spatial and one for temporal attention components, to improve performance and interpretability of our model; (3) We illustrate state-of-the-art performance on three standard publicly available datasets. We also explore the importance of our modeling choices through ablation experiments. (4) Finally, we qualitatively and quantitatively show that our spatiotemporal attention is able to localize discriminative regions and important frames, despite being trained in a purely weakly-supervised manner with only classiﬁcation labels.
Revised: We introduce three different regularizers, two for spatial and one for temporal attention components, to improve performance and interpretability of our model; (3) We demonstrate the efﬁcacy of our model for video action recognition in three public datasets and explore the importance of our modeling choices through ablation experiments; (4) Finally, we qualitatively and quantitatively show that our spatio-temporal attention is able to localize discriminative regions and important frames, despite being trained in a purely weakly-supervised manner with only classiﬁcation labels.

Original:  R ELATED WORK 2. S PATIAL ATTENTION
Revised:  R ELATED WORK 2. N ETWORK INTERPRETATION

Original: Sharma et al.
Revised: Various methods have been proposed to try to explain neural networks (Zeiler & Fergus, 2014; Springenberg et al., 2014; Mahendran & Vedaldi, 2016; Zhou et al., 2016; Zhang et al., 2016; Simonyan et al., 2013; Ramprasaath et al., 2016; Ribeiro et al., 2016; 2018; Chang et al., 2018) in various of ways, including visualizing the gradients, perturbing the inputs, and bridging relations with other well-studied systems.

Original: (2015) develop an attention-driven LSTM by highlighting important spatial locations for action recognition.
Revised: Visual attention is also one way that tries to explain which part of the image is responsible for the network’s decision (Li et al., 2018a; Jetley et al., 2018). Besides the explanation, Li et al. (2018a) build up an end-to-end model to provide supervision directly on these explanations, speciﬁcally network’s attention. 2. V ISUAL ATTENTION FOR VIDEO ACTION RECOGNITION

Original: However, it only focuses on the crucial spatial locations ofeach image, without considering the temporal importance of different video frames in a video sequence.
Revised: For video action recognition, visualizing which part of the frame and which frame of the video sequence that the model was attending to provides valuable insight into the model’s behavior.

Original: Wang et al.
Revised: Sharma et al.

Original: (2016b) propose a Hierarchical Attention Network (HAN), which enables to incorporate attention of both spatial and motion stream into action recognition.
Revised: (2015) develop an attention-driven LSTM by highlighting important spatial locations for action recognition.

Original: Girdhar & Ramanan (2017) introduce an attention mechanism based on a derivation of bottom-up and top-down attention as low-rank approximations of bilinear pooling methods.
Revised: Girdhar & Ramanan (2017) introduce an attention mechanism based on a derivation of bottom-up and top-down attention as low-rank approximations of bilinear pooling methods.

Original: However, it only uses a single frame to predict and apply spatial attention, without consideringlong-term temporal relations among different frames.
Revised: However, these work only focus on the crucial spatial locations of each image, without considering temporal relations among different frames in a video sequence.

Original: Our method, however, uses asingle frame to both predict and apply spatial attention, making it amenable to both single image and video based use cases. 2. T EMPORAL ATTENTION
Revised: To alleviate this shortcoming, visual attention is incorporated in the motion stream (Wang et al., 2016b; Li et al., 2018b; Du et al., 2018). However, the motion stream only employs the optical ﬂow frames generated from consequent frames, cannot consider the long-term temporal relations among different frames in a video sequence.

Original: Visualizing where in the input the model was attending to at each output time-step produces an inputoutput alignment which provides valuable insight into the model’s behavior. Soft attention inspects every entry of the memory at each output time-step, effectively allowing the model to condition on an arbitrary input sequence entry. Bahdanau et al.
Revised: Moreover, motion stream needs additional optical ﬂow frames as input, which imposes burden due to additional optical ﬂow extraction, storage and computation and is especially severe for large datasets.

Original: (2015) propose to a soft attention RNN model to automatically (soft-)search for parts of a source sentence that are relevant to predicting a target word. Torabi & Sigal (2017) propose an attention based LSTM model to hightlight frames in videos.
Revised: Torabi & Sigal (2017) propose an attention based LSTM model to hightlight frames in videos, but spatial information is not used for temporal attention. An end-to-end spatial

Original: [Table]
Revised: [Table]

Original: +wflw6OWjlPFsMliEatOQDUKLrFpuBHYSRTSKBDYDsY3s3r7CZXmsXw0kwT9iA4lDzmjxloPd33eL1fcqjsXWQUvhwrkavTLX71BzNIIpWGCat313MT4GVWGM4HTUi/VmFA2pkPsWpQ0Qu1n81Wn5Mw6AxLGyj5pyNz9PZHRSOtJFNjOiJqRXq7NzP9q3dSEV37GZZIalGzxUZgKYmIyu5sMuEJmxMQCZYrbXQkbUUWZsemUbAje8smr0LqoepbvLyv16zyOIpzAKZyDBzWowy00oAkMhvAMr/DmCOfFeXc+Fq0FJ585hj9yPn8AHdyNqQ==</latexit>

Original: N ETWORK INTERPRETATION

Original: Various methods have been proposed to try to explain neural networks (Zeiler & Fergus, 2014; Springenberg et al., 2014; Mahendran & Vedaldi, 2016; Zhou et al., 2016; Zhang et al., 2016; Simonyan et al., 2013; Ramprasaath et al., 2016; Ribeiro et al., 2016; 2018) in various ways. Visual attention  is also one way that tries to explain which part of the image is responsible for the network’s decision (Li et al., 2018; Jetley et al., 2018). Besides the explanation, Li et al.
Revised: +wflw6OWjlPFsMliEatOQDUKLrFpuBHYSRTSKBDYDsY3s3r7CZXmsXw0kwT9iA4lDzmjxloPd33eL1fcqjsXWQUvhwrkavTLX71BzNIIpWGCat313MT4GVWGM4HTUi/VmFA2pkPsWpQ0Qu1n81Wn5Mw6AxLGyj5pyNz9PZHRSOtJFNjOiJqRXq7NzP9q3dSEV37GZZIalGzxUZgKYmIyu5sMuEJmxMQCZYrbXQkbUUWZsemUbAje8smr0LqoepbvLyv16zyOIpzAKZyDBzWowy00oAkMhvAMr/DmCOfFeXc+Fq0FJ585hj9yPn8AHdyNqQ==</latexit> and temporal attention model is proposed in ( et al., 2017) for human action recognition, but additional skeleton data is needed.

Original: (2018) build up an end-to-end model to provide supervision directly on these explanations, speciﬁcally network’s attention.

Original:  S PATIAL - TEMPORAL ATTENTION MECHANISM
Revised:  S PATIAL - TEMPORAL ATTENTION MECHANISM

Original: Our overall model is an Recurrent Neural Network (RNN) that aggregates frame-based convolutional features across the video to make action predictions as shown in Fig.
Revised: Our overall model is an Recurrent Neural Network (RNN) that aggregates frame-based convolutional features across the video to make action predictions as shown in Fig.

Original: The convolutional features are attended over both spatially, in each frame, and subsequently temporally.
Revised: The convolutional features are attended over both spatially, in each frame, and subsequently temporally.

Original: Both attentions are soft, meaning that the effective ﬁnal representation at time t of an RNN, used to make the prediction, is a spatio-temporally weighted aggregation of convolutional features across the video along with the past hidden state from t − 1 .
Revised: Both attentions are soft, meaning that the effective ﬁnal representation at time t of an RNN, used to make the prediction, is a spatio-temporally weighted aggregation of convolutional features across the video along with the past hidden state from t − 1 .

Original: The core novelty is the overall form of our attention mechanism and the additional terms of the loss function that induce sensible spatial and temporal attention priors.
Revised: The core novelty is the overall form of our attention mechanism and the additional terms of the loss function that induce sensible spatial and temporal attention priors.

Original: 3. C ONVOLUTIONAL FRAME FEATURES
Revised: 3. C ONVOLUTIONAL FRAME FEATURES

Original: We use the last convolutional layer output extracted by ResNet50  (He et al., 2016), pretrained on the ImageNet (Deng et al., 2009) dataset and ﬁne-tuned for the target dataset, as our frame feature representation.
Revised: We use the last convolutional layer output extracted by ResNet50 or ResNet101 (He et al., 2016), pretrained on the ImageNet (Deng et al., 2009) dataset and ﬁne-tuned for the target dataset, as our frame feature representation.

Original: We acknowledge that more accurate feature extractors (for instance, network with more parameters such as ResNet-154 or higher performance networks such as DenseNet (Huang et al., 2017) or SENet (Hu et al., 2018)) and/or optical ﬂow features will likely lead to better overall performance.
Revised: We acknowledge that more accurate feature extractors (for instance, network with more parameters such as ResNet-152 or higher performance networks such as DenseNet (Huang et al., 2017) or SENet (Hu et al., 2018)) and optical ﬂow features will likely lead to better overall performance.

Original: Our primary purpose in this paper is to prove the efﬁcacy of our  attention mechanism.
Revised: Our primary purpose in this paper is to prove the efﬁcacy of our spatial-temporal attention mechanism.

Original: Hence we kept the features relatively simple.
Revised: Hence we kept the features relatively simple.

Original: 3. S PATIAL ATTENTION WITH IMPORTANCE MASK
Revised: 3. S PATIAL ATTENTION WITH IMPORTANCE MASK

Original: We apply an importance mask M i to the i -th image features X
Revised: We apply an importance mask M i to the i -th image features X

Original: i to obtain attended image features by element-wise multiplication:
Revised: i to obtain attended image features by element-wise multiplication:

Original: [Equation]
Revised: [Equation]

Original: for 1 ≤ i ≤ n .
Revised: for 1 ≤ i ≤ n .

Original: This operation attenuates certain regions of the feature map based on their estimated importance.
Revised: This operation attenuates certain regions of the feature map based on their estimated importance.

Original: Here we simply use three convolutional layers to learn the importance mask.
Revised: Here we simply use three convolutional layers to learn the importance mask (please refer to Appendix B.2 for network architecture details).

Original: Fig.illustrates our spatial attention mechanism.
Revised: Fig. 2 illustrates our spatial attention mechanism.

Original: However, if left uncontrolled, an arbitrarily structured mask could be learned, leading to possible overﬁtting.
Revised: However, if left uncontrolled, an arbitrarily structured mask could be learned, leading to possible overﬁtting.

Original: We posit that, in practice, it is often useful to attend to a few important larger regions (e.g., objects, elements of the scene).
Revised: We posit that, in practice, it is often useful to attend to a few important larger regions (e.g., objects, elements of the scene).

Original: To induce this behavior, we encourage smoothness of the mask by introducing total variation loss on the spatial attention, as will be described in Sec. 3.4.[Table]
Revised: To induce this behavior, we encourage smoothness of the mask by introducing total variation loss on the spatial attention, as will be described in Sec. 3.4.

Original: /JzIaGTOJAtsZURyZ5drM/K/WTTG88jOhkhS5YouPwlQSjMnsbjIQmjOUEwuUaWF3JWxENWVo0ynZELzlk1ehdVH1LN9dVurXeRxFOIFTOAcPalCHW2hAExgM4Rle4c2Rzovz7nwsWgtOPnMMf+R8/gBA0I3A</latexit> <latexit sha1_base64="8+8QeEk4ypPqyMWL3xIIb/LkUA8=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEepJCl48VrQf0oay2W7apZtN2J0IJfQnePGgiFd/kTf/jds2B219YeHhnRl25g0SKQy67rdTWFvf2Nwqbpd2dvf2D8qHRy0Tp5rxJotlrDsBNVwKxZsoUPJOojmNAsnbwfhmVm8/cW1ErB5wknA/okMlQsEoWuv+sY/9csWtunORVfByqECuRr/81RvELI24QiapMV3PTdDPqEbBJJ+WeqnhCWVjOuRdi4pG3PjZfNUpObPOgISxtk8hmbu/JzIaGTOJAtsZURyZ5drM/K/WTTG88jOhkhS5YouPwlQSjMnsbjIQmjOUEwuUaWF3JWxENWVo0ynZELzlk1ehdVH1LN9dVurXeRxFOIFTOAcPalCHW2hAExgM4Rle4c2Rzovz7nwsWgtOPnMMf+R8/gBA0I3A</latexit

Original: [Figure]

Original: 3. T EMPORAL ATTENTION
Revised: 3. T EMPORAL ATTENTION

Original: Inspired by attention for neural machine translation (Bahdanau et al., 2015), we introduce the temporal attention mechanism which generates energy for each attended frame
Revised: Inspired by attention for neural machine translation (Bahdanau et al., 2015), we introduce the temporal attention mechanism which generates energy for each attended frame

Original: ˜ X
Revised: ˜ X

Original: i at each time step t ,
Revised: i at each time step t ,

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: /JzIaGTOJAtsZURyZ5drM/K/WTTG88jOhkhS5YouPwlQSjMnsbjIQmjOUEwuUaWF3JWxENWVo0ynZELzlk1ehdVH1LN9dVurXeRxFOIFTOAcPalCHW2hAExgM4Rle4c2Rzovz7nwsWgtOPnMMf+R8/gBA0I3A</latexit> <latexit sha1_base64="8+8QeEk4ypPqyMWL3xIIb/LkUA8=">AAAB6nicbZBNS8NAEIYn9avWr6pHL4tF8FQSEepJCl48VrQf0oay2W7apZtN2J0IJfQnePGgiFd/kTf/jds2B219YeHhnRl25g0SKQy67rdTWFvf2Nwqbpd2dvf2D8qHRy0Tp5rxJotlrDsBNVwKxZsoUPJOojmNAsnbwfhmVm8/cW1ErB5wknA/okMlQsEoWuv+sY/9csWtunORVfByqECuRr/81RvELI24QiapMV3PTdDPqEbBJJ+WeqnhCWVjOuRdi4pG3PjZfNUpObPOgISxtk8hmbu/JzIaGTOJAtsZURyZ5drM/K/WTTG88jOhkhS5YouPwlQSjMnsbjIQmjOUEwuUaWF3JWxENWVo0ynZELzlk1ehdVH1LN9dVurXeRxFOIFTOAcPalCHW2hAExgM4Rle4c2Rzovz7nwsWgtOPnMMf+R8/gBA0I3A</latexit

Original: (2) where H t − 1 represents the ConvLSTM hidden state at time t − 1 that implicitly contains all previous information up to time step t − 1 ,
Revised: [Figure] where H t − 1 represents the ConvLSTM hidden state at time t − 1 that implicitly contains all previous information up to time step t − 1 ,

Original: ˜ X i represents the i -th frame masked features.
Revised: ˜ X i represents the i -th frame masked features.

Original: Φ =
Revised: Φ =

Original: Φ H ( H t − 1 ) + Φ X ( ˜ X i ) , where Φ H and Φ X are feedforward neural networks which are jointly trained with all other components of the proposed system.
Revised: Φ H ( H t − 1 ) + Φ X ( ˜ X i ) , where Φ H and Φ X are feed-forward neural networks which are jointly trained with all other components of the proposed system.

Original: This temporal attention model directly computes soft attention weight for each frame at each time t as shown in Fig.
Revised: This temporal attention model directly computes soft attention weight for each frame at each time t as shown in Fig.

Original: It allows the gradient of the cost function to be backpropagated through.
Revised: It allows the gradient of the cost function to be backpropagated through.

Original: This gradient can be used to train the entire spatial-temporal attention model jointly.
Revised: This gradient can be used to train the entire spatial-temporal attention model jointly.

Original: The importance weight w ti for each frame is:
Revised: The importance weight w ti for each frame is:

Original: [Equation]
Revised: [Equation]

Original: for 1 ≤ i ≤ n, 1 ≤ t ≤ T .
Revised: for 1 ≤ i ≤ n, 1 ≤ t ≤ T .

Original: This importance weighting mechanism decides which frame of the video to pay attention to.
Revised: This importance weighting mechanism decides which frame of the video to pay attention to.

Original: The ﬁnal feature map Y t to the ConvLSTM is a weighted sum of the feature from all of the frames as the ConvLSTM cell inputs:
Revised: The ﬁnal feature map Y t to the ConvLSTM is a weighted sum of the feature from all of the frames as the ConvLSTM cell inputs:

Original: [Equation]
Revised: [Equation]

Original: where ˜ X i denotes the i -th masked frame of each video, n represents the total number of frames for each video.
Revised: where ˜ X i denotes the i -th masked frame of each video, n represents the total number of frames for each video.

Original: For RNN, instead of using conventional LSTM (Graves, 2013), we use Convolutional LSTM (ConvLSTM)
Revised: For RNN, instead of using conventional LSTM (Graves, 2013), we use Convolutional LSTM (ConvLSTM)

Original: (Shi et al., 2015) instead.
Revised: (Shi et al., 2015) instead.

Original: The drawback of conventional LSTM is its use of full connections in the input-to-state and state-to-state transitions in which no spatial information is encoded.
Revised: The drawback of conventional LSTM is its use of full connections in the input-to-state and state-to-state transitions in which no spatial information is encoded.

Original: In contrast,in ConvLSTM, each input Y t , cell output O t , hidden state H t and gateI t , F t , O t are 3D tensors whose last two dimensions are spatial dimensions (rows and columns). [Equation] O t = σ ( W xo ∗ Y t + W ho ∗ H t − 1 + b o ) ,
Revised: In contrast, each input, cell output, hidden state, gate are 3D tensors whose last two dimensions are spatial dimensions which can preserve spatial information, which is more suitable for image inputs.

Original: H t =

Original: O t ◦ tanh( C t ) .

Original: We use the following initialization strategy for the ConvLSTM cell state and hidden state for faster convergence:
Revised: We use the following initialization strategy for the ConvLSTM cell state and hidden state for faster convergence:

Original: [Equation]
Revised: [Equation]

Original: where g c and g h are two layer convolutional networks with batch normalization (Ioffe & Szegedy, 2015).
Revised: where g c and g h are two layer convolutional networks with batch normalization (Ioffe & Szegedy, 2015).

Original: We calculate the average hidden states of ConvLSTM over time length T ,
Revised: We calculate the average hidden states of ConvLSTM over time length T ,

Original: [Equation]
Revised: [Equation]

Original: and send it to a fully connected classiﬁcation layer for the ﬁnal video action classiﬁcation.
Revised: and send it to a fully connected classiﬁcation layer for the ﬁnal video action classiﬁcation.

Original: 3. L OSS FUNCTION
Revised: 3. L OSS FUNCTION

Original: Considering the spatial and temporal nature of our video action recognition; we would like to learn (1) a sensible attention mask for spatial attention, (2) reasonable importance weighting scores for different frames, and (3) improve the action recognition accuracy at the same time.
Revised: Considering the spatial and temporal nature of our video action recognition; we would like to learn (1) a sensible attention mask for spatial attention, (2) reasonable importance weighting scores for different frames, and (3) improve the action recognition accuracy at the same time.

Original: Therefore, our loss function L :
Revised: Therefore, our loss function L :

Original: [Equation]
Revised: [Equation]

Original: where L CE is the cross entropy loss for classiﬁcation, L TV represents the total variation regularization (Rudin et al., 1992); L contrast represents the mask and background contrast regularizer; and
Revised: where L CE is the cross entropy loss for classiﬁcation, L TV represents the total variation regularization (Rudin et al., 1992); L contrast represents the mask and background contrast regularizer; and

Original: L unimodal represents unimodality regularizer.
Revised: L unimodal represents unimodality regularizer.

Original: λ
Revised: λ

Original: TV , λ contrast and λ unimodal are the weights for corresponding regularizers.
Revised: TV , λ contrast and λ unimodal are the weights for corresponding regularizers.

Original: The total variation regularization L TV of the learnable attention mask encourages spatial smoothness of the mask and is deﬁned as:
Revised: The total variation regularization L TV of the learnable attention mask encourages spatial smoothness of the mask and is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where M i is the mask for the i -th frame, and M j,ki is entry at the ( j, k ) -th spatial location of the mask.
Revised: where M i is the mask for the i -th frame, and M j,ki is entry at the ( j, k ) -th spatial location of the mask.

Original: Different from the total variation of the mask of using L 2 loss in Dabkowski & Gal (2017), we use L 1 loss instead.
Revised: Different from the total variation of the mask of using L 2 loss in Dabkowski & Gal (2017), we use L 1 loss instead.

Original: The contrast regularization L contrast of learnable attention mask is to suppress the irrelevant information and highlight important information:
Revised: The contrast regularization L contrast of learnable attention mask is to suppress the irrelevant information and highlight important information:

Original: [Equation]
Revised: [Equation]

Original: where B i =
Revised: where B i =

Original: I { M i > 0 . 5 } represents the binarized mask, I is the indicator function applied elementwise.
Revised: I { M i > 0 . 5 } represents the binarized mask, I is the indicator function applied elementwise.

Original: The unimodality regularizer L unimodal encourages the temporal attention weights to be unimodal, biasing against spurious temporal weights.
Revised: The unimodality regularizer L unimodal encourages the temporal attention weights to be unimodal, biasing against spurious temporal weights.

Original: This stems from our observation that in most cases only one activity would be present in the considered frame window, with possible irrelevant information on either or both sides.
Revised: This stems from our observation that in most cases only one activity would be present in the considered frame window, with possible irrelevant information on either or both sides.

Original: Here we use the log concave distribution to encourage the unimodal pattern of temporal attention weights:
Revised: Here we use the log concave distribution to encourage the unimodal pattern of temporal attention weights:

Original: [Equation]
Revised: [Equation]

Original: where T represents the ConvLSTM time sequence length and n is the number of frames for each video.
Revised: where T represents the ConvLSTM time sequence length and n is the number of frames for each video.

Original: More details on this log concave sequence please refer to Appendix A.
Revised: More details on this log concave sequence please refer to Appendix A.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we ﬁrst conduct experiments to evaluate our proposed method on video action recognition task on three public available datasets.
Revised: In this section, we ﬁrst conduct experiments to evaluate our proposed method on video action recognition task on three public available datasets.

Original: Then we evaluate our spatial attention mechanism on the spatial localization task and our temporal attention mechanism on the temporal localization task respectively.[Table]
Revised: Then we evaluate our spatial attention mechanism on the spatial localization task and our temporal attention mechanism on the temporal localization task respectively.

Original: 4. V IDEO ACTION RECOGNITION
Revised: 4. V IDEO ACTION RECOGNITION

Original: We ﬁrst conduct extensive studies on the widely used HMDB51 and UCF101 datasets.
Revised: We ﬁrst conduct extensive studies on the widely used HMDB51 and UCF101 datasets.

Original: The purpose of these experiments is mainly for ablation study to examine the effects of different sub-components.
Revised: The purpose of these experiments is mainly for ablation study to examine the effects of different sub-components.

Original: Then we show that our method can be applied to the challenging large-scale Moments in Time dataset.
Revised: Then we show that our method can be applied to the challenging large-scale Moments in Time dataset.

Original: Datasets.
Revised: Datasets.

Original: HMDB51 dataset (Kuehne et al., 2011) contains 51 distinct action categories, each containing at least 101 clips for a total of 6,766 video clips extracted from a wide range of sources.
Revised: HMDB51 dataset (Kuehne et al., 2011) contains 51 distinct action categories, each containing at least 101 clips for a total of 6,766 video clips extracted from a wide range of sources.

Original: These videos include general facial actions, general body movements, body movements with object interaction, body movements for human interaction.
Revised: These videos include general facial actions, general body movements, body movements with object interaction, body movements for human interaction.

Original:  UCF101 dataset (Soomro et al., 2012) is an action recognition dataset of realistic action videos, collected from YouTube, with101 action categories.
Revised: [Table] UCF101 dataset (Soomro et al., 2012) is an action recognition dataset of realistic action videos, collected from YouTube, with

Original: Moments in Time Dataset (Monfort et al., 2018) is a collection of one million short videos with one action label per video and 339 different action classes.
Revised: Moments in Time Dataset (Monfort et al., 2018) is a collection of one million short videos with one action label per video and 339 different action classes.

Original: As there could be more than one action taking place in a video, action recognition models may predict an action correctly yet be penalized because the ground truth does not include that action.
Revised: As there could be more than one action taking place in a video, action recognition models may predict an action correctly yet be penalized because the ground truth does not include that action.

Original: Therefore, it is believed that top 5 accuracy measure will be more meaningful for this dataset.
Revised: Therefore, it is believed that top 5 accuracy measure will be more meaningful for this dataset.

Original: Experimental setup.
Revised: Experimental setup.

Original: We use the same parameters for HMDB51 and UCF101: single Convolutional
Revised: We use the same parameters for HMDB51 and UCF101: single Convolutional

Original: LSTM layer with hidden-state dimension 512, sequence length T = 50 , λ TV = 10 − 5 , λ contrast = 10 − 4 , λ unimodal = 1 .
Revised: LSTM layer with hidden-state dimension 512, sequence length T = 25 , λ TV = 10 − 5 , λ contrast = 10 − 4 , λ unimodal = 1 .

Original: For the Moments in Time dataset, we use time sequence length T = 15 .
Revised: For the Moments in Time dataset, we use time sequence length T = 15 .

Original: For more details on the experimental setup please refer to Appendix B.1.
Revised: For more details on the experimental setup please refer to Appendix B.1.

Original: Quantitative results.
Revised: Quantitative results.

Original: We show the top-1 video action classiﬁcation accuracy for HMDB51 and
Revised: We show the top-1 video action classiﬁcation accuracy for HMDB51 and

Original: UCF101 dataset in Table 1.
Revised: UCF101 dataset in Table 1.

Original: It is clear from the table that the proposed method outperforms the conventional ResNet50-ImageNet and visual attention  (Sharma et al., 2015).
Revised: Our proposed model outperforms previous attention based model (Sharma et al., 2015; Li et al., 2018b; Girdhar & Ramanan, 2017) and conventional ResNet101ImageNet.

Original: From the ablation experiments, it demonstrates that all the sub-components of the proposed method contribute to improving the ﬁnal performance.
Revised: From the ablation experiments, it demonstrates that all the sub-components of the proposed method contribute to improving the ﬁnal performance.

Original: The results on the Moments in Time dataset are reported in Table 2.
Revised: The results on the Moments in Time dataset are reported in Table 2.

Original: Our method achieves the best accuracy comparing to other single-modality-based methods, and obtains better or comparative results comparing to the methods which uses more than one modality.
Revised: Our method achieves the best accuracy comparing to other single-modality-based methods, and obtains better or comparative results comparing to the methods which uses more than one modality.

Original: TRN-Multiscale (Zhou et al., 2018), which uses both RGB and optical ﬂow images, has better performance than ours, however, extracting optical ﬂow images for such large datasets is very time-consuming and needs the same order of magnitude of storage as RGB images.
Revised: TRN-Multiscale (Zhou et al., 2018), which uses both RGB and optical ﬂow images [Table] has better performance than ours, however, extracting optical ﬂow images for such large datasets is very time-consuming and needs the same order of magnitude of storage as RGB [Table]

Original: Qualitative results.
Revised: Qualitative results.

Original: We visualize the spatial attention and temporal attention results in Fig.
Revised: We visualize the spatial attention and temporal attention results in Fig.

Original: We can see that the spatial attention can correctly focus on important spatial area of the image, and the temporal attention shows a unimodal distribution for the entire action from starting the action to completing the action.
Revised: We can see that the spatial attention can correctly focus on important spatial area of the image [Table] and the temporal attention shows a unimodal distribution for the entire action from starting the action to completing the action.

Original: More results are shown in Appendix C.1.
Revised: More results are shown in Appendix C.1.

Revised: [Figure]

Revised: [Figure]

Original: 4. W EAKLY SUPERVISED LOCALIZATION
Revised: 4. W EAKLY SUPERVISED LOCALIZATION

Original: Due to the existence of spatial and temporal attention mechanisms, our model can not only classify the action of the video, but also give a better interpretability of the results, i.e. telling which region and frames contribute more to the prediction.
Revised: Due to the existence of spatial and temporal attention mechanisms, our model can not only classify the action of the video but also give a better interpretability of the results i e. telling which region and frames contribute more to the prediction.

Original: In other words, our proposed model can also localize [Table] [Figure] [Figure]
Revised: In other words our proposed model can also localize the most discriminant region and frames at the same time.

Original: the most discriminant region and frames at the same time.

Original: To verify this, we conduct the spatial localization and temporal localization experiments.
Revised: To verify this we conduct the spatial localization and temporal localization experiments.

Original: 4.2. S PATIAL ACTION LOCALIZATION
Revised: 4.2. S PATIAL ACTION LOCALIZATION

Original: Dataset.
Revised: Dataset.

Original: UCF101-24 is a subset of 24 classes out of 101 classes of UCF101 that comes with spatio-temporal localization annotation, released as bounding box annotations of humans with THUMOS2013 and THUMOS2014 challenge (Jiang et al., 2014).
Revised: UCF101-24 is a subset of 24 classes out of 101 classes of UCF101 that comes with spatio-temporal localization annotation, released as bounding box annotations of humans with THUMOS2013 and THUMOS2014 challenge (Jiang et al., 2014).

Original: Experimental setup.
Revised: Experimental setup.

Original: We use the same hyper-parameters for UCF101-24 as HMDB51 and UCF in the previous section.

Original: The model we used for feature extraction is our pretrained ResNet50 model on UCF101 in the previous experiment.

Original: For training, we only use the classiﬁcation labels without spatial bounding box labels.
Revised: For training, we only use the classiﬁcation labels without spatial bounding box labels.

Original: For evaluation, we threshold the produced saliency mask at 0.5 and the tightest bounding box that contains the thresholded saliency map is set as the predicted localization box for each frame.
Revised: For evaluation, we threshold the produced saliency mask at 0.5 and the tightest bounding box that contains the thresholded saliency map is set as the predicted localization box for each frame.

Original: Then these predicted localization boxes are compared with the ground truth bounding boxes at different Intersection Over Union (IOU) levels.[Table] [Table]
Revised: Then these predicted localization boxes are compared with the ground truth bounding boxes at different Intersection Over Union (IOU) levels.

Original: Qualitative results.
Revised: Qualitative results.

Original: We show some qualitative results in Fig.
Revised: We show some qualitative results in Fig.

Original: Our spatial attention can attend to important action areas.
Revised: Our spatial attention can attend to important action areas.

Original: The ground truth bounding boxes include all the entire human actions, while our attention could attend to crucial parts of an action such as in Fig.5 (d) and (e).
Revised: The ground truth bounding boxes include all the entire human actions, while our attention could attend to crucial parts of an action such as in Fig.5 (d) and (e).

Original: Furthermore, our attention mechanism is able to attend to areas with multiple human actions.
Revised: Furthermore, our attention mechanism is able to attend to areas with multiple human actions.

Original: For instance, in Fig.
Revised: For instance, in Fig.

Original: (f) the ground truth only includes one person bicycling [Table] but our attention can include both people bicycling.
Revised: (f) the ground truth only includes one person bicycling, but our attention can include both people bicycling.

Original: More qualitative results including failure cases are included in Appendix C.2.
Revised: More qualitative results including failure cases are included in Appendix C.2.

Original: Quantitative results.
Revised: Quantitative results.

Original: With our spatial-temporal attention mechanism, the action recognition accuracy for UCF101-24 is 93 .

Original: 04% , which is a 6 . 12% performance boost compared with 86 .

Original: 92% by averaging the predictions of ResNet-50 over 50 frames.

Original: Table 3 shows the quantitative results for UCF101-24 spatial localization results.
Revised: Table 3 shows the quantitative results for UCF101-24 spatial localization results.

Original: Our attention mechanism works better compared with the baseline methods when the IoU threshold is lower mainly because our model only focuses on important spatial areas rather than the entire human action annotated by bounding boxes. 
Revised: Our attention mechanism works better compared with the baseline methods when the IoU threshold is lower mainly because our model only focuses on important spatial areas rather than the entire human action annotated by bounding boxes. Compared with the baseline methods train- [Table]

Original: Compared with the baseline methods training with ground truth bounding boxes, we only use the action classiﬁcation label, no ground truth bounding boxes are used.
Revised: [Table] [Table] ing with ground truth bounding boxes, we only use the action classiﬁcation label, no ground truth bounding boxes are used.

Original: 4.2. T EMPORAL ACTION LOCALIZATION
Revised: 4.2. T EMPORAL ACTION LOCALIZATION

Original: Dataset.
Revised: Dataset.

Original: The action detection task of THUMOS14 (Jiang et al., 2014) consists of 20 classes of sports activities, and contains 2765 trimmed videos for training, while 200 and 213 untrimmed videos for validation and test respectively.
Revised: The action detection task of THUMOS14 (Jiang et al., 2014) consists of 20 classes of sports activities, and contains 2765 trimmed videos for training, while 200 and 213 untrimmed videos for validation and test respectively.

Original: More details on this dataset and pre-processing are included in
Revised: More details on this dataset and pre-processing are included in

Original: Appendix B.1.
Revised: Appendix B.1.

Original: Experimental setup.
Revised: Experimental setup.

Original: We use the same hyperparameters for THUMOS14 as HMDB51, UCF and UCF101-24.
Revised: We use the same hyperparameters for THUMOS14 as HMDB51, UCF and UCF101-24.

Original: For training, we only use the classiﬁcation labels without temporal annotation labels.
Revised: For training, we only use the classiﬁcation labels without temporal annotation labels.

Original: For evaluation, we threshold the normalized temporal attention importance weight at 0.5.
Revised: For evaluation, we threshold the normalized temporal attention importance weight at 0.5.

Original: Then these predicted temporal localization frames are compared with the ground truth annotation at different IoU thresholds.
Revised: Then these predicted temporal localization frames are compared with the ground truth annotation at different IoU thresholds.

Original: Qualitative results.
Revised: Qualitative results.

Original: We ﬁrst visualize some examples of learned attention weights on the test data of
Revised: We ﬁrst visualize some examples of learned attention weights on the test data of

Original: THUMOS14 in Fig.
Revised: THUMOS14 in Fig.

Original: We see that our temporal attention module is able to automatically highlight important frames and to avoid irrelevant frames corresponding to background or non-action human poses.
Revised: We see that our temporal attention module is able to automatically highlight important frames and to avoid irrelevant frames corresponding to background or non-action human poses.

Original: More qualitative results are included in Appendix C.3.
Revised: More qualitative results are included in Appendix C.4.

Original: Quantitative results.
Revised: Quantitative results.

Original: With our spatial temporal attention mechanism, the video action classiﬁcation accuracy for the THUMOS’14 20 classes improved from 74 .
Revised: With our spatial temporal attention mechanism, the video action classiﬁcation accuracy for the THUMOS’14 20 classes improved from 74 .

Original: 45% to 78 .
Revised: 45% to 78 .

Original: 33% : a 3 . 88% increase.
Revised: 33% : a 3 . 88% increase.

Original: Besides improving the classiﬁcation accuracy, we show our temporal attention mechanism is able to highlight discriminative frames quantitatively in Table 4.
Revised: Besides improving the classiﬁcation accuracy, we show our temporal attention mechanism is able to highlight discriminative frames quantitatively in Table 4.

Original: Compared with strongly supervised method (Yeung et al., 2016) and weakly supervised method (Wang et al., 2017), our method achieves the best accuracy in terms of different levels of IoU thresholds. 
Revised: Compared with reinforcement learning based method (Yeung et al., 2016) and weakly supervised method (Wang et al., 2017), our method achieves the best accuracy in terms of different levels of IoU thresholds. [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work we develop a novel spatial-temporal attention mechanism for the task of video action recognition, demonstrating the efﬁcacyand showing state-of-the-art performance across three publicly available datasets.
Revised: In this work we develop a novel spatial-temporal attention mechanism for the task of video action recognition, demonstrating the efﬁcacy across three publicly available datasets.

Original: Also, we introduce a set of regularizers that ensure our attention mechanism attends to coherent regions in space and time, further improving the performance and increasing the model interpretability.
Revised: Also, we introduce a set of regularizers that ensure our attention mechanism attends to coherent regions in space and time, further improving the performance and increasing the model interpretability.

Original: Moreover, we qualitatively and quantitatively show that our spatio-temporal attention is able to localize discriminative regions and important frames, despite being trained in a purely weakly-supervised manner with only classiﬁcation labels.
Revised: Moreover, we qualitatively and quantitatively show that our spatio-temporal attention is able to localize discriminative regions and important frames, despite being trained in a purely weakly-supervised manner with only classiﬁcation labels.

Original: A L OG - CONCAVE SEQUENCE
Revised: A L OG - CONCAVE SEQUENCE

Original: In probability and statistics a unimodal distribution is a probability distribution that has a single peak or mode.
Revised: In probability and statistics, a unimodal distribution is a probability distribution that has a single peak or mode.

Original: If a distribution has more modes it is called multimodal.
Revised: If a distribution has more modes it is called multimodal.

Original: The temporal attention weights are a univariate discrete distribution over the frames, indicating the importance of the frames for the task of classiﬁcation.
Revised: The temporal attention weights are a univariate discrete distribution over the frames, indicating the importance of the frames for the task of classiﬁcation.

Original: In the context of activity recognition, it is reasonable to assume that the frames that contain salient information should be consecutive, instead of scattered around.
Revised: In the context of activity recognition, it is reasonable to assume that the frames that contain salient information should be consecutive, instead of scattered around.

Original: Therefore, we would like to design a regularizer that encourages unimodality.
Revised: Therefore, we would like to design a regularizer that encourages unimodality.

Original: To this end, we introduce a mathematical concept called the log-concave sequence and deﬁne the regularizer based on it.
Revised: To this end, we introduce a mathematical concept called the log-concave sequence and deﬁne the regularizer based on it.

Original: We ﬁrst give a formal deﬁnition of the unimodal sequence.
Revised: We ﬁrst give a formal deﬁnition of the unimodal sequence.

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: A sequence { a i } ni =1 is unimodal if for some integer m,
Revised: A sequence { a i } ni =1 is unimodal if for some integer m,

Original: [Equation]
Revised: [Equation]

Original: A univariate discrete distribution is unimodal, if its probability mass function forms a unimodal sequence.
Revised: A univariate discrete distribution is unimodal, if its probability mass function forms a unimodal sequence.

Original: The log-concave sequence is deﬁned as follows.
Revised: The log-concave sequence is deﬁned as follows.

Original: Deﬁnition 2.
Revised: Deﬁnition 2.

Original: A non-negative sequence { a i } ni =1 is log-concave if a 2 i ≥ a i − 1 a
Revised: A non-negative sequence { a i } ni =1 is log-concave if a 2 i ≥ a i − 1 a

Original: i +1 .
Revised: i +1 .

Original: This property gets its name from the fact that if { a i } ni =1 is log-concave, then the sequence { log a k } ni =1 is concave.
Revised: This property gets its name from the fact that if { a i } ni =1 is log-concave, then the sequence { log a k } ni =1 is concave.

Original: The connection between unimodality and log-concavity is given by the following proposition.
Revised: The connection between unimodality and log-concavity is given by the following proposition.

Original: Proposition 1.
Revised: Proposition 1.

Original: A log-concave sequence is unimodal.
Revised: A log-concave sequence is unimodal.

Original: Proof.
Revised: Proof.

Original: Rearranging the deﬁning inequality for log-concavity, we see that
Revised: Rearranging the deﬁning inequality for log-concavity, we see that

Original: [Equation]
Revised: [Equation]

Original: so the ratio of consecutive terms is decreasing.
Revised: so the ratio of consecutive terms is decreasing.

Original: Until the ratios decrease below 1, the sequence is increasing, and after this point, the sequence is decreasing, so it is unimodal.
Revised: Until the ratios decrease below 1, the sequence is increasing, and after this point, the sequence is decreasing, so it is unimodal.

Original: Given the deﬁnition of log-concavity, it is straightforward to design a regularization term that encourages log-concavity:
Revised: Given the deﬁnition of log-concavity, it is straightforward to design a regularization term that encourages log-concavity:

Original: [Equation]
Revised: [Equation]

Original: By Proposition 1, this regularizer also encourages unimodality.
Revised: By Proposition 1, this regularizer also encourages unimodality.

Original: B M ORE DATASETS AND IMPLEMENTATION DETAILS B. M ORE DETAILS ON THE DATASET AND EXPERIMENTAL SETUP
Revised: B M ORE DATASETS AND IMPLEMENTATION DETAILS B. M ORE DETAILS ON THE DATASET AND EXPERIMENTAL SETUP

Original: HMDB51 and UCF

Revised: HMDB51 and UCF

Original: The dataset pre-processing and data augumentation are the same as the
Revised: The dataset pre-processing and data augumentation are the same as the

Original: ResNet ImageNet experiment (He et al., 2016).
Revised: ResNet ImageNet experiment (He et al., 2016).

Original: All the videos are resized to 224 × 224 resolution and fed into a ResNet-50 pretrained on ImageNet.
Revised: All the videos are resized to 224 × 224 resolution and fed into a ResNet-50 pretrained on ImageNet.

Original: The last convolutional layer feature map size is 2048 × 7 × 7 .
Revised: The last convolutional layer feature map size is 2048 × 7 × 7 .

Original: The experimental setup for the Moments in Time dataset is the same as HMDB51 and
Revised: The experimental setup for the Moments in Time dataset is the same as HMDB51 and

Original: UCF101 except time sequence and image resolution.
Revised: UCF101 except time sequence and image resolution.

Original: Moments in Time For the Moments in Time dataset (Monfort et al., 2018), the videos only haveseconds, much shorter than HMDB51 and UCF101.
Revised: Moments in Time For the Moments in Time dataset (Monfort et al., 2018), the videos only haveseconds, much shorter than HMDB51 and UCF101.

Original: We extract RGB frames from the raw videos at 5 fps.
Revised: We extract RGB frames from the raw videos at 5 fps.

Original: Therefore, the sequence length is T = 15 .
Revised: Therefore, the sequence length is T = 15 .

Original: Following the practice in (Monfort et al., 2018) to make all the videos uniform resolution, we resize the RGB frames to 340 × 256 pixels.
Revised: Following the practice in (Monfort et al., 2018) to make all the videos uniform resolution, we resize the RGB frames to 340 × 256 pixels.

Original: When extracting features, we use the ResNet-50 pretrained on ImageNet model using resized images with resolution of 256 × 256 pixels.
Revised: When extracting features, we use the ResNet-50 pretrained on ImageNet model using resized images with resolution of 256 × 256 pixels.

Original: The data augmentation is the same as the ResNet ImageNet experiment (He et al., 2016).
Revised: The data augmentation is the same as the ResNet ImageNet experiment (He et al., 2016).

Original: The feature map size of the last convolutional layer is 2048 × 8 × 8 .
Revised: The feature map size of the last convolutional layer is 2048 × 8 × 8 .

Original: THUMOS

Revised: THUMOS

Original: The action detection task of THUMOS’14 (Jiang et al., 2014) consists of 20 classes of sports activities, and contains 2765 trimmed videos for training, while 200 and 213 untrimmed videos for validation and test respectively.
Revised: The action detection task of THUMOS’14 (Jiang et al., 2014) consists of 20 classes of sports activities, and contains 2765 trimmed videos for training, while 200 and 213 untrimmed videos for validation and test respectively.

Original: Following the standard practice (Yeung et al., 2016; Zhao et al., 2017), we use the validation set as training and evaluate on the testing set.
Revised: Following the standard practice (Yeung et al., 2016; Zhao et al., 2017), we use the validation set as training and evaluate on the testing set.

Original: 220 videos in validation set and 212 videos in test set have temporal annotations in 20 classes.

Original: This dataset is particularly challenging as it consists of very long videos (up to a few hundreds of seconds) with multiple activity instances of very small duration (up to few tens of seconds).

Original: Most videos contain multiple activity instances of the same activity class.

Original: In addition, some videos contain activity segments from different classes.

Original: Following the standard practice (Xu et al., 2017) to avoid the training ambiguity, we remove the videos with multiple labels.
Revised: Following the standard practice (Xu et al., 2017) to avoid the training ambiguity, we remove the videos with multiple labels.

Original: We extract RGB frames from the raw videos at 10 fps.
Revised: We extract RGB frames from the raw videos at 10 fps.

Original: We use the ResNet-50 pretrained on ImageNet and ﬁne-tuned on Thumos14, and the preprocessing and data augmentation of Thumos14 is the same as ImageNet.

Original: The last convolutional layer feature map size is 2048 × 7 × 7 .
Revised: The last convolutional layer feature map size is 2048 × 7 × 7 .

Revised: B. S PATIAL ATTENTION NETWORK ARCHITECTURE

Revised: The detailed architecture of spatial attention network described in Section 3.2 is listed in the Table 5.

Revised: [Table]

Original: B. M ORE IMPLEMENTATION DETAILS
Revised: B. M ORE IMPLEMENTATION DETAILS

Original: All the experiments are evaluated on machines with a single Nvidia GeForce GTX 1080Ti GPU.
Revised: All the experiments are evaluated on machines with a single Nvidia GeForce GTX 1080Ti GPU.

Original: The networks are implemented using the Pytorch library and our code will be publicly available with this paper.
Revised: The networks are implemented using the Pytorch library and our code will be publicly available with the paper.

Original: C M ORE RESULTS C. M ORE SPATIAL TEMPORAL ATTENTION RESULTS
Revised: C M ORE RESULTS C. M ORE SPATIAL TEMPORAL ATTENTION RESULTS

Original: Fig.
Revised: Fig.

Original: 7 shows more results on spatial temporal attention.
Revised: 7 shows more results on spatial temporal attention.

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: C. M ORE SPATIAL LOCALIZATION RESULTS
Revised: C. M ORE SPATIAL LOCALIZATION RESULTS

Original: Fig.
Revised: Fig.

Original: 8 shows more spatial localization results.
Revised: 8 shows more spatial localization results.

Original: Fig.
Revised: Fig.

Original: 9 shows some failure cases
Revised: 9 shows some failure cases.

Revised: [Figure]

Revised: [Table]

Original: . C. M ORE TEMPORAL LOCALIZATION RESULTS
Revised: C

Revised: M ORE ACTION RECOGNITION RESULTS

Revised: Table 6 shows results of our spatial-temporal attention model with different base networks.

Revised: Our spatial-temporal attention mechanism is a easy plug-in model which could be based on different network architectures, and can boost performance.

Revised: C. M ORE TEMPORAL LOCALIZATION RESULTS

Original: Fig.
Revised: Fig.

Original: 10 shows more results on temporal localization with our temporal attention.
Revised: 10 shows more results on temporal localization with our temporal attention.