Original: L EARNING S ELF -S IMILARITY IN S PACE AND T IME AS A G ENERALIZED M OTION FOR A CTION R ECOGNITION
Revised: L EARNING S ELF -S IMILARITY IN S PACE AND T IME AS A G ENERALIZED M OTION FOR A CTION R ECOGNITION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Spatio-temporal convolution often fails to learn motion dynamics in videos and thus an effective motion representation is required for video understanding in the wild.
Revised: Spatio-temporal convolution often fails to learn motion dynamics in videos and thus an effective motion representation is required for video understanding in the wild.

Original: In this paper, we propose a rich and robust motion representation method based on spatio-temporal self-similarity (STSS).
Revised: In this paper, we propose a rich and robust motion representation method based on spatio-temporal self-similarity (STSS).

Original: Given a sequence of frames,
Revised: Given a sequence of frames,

Original: STSS represents each local region as similarities to its neighbors in space and time.
Revised: STSS represents each local region as similarities to its neighbors in space and time.

Original: By converting appearance features into relational values, it enables the learner to better recognize structural patterns in space and time.
Revised: By converting appearance features into relational values, it enables the learner to better recognize structural patterns in space and time.

Original: We leverage the whole volume of STSS and let our model learn to extract an effective motion representation from it.
Revised: We leverage the whole volume of STSS and let our model learn to extract an effective motion representation from it.

Original: The proposed method is implemented as a neural block, dubbed SELFY , that can be easily inserted into neural architectures and learned end-to-end without additional supervision.
Revised: The proposed method is implemented as a neural block, dubbed SELFY , that can be easily inserted into neural architectures and learned end-to-end without additional supervision.

Original: With a sufﬁcient volume of the neighborhood in space and time, it effectively captures long-term interaction and fast motion in the video, leading to robust action recognition.
Revised: With a sufﬁcient volume of the neighborhood in space and time, it effectively captures long-term interaction and fast motion in the video, leading to robust action recognition.

Original: Our experimental analysis demonstrates its superiority over previous methods for motion modeling as well as its complementarity to spatio-temporal features from direct convolution.
Revised: Our experimental analysis demonstrates its superiority over previous methods for motion modeling as well as its complementarity to spatio-temporal features from direct convolution.

Original: On the standard action recognition benchmarks, Something-Something-V1 & V2, Diving-48, and FineGym, the proposed method achieves the state-of-the-art results
Revised: On the standard action recognition benchmarks, Something-Something-V1 & V2, Diving-48, and FineGym, the proposed method achieves the state-of-the-art results

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Learning spatio-temporal dynamics is the key to video understanding.
Revised: Learning spatio-temporal dynamics is the key to video understanding.

Original: To this end, extending convolutional neural networks (CNNs) with spatio-temporal convolution has been actively investigated in recent years (Tran et al., 2015; Carreira & Zisserman, 2017; Tran et al., 2018).
Revised: To this end, extending convolutional neural networks (CNNs) with spatio-temporal convolution has been actively investigated in recent years (Tran et al., 2015; Carreira & Zisserman, 2017; Tran et al., 2018).

Original: The empirical results so far indicate that spatio-temporal convolution alone is not sufﬁcient for grasping the whole picture; it often learns irrelevant context bias rather than motion information (Materzynska et al., 2020) and thus the additional use of optical ﬂow turns out to boost the performance in most cases (Carreira &
Revised: The empirical results so far indicate that spatio-temporal convolution alone is not sufﬁcient for grasping the whole picture; it often learns irrelevant context bias rather than motion information (Materzynska et al., 2020) and thus the additional use of optical ﬂow turns out to boost the performance in most cases (Carreira &

Original: Zisserman, 2017; Lin et al., 2019).
Revised: Zisserman, 2017; Lin et al., 2019).

Original: Motivated by this, recent action recognition methods learn to extract explicit motion, i.e. , ﬂow or correspondence, between feature maps of adjacent frames and they improve the performance indeed
Revised: Motivated by this, recent action recognition methods learn to extract explicit motion, i.e. , ﬂow or correspondence, between feature maps of adjacent frames and they improve the performance indeed

Original: (Li et al., 2020c; Kwon et al., 2020).
Revised: (Li et al., 2020c; Kwon et al., 2020).

Original: But, is it essential to extract such an explicit form of ﬂows or correspondences?
Revised: But, is it essential to extract such an explicit form of ﬂows or correspondences?

Original: How can we learn a richer and more robust form of motion information for videos in the wild?
Revised: How can we learn a richer and more robust form of motion information for videos in the wild?

Original: In this paper, we propose to learn spatio-temporal self-similarity (STSS) representation for video understanding.
Revised: In this paper, we propose to learn spatio-temporal self-similarity (STSS) representation for video understanding.

Original: Self-similarity is a relational descriptor for an image that effectively captures intrastructures by representing each local region as similarities to its spatial neighbors (Shechtman &
Revised: Self-similarity is a relational descriptor for an image that effectively captures intrastructures by representing each local region as similarities to its spatial neighbors (Shechtman &

Original: Irani, 2007).
Revised: Irani, 2007).

Original: Given a sequence of frames, i.e. , a video, it extends along the temporal dimension and thus represents each local region as similarities to its neighbors in space and time.
Revised: Given a sequence of frames, i.e. , a video, it extends along the temporal dimension and thus represents each local region as similarities to its neighbors in space and time.

Original: By converting appearance features into relational values, STSS enables a learner to better recognize structural patterns in space and time.
Revised: By converting appearance features into relational values, STSS enables a learner to better recognize structural patterns in space and time.

Original: For neighbors at the same frame it computes a spatial self-similarity map, while for neighbors at a different frame it extracts a motion likelihood map.
Revised: For neighbors at the same frame it computes a spatial self-similarity map, while for neighbors at a different frame it extracts a motion likelihood map.

Original: If we ﬁx our attention to the similarity map to the very next frame within STSS and attempt to extract a single displacement vector to the most likely position at the frame, the problem reduces to optical ﬂow, which is a particular type of motion information.
Revised: If we ﬁx our attention to the similarity map to the very next frame within STSS and attempt to extract a single displacement vector to the most likely position at the frame, the problem reduces to optical ﬂow, which is a particular type of motion information.

Original: In contrast, we leverage the whole volume of
Revised: In contrast, we leverage the whole volume of

Original: STSS and let our model learn to extract an effective motion representation from it in an end-toend manner.
Revised: STSS and let our model learn to extract an effective motion representation from it in an end-toend manner.

Original: With a sufﬁcient volume of neighborhood in space and time, it effectively captures long-term interaction and fast motion in the video, leading to robust action recognition.
Revised: With a sufﬁcient volume of neighborhood in space and time, it effectively captures long-term interaction and fast motion in the video, leading to robust action recognition.

Original: We introduce a neural block for STSS representation, dubbed SELFY , that can be easily inserted into neural architectures and learned end-to-end without additional supervision.
Revised: We introduce a neural block for STSS representation, dubbed SELFY , that can be easily inserted into neural architectures and learned end-to-end without additional supervision.

Original: Our experimental analysis demonstrates its superiority over previous methods for motion modeling as well as its complementarity to spatio-temporal features from direct convolutions.
Revised: Our experimental analysis demonstrates its superiority over previous methods for motion modeling as well as its complementarity to spatio-temporal features from direct convolutions.

Original: On the standard benchmarks for action recognition, Something-Something V1&V2, Diving-48, and FineGym, the proposed method achieves the state-of-the-art results.
Revised: On the standard benchmarks for action recognition, Something-Something V1&V2, Diving-48, and FineGym, the proposed method achieves the state-of-the-art results.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Video action recognition .
Revised: Video action recognition .

Original: Video action recognition is a task to categorize videos into pre-deﬁned action classes.
Revised: Video action recognition is a task to categorize videos into pre-deﬁned action classes.

Original: One of the conventional topics in action recognition is to capture temporal dynamics in videos.
Revised: One of the conventional topics in action recognition is to capture temporal dynamics in videos.

Original: In deep learning, many approaches attempt to learn temporal dynamics in different ways: Two-stream networks with external optical ﬂows (Simonyan & Zisserman, 2014;
Revised: In deep learning, many approaches attempt to learn temporal dynamics in different ways: Two-stream networks with external optical ﬂows (Simonyan & Zisserman, 2014;

Original: Wang et al., 2016), recurrent networks (Donahue et al., 2015), and 3D CNNs (Tran et al., 2015; Carreira &
Revised: Wang et al., 2016), recurrent networks (Donahue et al., 2015), and 3D CNNs (Tran et al., 2015; Carreira &

Original: Zisserman, 2017).
Revised: Zisserman, 2017).

Original: Recent approaches have introduced the advanced 3D CNNs (Tran et al., 2018; 2019; Feichtenhofer, 2020; Lin et al., 2019; Fan et al., 2020) and show the effectiveness of capturing spatio-temporal features, so that 3D CNNs now become a de facto approach to learn temporal dynamics in the video.
Revised: Recent approaches have introduced the advanced 3D CNNs (Tran et al., 2018; 2019; Feichtenhofer, 2020; Lin et al., 2019; Fan et al., 2020) and show the effectiveness of capturing spatio-temporal features, so that 3D CNNs now become a de facto approach to learn temporal dynamics in the video.

Original: However, spatio-temporal convolution is vulnerable unless relevant features are well aligned across frames within the ﬁxed-sized kernel.
Revised: However, spatio-temporal convolution is vulnerable unless relevant features are well aligned across frames within the ﬁxed-sized kernel.

Original: To address this issue, a few methods adaptively translate the kernel offsets with deformable convolutions (Zhao et al., 2018; Li et al., 2020a), while several methods (Feichtenhofer et al., 2019; Li et al., 2020b) modulate the other hyper-parameters, e.g. , higher frame rate or larger spatial receptive ﬁelds.
Revised: To address this issue, a few methods adaptively translate the kernel offsets with deformable convolutions (Zhao et al., 2018; Li et al., 2020a), while several methods (Feichtenhofer et al., 2019; Li et al., 2020b) modulate the other hyper-parameters, e.g. , higher frame rate or larger spatial receptive ﬁelds.

Original: Unlike these methods, we address the problem of the spatio-temporal convolution by a sufﬁcient volume of STSS, capturing far-sighted spatio-temporal relations.
Revised: Unlike these methods, we address the problem of the spatio-temporal convolution by a sufﬁcient volume of STSS, capturing far-sighted spatio-temporal relations.

Original: Learning motion features.
Revised: Learning motion features.

Original: Since using the external optical ﬂow beneﬁts 3D CNNs to improve the action recognition accuracy (Carreira & Zisserman, 2017; Zolfaghari et al., 2018; Tran et al., 2018), several approaches try to learn frame-by-frame motion features from RGB sequences inside neural architectures.
Revised: Since using the external optical ﬂow beneﬁts 3D CNNs to improve the action recognition accuracy (Carreira & Zisserman, 2017; Zolfaghari et al., 2018; Tran et al., 2018), several approaches try to learn frame-by-frame motion features from RGB sequences inside neural architectures.

Original: Fan et al.
Revised: Fan et al.

Original: ; Piergiovanni & Ryoo (2019) internalize TV-L1 (Zach et al., 2007) optical ﬂows into the CNN.
Revised: ; Piergiovanni & Ryoo (2019) internalize TV-L1 (Zach et al., 2007) optical ﬂows into the CNN.

Original: Frame-wise feature differences (Sun et al., 2018b; Lee et al., 2018;
Revised: Frame-wise feature differences (Sun et al., 2018b; Lee et al., 2018;

Original: Jiang et al., 2019; Li et al., 2020c) are also utilized as the motion features.
Revised: Jiang et al., 2019; Li et al., 2020c) are also utilized as the motion features.

Original: Recent correlation-based methods (Wang et al., 2020; Kwon et al., 2020) adopt a correlation operator (Dosovitskiy et al., 2015; Sun et al., 2018a; Yang & Ramanan, 2019) to learn motion features between adjacent frames.
Revised: Recent correlation-based methods (Wang et al., 2020; Kwon et al., 2020) adopt a correlation operator (Dosovitskiy et al., 2015; Sun et al., 2018a; Yang & Ramanan, 2019) to learn motion features between adjacent frames.

Original: However, these methods compute frame-by-frame motion features between two adjacent frames and then rely on stacked spatio-temporal convolutions for capturing long-range motion dynamics.
Revised: However, these methods compute frame-by-frame motion features between two adjacent frames and then rely on stacked spatio-temporal convolutions for capturing long-range motion dynamics.

Original: We propose to learn STSS features, as generalized motion features, that enable to capture both shortterm and long-term interactions in the video.
Revised: We propose to learn STSS features, as generalized motion features, that enable to capture both shortterm and long-term interactions in the video.

Original: Self-similarity.
Revised: Self-similarity.

Original: Self-similarity represents an internal geometric layout of images.
Revised: Self-similarity represents an internal geometric layout of images.

Original: It is widely used in many computer vision tasks, such as object detection (Shechtman & Irani, 2007), image retrieval (H¨orster & Lienhart, 2008), and semantic correspondence matching (Kim et al., 2015; 2017).
Revised: It is widely used in many computer vision tasks, such as object detection (Shechtman & Irani, 2007), image retrieval (H¨orster & Lienhart, 2008), and semantic correspondence matching (Kim et al., 2015; 2017).

Original: In the video domain, Shechtman & Irani (2007) ﬁrstly introduce the concept of STSS and transforms the STSS to a hand-crafted local descriptor for action detection.
Revised: In the video domain, Shechtman & Irani (2007) ﬁrstly introduce the concept of STSS and transforms the STSS to a hand-crafted local descriptor for action detection.

Original: Inspired from this work, early methods adopt self-similarities for capturing view-invariant temporal patterns (Junejo et al.,
Revised: Inspired from this work, early methods adopt self-similarities for capturing view-invariant temporal patterns (Junejo et al.,

Original: 2008; 2010; K¨orner & Denzler, 2013), but they use temporal self-similarities only due to computational costs.
Revised: 2008; 2010; K¨orner & Denzler, 2013), but they use temporal self-similarities only due to computational costs.

Original: Recently, there are several non-local approaches (Wang et al., 2018;
Revised: Recently, there are several non-local approaches ( et et al

Original: Liu et al., 2019) that utilize STSS for capturing long-range dynamics of videos.
Revised: , 2019) that utilize STSS for capturing long-range dynamics of videos.

Original: However, they use STSS for reweighting or aligning visual features, which is an indirect way of using STSS.
Revised: However, they use STSS for reweighting or aligning visual features, which is an indirect way of using STSS.

Original: Different from these methods, our method leverages full STSS directly as generalized motion information and learns an effective representation for action recognition within a video-processing architecture.
Revised: Different from these methods, our method leverages full STSS directly as generalized motion information and learns an effective representation for action recognition within a video-processing architecture.

Original: To the best of our knowledge, our work is the ﬁrst in learning STSS representation using modern CNNs.
Revised: To the best of our knowledge, our work is the ﬁrst in learning STSS representation using modern CNNs.

Original: The contribution of our paper can be summarized as follows.
Revised: The contribution of our paper can be summarized as follows.

Original: First, we revisit the notion of selfsimilarity and propose to learn a generalized, far-sighted motion representations from STSS.
Revised: First, we revisit the notion of selfsimilarity and propose to learn a generalized, far-sighted motion representations from STSS.

Original: Second, we implement STSS representation learning as a neural block, dubbed SELFY , that can be integrated into existing neural architectures.
Revised: Second, we implement STSS representation learning as a neural block, dubbed SELFY , that can be integrated into existing neural architectures.

Original: Third, we provide comprehensive evaluations on SELFY, achieving the state-of-the-art on benchmarks: Something-Something V1&V2, Diving-48, and FineGym.
Revised: Third, we provide comprehensive evaluations on SELFY, achieving the state-of-the-art on benchmarks: Something-Something V1&V2, Diving-48, and FineGym.

Original:  O UR A PPROACH
Revised:  O UR A PPROACH

Original: In this section, we ﬁrst revisit the notions of self-similarity and discuss its relation to motion.
Revised: In this section, we ﬁrst revisit the notions of self-similarity and discuss its relation to motion.

Original: We then introduce our method for learning effective spatio-temporal self-similarity representation, which can be easily integrated into video-processing architectures and learned end-to-end.
Revised: We then introduce our method for learning effective spatio-temporal self-similarity representation, which can be easily integrated into video-processing architectures and learned end-to-end.

Original: 3. S ELF -S IMILARITY R EVISITED
Revised: 3. S ELF -S IMILARITY R EVISITED

Original: Self-similarity is a relational descriptor that suppresses variations in appearance and reveals structural patterns in images or videos (Shechtman & Irani, 2007).
Revised: Self-similarity is a relational descriptor that suppresses variations in appearance and reveals structural patterns in images or videos (Shechtman & Irani, 2007).

Original: Given an image feature map I ∈ R X
Revised: Given an image feature map I ∈ R X

Original: × Y × C , self-similarity transformation of I results in a 4D tensor
Revised: × Y × C , self-similarity transformation of I results in a 4D tensor

Original: S ∈ R X
Revised: S ∈ R X

Original: × Y × U × V , whose elements are deﬁned as
Revised: × Y × U × V , whose elements are deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where sim( · , · ) is a similarity function, e.g. , cosine similarity.
Revised: where sim( · , · ) is a similarity function, e.g. , cosine similarity.

Original: Here, ( x, y ) is a query coordinate while ( u, v ) is a spatial offset from it.
Revised: Here, ( x, y ) is a query coordinate while ( u, v ) is a spatial offset from it.

Original: To impose a locality, the offset is restricted to its neighborhood: ( u, v ) ∈ [ − d U , d U ] × [ − d V , d V ] , so that U =
Revised: To impose a locality, the offset is restricted to its neighborhood: ( u, v ) ∈ [ − d U , d U ] × [ − d V , d V ] , so that U =

Original: 2 d U + 

Revised: 2 d U +

Original: and V = 2 d V + 1 , respectively.
Revised: and V = 2 d V + 1 , respectively.

Original: By converting C -dimensional appearance feature I x,y into UV -dimensional relational feature S x,y , it suppresses variations in appearance and reveals spatial structures in the image.
Revised: By converting C -dimensional appearance feature I x,y into UV -dimensional relational feature S x,y , it suppresses variations in appearance and reveals spatial structures in the image.

Original: Note that the selfsimilarity transformation closely relates to conventional cross-similarity (or correlation) across two different feature maps ( I , I (cid:48)
Revised: Note that the selfsimilarity transformation closely relates to conventional cross-similarity (or correlation) across two different feature maps ( I , I (cid:48)

Original: ∈ R X × Y × C ), which can be deﬁned as
Revised: ∈ R X × Y × C ), which can be deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Given two images of a moving object, the cross-similarity transformation effectively captures motion information and thus is commonly used in optical ﬂow and correspondence estimation (Dosovitskiy et al., 2015; Sun et al., 2018a; Yang & Ramanan, 2019).
Revised: Given two images of a moving object, the cross-similarity transformation effectively captures motion information and thus is commonly used in optical ﬂow and correspondence estimation (Dosovitskiy et al., 2015; Sun et al., 2018a; Yang & Ramanan, 2019).

Original: For a sequence of frames, i.e. , a video, one can naturally extend the spatial self-similarity along the temporal axis.
Revised: For a sequence of frames, i.e. , a video, one can naturally extend the spatial self-similarity along the temporal axis.

Original: Let V ∈ R T
Revised: Let V ∈ R T

Original: ×
Revised: ×

Original: X × Y × C denote a feature map of the video with T frames.
Revised: X × Y × C denote a feature map of the video with T frames.

Original: Spatiotemporal self-similarity (STSS) transformation of V results in a 6D tensor S ∈ R T ×
Revised: Spatiotemporal self-similarity (STSS) transformation of V results in a 6D tensor S ∈ R T ×

Original: X × Y ×
Revised: X × Y ×

Original: L × U × V , whose elements are deﬁned as
Revised: L × U × V , whose elements are deﬁned as

Original: [Equation]
Revised: [Equation]

Original:  where ( t, x, y ) is the spatio-temporal coordinate and ( l, u, v ) is a spatio-temporal offset from it.
Revised: [Figure] where ( t, x, y ) is the spatio-temporal coordinate and ( l, u, v ) is a spatio-temporal offset from it.

Original: In addition to the locality of spatial offsets above, the temporal offset l is also restricted to its temporal neighborhood: l ∈ [ − d L , d L ] , so that L = 2 d L + 1 .[Figure]
Revised: In addition to the locality of spatial offsets above, the temporal offset l is also restricted to its temporal neighborhood: l ∈ [ − d L , d L ] , so that L = 2 d L + 1 .

Original: What types of information does STSS describe?
Revised: What types of information does STSS describe?

Original: Interestingly, for each time t , the STSS tensor S can be decomposed along temporal offset l into a single spatial self-similarity tensor (when l = 0 ) and 2 d L spatial cross-similarity tensors (when l (cid:54) = 0 ); the partial tensors with a small offset ( e.g. , l = − 1 or +1 ) collect motion information from adjacent frames and those with larger offsets capture it from further frames both forward and backward in time.
Revised: Interestingly, for each time t , the STSS tensor S can be decomposed along temporal offset l into a single spatial self-similarity tensor (when l = 0 ) and 2 d L spatial cross-similarity tensors (when l (cid:54) = 0 ); the partial tensors with a small offset ( e.g. , l = − 1 or +1 ) collect motion information from adjacent frames and those with larger offsets capture it from further frames both forward and backward in time.

Original: Unlike previous approaches to learn motion (Dosovitskiy et al., 2015; Wang et al., 2020; Kwon et al., 2020), which rely on crosssimilarity between adjacent frames, STSS allows to take a generalized, far-sighted view on motion, i.e. , both short-term and long-term, both forward and backward, as well as spatial self-motion.
Revised: Unlike previous approaches to learn motion (Dosovitskiy et al., 2015; Wang et al., 2020; Kwon et al., 2020), which rely on crosssimilarity between adjacent frames, STSS allows to take a generalized, far-sighted view on motion, i.e. , both short-term and long-term, both forward and backward, as well as spatial self-motion.

Original: 3. S PATIO - TEMPORAL SELF - SIMILARITY REPRESENTATION LEARNING
Revised: 3. S PATIO - TEMPORAL SELF - SIMILARITY REPRESENTATION LEARNING

Original: By leveraging the rich information in STSS, we propose to learn a generalized motion representation for video understanding.
Revised: By leveraging the rich information in STSS, we propose to learn a generalized motion representation for video understanding.

Original: To achieve this goal without additional supervision, we design a neural block, dubbed SELFY, which can be inserted into a video-processing architectures and learned end-to-end.
Revised: To achieve this goal without additional supervision, we design a neural block, dubbed SELFY, which can be inserted into a video-processing architectures and learned end-to-end.

Original: The overall structure is illustrated in Fig.
Revised: The overall structure is illustrated in Fig.

Original: It consists of three steps: self-similarity transformation , feature extraction , and feature integration .
Revised: It consists of three steps: self-similarity transformation , feature extraction , and feature integration .

Original: Given the input video feature tensor V , the self-similarity transformation step converts it to the
Revised: Given the input video feature tensor V , the self-similarity transformation step converts it to the

Original: STSS tensor S as in Eq.(1).
Revised: STSS tensor S as in Eq.(1).

Original: In the following, we describe feature extraction and integration steps.
Revised: In the following, we describe feature extraction and integration steps.

Original: 3.2. F EATURE EXTRACTION
Revised: 3.2. F EATURE EXTRACTION

Original: From the STSS tensor S ∈ R T × X × Y ×
Revised: From the STSS tensor S ∈ R T × X × Y ×

Original: L × U × V , we extract a C F -dimensional feature for each spatiotemporal position ( t, x, y ) and temporal offset l so that the resultant tensor is F ∈ R T ×
Revised: L × U × V , we extract a C F -dimensional feature for each spatiotemporal position ( t, x, y ) and temporal offset l so that the resultant tensor is F ∈ R T ×

Original: X × Y ×
Revised: X × Y ×

Original: L × C F , which is equivariant to translation in space, time, and temporal offset.
Revised: L × C F , which is equivariant to translation in space, time, and temporal offset.

Original: The dimension of L is preserved to extract motion information across different temporal offsets in a consistent manner.
Revised: The dimension of L is preserved to extract motion information across different temporal offsets in a consistent manner.

Original: While there exist many design choices, we introduce three methods for feature extraction in this work.
Revised: While there exist many design choices, we introduce three methods for feature extraction in this work.

Original: Soft-argmax.
Revised: Soft-argmax.

Original: The ﬁrst method is to compute explicit displacement ﬁelds using S , which previous motion learning methods adopt using spatial cross-similarity (Dosovitskiy et al., 2015; Sun et al., 2018a; Yang & Ramanan, 2019).
Revised: The ﬁrst method is to compute explicit displacement ﬁelds using S , which previous motion learning methods adopt using spatial cross-similarity (Dosovitskiy et al., 2015; Sun et al., 2018a; Yang & Ramanan, 2019).

Original: One may extract the displacement ﬁeld by indexing the positions with the highest similarity value via arg max ( u,v ) , but it is not differentiable.
Revised: One may extract the displacement ﬁeld by indexing the positions with the highest similarity value via arg max ( u,v ) , but it is not differentiable.

Original: We instead use softargmax (Chapelle & Wu, 2010), which aggregates displacement vectors with softmax weighting (Fig. 3a).
Revised: We instead use softargmax (Chapelle & Wu, 2010), which aggregates displacement vectors with softmax weighting (Fig. 3a).

Original: The soft-argmax feature extraction can be formulated as
Revised: The soft-argmax feature extraction can be formulated as

Original: [Equation]
Revised: [Equation]

Original: which results in a feature tensor F ∈ R T ×
Revised: which results in a feature tensor F ∈ R T ×

Original: X × Y ×
Revised: X × Y ×

Original: L × 2 .
Revised: L × 2 .

Original: The temperature factor τ adjusts the softmax distribution, and we set τ = 0 .
Revised: The temperature factor τ adjusts the softmax distribution, and we set τ = 0 .

Original: 01 in our experiments.
Revised: 01 in our experiments.

Original: Multi-layer perceptron (MLP).
Revised: Multi-layer perceptron (MLP).

Original: The second method is to learn an MLP that converts self-similarity values into a feature.
Revised: The second method is to learn an MLP that converts self-similarity values into a feature.

Original: For this, we ﬂatten the ( U, V ) volume into UV -dimensional vectors, and apply an MLP to them (Fig. 3b).
Revised: For this, we ﬂatten the ( U, V ) volume into UV -dimensional vectors, and apply an MLP to them (Fig. 3b).

Original: For the reshaped tensor S ∈ R T × X × Y × L × UV , a perceptron f ( · ) can be expressed as
Revised: For the reshaped tensor S ∈ R T × X × Y × L × UV , a perceptron f ( · ) can be expressed as

Original: [Equation]
Revised: [Equation]

Original: where × n denotes the n -mode tensor product, W φ ∈ R C (cid:48) × UV is the perceptron parameters, and the output is f ( S )
Revised: where × n denotes the n -mode tensor product, W φ ∈ R C (cid:48) × UV is the perceptron parameters, and the output is f ( S )

Original: ∈ R T ×
Revised: ∈ R T ×

Original: [Equation]
Revised: X × Y ×

Original:  × C (cid:48) .
Revised: L × C (cid:48) .

Original: The MLP feature extraction can thus be formulated as
Revised: The MLP feature extraction can thus be formulated as

Original: [Equation]
Revised: [Equation]

Original: which produces a feature tensor F ∈ R T ×
Revised: which produces a feature tensor F ∈ R T ×

Original: X × Y ×
Revised: X × Y ×

Original: L × C F .
Revised: L × C F .

Original: This method is more ﬂexible and may also be more effective than the soft-argmax because not only can it encode displacement information but also it can directly access the similarity values, which may be helpful for learning motion distribution.
Revised: This method is more ﬂexible and may also be more effective than the soft-argmax because not only can it encode displacement information but also it can directly access the similarity values, which may be helpful for learning motion distribution.

Original: Convolution.
Revised: Convolution.

Original: The third method is to learn convolution kernels over ( L, U, V ) volume of S (Fig. 3c).
Revised: The third method is to learn convolution kernels over ( L, U, V ) volume of S (Fig. 3c).

Original: When we regard S as a 7D tensor S ∈ R T × X × Y ×
Revised: When we regard S as a 7D tensor S ∈ R T × X × Y ×

Original: L × U × V × C with C = 1 , the convolution layer g can be expressed using S (cid:48)
Revised: L × U × V × C with C = 1 , the convolution layer g can be expressed using S (cid:48)

Original: =
Revised: =

Original: g ( S ) , whose elements are computed by
Revised: g ( S ) , whose elements are computed by

Original: [Equation]
Revised: [Equation]

Original: where K ∈ R L κ × U κ × V κ × C × C (cid:48) is a multi-channel convolution kernel, ( l κ , u κ , v κ ) is the kernel parameter indices, and ( c, c (cid:48) ) is the channel indices.
Revised: where K ∈ R L κ × U κ × V κ × C × C (cid:48) is a multi-channel convolution kernel, ( l κ , u κ , v κ ) is the kernel parameter indices, and ( c, c (cid:48) ) is the channel indices.

Original: The indices (ˆ l κ , ˆ u κ , ˆ v κ ) are centered as ˆ l κ = l κ − L κ / 2 , ˆ u κ = u κ − U κ / 2 , ˆ v κ = v κ − V κ / 2 .
Revised: The indices (ˆ l κ , ˆ u κ , ˆ v κ ) are centered as ˆ l κ = l κ − L κ / 2 , ˆ u κ = u κ − U κ / 2 , ˆ v κ = v κ − V κ / 2 .

Original: Starting from R T × X × Y ×
Revised: Starting from R T × X × Y ×

Original: L × U × V × 1 , we gradually downsample (U,V) and expand channels via multiple convolutions with strides, ﬁnally resulting in R T ×
Revised: L × U × V × 1 , we gradually downsample (U,V) and expand channels via multiple convolutions with strides, ﬁnally resulting in R T ×

Original: X × Y ×
Revised: X × Y ×

Original: L × 1 × 1 × C F .
Revised: L × 1 × 1 × C F ; we preserve the L dimension, since maintaining ﬁne temporal resolution is shown to be effective for capturing detailed motion information (Lin et al., 2019; Feichtenhofer et al., 2019).

Original: The convolutional feature extraction with n layers can thus be formulated as
Revised: The convolutional feature extraction with n layers can thus be formulated as

Original: [Equation]
Revised: [Equation]

Original: ) which results in a feature tensor F ∈ R T ×
Revised: which results in a feature tensor F ∈ R T ×

Original: X × Y ×
Revised: X × Y ×

Original: L × C F .
Revised: L × C F .

Original: This method is effective in learning structural patterns with their convolution kernels, thus outperforming the former methods as will be seen in our experiments.
Revised: This method is effective in learning structural patterns with their convolution kernels, thus outperforming the former methods as will be seen in our experiments.

Original: 3.2. F EATURE INTEGRATION
Revised: 3.2. F EATURE INTEGRATION

Original: In this step, we integrate the extracted STSS features F ∈ R T ×
Revised: In this step, we integrate the extracted STSS features F ∈ R T ×

Original: X × Y ×
Revised: X × Y ×

Original: L × C F to feed them back to the original input stream with ( T, X, Y, C ) volume.
Revised: L × C F to feed them back to the original input stream with ( T, X, Y, C ) volume.

Original: We ﬁrst use 3 × 3 spatial convolution kernels along ( x, y ) dimension of F .
Revised: We ﬁrst use 3 × 3 spatial convolution kernels along ( x, y ) dimension of F .

Original: The spatial convolution layer h can be expressed using F (cid:48)
Revised: The spatial convolution layer h can be expressed using F (cid:48)

Original: =
Revised: =

Original: h ( F ) , whose elements are computed by
Revised: h ( F ) , whose elements are computed by

Original: [Equation]
Revised: [Equation]

Original: where K ∈ R X κ × Y κ × C F × C (cid:48)
Revised: where K ∈ R X κ × Y κ × C F × C (cid:48)

Original: F is the multi-channel convolution kernel, ( x κ , y κ ) is the kernel parameter indices, and ( c F , c (cid:48) F ) is the channel indices.
Revised: F is the multi-channel convolution kernel, ( x κ , y κ ) is the kernel parameter indices, and ( c F , c (cid:48) F ) is the channel indices.

Original: (ˆ x κ , ˆ y κ ) is centered as ˆ x κ = x κ − X κ / 2 , ˆ y κ = y κ − Y κ / 2 .
Revised: (ˆ x κ , ˆ y κ ) is centered as ˆ x κ = x κ − X κ / 2 , ˆ y κ = y κ − Y κ / 2 .

Original: This type of spatial convolutions integrate the original features by extending receptive ﬁelds along ( x, y ) dimension.
Revised: This type of spatial convolutions integrate the original features by extending receptive ﬁelds along ( x, y ) dimension.

Original: The resultant features F (cid:63) ∈ R T ×
Revised: The resultant features F (cid:63) ∈ R T ×

Original: X × Y ×
Revised: X × Y ×

Original: L × C (cid:63)F is deﬁned as
Revised: L × C (cid:63)F is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: We then ﬂatten the ( L, C (cid:63) F ) volume into LC (cid:63) F -dimensional vectors to obtain F (cid:63) ∈ R T × X × Y × LC (cid:63)F , and apply an 1 × 1 × 1 convolution layer to obtain the ﬁnal output.
Revised: We then ﬂatten the ( L, C (cid:63)F ) volume into LC (cid:63)F -dimensional vectors to obtain F (cid:63) ∈ R T × X × Y × LC (cid:63)F , and apply an 1 × 1 × 1 convolution layer to obtain the ﬁnal output.

Original: This convolution layer integrates features from different temporal offsets and also adjusts its channel dimension to ﬁt that of the original input V .
Revised: This convolution layer integrates features from different temporal offsets and also adjusts its channel dimension to ﬁt that of the original input V .

Original: We adopt the identity mapping of the input for residual learning (He et al., 2016).
Revised: We adopt the identity mapping of the input for residual learning (He et al., 2016).

Original: The ﬁnal output tensor Z is expressed as
Revised: The ﬁnal output tensor Z is expressed as

Original: [Equation]
Revised: [Equation]

Original: where × n is the n -mode tensor product and W θ ∈ R C × LC (cid:63)F is the weights of the convolution layer.
Revised: where × n is the n -mode tensor product and W θ ∈ R C × LC (cid:63)F is the weights of the convolution layer.

Original:  E XPERIMENTS 4. D ATASETS & I MPLEMENTATION DETAILS
Revised:  E XPERIMENTS 4. D ATASETS & I MPLEMENTATION DETAILS

Original: For evaluation, we use benchmarks that contain ﬁne-grained spatio-temporal dynamics in videos. 
Revised: For evaluation, we use benchmarks that contain ﬁne-grained spatio-temporal dynamics in videos. [Table] [Table] [Table]

Revised: [Table]

Original:  Something-Something V1 & V2 (SS-V1 & V2)
Revised: [Table] Something-Something V1 & V2 (SS-V1 & V2)

Original: (Goyal et al., 2017b), which are both large-scale action recognition datasets, contain ∼ 108k and ∼ 220k video clips, respectively.
Revised: (Goyal et al., 2017b), which are both large-scale action recognition datasets [Table] contain ∼ 108k and ∼ 220k video clips, respectively.

Original: Both datasets share the same 174 action classes that are labeled, e.g. , ‘pretending to put something next to something.’
Revised: Both datasets share the same 174 action classes that are labeled, e

Original: [Table]
Revised: [Table]

Original: [Table] [Table] [Table]
Revised: g. , ‘pretending to put something next to something.’

Original: Diving-48 (Li et al., 2018), which contains ∼ 18k videos with 48 different diving action classes [Table] is an action recognition dataset that minimizes contextual biases,
Revised: Diving-48 (Li et al., 2018), which contains ∼ 18k videos with 48 different diving action classes, is an action recognition dataset that minimizes contextual biases,

Original: i
Revised: i

Original: [Table]
Revised: [Table]

Original: e. , scenes or objects
Revised: e. , scenes or objects.

Original: [Table] FineGym (Shao et al., 2020) is a ﬁne-grained action dataset built on top of gymnastic videos.
Revised: FineGym (Shao et al., 2020) is a ﬁne-grained action dataset built on top of gymnastic videos.

Original: We adopt the Gym288 and Gym99 sets for experiments that contains 288 and 99 classes, respectively.
Revised: We adopt the Gym288 and Gym99 sets for experiments that contains 288 and 99 classes, respectively.

Original: Action recognition architecture.
Revised: Action recognition architecture.

Original: We employ TSN ResNets (Wang et al., 2016) as 2D CNN backbones and
Revised: We employ TSN ResNets (Wang et al., 2016) as 2D CNN backbones and

Original: TSM ResNets
Revised: TSM ResNets

Original: (Lin et al., 2019) as 3D CNN backbones.
Revised: (Lin et al., 2019) as 3D CNN backbones.

Original: TSM enables to obtain the effect of spatio-temporal convolutions using spatial convolutions by shifting a part of input channels along the temporal axis before the convolution operation.
Revised: TSM enables to obtain the effect of spatio-temporal convolutions using spatial convolutions by shifting a part of input channels along the temporal axis before the convolution operation.

Original: TSM is added into each residual block of the
Revised: TSM is added into each residual block of the

Original: ResNet.
Revised: ResNet.

Original: We adopt ImageNet pre-trained weights for our backbones.
Revised: We adopt ImageNet pre-trained weights for our backbones.

Original: To transform the backbones to the self-similarity network (SELFYNet), we insert a single SELFY block after the third stage in the backbones.
Revised: To transform the backbones to the self-similarity network (SELFYNet), we insert a single SELFY block after the third stage in the backbones.

Original: For SELFY block, we use the convolution method as a default feature extraction method and use multi-channel 1 × 3 × 3 convolution kernels.
Revised: For SELFY block, we use the convolution method as a default feature extraction method and use multi-channel 1 × 3 × 3 convolution kernels.

Original: For more details, please refer Appendix A, B.
Revised: For more details, please refer Appendix A, B.

Original: Training & testing.
Revised: Training & testing.

Original: For training, we sample a clip of 8 or 16 frames from each video by using segment-based sampling (Wang et al., 2016).
Revised: For training, we sample a clip of 8 or 16 frames from each video by using segment-based sampling (Wang et al., 2016).

Original: The spatio-temporal matching region ( L, U, V ) of
Revised: The spatio-temporal matching region ( L, U, V ) of

Original: SELFY block is set as (5 , 9 , 9) or (9 , 9 , 9) when using 8 or 16 frames, respectively.
Revised: SELFY block is set as (5 , 9 , 9) or (9 , 9 , 9) when using 8 or 16 frames, respectively.

Original: For testing, we sample one or two clips from a video, crop their center, and evaluate the averaged prediction of the sampled clips.
Revised: For testing, we sample one or two clips from a video, crop their center, and evaluate the averaged prediction of the sampled clips.

Original: For more details, please refer Appendix A. 
Revised: For more details, please refer Appendix A. [Table]

Revised: [Table]

Original: 4. C OMPARISON WITH THE STATE - OF - THE - ART METHODS
Revised: 4. C OMPARISON WITH THE STATE - OF - THE - ART METHODS

Original: For a fair comparison, we compare our model with other models that are not pre-trained on additional large-scale video datasets, e.g. , Kinetics (Kay et al., 2017) or Sports1M (Karpathy et al., 2014) in the following experiments.[Table] [Table] [Table]
Revised: For a fair comparison, we compare our model with other models that are not pre-trained on additional large-scale video datasets, e.g. , Kinetics (Kay et al., 2017) or Sports1M (Karpathy et al., 2014) in the following experiments.

Original: [Table] Table 1 summarizes the results on SS-V1&V2.
Revised: Table 1 summarizes the results on SS-V1&V2.

Original: The ﬁrst and second compartment of the table shows the results of other 2D CNN and (pseudo-) 3D CNN models, respectively.
Revised: The ﬁrst and second compartment of the table shows the results of other 2D CNN and (pseudo-) 3D CNN models, respectively.

Original: The last part of each compartment shows the results of SELFYNet.
Revised: The last part of each compartment shows the results of SELFYNet.

Original: SELFYNet with TSN-ResNet (SELFYNet-TSN-R50) achieves 50.7% and 62.7% at top-1 accuracy, respectively, which outperforms other 2D models using 8 frames only.
Revised: SELFYNet with TSN-ResNet (SELFYNet-TSN-R50) achieves 50.7% and 62.7% at top-1 accuracy, respectively, which outperforms other 2D models using 8 frames only.

Original: When we adopt TSM ResNet (TSM-R50) as our backbone and use 16 frames, our method (SELFYNet-TSM-R50) achieves 54.3% and 65.7% at top-1 accuracy, respectively, which is the best among the single models.
Revised: When we adopt TSM ResNet (TSM-R50) as our backbone and use 16 frames, our method (SELFYNet-TSM-R50) achieves 54.3% and 65.7% at top-1 accuracy, respectively, which is the best among the single models.

Original: Compared to TSM-R50, a single SELFY block obtains a signiﬁcant gain of 7.0%p and 4.5%p at top-1 accuracy, respectively; our method is more accurate than
Revised: Compared to TSM-R50, a single SELFY block obtains a signiﬁcant gain of 7.0%p and 4.5%p at top-1 accuracy, respectively; our method is more accurate than

Original: TSM-R50 Two-stream on both datasets.
Revised: TSM-R50 Two-stream on both datasets.

Original: Finally, our ensemble model (SELFYNet-TSM-R50 EN ) with 2-clip evaluation sets a new state of the art on both datasets by achieving 56.6% and 67.7% at top-1 accuracy, respectively.
Revised: Finally, our ensemble model (SELFYNet-TSM-R50 EN ) with 2-clip evaluation sets a new state of the art on both datasets by achieving 56.6% and 67.7% at top-1 accuracy, respectively.

Original: Table 2 summarizes the results on Diving-48 & FineGym.
Revised: Table 2 summarizes the results on Diving-48 & FineGym.

Original: For Diving-48, TSM-R50 using 16 frames shows 38.8% in top-1 accuracy in our implementation.
Revised: For Diving-48, TSM-R50 using 16 frames shows 38.8% in top-1 accuracy in our implementation.

Original: SELFYNet-TSM-R50 outperforms TSMR50 by 2.8%p in accuracy so that it sets a new state-of-the-art top-1 accuracy as 41.6% on Diving48.
Revised: SELFYNet-TSM-R50 outperforms TSMR50 by 2.8%p in accuracy so that it sets a new state-of-the-art top-1 accuracy as 41.6% on Diving48.

Original: For FineGym, SELFYNet-TSM-R50 achieves 49.5% and 87.7% at given 288 and 99 classes, respectively, surpassing all the other models reported in Shao et al.
Revised: For FineGym, SELFYNet-TSM-R50 achieves 49.5% and 87.7% at given 288 and 99 classes, respectively, surpassing all the other models reported in Shao et al.

Original: 4. A BLATION STUDIES
Revised: 4. A BLATION STUDIES

Original: We conduct ablation experiments to demonstrate the effectiveness of the proposed method.
Revised: We conduct ablation experiments to demonstrate the effectiveness of the proposed method.

Original: All experiments are performed on SS-V1 by using 8 frames.
Revised: All experiments are performed on SS-V1 by using 8 frames.

Original: Unless otherwise speciﬁed, we set ImageNet pre-trained TSM ResNet-18 (TSM-R18) with the single SELFY block of which ( L, U, V ) =
Revised: Unless otherwise speciﬁed, we set ImageNet pre-trained TSM ResNet-18 (TSM-R18) with the single SELFY block of which ( L, U, V ) =

Original: (5 , 9 , 9) , as our default SELFYNet.
Revised: (5 , 9 , 9) , as our default SELFYNet.

Original: Types of similarity.
Revised: Types of similarity.

Original: In Table 3a, we investigate the effect of different types of similarity by varying the set of temporal offset l on both TSN-ResNet-18 (TSN-R18) and TSM-R18.
Revised: In Table 3a, we investigate the effect of different types of similarity by varying the set of temporal offset l on both TSN-ResNet-18 (TSN-R18) and TSM-R18.

Original: Interestingly, learning spatial self-similarity ( { 0 } ) improves accuracy on both backbones, which implies that selfsimilarity features help capture structural patterns of visual features.
Revised: Interestingly, learning spatial self-similarity ( { 0 } ) improves accuracy on both backbones, which implies that selfsimilarity features help capture structural patterns of visual features.

Original: Learning cross-similarity with a short temporal range ( { 1 } ) shows a noticeable gain in accuracy on both backbones, indicating the signiﬁcance of motion features.
Revised: Learning cross-similarity with a short temporal range ( { 1 } ) shows a noticeable gain in accuracy on both backbones, indicating the signiﬁcance of motion features.

Original: Learning STSS outperforms other types of similarity, and the accuracy of SELFYNet increases as the temporal range becomes longer. 
Revised: Learning STSS outperforms other types of similarity, and the accuracy of SELFYNet increases as the temporal range becomes longer. When STSS takes a far-sighted [Figure]

Original: When STSS takes a far-sighted view on motion, STSS learns both short-term and long-term interactions in videos, as well as spatial self-similarity.
Revised: [Table] [Figure] view on motion, STSS learns both short-term and long-term interactions in videos, as well as spatial self-similarity.

Original: Qualitative results in Appendix C show that SELFYNet with a long temporal range ( {− 3 , · · · , 3 } ) effectively captures long-term interactions in videos.
Revised: Qualitative results in Appendix D show that SELFYNet with a long temporal range ( {− 3 , · · · , 3 } ) effectively captures long-term interactions in videos.

Original: Feature extraction  methods.
Revised: Feature extraction and integration methods.

Original: In Table 3b, we compare the  different  feature extraction methods introduced in Section 3.2.1.
Revised: In Table 3b, we compare the performance of different combinations of feature extraction and integration methods.

Revised: From the 2 nd to the 4 th rows, different feature extraction methods are compared ﬁxing the feature integration methods to a single fully-connected (FC) layer.

Original: The soft-argmax, which extracts spatial displacement features, improves top-1 accuracy by 3.1%p.
Revised: Compared to the baseline, the use of soft-argmax, which extracts spatial displacement features, improves top-1 accuracy by 1.0%p.

Original: When we replace the soft-argmax with the MLP, we obtain an additional gain of 1.4%p at top-1 accuracy, showing the effectiveness of directly using similarity values. 
Revised: Replacing soft-argmax with MLP provides an additional gain of 1.9%p at top-1 accuracy, showing the effectiveness of directly using similarity values. When using the convolution method for feature extraction, we achieve 46.7% at top-1 accuracy; the multi-channel convolution kernel is more effective in capturing structural patterns along ( u, v ) dimensions than MLP. From the 4 th to the 6 th rows, different feature integration methods are compared ﬁxing the feature extraction method to convolution. Replacing the single

Original: Lastly, when we use the convolution method for feature extraction, we achieve 48.2% at top-1 accuracy. Compared to the MLP, the multi-channel convolution kernel is more effective to capture structural patterns along ( u, v ) dimension.
Revised: FC layer with MLP improves the top-1 accuracy by 0.6 % p. Replacing MLP with convolutional layers further improves and achieves 48.4 %p at top-1 accuracy. These results demonstrate that our design choice of using convolutions along ( U, V ) and ( H, W ) is the most effective in learning the geometry-aware STSS representation.

Original: For more ablation experiments, please refer to Appendix B.
Revised: For more ablation experiments, please refer to Appendix B.

Original: 4. C OMPLEMENTARITY BETWEEN SPATIO - TEMPORAL FEATURES AND STSS FEATURES
Revised: 4. C OMPLEMENTARITY BETWEEN SPATIO - TEMPORAL FEATURES AND STSS FEATURES

Original: We conduct experiments for analyzing different meanings of spatio-temporal features and STSS features.
Revised: We conduct experiments for analyzing different meanings of spatio-temporal features and STSS features.

Original: We organize two basic blocks for representing two different features: spatio-temporal convolution block (STCB) that consists of several spatial-temporal convolutions (Fig. 4a) and SELFY-s block, light-weighted version of the SELFY block by removing spatial convolution layers (Fig. 4b).
Revised: We organize two basic blocks for representing two different features: spatio-temporal convolution block (STCB) that consists of several spatial-temporal convolutions (Fig. 4a) and SELFY-s block, light-weighted version of the SELFY block by removing spatial convolution layers (Fig. 4b).

Original: Both blocks have the same receptive ﬁelds and a similar number of parameters for a fair comparison.
Revised: Both blocks have the same receptive ﬁelds and a similar number of parameters for a fair comparison.

Original: Different combinations of the basic blocks are inserted after the third stage of TSN-ResNet-18.
Revised: Different combinations of the basic blocks are inserted after the third stage of TSN-ResNet-18.

Original: Table 5 summarizes the results on SS-V1.
Revised: Table 5 summarizes the results on SS-V1.

Original: STSS features (Fig. 4b, 4d) are more effective than

Original: [Figure]

Original: [Table] [Figure] spatio-temporal features (Fig. 4a, 4c) in top-1 and top-5 accuracy when the same number of blocks are inserted.
Revised: STSS features (Fig. 4b, 4d) are more effective than spatio-temporal features (Fig. 4a, 4c) in top-1 and top-5 accuracy when the same number of blocks are inserted.

Original: Interestingly, the combination of two different features (Fig.
Revised: Interestingly, the combination of two different features (Fig.

Original: 4e, 4f) shows better results in top-1 and top-5 accuracy compared to the single feature cases (Fig.
Revised: 4e, 4f) shows better results in top-1 and top-5 accuracy compared to the single feature cases (Fig.

Original: 4c, 4d), which demonstrate that both features complement each other.
Revised: 4c, 4d), which demonstrate that both features complement each other.

Original: We conjecture that this complementarity comes from different characteristics of the two features; while spatio-temporal features are obtained by directly encoding appearance features, STSS features are obtained by suppressing variations in appearance and focusing on the relational features in space and time.
Revised: We conjecture that this complementarity comes from different characteristics of the two features; while spatio-temporal features are obtained by directly encoding appearance features, STSS features are obtained by suppressing variations in appearance and focusing on the relational features in space and time.

Original: 4. I MPROVING ROBUSTNESS WITH STSS
Revised: 4. I MPROVING ROBUSTNESS WITH STSS

Original: In this section, we demonstrate that STSS representation helps video-processing models to be more robust to video corruptions.
Revised: In this section, we demonstrate that STSS representation helps video-processing models to be more robust to video corruptions.

Original: We test two corruptions that are likely to happen in the real world videos: occlusion and motion blur.
Revised: We test two corruptions that are likely to happen in the real world videos: occlusion and motion blur.

Original: To induce the corruptions, We either cut out a rectangle patch of a particular frame or generate a motion blur (Hendrycks & Dietterich, 2019).
Revised: To induce the corruptions, We either cut out a rectangle patch of a particular frame or generate a motion blur (Hendrycks & Dietterich, 2019).

Original: We corrupt a single center-frame for every clip of SS-V1 at the testing phase and gradually increase the severity of corruptions.
Revised: We corrupt a single center-frame for every clip of SS-V1 at the testing phase and gradually increase the severity of corruptions.

Original: We compare the results of TSM-R18 and SELFYNet variants of Table 3a.
Revised: We compare the results of TSM-R18 and SELFYNet variants of Table 3a.

Original: Fig.
Revised: Fig.

Original: 5a, 5b summarizes the results of two corruptions, respectively.
Revised: 5a, 5b summarizes the results of two corruptions, respectively.

Original: The top-1 accuracy of TSM-R18 and SELFYNets with the short temporal range ( { 0 } , { 1 } , and {− 1 , 0 , 1 } ) signiﬁcantly drops as the severity of corruptions becomes harder.
Revised: The top-1 accuracy of TSM-R18 and SELFYNets with the short temporal range ( { 0 } , { 1 } , and {− 1 , 0 , 1 } ) signiﬁcantly drops as the severity of corruptions becomes harder.

Original: We conjecture that the features of the corrupted frame propagate through the stacked TSMs, confusing the entire network.
Revised: We conjecture that the features of the corrupted frame propagate through the stacked TSMs, confusing the entire network.

Original: However, the SELFYNets with the long temporal range ( {− 2 , · · · , 2 } and {− 3 , · · · , 3 } ) show more robust performance than the other models.
Revised: However, the SELFYNets with the long temporal range ( {− 2 , · · · , 2 } and {− 3 , · · · , 3 } ) show more robust performance than the other models.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 5a, 5b, the accuracy gap between SELFYNets with the long temporal range and the others increases as the severity of corruptions becomes higher, indicating that the larger size of STSS features can improve the robustness on action recognition.
Revised: 5a, 5b, the accuracy gap between SELFYNets with the long temporal range and the others increases as the severity of corruptions becomes higher, indicating that the larger size of STSS features can improve the robustness on action recognition.

Original: We also present some qualitative results (Fig. 5c) where two SELFYNets with different temporal ranges, { 1 } and {− 3 , · · · , 3 } , both answer correctly without corruption, while the SELFYNet with { 1 } fails for the corrupted input.
Revised: We also present some qualitative results (Fig. 5c) where two SELFYNets with different temporal ranges, { 1 } and {− 3 , · · · , 3 } , both answer correctly without corruption, while the SELFYNet with { 1 } fails for the corrupted input.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we propose to learn a generalized, far-sighted motion representation from STSS for video understanding.
Revised: In this paper, we have proposed to learn a generalized, far-sighted motion representation from STSS for video understanding.

Original: The comprehensive analyses on the STSS demonstrate that STSS features effectively capture both short-term and long-term interactions, complement spatio-temporal features, and improve the robustness of video-processing models.
Revised: The comprehensive analyses on the STSS demonstrate that STSS features effectively capture both short-term and long-term interactions, complement spatio-temporal features, and improve the robustness of video-processing models.

Original: Our method outperforms other state-
Revised: Our method outperforms other state-of-the-art methods on the three benchmarks for video action recognition.

Original: A I MPLEMENTATION DETAILS
Revised: A I MPLEMENTATION DETAILS

Original: Architecture details.
Revised: Architecture details.

Original: We use TSN-ResNet and TSM-ResNet as our backbone (see Table 6) and initialize them with ImageNet pre-trained weights.
Revised: We use TSN-ResNet and TSM-ResNet as our backbone (see Table 6) and initialize them with ImageNet pre-trained weights.

Original: We insert a single SELFY block into right after res 3 and use the convolution method as a default feature extraction method.
Revised: We insert a single SELFY block into right after res 3 and use the convolution method as a default feature extraction method.

Original: We set the spatiotemporal matching region of SELFY block, ( L, U, V ) , as (5 , 9 , 9) or (9 , 9 , 9) when using 8 orinput frames, respectively.
Revised: We set the spatiotemporal matching region of SELFY block, ( L, U, V ) , as (5 , 9 , 9) or (9 , 9 , 9) when using 8 orinput frames, respectively.

Original: We stack four 1 × 3 × 3 convolution layers along ( l, u, v ) dimension for the feature extraction method, and use four 3 × 3 convolution layer along ( x, y ) dimension for the feature enhancement.
Revised: We stack four 1 × 3 × 3 convolution layers along ( l, u, v ) dimension for the feature extraction method, and use four 3 × 3 convolution layer along ( x, y ) dimension for the feature enhancement.

Original: We reduce a spatial resolution of video feature tensor, V , as 14 × 14 for computation efﬁciency before the self-similarity transformation.
Revised: We reduce a spatial resolution of video feature tensor, V , as 14 × 14 for computation efﬁciency before the self-similarity transformation.

Original: After the feature enhancement, we upsample the enhanced feature tensor, G (cid:63) , as 28 × 28 for the residual connection.
Revised: After the feature enhancement, we upsample the enhanced feature tensor, G (cid:63) , as 28 × 28 for the residual connection.

Original: Training.
Revised: Training.

Original: We sample a clip of 8 or 16 frames from each video by using segment-based sampling (Wang et al., 2016).
Revised: We sample a clip of 8 or 16 frames from each video by using segment-based sampling (Wang et al., 2016).

Original: We resize the sampled clips into 240 × 320 images and apply random scaling and horizontal ﬂipping for data augmentation.
Revised: We resize the sampled clips into 240 × 320 images and apply random scaling and horizontal ﬂipping for data augmentation.

Original: When applying the horizontal ﬂipping on
Revised: When applying the horizontal ﬂipping on

Original: SS-V1&V2 (Goyal et al., 2017b), we do not ﬂipclips of which class labels include ‘left’ or
Revised: SS-V1&V2 (Goyal et al., 2017b), we do not ﬂip labels We ﬁt into , we set we use - ule

Revised: We set the rate to

Original: Testing.
Revised: Testing.

Original: Given a video, we sample 1 or 2 clips, resize them into 240 × 320 images, and crop their centers as 224 × 224.
Revised: Given a video, we sample 1 or 2 clips, resize them into 240 × 320 images, and crop their centers as 224 × 224.

Original: We evaluate an average prediction of the sampled clips.
Revised: We evaluate an average prediction of the sampled clips.

Original: We report top-1 and top-5 accuracy for SS-V1&V2 and Diving-48, and mean-class accuracy for FineGym.
Revised: We report top-1 and top-5 accuracy for SS-V1&V2 and Diving-48, and mean-class accuracy for FineGym.

Original: Frame corruption details.
Revised: Frame corruption details.

Original: We adopt two corruptions, occlusion and motion blur, to test the robustness of SELFYNet.
Revised: We adopt two corruptions, occlusion and motion blur, to test the robustness of SELFYNet.

Original: We only corrupt a single center-frame for every validation clip of SS-V1; we corrupt the 4 th frame amongst 8 input frames.
Revised: We only corrupt a single center-frame for every validation clip of SS-V1; we corrupt the 4 th frame amongst 8 input frames.

Original: For the occlusion, we cut out a rectangle region from the center of the frame.
Revised: For the occlusion, we cut out a rectangle region from the center of the frame.

Original: For the motion blur, we adopt ImageNet-C implementation, which is available online 1 .
Revised: For the motion blur, we adopt ImageNet-C implementation, which is available online 1 .

Original: We set 6 levels of severity for each corruption.
Revised: We set 6 levels of severity for each corruption.

Original: We set the side length of the occluded region as 40px, 80px, 120px, 160px, 200px and 224px from the level 1 to 6.
Revised: We set the side length of the occluded region as 40px, 80px, 120px, 160px, 200px and 224px from the level 1 to 6.

Original: For the motion blur, we set ( radius , sigma ) tuple arguments as (15 , 5) , (10 , 8) , (15 , 12) , (20 , 15) , (25 , 20) , and (30 , 25) , respectively.
Revised: For the motion blur, we set ( radius , sigma ) tuple arguments as (15 , 5) , (10 , 8) , (15 , 12) , (20 , 15) , (25 , 20) , and (30 , 25) , respectively.

Original: [Table]
Revised: [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: B A DDITIONAL EXPERIMENTS
Revised: B A DDITIONAL EXPERIMENTS

Original: We conduct additional ablation experiments to identify the behaviors of the proposed method.
Revised: We conduct additional ablation experiments to identify the behaviors of the proposed method.

Original: All experiments are performed on SS-V1 by using 8 frames.
Revised: All experiments are performed on SS-V1 by using 8 frames.

Original: Unless otherwise speciﬁed, we set ImageNet pre-trained TSM ResNet-18 (TSM-R18) with a single SELFY block of which ( L, U, V ) =
Revised: Unless otherwise speciﬁed, we set ImageNet pre-trained TSM ResNet-18 (TSM-R18) with a single SELFY block of which ( L, U, V ) =

Original: (5 , 9 , 9) , as our default SELFYNet.
Revised: (5 , 9 , 9) , as our default SELFYNet.

Original: Comparison with non-local methods.
Revised: Comparison with non-local methods.

Original: We compare our method with popular non-local methods (Wang et al., 2018; Liu et al., 2019), whichalso capture the long-range dynamics of videos.
Revised: We compare our method with popular non-local methods (Wang et al., 2018; Liu et al., 2019) [Table] which capture the long-range dynamics of videos.

Original: Both methods construct the global STSS matrix and use the similarity values  for re-weighting visual features  or aligning top- k corresponded visual features.
Revised: While computing the self-similarity values as ours, both methods use them as attention weights for feature aggregation by multiplying them to the visual features (Wang et al., 2018) or aligning top- K corresponding features (Liu et al., 2019); they both do not use STSS itself as a relational representation. In contrast, our method does it indeed and learns a more powerful relational feature from STSS. The difference between our method and non-local methods are illustrated in Fig. We have conducted experiments for performance comparison, and the results are shown in Table 7a.

Original: We re-implement the non-local block and the CP module in Pytorch based on their ofﬁcial codes 23 .
Revised: We re-implement the non-local block and the CP module in Pytorch based on their ofﬁcial codes 23 .

Original: For a fair comparison, we insert a single block or module at the same position (after res 3 of ResNet-18), and the results are shown in Table 7b.
Revised: For a fair comparison, we insert a single block or module at the same position (after res 3 of ResNet18).

Original: Note that our method downsamples a spatial resolution of V to 14 × 14 before STSS transformation, whereas the others do not.
Revised: Note that our method downsamples a spatial resolution of V to 14 × 14 before STSS transformation, whereas the others do not.

Original: Compared to the non-local block and the CP module, SELFY block improves top-1 accuracy by 4.4%p and 1.5%p, while computing less ﬂoating-point operations as 7.5 GFLOPs and 8.3 GFLOPs, respectively.
Revised: Compared to the non-local block and the CP module, SELFY block improves top-1 accuracy by 4.4%p and 1.5%p, while computing less ﬂoating-point operations as 7.5 GFLOPs and 8.3 GFLOPs, respectively.

Original: It demonstrates that the direct integration of STSS features is more effective for action recognition than the indirect ways of using STSS, e.g. , re-weighting visual-semantic features or learning correspondences. 
Revised: It demonstrates that the direct integration of STSS features is more effective for action recognition than the indirect ways of using STSS, e.g. , re-weighting visual-semantic features or learning correspondences. Comparison with correlation-based methods.

Revised: We also compare our method with correlationbased methods (Kwon et al., 2020; Wang et al., 2020).

Revised: While correlation-based methods extract motion features between two adjacent frames only and are thus limited to short-term motion, our method effectively captures bi-directional and long-term motion information via learning with the sufﬁcient volume of STSS.

Original: Multi-channel 3 × 3 × 3 kernel for feature extraction.
Revised: Our method can also exploit richer information from the self-similarity values than other methods. MS module (Kwon et al., 2020) only focuses on the maximal similarity value of the ( U, V ) dimension to extract ﬂow information, and Correlation block (Wang et al., 2020) uses an 1 × 1 convolution layer for extracting motion features from the similarity values. In contrast to the two methods, we introduce a generalized motion learning framework using the self-similarity tensor at Sec 3.2 of our main paper. The difference between our method and correlation-based methods are illustrated in Fig.

Revised: We also have conducted experiments to compare our method with MSNet (Kwon et al., 2020), one of the correlation-based methods.

Revised: For an apple-to-apple comparison, we apply kernel soft-argmax and max pooling operation ( KS + CM in (Kwon et al., 2020)) to our feature extraction method by following their ofﬁcial codes 4 .

Revised: Please note that, when we restrict the temporal offset l to { 1 } , the

Revised: SELFY block using KS + CM is equivalent to the MS module of which feature transformation layers are the standard 2D convolutional layers.

Revised: Table 7b summarizes the results.

Revised: KS+CM method achieves 46.1 %p at top-1 accuracy.

Revised: As we enlarge the temporal window L to 5, we obtain an additional gain as 1.3 %p.

Revised: The learnable convolution layers improve top-1 accuracy by 1.0 %p in both cases.

Revised: The results demonstrates the effectiveness of learning geometric patterns within the sufﬁcient volume of

Original:  Wealso investigate the effect of the convolution method for STSS feature extraction, when we use multi-channel 3 × 3 × 3 kernels.
Revised: STSS tensors for learning abundant motion features. Multi -channel 3 × 3 × 3 kernel for feature extraction. We investigate the effect of the convolution method for STSS feature extraction, when we use multi-channel 3 × 3 × 3 kernels.

Original: For the experiment, we stack four 3 × 3 × 3 convolution layers followed by the feature integration step, which are the same as in Section 3.2.2.
Revised: For the experiment, we stack four 3 × 3 × 3 convolution layers followed by the feature integration step, which are the same as in Section 3.2.2.

Original: Table 7a summarizes the results.
Revised: Table 7c summarizes the results.

Original: Note that we do not report models of which temporal window L = 1 , e.g. , { 0 } and { 1 } .
Revised: Note that we do not report models of which temporal window L = 1 , e.g. , { 0 } and { 1 } .

Original: As shown in the table, indeed, the long temporal range gives the higher accuracy.
Revised: As shown in the table, indeed, the long temporal range gives the higher accuracy.

Original: However, the effect of the 3 × 3 × 3 kernel is comparable to that of the 1 × 3 × 3 kernel in Table 3a in terms of accuracy.
Revised: However, the effect of the 3 × 3 × 3 kernel is comparable to that of the 1 × 3 × 3 kernel in Table 3a in terms of accuracy.

Original: Considering the accuracy-computation trade-off, we choose to ﬁx the kernel size, L κ ×
Revised: Considering the accuracy-computation trade-off, we choose to ﬁx the kernel size, L κ ×

Original: U κ × V κ , as 1 × 3 × 3 for the STSS feature extraction.
Revised: U κ × V κ , as 1 × 3 × 3 for the STSS feature extraction.

Original: Block position.
Revised: Spatial matching region. In Table 7d, we compare a single SELFY block with different spatial matching regions, ( U, V ) . As a result, indeed, the larger spatial matching region leads the better accuracy. Considering the accuracy-computation trade-off, we set our spatial matching region, ( U, V ) , as (9 , 9) as a default.

Revised: Fusing STSS with visual features.

Revised: We evaluate SELFYNet purely based on STSS features to see how much the ordinary visual feature V contributes for the ﬁnal prediction.

Revised: That is, we pass the

Revised: STSS features, ReLU( F (cid:63) × 5 W θ ) , into the downstream layers without the visual features V

Revised: (Eq.in our main paper).

Revised: For the simplicity of description, we denote the relational feature ReLU( F (cid:63) ×

Revised: W θ ) by R .

Revised: Table 7e compares the results of using different cases of the output tensor Z ( Z = V ,

Revised: Z = R , and Z = R + V ) on SS-V1.

Revised: Interestingly, SELFYNet using only R achieves 45.5 %p at top1 accuracy, which is higher as 2.5 %p than the baseline.

Revised: As we add V to R , we obtain an additional gain as 2.9 %

Revised: p.

Revised: It indicates that the STSS features and the visual features are complementary to each other.

Original:  From the 2 nd to the 4 th row of Table 7c, we identify the effect of different positions of SELFY block in the backbone.
Revised: Block position. From the 2 nd to the 6 th row of Table 7f, we identify the effect of different positions of SELFY block in the backbone.

Original: We resize the spatial resolution of the video tensor, ( X, Y ) , into 14 × 14, and ﬁx the matching region, ( L, U, V ) , as (5 , 9 , 9) for all the cases maintaining the similar computational cost.
Revised: We resize the spatial resolution of the video tensor, ( X, Y ) , into 14 × 14, and ﬁx the matching region, ( L, U, V ) , as (5 , 9 , 9) for all the cases maintaining the similar computational cost.

Original: SELFY after the res 3 shows the best trade-off by achieving the highest accuracy among the cases.
Revised: SELFY after the res 3 shows the best trade-off by achieving the highest accuracy among the cases.

Original: The last row in Table 7c shows that the multiple SELFY blocks improve accuracy compared to the single block. 
Revised: The last row in Table 7f shows that the multiple SELFY blocks improve accuracy compared to the single block. [Table]

Original: Spatial matching region.
Revised: The local self-attention (Hu et al., 2019; Ramachandran et al., 2019;

Original: In Table 7d, we compare a single SELFY block with different spatial matching regions, ( U, V ) .
Revised: Zhao et al., 2020) and our method have a common denominator of using the self-similarity tensor but use it in a very different way and purpose.

Original: As a result, indeed, the larger spatial matching region leads the better accuracy.
Revised: The local self-attention mechanism aims to aggregate the local context features using the self-similarity tensor and it thus uses the self-similarity values as attention weights for feature aggregation. However, our method aims to learn a generalized motion representation from the local STSS, so the ﬁnal STSS representation is directly fed into the neural network instead of multiplying it to local context features. For an empirical comparison, we conduct an ablation experiment as follows. We extend the local self attention layer (Ramachandran et al., 2019) to temporal dimension, and then add the spatio-temporal local self-attention layer, which is followed by feature integration layers, after res 3 .

Original: Considering the accuracy-computation trade-off, we set our spatial matching region, ( U, V ) , as (9 , 9) as a default.
Revised: All experimental details are the same as those in Appendix A, except that we reduce the channel dimension C ‘ of appearance feature V to 32. Table 8 summarizes the results on SS-V1. The spatio-temporal local self-attention layer is accurate as 43.8 %p at top-1 accuracy, and both of SELFY blocks using the embedded Gaussian and the cosine similarity outperform the local self-attention by achieving top-1 accuracy as 47.6 %p and 47.8 %p, respectively. These results are in alignment with the prior work (Liu et al., 2019), which reveals that the self-attention mechanism hardly captures motion features in video.

Original: C V ISUALIZATIONS
Revised: D V ISUALIZATIONS

Original: In Fig.
Revised: In Fig.

Original: 6, we visualize some qualitative results of two different SELFYNet-TSM-R18 ( { 1 } and {− 3 , · · · , 3 } ) on SS-V1.
Revised: 7, we visualize some qualitative results of two different SELFYNet-TSM-R18 ( { 1 } and {− 3 , · · · , 3 } ) on SS-V1.

Original: We show the different predictions of the two models with 8 input frames.
Revised: We show the different predictions of the two models with 8 input frames.

Original: We also overlay Grad-CAMs (Selvaraju et al., 2017) on the input frames to see whether a larger volume of STSS beneﬁts to capture long-term interactions in videos.
Revised: We also overlay Grad-CAMs (Selvaraju et al., 2017) on the input frames to see whether a larger volume of STSS beneﬁts to capture long-term interactions in videos.

Original: We take Grad-CAMs of features which is right before a global average pooling layer.
Revised: We take Grad-CAMs of features which is right before a global average pooling layer.

Original: As shown in the ﬁgure, the STSS with the sufﬁcient volume helps to learn more enriched context of temporal dynamics in the video; in Fig.
Revised: As shown in the ﬁgure, the STSS with the sufﬁcient volume helps to learn more enriched context of temporal dynamics in the video; in Fig.

Original: 6a, for example, SELFYNet with the range of ( {− 3 , · · · , 3 } ) focuses on not only regions on which an action occurs but also focuses on the white-stain after the action to verify whether the stain is wiped off or not.
Revised: 7a, for example, SELFYNet with the range of ( {− 3 , · · · , 3 } ) focuses on not only regions on which an action occurs but also focuses on the white-stain after the action to verify whether the stain is wiped off or not.

Original: [Figure]
Revised: [Figure]

Revised: [Figure]