C AN AN I MAGE C LASSIFIER S UFFICE FOR A CTION R ECOGNITION ?
Anonymous authors
Paper under double-blind review
A BSTRACT
We explore a new perspective on video understanding by casting the video recognition problem as an image recognition task.
Our approach rearranges input video frames into super images, which allow for training an image classiﬁer directly to fulﬁll the task of action recognition, in exactly the same way as image classiﬁcation.
With such a simple idea, we show that transformer-based image classiﬁers alone can sufﬁce for action recognition.
In particular, our approach demonstrates strong and promising performance against SOTA methods on several public datasets including Kinetics400, Moments
In Time, Something-Something V2 (SSV2), Jester and Diving48.
We also experiment with the prevalent ResNet image classiﬁers in computer vision to further validate our idea.
The results on both Kinetics400 and
SSV2 are comparable to some of the best-performed CNN approaches based on spatio-temporal modeling.
Our codes and models will be publicly available.
I NTRODUCTION
[Figure]
The recent advances in convolutional neural networks (CNNs) (He et al., 2016; Tan & Le, 2019), along with the availability of large-scale video benchmark datasets (Kay et al., 2017; Monfort et al., 2019; Damen et al., 2020), have signiﬁcantly improved action recognition, one of the fundamental problems of video understanding.
Many existing approaches for action recognition naturally extend or borrow ideas from image recognition.
At the core of these approaches is spatio-temporal modeling , which regards time as an additional dimension and jointly models it with space by extending image models (i.e., 3D CNNs)
(Tran et al., 2015; Carreira et al., 2017; Feichtenhofer, 2020) or fuses temporal information with spatial information processed separately by 2D CNN models (Lin et al., 2019;
Fan et al., 2019).
CNN-based approaches demonstrate strong capabilities in learning saptio-temporal feature representations from video data.
Videos present long-range pixel interactions in both space and time.
It’s known in approaches like non-local networks (Wang et al., 2018) that modeling such relationships helps action recognition.
The recently emerging Vision Transformers naturally own the strength of capturing long-range dependencies in data, making them very suitable for video understanding.
Several approaches (Bertasius et al., 2021a; Li et al., 2021; Arnab et al., 2021) have applied ViTs for action recognition and shown better performance than their CNN counterparts.
However, these approaches are still following the conventional paradigm of video action recognition, and perform temporal modeling in a similar way to CNN-based approaches using dedicated self-attention modules.
In this work, we explore a different perspective for action recognition by casting the problem as an image recognition task.
We ask if it is possible to model temporal information with ViT directly without using dedicated temporal modules.
In other words, can an image classiﬁer alone sufﬁce for action recognition?
To this end, we ﬁrst propose a simple idea to turn a 3D video into a 2D image.
Given a sequence of input video frames, we rearrange them into a super image according to a pre-deﬁned spatial layout, as illustrated in Fig.
The super image encodes 3D spatio-temporal patterns in a video into 2D spatial image patterns.
We then train an image classiﬁer to fulﬁll the task of action recognition, in exactly the same way as image classiﬁcation.
Without surprise, based on the concept of super images, any image classiﬁer can be re-purposed for action recognition.
For convenience, we dub our approach SIFAR , short for S uper I mage f or A ction R ecognition.
We validate our idea by using Swin Transformer (Liu et al., 2021), a recently developed vision transformer that has demonstrated good performance on both image classiﬁcation and object detection.
Since a super image has a larger size than an input frame, we modify Swin Transformer to allow for full self-attention in the last layer of the model, which further strengthens the model’s ability in capturing long-range temporal relations across frames in the super image.
With such a change, we show that SIFAR produces strong performance against the existing SOTA approaches (Fig. 1) on several benchmarks including Kinetics400 (Kay et al., 2017), Moments in Time (Monfort et al., 2019) , Something-Something V2 (SSV2) Goyal et al.
(2017), Jester (Materzynska et al., 2019)
and
Diving48 (Li et al., 2018).
SIFAR also enjoys efﬁciency in computation as well as in parameters.
We further study the potential of CNN-based classiﬁers directly used for action recognition under the proposed SIFAR framework.
Surprisingly, they achieve very competitive results on Kinetics against existing CNN-based approaches that rely on much more sophisticated spatio-temporal modeling.
Since 3 × 3 convolutions focus on local pixels only, CNN-based SIFAR handles temporal actions on Something-Something less effectively.
We experiment with larger kernel sizes to expand the temporal receptive ﬁeld of CNNs, which substantially improves CNN-based SIFAR by 4%-6.8% with ResNet50.
SIFAR brings several advantages compared to the traditional spatio-temporal action modeling.
Firstly, it is simple but effective.
With one single line of code change in pytorch, SIFAR can use any image classiﬁer for action recognition.
We expect that similar ideas can also work well with other video tasks such as video object segmentation (Duke et al., 2021).
Secondly, SIFAR makes action modeling easier and more computationally efﬁcient as it doesn’t require dedicated modules for temporal modeling.
Nevertheless, we do not tend to underestimate the signiﬁcance of temporal modeling for action recognition.
Quite opposite, SIFAR highly relies on the ability of its backbone network to model long-range temporal dependencies in super images for more efﬁcacy.
Lastly, but not the least, the perspective of treating action recognition the same as image recognition unleashes many possibilities of reusing existing techniques in a more mature image ﬁeld to improve video understanding from various aspects.
For example, better model architectures (Tan & Le, 2019), model pruning (Liu et al., 2017) and interpretability (Desai & Ramaswamy, 2020), to name a few.
R ELATED W ORK
Action Recognition from a Single Image.
One direction for video action recognition is purely based on a single image (Davis & Bobick, 1997; Zhao et al., 2017; Safaei & Foroosh, 2019; Bilen et al., 2016).
In (Davis & Bobick, 1997), multiple small objects are ﬁrst identiﬁed in a still image
and then the target action is inferred from the relationship among the objects.
Other approaches such as (Safaei & Foroosh, 2019) propose to predict the missing temporal information in still images and then combine it with spatial information for action classiﬁcation.
There are also approaches that attempt to summarize RGB or motion information in a video into a representation image for action recognition.
For instance, motion-energy image (MEI) (Davis & Bobick, 1997), Dynamic Image Network (Bilen et al., 2016), Informative Frame Synthesis (IFS) (Qiu et al., 2021), Adaptive Weighted Spatio-temporal Distillation (AWSD) (Tavakolian et al., 2019b) and Adversarial Video Distillation (AVD) (Tavakolian et al., 2019a).
Nonetheless, our approach does not attempt to understand a video from a single input image or a summarization image. Instead the approach composites the video into a super image, and then classiﬁes the image with an image classiﬁer directly.
Action Recognition with CNNs.
Action recognition is dominated by CNN-based models recently (Feichtenhofer et al., 2018; Carreira et al., 2017; Fan et al., 2019; Feichtenhofer, 2020;
Chen et al., 2021; Lin et al., 2019; Wang et al., 2016; Zhou et al., 2018; Liu et al., 2020; Jiang et al., 2019a; Tran et al., 2019).
These models process the video as a cube to extract spatial-temporal features via the proposed temporal modeling methods.
E.g., SlowFast (Feichtenhofer et al., 2018) proposes two pathways whose speed is different to capture short-range and long-range time dependencies.
TSM (Lin et al., 2019) applies a temporal shifting module to exchange information between neighboring frames and TAM (Fan et al , 2019) further enhances TSM by determining the of information to be shifted and blended.
On the other hand, another thread of work attempts to select the key frame of an activity for faster recognition ( al Meng E.g
) employs a policy network to a network determines what resolution to simply uses an image classiﬁer as a video classiﬁer by laying out a video to a without
Action Recognition with Transformer.
Following the vision transformer (ViT) (Dosovitskiy et al., 2021), which demonstrates competitive performance against CNN models on image classiﬁcation, many recent works attempt to extend the vision transformer for action recognition (Neimark et al., 2021;
Li et al., 2021;
Bertasius et al., 2021b; Arnab et al., 2021; Fan et al., 2021).
VTN (Neimark et al., 2021), VidTr (Li et al., 2021), TimeSformer (Bertasius et al., 2021b) and ViViT (Arnab et al., 2021) share the same concept that inserts a temporal modeling module into the existing ViT to enhance the features from the temporal direction.
E.g., VTN (Neimark et al., 2021) processes each frame independently and then uses a longformer to aggregate the features across frames.
On the other hand, divided-space-time modeling in TimeSformer (Bertasius et al., 2021a) inserts a temporal attention module into each transformer encoder for more ﬁne-grained temporal interaction.
MViT (Fan et al., 2021) develops a compact architecture based on the pyramid structure for action recognition.
It further proposes a pooling-based attention to mix the tokens before computing the attention map so that the model can focus more on neighboring information.
Nonetheless, our method is straightforward and applies the Swin (Liu et al., 2021) model to classify super images composed from input frames.
Note that the joint-space-time attention in TimeSformer (Bertasius et al., 2021a) is a special case of our approach since their method can be considered as ﬂattening all tokens into one plane and then performing self-attention over all tokens.
However, the memory complexity of such an approach is prohibitively high, and it is only applicable to the vanilla ViT (Dosovitskiy et al., 2021) without inductive bias.
On the other hand, our SIFAR is general and applicable to any image classiﬁers.
A PPROACH 3. O VERVIEW OF OUR A PPROACH
The key insight of SIFAR is to turn spatio-temporal patterns in video data into purely 2D spatial patterns in images.
Like their 3D counterparts, these 2D patterns may not be visible and recognizable by human.
However, we expect they are characteristic of actions and thus identiﬁable by powerful neural network models.
To that end, we make a sequence of input frame images from a video into a super image, as illustrated in Fig. 2, and then apply an image classiﬁer to predict the label of the video.
Note that the action patterns embedded in a super image can be complex and may involve both local (i.e., spatial information in a video frame) and global contexts (i.e., temporal dependencies across frames).
It is thus understandable that effective learning can only be ensured by image classiﬁers with strong capabilities in modeling short-range and long-range spatial dependencies in super images.
For this reason, we explore the recently developed vision transformers based on self-attention to validate our proposed idea.
These methods come naturally with the ability to model global image contexts and have demonstrated competitive performance against the best-performed CNN-based approaches on image classiﬁcation as well as action recognition.
Next we brieﬂy describe Swin Transformer (Liu et al., 2021), an efﬁcient approach that we choose to implement our main idea in this work.
Preliminary.
The Vision Transformer (ViT) [13] is a purely attention-based classiﬁer borrowed from NLP.
It consists of stacked transformer encoders, each of which is featured with a multi-head self-attention module (MSA) and a feed-forward network (FFN).
While demonstrating promising results on image classiﬁcation, ViT uses an isotropic structure and has a quadruple complexity w.r.t image resolution in terms of memory and computation.
This signiﬁcantly limits the application of
ViT to many vision applications that requires high-resolution features such as object detection and segmentation.
In light of this issue, several approaches (Liu et al., 2021; Chu et al., 2021; Zhang et al., 2021) have been proposed to perform region-level local self-attention to reduce memory usage and computation, and Swin Transformer is one of such improved vision transformers.
Swin Transformer (Liu et al., 2021)
ﬁrst adopts a pyramid structure widely used in CNNs to reduce computation and memory.
At the earlier layers, the network keeps high image resolution with fewer feature channels to learn ﬁne-grained information.
As the network goes deeper, it gradually reduces spatial resolution while expanding feature channels to model coarse-grained information.
To further save memory, Swin Transformer limits self-attention to non-overlapping local windows (W-MSA) only.
The communications between W-MSA blocks is achieved through shifting them in the succeeding transformer encoder.
The shifted W-MSA is named as SW-MSA.
Mathematically, the two consecutive blocks can be expressed as:
[Equation]
where x l is the features at the l th layer and FFN and LN are feed-forward network and layer normalization, respectively.
SIFAR.
In our case, SIFAR learns action patterns by sliding window, as illustrated in Fig 3.
When the sliding window (blue box) is within a frame, spatial dependencies are learned.
On the other hand, when the window (red box) spans across frames, temporal dependencies between them are effectively captured.
The spatial pooling further ensures longer-range dependencies across frames captured.
cies ent frames (
), temporal dependencies between them are effectively further ensures longer-range dependencies to be learnt.
Best viewed
7 a)
[Figure]
Creation of Super Image.
Given a set of video frames, we order them by a given layout (Fig. 4) to form a large super image.
Different layouts give different spatial patterns for an action class.
We hypothesize that a more compact structure such as a square grid may facilitate a model to learn temporal dependencies across frames as such a shape provides the shortest maximum distance between any two images.
Given n input frames, we create a super image by placing all the frames in order onto a grid of size ( m − 1)
× m when n < ( m − 1) × m or m × m when n ≥ ( m − 1) × m where m =
(cid:100)√ n (cid:101) .
Empty images are padded at the end if the grid is not full.
With this method, for example, 12 frames will be ﬁt into a 3 × 4 grid while 14 frames into a 4 × 4 grid.
In the default setting, we use a 3 × 3 layout for 8 images and a 4 × 4 one for 16 images, respectively.
There are other spatial arrangements as well (see Fig.
4 for more examples).
However our experiments empirically show that a square grid performs the best.
our approach has linear computational complexity w.r.t the number of input frames.
As described above, the size of a super image is m ( m = (cid:100)√ n (cid:101) ) times as large as the size of a frame image, suggesting that the total number of tokens (or image patches) in Swin grows linearly by n .
[Table]
Sliding Window.
As previously mentioned,
Swin Transformer performs self-attention within a small local window to save memory.
It uses a uniform window size across all layers, and the default window size is 7 in the original paper.
Since a larger window leads to more interactions across frames, which is beneﬁcial for SIFAR to learn long-range temporal dependencies in super images, we slightly modify the architecture of
Swin Transformer (Liu et al., 2021) for it to take different window sizes ﬂexibly in self-attention.
In particular, we keep the same window size for all the layers except the last one, whose window is as large as its image resolution, implying a global self-attention including all the tokens.
Since the last layer has only two transformer encoders, the computational overhead imposed by an increased window size is quite small, as indicated in Table 1.
The change of window size may result in adjustment of the input image size as the image resolution at each layer must be divisible by the window size in Swin Transformer.
As noted in Table 1, SIFAR-Bkeeps the vanilla architecture of Swin-B. SIFAR-B-12 is more efﬁcient than SIFAR-B-7 because
SIFAR-B-12 takes smaller images (192 2 ) as input.
We demonstrate later in Sec.
4 that a larger window is critical for SIFAR to achieve good performance on more temporal datasets such as SSV2.
Implementation.
Once the spatial layout for the input frames is determined, implementing our idea in pytorch is as simple as inserting into an image classiﬁer the following line of code, which changes the input of a video to a super image.
[Equation]
The trivial code change described above transforms an image classiﬁer into an video action classiﬁer.
Our experiments show that the same training and evaluation protocols for action models can be still applied to the repurposed image classiﬁer.
E XPERIMENTS 4. D ATASETS AND E XPERIMENTAL S ETUP
Datasets.
We use Kinetics400 (K400)
(Kay et al., 2017), Something-Something V2 (SSV2) (Goyal et al., 2017), Moments-in-time (MiT) (Monfort et al., 2019), Jester (Materzynska et al., 2019), and Diving48 (Li et al., 2018) in our evaluation.
Kinetics400 is a widely-used benchmark for action recognition, which includes ∼ 240k training videos and 20k validation videos in 400 classes.
SSV2 contains 220k videos of 174 types of predeﬁned human-object interactions with everyday objects.
This dataset is known for its high temporal dynamics.
MiT is a fairly large collection of one million 3-second labeled video clips, involving actions not only from humans, but also from animals, objects and natural phenomena.
The dataset includes around 800k training videos and 33,900 validation videos in 339 classes.
Jester contains actions of predeﬁned hand gestures, with 118,562 and 14, training and validation videos over 27 classes, respectively.
Diving48 is action recognition dataset without representation bias, which includes 15,943 training videos and 2,096 validation videos over 48 classes.
Training .
We employ uniform sampling to generate video input for our models.
Such a sampling strategy divides a video into multiple segments of equal length.
During training, one frame from each segment is randomly selected to form the input sequence to a model while in test, a frame at a ﬁxed position of each segment (usually the center frame) is picked.
Uniform sampling has shown to be effective on both Kinetics400 and SSV2.
We train all our models by ﬁnetuning a Swin-B model (Liu et al., 2021) pretrained on ImageNet-21K (Deng et al.,
2009), except for those SSV2 models, which are ﬁne tuned from the corresponding Kinetics400 models in Table 3.
Our training recipes and augmentations closely follow DeiT (Touvron et al., 2020).
First, we apply multi-scale jitter to augment the input (Wang et al., 2016) with different scales and then randomly crop a target input size (e.g. 8 × 224 × 224 for SIFAR-B-7).
We then use Mixup (Zhang et al., 2018) and CutMix (Yun et al., 2019) to augment the data further, with their values set to 0.8 and 1.0, respectively.
After that, we rearrange the image crops as a super image.
Furthermore, we apply drop path (Tan & Le, 2019) with a rate of 0.1, and enable label smoothing (Szegedy et al., 2016) at a rate of 0.1.
All our models were trained using V100 GPU cards with 16G or 32G memory.
Depending on the size of a model, we use a batch size of 96, 144 or 192 to train the model for 15 epochs on MiT or 30 epochs on other datasets, including 5 warming-up epochs.
The optimizer used in our training is AdamW (Loshchilov & Hutter, 2019) with a weight decay of 0.05, and the scheduler is Cosine (Loshchilov & Hutter, 2017) with a base linear learning rate of 0.0001.
Inference.
We ﬁrst scale the shorter side of an image to the model input size and then take three crops (top-left, center and bottom-right) for evaluation.
The average of the three predictions is used as the ﬁnal prediction.
We report results by top-1 and top-5 classiﬁcation accuracy (%) on validation data, the total computational cost in FLOPs and the model size in number of parameters.
4. M AIN R ESULTS
[Table]
Comparison with Baselines.
We ﬁrst compare our ap- proach with several representative CNN-based methods including I3D (Carreira et al., 2017), TSM (Lin et al., 2019) and TAM (Fan et al., 2019).
Also included in the comparison are two TimeSformer models (Bertasius et al., 2021a) based on the same backbone Swin-B
(Liu et al., 2021) as used by our models.
All the models considered take 8 frames as input.
As can be seen from Table 2, our approach substantially outperforms the CNN baselines on Kinetics400 while achieving comparable results on SSV2.
Our approach is also better than TimeSformer on both datasets.
These results clearly demonstrate that a powerful image classiﬁer like Swin Transformer can learn expressive spatio-temporal patterns effectively from super images for action recognition.
In other words, an image classiﬁer can sufﬁce video understanding without explicit temporal modeling.
The results also conﬁrm that a larger sliding window is more helpful in capturing temporal dependencies on temporal datasets like SSV2.
Our approach performs global self-attention in the last layer of a model only (see
Table 1).
This substantially mitigates the memory issue in training SIFAR models.
Kinetics400.
We report our Kinetics400 results in Table 3 and compare them with SOTA approaches.
Our 8-frame models (SIFAR-12 and SIFAR-14) achieve 80 .
% and 80 .
4% top-1 accuracies, outperforming all the CNN-based approaches while being more efﬁcient than the majority of them.
SIFAR-B-14 † further gains ∼ 1 .
8% improvement, beneﬁting from more input frames.
Especially, SIFAR-L-12 ‡ yields an accuracy of 84.2%, the best among all the very recently developed approaches based on vision transformers including TimeSformer (Bertasius et al., 2021b) and MViT-B (Fan et al., 2021).
Our proposed approach also offers clear advantages in terms of FLOPs and model parameters compared to other approaches except MViT-B.
For example, SIFAR-B-12 ‡ has 5 × and 37 × fewer FLOPs than TimeSformer-L and ViViT-L, respectively, while being 1.4 × and 3.6 × smaller in model size.
SSV2.
Table 4 lists the results of our models and the SOTA approaches on SSV2.
With the same number of input frames, our approach is 1 ∼ 2% worse than the best-performed CNN methods.
However, our approach performs on par with other transformer-based method such as TimeSformer (Bertasius et al., 2021a) and VidTr-L (Li et al., 2021) under the similar setting.
Note that ViViT-L (Arnab et al., 2021) achieves better results with a larger and stronger backbone ViT-L (Dosovitskiy et al., 2021).
MViT-B (Fan et al., 2021) is an efﬁcient multi-scale architecture, which can process much longer input sequences to capture ﬁne-grained motion patterns in SSV data.
Training SIFAR models with more than 16 frames still remains computationally challenging, especially for models like SIFAR-B-14 and SIFAR-L-14 † , which need a larger sliding window size.
Our results suggest that developing more efﬁcient architectures of vision transformer be an area of improvement and future work for SIFAR to take advantage of more input frames on SSV2.
MiT. MiT is a large diverse dataset containing label noise.
As seen from Table 5, with the same backbone ViT-L, SIFAR-L-12 ‡ is % better than ViViT-L (Arnab et al., 2021), et ., 2020) based on neural architecture search by a considerable margin of 8%.
Jester and Diving48.
We further evaluate our approach on two other popular benchmarks: Jester (Materzynska et al., 2019) and Diving48 (Li et al., 2018).
Here we only consider the best single models from other approaches [Table] [Table] for fair comparison.
As shown in Table 6, SIFAR achieves competitive results again on both datasets, surpassing all other models in comparison.
Note that the Diving48 benchmark contains videos with similar background and objects but different action categories, and is generally considered as an unbiased benchmark.
Our model SIFAR-B-14 † outperforms TimeSformer-L by a large margin of 6%.
Classiﬁcation by CNNs.
We also test our proposed approach using the ResNet image classiﬁers on both SSV2 and Kinetics400 datasets.
For fairness, the ResNet models are pretrained on ImageNet-21K. Table 7 shows the results.
Our models clearly outperform the traditional CNN-based models for action recognition on Kinetics400.
[Table]
[Table]
Especially, with a strong backbone R152x2 (a model 2 × wider than Resnet152), SIFAR-R152x2 achieves a superior accuracy of 79.0%, which is surprisingly comparable to the best CNN results (SlowFast16×8+NL: 79.8%) reported in Table
[Table] [Table]
On SSV2, the results of CNN-based SIFAR are less satisfactory but reasonable.
This is because 3x3 convolutions are local with a small receptive ﬁeld, thus failing to capturing long-range temporal dependencies in super images.
We hypothesize that a larger ker- nel size with a wider receptive ﬁeld may address this limitation and potentially improve the performance of CNN-based SIFAR models.
To validate this, we perform additional experiments by adding one or two more residual blocks to the end of ResNet models with larger kernel sizes, i.e. replacing the second con- volution in those new blocks by a 7x7, 11x11 or 21x kernel.
These models are indicated by names ending with “C7” (7x7), “C11” (11x11) or “C21” (21x21) in Table 7.
As seen from the table, using larger kernel sizes consistently improves the results on both ResNet50 and ResNet101 models.
For example, we obtain an absolute 5.0% improvement over original ResNet50 and 2.0% over original ResNet101 respectively, using one more block with a kernel size of 21x21.
When adding another block with a kernel size of 11x11 (i.e. SIFA-R50-C21-11 and SIFA-R101-C21-11), it further boosts the performance up to 6.8% with ResNet50 and 2.7% with ResNet101.
These results strongly suggest that expanding the receptive ﬁeld of CNNs be a promising direction to design better CNN-based SIFAR models.
We leave this as future work.
4. A BLATION S TUDIES
In this section, we conduct ablation studies to provide more insights about our approach.
How does an image layout affect the performance?
The layout of a super image determines how spatio- temporal patterns are embedded in it.
We hypothesize that the layout could affect the learning effectiveness of a model.
To analyze this, we trained a SIFAR model on SSV2 for each layout illustrated in Fig.
As shown in Table 8a, a strip layout performs the worst while a grid layout produces the best results, which conﬁrms our hypothesis. [Table]
Does absolute positioning embedding help?
Absolute Position Embedding (APE) assigns ﬁxed or learnable position information to each token in a transformer model, and it has been proven helpful to vision transformers such as ViT (Dosovitskiy et al., 2021).
However, the Swin paper (Liu et al., 2021) shows that when relative position bias are added, APE is only moderately beneﬁcial for classiﬁcation, but not for object detection and segmentation.
They thus conclude that inductive bias that encourages certain translation invariance is still important for vision tasks.
To ﬁnd out whether or not APE is effective in our approach, we add APE to each frame rather than each token.
The results in Table 8b indicate that APE slightly improves model accuracy on SSV2, but is harmful to Kinetics400.
In our main results, we thus apply APE to SSV2 only.
[Table]
Does the temporal order of input matter?
We evaluate the SIFARB-12 model ( trained with input frames of normal order) by using three types of input with different temporal orders, i.e the reverse, random and normal.
As can be seen from Table 9, SIFAR is not sensitive to the input order on Kinetics400, whereas on SSV2, changing the input order results in a signiﬁcant performance drop.
This is consistent with the ﬁnding in the S3D paper (Xie et al., 2018), indicating that for datasets like SSV2 where there are visually similar action categories, the order of input frames matters in model learning.
What does SIFAR learn?
One big advantage of our proposed approach is that many techniques developed in the image domain now can be directly used for video understanding without re-invention.
Here we apply ablation cam (Desai & Ramaswamy, 2020), an image model interpretability technique, to understand what our models learn.
Fig.
5 shows the Class Activation Maps (CAM) of 4 actions correctly predicted by SIFAR-B-12.
Not surprisingly, the model learns to attend to objects relevant to the target action such as the hula hoop in a) and soccer ball in b).
In c) and d), the model seems to correctly focus on where meaningful motion happens.
C ONCLUSION
We have presented a new perspective for action recognition by casting the problem as an image recognition task.
Our idea is simple but effective, and with one line of code to transform an sequence of input frames into a super image, it can re-purpose any image classiﬁer for action recognition.
We have implemented our idea with both CNN-based and transformer-based image classiﬁers, both of which show promising and competitive results on several popular public video benchmarks.
Our experiments and results show that applying super images for video understanding is an interesting direction worth further exploration.