Lightweight Action Recognition in Compressed Videos
Abstract.
Most existing action recognition models are large convolutional neural networks (CNNs) that work only with raw RGB frames as input.
However, practical applications require lightweight models that directly process compressed videos.
In this work, for the ﬁrst time, such a model is developed, which is lightweight enough to run in real-time on embedded AI devices (e.g., 40FPS on a Jetson TX2) without sacriﬁces in recognition accuracy.
Compared to existing compressed video action recognition models, it is much more compact and faster thanks to adopting a lightweight CNN backbone.
Further, a number of novel components are introduced to improve the eﬀectiveness of the model: (1) A new Aligned Temporal Trilinear Pooling (ATTP) module is formulated to fuse three modalities in a compressed video namely I-frames, motion vectors, and residuals.
(2) To remedy the weaker motion vectors (compared to optical ﬂow computed from raw RGB streams) for representing dynamic content, we introduce a temporal fusion method to explicitly induce the temporal context, as well as knowledge distillation from a model trained with optical ﬂows via feature alignment.
Importantly, in contrast to existing models that either ignore B-frames or use them incorrectly, our ATTP model employs correct but more complicated B-frame modeling, thus being compatible with a wider range of contemporary codecs.
Extensive experiments show that our ATTP outperforms the state-ofthe-art alternatives in both eﬃciency and accuracy.
Introduction
Video analysis has drawn great attention from the computer vision community recently due to an increasing demand for automated video content understanding.
In particular, videos account for more than 75% of the global IP trafﬁc everyday [11].
With the advancements of deep learning methods, promising performance has been achieved on a variety of video analysis tasks, including action recognition [19,32,6,37,10,40,8,9,2,29,38,39,42,46,51,7,41], semantic segmentation [24,17], action localization [26,1], and deception detection [5,48].
However, existing convolutional neural network (CNN) based video analysis models still do not meet the requirements of many real-world applications.
This is primarily due to two reasons.
First, videos such as those on social media sites like YouTube or on smartphones are stored in a compression format to save space.
Nevertheless, most existing models work only with uncompressed raw RGB frames.
This means that the compressed videos must be decoded ﬁrst, leading to extra costs on both processing time and storage.
Second, most deep video models, based on either two-streams [32,40] or 3D CNNs [37,2,29] are slow (e.g., with high latency for calculating optical ﬂows) and heavy (having a large number of parameters), therefore unsuitable for either processing the vast amount of videos produced everyday, or running on embedded AI devices (e.g., smartphones).
Although there are some recent eﬀorts on lightweight video models [3,25], these models do not work on compressed videos.
Recently, researchers start to address the problem of compressed video action recognition [45,30,49,50].
These models take compressed videos (e.g., MPEG-4) directly as input.
Such a video contains only a few key frames and their oﬀsets (i.e., motion vectors and residual errors) for storage reduction.
However, none of these models is lightweight, and thus they still incur high latency and cannot run on embedded AI devices for edge computing.
Apart from the eﬃciency limitation, existing models are also ineﬀective due to the fact that they either ignore Bframes or use them incorrectly for extracting motion vectors and residuals.
This is despite the fact that in most video bit-streams, most of the frames (more than 60%) are encoded as B-frames to achieve the best compression rate.
In this paper, for the ﬁrst time, a challenging video analysis task, called
Lightweight Action Recognition in Compressed Videos , is tackled to ﬁll a gap in video analysis – to the best of our knowledge, this task has not been studied before.
The key challenges of lightweight compressed video action recognition are: 1) how to design a lightweight yet highly eﬀective deep CNN model for action recognition; 2) how to extract meaningful representations from compressed videos that contain far less information than the raw RGB frames
To address these challenges, we propose a lightweight model for compressed video action recognition.
Speciﬁcally, we adopt EﬃcientNet [36] as the backbone network to process the multiple modalities extracted from a compressed video (including RGB I-frames I , motion vectors MV , and residuals R ).
To fuse these modalities, we propose a novel Aligned Temporal Trilinear Pooling (ATTP) scheme to exploit the complementary information contained in them.
To remedy the weaker motion vectors (compared to optical ﬂow computed from raw RGB streams) for representing the dynamic content, we introduce a temporal fusion method to induce the temporal context explicitly.
Further, since motion vectors are much coarser than optical ﬂow vectors, we adopt a knowledge distillation strategy via feature alignment between our model and a model trained with optical ﬂow extracted from uncompressed videos.
Finally, we overcome a limitation [Figure] of existing models [45,30,49,50] in that they either ignore the B-frames or utilize them incorrectly.
This is because computing motion vectors and residuals from the B-frames is far more challenging than from the alternatives (i.e., P-frames).
In this work, by employing correct but more complicated B-frame modeling, our proposed model is applicable to all modern video codecs (e.g., H.264 & HEVC).
The result of introducing these new components is a lightweight yet powerful action recognition model: it outperforms the state-of-the-art alternatives in both eﬃciency and accuracy, as shown in Fig. 1.
Our contributions are four-fold: (1) For the ﬁrst time, we address the challenging problem of lightweight compressed video action recognition.
(2) We propose a trilinear pooling module for fusing the multiple modalities extracted from compressed videos.
(3) To work with the weak motion vectors, we propose a temporal fusion method to explicitly induce the temporal context, and also boost the backbone trained with motion vectors by distillation with feature alignment.
We are the ﬁrst to exploit both B-frames and P-frames from compressed videos in the correct manner, making our model more compatible with contemporary video codecs.
The eﬃciency test on a fast embedded AI computing device (i.e.,
Jetson TX2) indicates that our ATTP model can perform video action recognition at about 40FPS (see Table 1).
Moreover, as shown in Fig. 1, our ATTP model achieves the best performance but with signiﬁcantly fewer parameters, as compared to the state-of-the-art methods.
This observation is supported by the extensive results reported on two benchmarks widely used for action recognition (see Table 2).
To our best knowledge, the proposed model is the ﬁrst end-to-end lightweight one that can perform real-time action recognition on resource-limited devices without sacriﬁces in recognition accuracy.
Related Work
Conventional Video Action Recognition Most recent action recognition models [19,32,6,37,10,40,8,9,2,29,38,39,42,46,51,7,41] are based on large CNNs [22].
One of the early models is the two-stream network [32], which is proposed to utilize two CNNs to model raw video frames and optical ﬂow, respectively.
Various improved versions [32,40,10,8,9], such as the Temporal Segment Network (TSN) [40], are designed to capture the long-range temporal structure, but they still rely on the optical ﬂow stream, which is expensive to compute.
C3D
[37] is proposed to model the temporal structure with 3D CNNs.
It avoids using optical ﬂow as input.
However, it is still much larger than a 2D CNN due to the 3D convolution operations.
I3D [2] integrates 3D convolution into a two-stream network and beneﬁts from 2D CNN pre-trained by inﬂating 2D CNN into a 3D one.
One of the key limitations of these models is that they are too heavy for eﬃcient large-scale video analysis.
This is particularly true when most videos are stored in compression formats (e.g., MPEG-4 [31] & H.264 [44]), and thus need to be decoded ﬁrst for these models to run.
The recent eﬀorts on lightweight action recognition model design [3,51,25] only partially address the problem, but still cannot work with compressed videos directly.
In contrast, our model is lightweight and also addresses the compressed video action recognition problem.
Compressed Video Action Recognition Due to the limitation of storage space and transfer speed, videos are generally stored and transmitted in a compressed data format.
The compression standards, including MPEG-4 [31],
H.264/AVC [44], and HEVC [35] (listed in chronological order), commonly use the motion compensation technique that reduces the video data size based on motion estimation from adjacent frames.
There are several approaches that leverage useful information from compressed videos for the action recognition task.
[18] developed highly eﬃcient video features using motion information based on handcrafted features.
DTMC-CNN [49,50] distills the knowledge from optical ﬂows to motion vectors, but the raw video frames are still used for action recognition.
CoViAR [45] takes only compressed videos as input, but the whole training process is not end-to-end since multiple modalities are handled separately.
Recently, DMC-Net [30] improves CoViAR [45] and achieves state-of-the-art results by adding an optical ﬂow generation network, but both models [45,30] employ the large ResNet-152 [13] as the CNN backbone, which has too many parameters with very high computational cost.
Note that both DMC-Net and CoViAR ignore B-frames when extracting motion vectors from compressed videos, while
B-frames cover more than 60 percent of total frames in videos.
They need to make transformation into old codecs and thus are not compatible with most recent video codecs.
Compared to these models, our proposed model is much more lightweight yet more eﬀective in terms of recognition accuracy.
It is also more generally applicable by correctly exploiting both B-frames and P-frames.
Note that DTMV-CNN [50] also uses B-frames, but in an incorrect way by treating
B-frames as P-frames and only considers forward reference.
Further, it needs access to both raw and compressed videos to compute motion vectors.
In contrast, using compressed videos alone, our ATTP is clearly superior (see Fig. 1).
P I
P
P P P
P
P P
P P
P P
I
B I B B
P
B B
P
B B
P
B B
I
Group of Pictures (GOP) (a) (b) ●●● ●●● ●●● ●●● ●●● Multi-Modal Pooling Pooling methods are required in two-stream networks [32,40] as well as other feature fusion models. [40] utilizes average pooling.
[27] proposed bilinear pooling to model local parts of an object: two feature representations are learned separately and then multiplied using the outer product to obtain the holistic representation.
[43] combines the two-stream network with a compact bilinear representation [12].
However, most of existing pooling models can combine only two features, while in compressed videos, more than two modalities exist.
To address such a challenging problem, we propose a novel Aligned
Temporal Trilinear Pooling module to exploit the complementary information contained in the three modalities extracted from compressed videos.
Lightweight Neural Networks Recently, lightweight neural networks including SqeezeNet [16], Xception [4], ShuﬄeNet [28], MobileNet [15] and EﬃcientNet [36] have been proposed, with the number of parameters and computational cost being reduced signiﬁcantly.
Since we focus on lightweight action recognition, they all can be used as the backbone network.
In this work, EﬃcientNet [36] is selected.
Note that a lightweight backbone typically leads to performance degradation in action recognition, and hence the temporal fusion and knowledge distillation model designs are introduced in our ATTP model.
Methodology 3. Modeling Compressed Representation
Modern compression codecs use motion compensation to convert successive raw video frames into several groups of pictures (GOPs), where each GOP contains one I-frame (Intra-coded frame), one or more P-frames (Predicted frames), and one or more B-frames (Bi-directional predicted frames).
From Fig.
2 (a), we can see an example GOP frame pattern used by modern compression codecs.
I-frame ( I ) is the ﬁrst frame in the GOP, which is compressed with image compression codecs (e.g., JPEG).
P-frames hold the changes in the images w.r.t.
the preceding frame and thus save the storage space, while B-frames save even more space by using diﬀerences between the current frame and both the preceding and following frames to represent their content.
Consequently, images from B-frames and P-frames are stored in a compressed format and are reconstructed using these encoded oﬀsets, namely motion vectors ( MV ) and residuals ( R ).
Let x , y , and z denote the coordinates in the three dimensions.
The relation between video frames, motion vectors, and residuals can be formalized as:
[Equation]
where g F ∈ R h × w × 3 denotes the reconstructed image of the F-frame (F = I, P, or B) and f F ∈ R h × w × 3 denotes its counterpart in raw image, with h and w being respectively the height and width.
Moreover, mv P (or mv B ) ∈ R h × w ×describes the predicted block-level motion trajectories from the reference frame to the current P-frame (or B-frame).
For each position ( x, y ) in mv , mv ( x, y, 0) and mv ( x, y, 1) depict the horizontal and vertical movements.
In addition, r P (or r B ) ∈ R h × w × 3 is the RGB-like image describing the residual error between the original P-frame (or B-frame) and its predicted frame.
f P ref and f B ref are frames referenced by P-frames and B-frames.
f P ref denotes the previous I or P-frame of the current P-frame, and its motion vector contains only backward reference.
Note that B-frame has both forward and backward motion information, and diﬀerent codecs choose diﬀerent f P ref .
Since both MV and R are available in compressed videos, we can readily extract them.
However, since B-frames require a higher computational cost to compress than P-frames, earlier codecs such as MPEG-4 [31] use only P-frames for low-cost applications by default, where P-frames substitute all B-frames since they are capable of getting the highest computational eﬃciency (see Fig.
2 (b)).
Because of the complicated modeling of B-frames, existing compressed video action recognition works [45,30] extract motion vectors and residual representations from only P-frames and thus are only applicable to the outdated MPEG-4 standard, and Zhang et al.
[49,50] simply treat B-frames as P-frames.
All these methods only consider forward reference, and at least half of the temporal information is thus ignored.
When applied to the modern video codecs such as H.264 [44] and
HEVC [35], these methods cannot take advantage of both forward and backward motion information induced by B-frames, which thus limits their performance.
Note that CoViAR [45] and DMC-Net [30] even need to re-encode the compressed videos using the MPEG-4 codec for avoiding handling B-frames, which leads to extra cost on both processing time and storage.
3. Lightweight Video Action Recognition with Multiple Modalities
We now formally deﬁne the lightweight video action recognition problem as follows.
Given the three modalities (i.e., I , MV , and R ) extracted from a
(b)
TP (a)
BrushingHair
TTP
LiteNet
I
I t+Δt I-Frame t+Δt
I t LiteNet I I-Frame t
LiteNet MV
MV t Motion Vector t
LiteNet R
R t Residual t
LiteNet OF
OF t Optical Flow t
Feature Alignment
TP
TP
F C S o ft m ax
Norm Layers
FC FC
FC compressed video, our goal is to perform high-speed action recognition using a model with fewer parameters. Recent works utilize diﬀerent CNNs to process the modalities independently. For example, in [45], ResNet-152 [13] is used to process I , which is eﬀective but has extremely high computational cost and require large storage space (see Table 1). In this paper, we adopt a lightweight network LiteNet instead: LiteNet I , LiteNet MV , and LiteNet R take I , MV , and
R as inputs, respectively.
Each of them has the same EﬃcientNet [36] network architecture.
Note that the three modalities are only fused at the test phase in [45], and by doing so, the interactions between diﬀerent modalities are not fully explored during training for action recognition.
More eﬀective multi-modal fusion is thus needed.
To that end, we propose our Aligned Temporal Trilinear
Pooling (ATTP) framework (see Fig. 3(a)), which is detailed next.
3. Trilinear Pooling
We propose a novel trilinear pooling module to model three-factor variations together.
This is motivated by bilinear pooling models [27,47], which are initially proposed to model two-factor variations, such as “style” and “content”.
For lightweight video action recognition, we generalize the bilinear pooling method to fuse the three modalities extracted from compressed videos.
Speciﬁcally, a feature vector is denoted as x ∈ R c , where c is the dimensions of the feature x .
The bilinear combination of two feature vectors with the same dimension x ∈ R c and y ∈ R c is deﬁned as xy T ∈ R c × c [27].
In general, given two representation matrices X =
[ x 1 ; x 2 ; · · · ; x K ] ∈ R c × K and Y =
[ y 1 ; y 2 ; · · · ; y K ] ∈ R c × K for two frames, a pooling layer takes the following bilinear combination:
[Equation]
It can be seen clearly that the above bilinear pooling allows the outputs ( X and Y ) of the feature extractor to be conditioned on each other by considering all their pairwise interactions in the form of a quadratic kernel expansion.
However, this results in very high-dimensional features with a large number of parameters involved.
To address this problem, Multi-modal Factorized Bilinear (MFB) [47] introduces an eﬃcient attention mechanism into the original bilinear pooling based on the Hadamard product.
Let D be the number of projection matrices.
The MFB model w.r.t.
projection i ( i = 1 , ..., D ) is deﬁned as:
[Equation]
where U i ∈ R c × d
and V i ∈
R c × d are projection matrices, (cid:12) is the Hadamard product, 1 ∈ R d is an all-one vector, and d denotes the dimension of these factorized matrices.
Therefore, we only need to learn U =
[ U 1 ; U 2 ; · · · ; U D ] ∈ R c
×
d × D and V =
[ V 1 ; V 2 ; · · · ; V D ] ∈ R c
×
d × D .
Inspired by MFB, we propose a novel trilinear pooling method, which aims to fuse three feature vectors ( x , y , and z ).
Unlike bilinear pooling that can combine only two feature vectors, our Trilinear Pooling method fuse x , y and z using the
Hadamard product:
[Equation]
where W is also a projection matrix W =
[ W 1 ; W [Equation] [Equation] W D ] ∈ R
[Equation]
×
[Equation] × [Equation] , and f TP denotes the output of trilinear pooling.
Note that our trilinear pooling becomes
MFB if all elements in W and z are ﬁxed as 1.
When the inputs are generalized to feature maps (i.e., X =
[ x i ] , Y =
[ y i ] , Z =
[ z i ] ∈ R c × K ), every position of these feature maps makes up one group of inputs, and the outputs of them are summed element-wised as follows:
[Equation]
We thus utilize trilinear pooling to obtain the multi-modal representation of the t -th GOP by fusing I-frame I t , motion vector MV t and residual R t (see Fig. 3(b)):
[Equation]
where I t , MV t and R t are the output feature maps from LiteNet I , LiteNet MV and LiteNet R , respectively.
We set LiteNet as EﬃcientNet.
For each GOP t , the I-frame is selected as I t , while one MV t and one R t are randomly selected.
As in [21], the trilinear vector is then processed with a signed square root step ( f ← sign( f ) (cid:112) | f | ), followed by l 2 normalization ( f ← f/
|| f || ).
3. Temporal Trilinear Pooling
Motion vector is initially introduced to represent the temporal structure as the optical ﬂow does.
However, compared to the high-resolution optical ﬂow, motion vector is much coarser:
Since it only describes the movement on macroblocklevel (e.g., 16*16 pixels), all values within the same macroblock are identical.
Although we have proposed to use trilinear pooling to address this drawback, the temporal information still needs to be explicitly explored.
We note that, because residuals represent the diﬀerence between frames, they are strongly correlated with motion vectors.
Therefore, we propose to model the motion vectors and the residuals jointly.
Note that the fusion of I t , MV t and R t within only one GOP is not enough to capture the temporal information.
We thus further choose to include the adjacent GOP’s information.
Speciﬁcally, in addition to calculating f TP (I t , MV t , R t ) by trilinear pooling, we also combine MV t and R t with I t + ∆t (i.e., the I-frame in the adjacent GOP).
The output of temporal trilinear pooling (TTP) is deﬁned as (see Fig. 3(a)):
[Equation]
where f TP ( t, ∆t ) denotes f TP (I t + ∆t , MV t , R t ) for notation simplicity.
In this paper, we sample the oﬀset ∆t from {− 1 , 1 } during the training stage.
During the test stage, ∆t is ﬁxed as 1 for the ﬁrst GOP and − 1 for other GOPs.
This temporal fusion method solves the temporal representation drawback without introducing extra parameters, which is eﬃcient and lightweight.
The TTP representation is further put into a fully connected layer to calculate the classiﬁcation scores s ( t ) =
P T f TTP ( t ), where
P ∈ R D
×
C is learnable parameters and
C is the number of categories.
3. Feature Alignment
Since motion vectors can only be regarded as a blurred version of optical ﬂow, we choose to boost them with optical ﬂow by feature alignment based knowledge distillation.
Speciﬁcally, we employ another lightweight network LiteNet OF which takes optical ﬂow information as input, and align the features generated by LiteNet MV to those by LiteNet OF , as illustrated in Fig. 3(a).
Diﬀerent from the original method [14] that transfers knowledge from complex models to simple models, our feature alignment sets LiteNet MV and LiteNet OF to be the same lightweight network.
Notably, we ﬁnd that our feature alignment performs better than conventional knowledge distillation based on classiﬁcation probability alignment.
Moreover, our feature alignment is also much more eﬃcient in the training phase since we do not need a large CNN as a teacher.
Our feature alignment module tries to minimize the diﬀerence between features generated by LiteNet MV and those by LiteNet OF .
Note that Zhang et al.
[49,50] utilized all layers for knowledge distillation from the teacher network to the student network, while we only exploit features before the fully-connected (FC) layer and those after the FC layer for feature alignment (which is more eﬃcient and thus more suitable for lightweight action recognition).
Let f (3 d ) MV (or f (3 d ) OF ) be features before the FC layer of LiteNet MV (or LiteNet OF ) and f ( fc ) MV (or f ( fc ) OF ) as features after the FC layer of LiteNet MV (or LiteNet OF ).
The loss for feature alignment is given by:
[Equation]
Given that video action recognition is essentially a multi-class classiﬁcation problem, we utilize the standard cross-entropy loss for training the TTP module:
[Equation]
where s gt ( t ) is the predicted score for t -th GOP with respect to its ground-truth class label.
The total loss of our Aligned Temporal Trilinear Pooling (ATTP) model is deﬁned as follows:
[Equation]
where λ is the weight parameter (we empirically set λ
= 1 in this work).
Experiments 4. Datasets and Settings
In this paper, the main results are reported on two widely-used benchmark datasets, namely HMDB-51 [23] and UCF-101 [33], as in [40,45,30] for direct comparison.
HMDB-51 contains 6,766 videos from 51 action categories, while
UCF-101 contains 13,320 videos from 101 action categories.
Both benchmark datasets have three oﬃcially given training/test splits.
In HMDB-51, each training/test split consists of 3,570 training clips and 1,530 testing clips.
In UCF-101, each training/test split consists of approximately 9,600 clips in the training split and 3,700 clips in the test split.
Since each video in these two datasets is a short clip belonging to a single category, we employ top-1 accuracy on video-level class predictions (for the test split of each dataset) as the evaluation metric.
As in [40,45], we resize frames in all videos to 340 × 256.
In order to implement our proposed model on resource-limited devices, we choose the EﬃcientNet
B1 [36] pre-trained on ImageNet as the core CNN module to extract the representations of I-frames, motion vectors, and residuals.
All the parameters of the projection layers are randomly initialized.
4. Eﬃciency Test Results
We ﬁrstly demonstrate the per-frame running time and FPS of our model in both limited-resource and suﬃcient-resource environments.
We make comparisons to the state-of-the-art CoViAR [45] and DMC-Net [30] since they also exploit compressed videos for action recognition.
Note that CoViAR does not use optical ﬂow, while DMC-Net uses optical ﬂow but only during the training phase like our model.
However, both CoViAR and DMC-Net utilize ResNetfor I-frames and two ResNet-18 for motion vectors and residuals independently, which means their models are much larger in size and slower to run.
We compare the eﬃciency of the three models (CoViAR, DMC-Net, and our
ATTP model) under precisely the same test setting.
On the Dell R730 platform, the preprocessing phase (including loading networks) is mainly run on two Intel
Xeon Silver 4110 CPUs, and the CNN forwarding phase (including extracting
[Table]
motion vectors and residuals) is mainly run on one TITAN
Xp GPU.
As shown in
Table 1, our ATTP model runs faster among three models in both preprocessing and CNN phases on the Dell R730 platform.
The preprocessing time on contrast is particularly stark.
This is due to that both CoViAR and DMC-Net have three large networks with lots of parameters, resulting in massive cost on loading the networks.
For the eﬃciency test on the resource-limited device, the experiments are conducted on the Nvidia Jetson TX2 platform.
The preprocessing phase runs on Dual-core Denver 2 64-bit CPU, and the CNN forward phase runs on the GPU.
The results in Table 1 demonstrate that CoViAR and DMC-Net are too large to be employable on this device [Table] while our ATTP framework ﬁts well in the embedded environment and runs very fast.
These results thus show clearly that our ATTP model outperforms the other two models on eﬃciency and is the only one that is suitable for embedded AI devices.
4. Comparative Results on Accuracy
Now we make a comprehensive comparison between our ATTP method and other state-of-the-art action recognition methods.
To this end, we compute not only the top-1 accuracy but also the eﬃciency (i.e., parameters and GFLOPs) as the evaluation metrics for action recognition.
In our experiments, the compared methods can be divided into two groups: 1) Raw-Video Based Methods :
LRCN [6] and Composite LSTM [34] utilize RNNs to process the optical ﬂow information, while Two-stream [32] adopts two-way CNNs to process the optical ﬂow information.
ResNet-152 [13], C3D [37], I3D (RGB-only) [2], P3D [29] and TSN (RGB-only) [40] employ large CNN models over RGB frames without using other information.
ECO [51] and TSM [25] make eﬀorts on lightweight video analysis models, but these models do not work on compressed videos. 2)
Compressed-Video Based Methods : DTMV-CNN [50] integrates both compressed videos and raw ones into a single two-stream network, while CoViAR [45] and DMC-Net [30] are among the most closely related models (w.r.t.
our ATTP model) that only exploit compressed videos for action recognition.
In particular, DMC-Net, DTMV-CNN, and our ATTP model use optical ﬂow during the training phase (but not during the test phase).
[Table]
The comparative results are shown in Table 2.
We have the following observations: (1) Our ATTP model is the most eﬃcient for video action recognition.
Speciﬁcally, it contains only 23.4 × 10 6 parameters and has only average 3.
GFLOPs over all frames.
(2) Among all compressed video action recognition methods (including CoViAR and DMC-Net), our model performs the best w.r.t.
both eﬃciency and accuracy.
(3) As compared to the strongest baseline DMCNet, although our model only achieves marginal improvement on accuracy, this is obtained with much fewer parameters and only one-tenth of GFLOPS.
This is mainly due to its ability to more eﬀectively fuse multiple modalities using the proposed trilinear pooling module.
(4) As compared to CoViAR, our model saves nearly 70% of the storage with a clear advantage on accuracy.
To better understand the reason why our lighter model can yield more accurate recognition, in our ablation study (to be presented next), it is noted that “I+MV+R” in Table 3 is essentially CoViAR by using EﬃcientNet as the core CNN module.
Under such a fair comparison setting (same CNN backbone), our model consistently yields accuracy improvements over CoViAR on all dataset splits.
(5) Our model achieves even higher (or comparable) accuracies w.r.t.
recent raw-video based models (TSM uses the external Kinetics [20] for pre-training), showing its potential for directly analyzing compressed video for action recognition.
4. Ablation Study
Ablation Study over Diﬀerent Modalities We conduct experiments to show the beneﬁts of using our ATTP model compared with single modality and other fusion options.
Speciﬁcally, we uniformly use three EﬃcientNet networks to process the three components ( I , MV , and R ) extracted from compressed videos
[Table]
and demonstrate all the ablative results on the two benchmarks by training with diﬀerent parts of our ATTP model.
For single-modality based models, “I”, “MV” and “R” denote the results obtained by using LiteNet I , LiteNet MV and
LiteNet R , respectively.
For the late-fusion based model, “I+MV+R” indicates that the output is fused by simply adding the score of the three CNNs together.
We also compare our ATTP model with existing bilinear pooling models, which are the simpliﬁed version of our Trilinear Pooling.
Since existing bilinear pooling methods cannot be directly adapted to the three modalities, we readily apply pairwise combination over them, and sum the three predicted classiﬁcation scores together like “I+MV+R”.
Note that conventional bilinear pooling [27] and factorized bilinear pooling [21] have too many parameters to be eﬃcient, we resort to compact bilinear pooling [12] (denoted as “BP”) with much fewer parameters.
Finally, “TP” denotes our Trilinear Pooling, “TTP” denotes our Temporal
Trilinear Pooling, and “ATTP” denotes our Aligned Temporal Trilinear Pooling.
As shown in Table 3, single-modality based models (i.e., I, MV, or R) could not achieve good results without using multi-modal information, indicating that the compressed video needs to be fully explored to obtain high accuracy.
I and
R yield similar results because they both contain the RGB data: the I-frames contain a small number of informative frames, while the residuals contain a large number of less informative frames.
Since the motion vectors only contain the motion information, MV could not perform as well as the other two.
Moreover, for multi-modal fusion, all bilinear/trilinear pooling methods outperform I+MV+R, showing the power of pooling methods instead of linearly late fusion.
Particularly, our TP method yields %1 gains over BP, validating the eﬀectiveness of our pooling method.
In addition, the improvements achieved by TTP over TP (and those achieved by ATTP over TTP) clearly show the importance of boosting motion vectors by inducing temporal context (and feature alignment).
Ablation Study over Diﬀerent Types of Frames To show the contribution of each type of frames to action recognition, we make comparison among four versions of the late fusion model: (1) I – only I-frames are used to obtain the single modality I ; (2) I+P – both I-frames and P-frames are used to extract the three modalities (i.e., I , MV , and R ); (3) I+B – both I-frames and B-frames are used to extract the three modalities; (4) I+P+B – all three types of frames
[Figure]
are used to extract the three modalities.
The ablative results in Fig.
4 (left) show that: (a) The performance of action recognition continuously increases when more types of frames are added, validating the contribution of each type of frames.
(b) The improvements achieved by I+B over I+P verify that B-frames are more important than P-frames for action recognition.
Ablation Study for Feature Alignment To conduct the ablation study for our proposed feature alignment method, we make comparisons among four related methods: (1) W/O Feature Alignment – features are directly extracted from motion vectors, without feature alignment; (2) Feature Alignment after FC
Layer – our proposed feature alignment method is used, but only the features after the FC layer are aligned; (3) Knowledge Distillation – the knowledge distillation [14] method is used for feature alignment; (4) Our Feature Alignment – our proposed feature alignment method deﬁned in Eq.
The ablative results are presented in Fig.
4 (right).
It can be seen that: (1) The three feature alignment methods outperform the ‘W/O Feature Alignment’ method, validating the eﬀectiveness of feature alignment.
Our proposed feature alignment method performs the best among the three feature alignment methods.
This suggests that aligning the features before and after the FC layer is more eﬀective than feature alignment after the FC Layer and even than knowledge distillation.
Conclusion
In this work, we address a key limitation of existing deep neural networksbased video action recognition methods.
That is, they either only work with raw RGB video frames instead of the compressed video directly, or heavy in model size and slow to run.
We therefore proposed to address a more challenging task, namely lightweight compressed video action recognition.
By employing
EﬃcientNet as the backbone, we proposed a novel ATTP model to fuse the multiple modalities for lightweight video action recognition.
Importantly, for the ﬁrst time, our ATTP models the B-frames correctly, therefore being compatible with a wider range of contemporary codecs.
The comparative results on three benchmark datasets show that our ATTP model outperforms the state-of-the-art alternatives in both eﬃciency and accuracy.