VA-RED 2 : V IDEO A DAPTIVE R EDUNDANCY R EDUCTION
A BSTRACT
Performing inference on deep learning models for videos remains a challenge due to the large amount of computational resources required to achieve robust recognition.
An inherent property of real-world videos is the high correlation of information across frames which can translate into redundancy in either temporal or spatial feature maps of the models, or both.
The type of redundant features depends on the dynamics and type of events in the video: static videos have more temporal redundancy while videos focusing on objects tend to have more channel redundancy.
Here we present a redundancy reduction framework, termed VA-RED 2 , which is input-dependent .
Speciﬁcally, our VA-RED 2 framework uses an input-dependent policy to decide how many features need to be computed for temporal and channel dimensions.
To keep the capacity of the original model, after fully computing the necessary features, we reconstruct the remaining redundant features from those using cheap linear operations.
We learn the adaptive policy jointly with the network weights in a differentiable way with a shared-weight mechanism, making it highly efﬁcient.
Extensive experiments on multiple video datasets and different visual tasks show that our framework achieves 20% − 40% reduction in computation (FLOPs) when compared to state-of-the-art methods without any performance loss.
Project page: http://people.csail.mit.edu/bpan/va-red/ .
I NTRODUCTION
Large computationally expensive models based on 2D/3D convolutional neural networks (CNNs) are widely used in video understanding (Tran et al., 2015; Carreira & Zisserman, 2017; Tran et al., 2018).
Thus, increasing computational efﬁciency is highly sought after (Feichtenhofer, 2020; Zhou et al., 2018c; Zolfaghari et al., 2018).
However, most of these efﬁcient approaches focus on architectural changes in order to maximize network capacity while maintaining a compact model (Zolfaghari et al., 2018; Feichtenhofer, 2020) or improving the way that the network consumes temporal information (Feichtenhofer et al., 2018; Korbar et al., 2019).
Despite promising results, it is well known that CNNs perform unnecessary computations at some levels of the network (Han et al., 2015a;
Howard et al., 2017; Sandler et al., 2018; Feichtenhofer, 2020; Pan et al., 2018), especially for video models since the high appearance similarity between consecutive frames results in a large amount of redundancy.
In this paper, we aim at dynamically reducing the internal computations of popular video CNN architectures.
Our motivation comes from the existence of highly similar feature maps across both time and channel dimensions in video models.
Furthermore, this internal redundancy varies depending on the input: for instance, static videos will have more temporal redundancy whereas videos depicting a single large object moving tend to produce a higher number of redundant feature maps.
To reduce the varied redundancy across channel and temporal dimensions, we introduce an input-dependent redundancy reduction framework called VA-RED 2 (Video Adaptive REDundancy REDuction) for efﬁcient video recognition (see Figure 1 for an illustrative example).
Our method is model-agnostic and hence can be applied to any state-of-the-art video recognition networks.
The key mechanism that VA-RED 2 uses to increase efﬁciency is to replace full computations of some redundant feature maps with cheap reconstruction operations.
Speciﬁcally, our framework avoids computing all the feature maps.
Instead, we choose to only calculate those non-redundant part of feature maps and reconstruct the rest using cheap linear operations from the non-redundant
Output layer “rafting” …… …… …… ……
Stem
L L
L N f f f f “adult+ male+ speaking”
Output layer f f f f
Stem …… …… …… ……
L
L N γ 1 C 1 out (1− γ 1 ) C 1 out γ N C Nout (1− γ N )
C Nout temporal
[Figure]
channel
[Figure]
temporal
[Figure]
convolution
[Figure] [Figure]
adaptive temporal
[Figure]
reduction Example#
[Figure]
: adaptive channel redundancy reduction
[Figure]
features maps.
In addition, VA-RED 2 makes decisions on a per-input basis: our framework learns an input-dependent policy that deﬁnes a ”full computation ratio” for each layer of a 2D/3D network.
This ratio determines the amount of features that will be fully computed at that layer, versus the features that will be reconstructed from the non-redundant feature maps.
Importantly, we apply this strategy on both time and channel dimensions.
We show that for both traditional video models such as I3D (Carreira & Zisserman, 2017), R(2+1)D
(Tran et al., 2018), and more advanced models such as X3D (Feichtenhofer, 2020), this method signiﬁcantly reduces the total ﬂoating point operations (FLOPs) on common video datasets without accuracy degradation.
The main contributions of our work includes: (1) A novel input-dependent adaptive framework for efﬁcient video recognition, VA-RED 2 , that automatically decides what feature maps to compute per input instance.
Our approach is in contrast to most current video processing networks, where feature redundancy across both time and channel dimensions is not directly mitigated.
(2) An adaptive policy jointly learned with the network weights in a fully differentiable way with a sharedweight mechanism, that allows us to make decisions on how many feature maps to compute.
Our approach is model-agnostic and can be applied to any backbones to reduce feature redundancy in both time and channel domains.
(3) Striking results of VA-RED 2 over baselines , with a 30% reduction in computation in comparison to R(2+1)D (Tran et al., 2018), a 40% over I3D-InceptionV2 (Carreira & Zisserman, 2017), and about 20% over the recently proposed X3D-M (Feichtenhofer, 2020) without any performance loss, for video action recognition task.
The superiority of our approach is extensively tested on three video recognition datasets (Mini-Kinetics-200, Kinetics-400 (Carreira & Zisserman, 2017), and Moments-In-Time (Monfort et al., 2019)) and one spatio-temporal action localization dataset (J-HMDB-21 (Jhuang et al., 2013)).
(4) A generalization of our framework to video action recognition, spatio-temporal localization, and semantic segmentation tasks, achieving promising results while offering signiﬁcant reduction in computation over competing methods.
R ELATED W ORK
Efﬁciency in Video Understanding Models.
Video understanding has made signiﬁcant progress in recent years, mainly due to the adoption of convolutional neural networks, in form of 2D CNNs (Karpathy et al., 2014; Simonyan & Zisserman, 2014; Ch´eron et al., 2015; Feichtenhofer et al., 2017;
Gkioxari & Malik, 2015; Wang et al., 2016; Zhou et al., 2018a; Lin et al., 2019; Fan et al., 2019) or 3D CNNs (Tran et al., 2015; Carreira & Zisserman, 2017; Hara et al., 2018; Tran et al., 2018).
Despite promising results on common benchmarks, there is a signiﬁcant interest in developing more efﬁcient techniques and smaller models with reasonable performance.
Previous works have shown reductions in computational complexity by using hybrid 2D-3D architectures (Xie et al., 2018;
Zhou et al., 2018c; Zolfaghari et al., 2018), group convolution (Tran et al., 2019) or selecting salient clips (Korbar et al., 2019).
Feichtenhofer et al.
, (Feichtenhofer et al., 2018) propose a dedicated low-framerate pathway.
Expansion of 2D architectures through a stepwise expansion approach over the key variables such as temporal duration, frame rate, spatial resolution, network width, is recently proposed in (
Feichtenhofer
Diba et al. Diba et al.
, 2019) learn motion dynamic of videos with a self-supervised task for video understanding.
Fan et al.
(Fan et al., 2020) incorporate a efﬁcient learnable 3D-shift module into a 3D video network.
Wang et al.
(Wang et al., 2020) devise a correlation module to learn correlation along temporal dimension.
Li et al.
(Li et al., 2020) encode the clip-level ordered temporal information with a CIDC network.
While these approaches bring considerable efﬁciency improvements, none of them dynamically calibrates the required feature map computations on a per-input basis.
Our framework achieves substantial improvements in average efﬁciency by avoiding redundant feature map computation depending on the input.
Adaptive Inference.
Many adaptive computation methods have been recently proposed with the goal of improving efﬁciency (Bengio et al., 2015; 2013; Veit & Belongie, 2018;
Wang et al., 2018; Graves, 2016; Meng et al., 2021).
Several works have been proposed that add decision branches to different layers of CNNs to learn whether to exit the network for faster inference (Yu et al., 2018;
Figurnov et al., 2017; McGill & Perona, 2017; Teerapittayanon et al., 2016)
Wang et al.
(Wang et al., 2018) propose to skip convolutional blocks on a per input basis using reinforcement learning and supervised pre-training.
Veit et al.
(Veit & Belongie, 2018) propose a block skipping method controlled by samples from a Gumbel softmax, while Wu et al.
(Wu et al., 2018) develop a reinforcement learning approach to achieve this goal.
Adaptive computation time for recurrent neural networks is also presented in (Graves, 2016).
SpotTune (Guo et al., 2019) learns to adaptively route information through ﬁnetuned or pre-trained layers.
A few works have been recently proposed for selecting salient frames conditioned on the input (Yeung et al., 2016; Wu et al., 2019; Korbar et al., 2019; Gao et al., 2019) while recognizing actions in long untrimmed videos.
Different from adaptive data sampling (Yeung et al., 2016; Wu et al., 2019; Korbar et al., 2019; Gao et al., 2019), in this paper, our goal is to remove feature map redundancy by deciding how many features need to be computed for temporal and channel dimensions per input basis, for efﬁcient video recognition.
AR-Net (Meng et al., 2020) recently learns to adaptively choose the resolution of input frames with several individual backbone networks for video inference.
In contrast, our method focuses on reducing the redundancy in both temporal and channel dimension, and is applicable to both 3D and 2D models, while AR-Net is only for 2D model and it is focused on spatial resolution.
Moreover, our method integrates all the inference routes into a single model which is in the almost same size to the original base model.
Thus our model is signiﬁcantly smaller than AR-Net in terms number of model parameters.
Neural Architecture Search.
Our network learns the best internal redundancy reduction scheme, which is similar to previous work on automatically searching architectures (Elsken et al., 2018).
Liu et al.
(Liu et al., 2018) formulate the architecture search task in a differentiable manner;
Cai et al.
(Cai et al., 2018) directly learn architectures for a target task and hardware, Tan et al.
(Tan & Le, 2019) design a compound scaling strategy that searches through several key dimensions for
CNNs (depth, width, resolution).
Finally, Tan et al.
(Tan et al., 2019) incorporate latency to ﬁnd efﬁcient networks adapted for mobile use.
In contrast, our approach learns a policy that chooses over full or reduced convolutions at inference time, effectively switching between various discovered subnetworks to minimize redundant computations and deliver high accuracy.
V IDEO A DAPTIVE R EDUNDANCY R EDUCTION
Our main goal is to automatically decide which feature maps to compute for each input video in order to classify it correctly with the minimum computation.
The intuition behind our proposed method is that there are many similar feature maps along the temporal and channel dimensions.
For each video instance, we estimate the ratio of feature maps that need to be fully computed along the temporal dimension and channel dimension.
Then, for the other feature maps, we reconstruct them from those pre-computed feature maps using cheap linear operations.
Approach Overview.
Without loss of generality, we start from a 3D convolutional network G , and denote its l th 3D convolution layer as f l , and the corresponding input and output feature maps as X l and Y l respectively.
For each 3D convolution layer, we use a very lightweight policy layer p l denoted as soft modulation gate to decide the ratio of feature maps along the temporal and channel dimensions which need to be computed.
As shown in Figure 2, for temporal-wise dynamic inference, we reduce the computation of 3D convolution layer by dynamically scaling the temporal stride of the 3D ﬁlter with a factor R =
p l ( X l )[0] .
Thus the shape of output Y (cid:48) l becomes C out × T o /R
× H o × W o .
To keep the same output shape, we reconstruct the remaining features based on Y (cid:48) l as
[Equation]
t avg.
pool 0 1 2 Temporal policy:1 2 3 4 5 6 7 8 9 2' 3' 4' 6' 7' 8' 10'11'12'
Conv.
stride: S × R S :
R : 2Φ t Φ t Φ t
C in × T × H × W
C in T ×1× c avg.
pool
C in × T × H × W
C in T ×1× 0 1 2 Channel policy:rC out × T × H × W
Spatial Conv.
Φ c Identity
[Figure]
C
[Figure]
Output features r :
[Figure]
where Y l [ j + iR ] represents the ( j + iR ) th feature map of Y l along the temporal dimension, Y (cid:48)
l [ i ] denotes the i th feature map of Y (cid:48) l , and Φ ti,j is the cheap linear operation along the temporal dimension.
The total computational cost of this process can be written as:
[Equation]
where the function C ( · ) returns the computation cost for a speciﬁc operation, and f tl represents our dynamic convolution process along temporal dimension.
Different from temporal-wise dynamic inference, we reduce the channel-wise computation by dynamically controlling the number of output channels.
We scale the output channel number with a factor r = ( 12 ) p l ( X l )[1] .
In this case, the shape of output Y (cid:48) l is rC out × T o × H
o × W o .
Same as before, we reconstruct the remaining features via cheap linear operations, which can be formulated as Y l =
[ Y (cid:48) l , Φ c ( Y (cid:48) l )] , where Φ c ( Y (cid:48) l ) ∈ R (1 − r ) C out × T o × H o × W
o
represents the cheaply generated feature maps along the channel dimension, and Y l ∈ R C out × T o × H o × W o is the output of the channel-wise dynamic inference.
The total computation cost of joint temporal-wise and channel-wise dynamic inference is:
[Equation]
where f t,cl is the adjunct process of temporal-wise and channel-wise dynamic inference.
Soft Modulation Gate for Differentiable Optimization.
We adopt an extremely lightweight policy layer p l called soft modulation gate for each convolution layer f l to modulate the ratio of features which need to be computed.
Speciﬁcally, the soft modulation gate takes the input feature maps X l as input and learns two probability vectors V lt ∈ R S t and V lc ∈ R S c , where S t and S c are the temporal search space size and the channel search space size respectively.
The V lt and V lc are learned by:
[Equation]
where F ( · , · ) denotes the fully-connected layer, N is the batch normalization, δ ( · ) represents the tanh( · ) function, G is the global pooling operation whose output shape is C in · T × 1 × 1 , φ ( · ) is the output activation function, here we just use max(tanh( · ) , 0) whose output range is [0 , 1) , and ω p, 1 ∈ R ( S t + S c ) × D h , ω p, 2 ∈ R D h × C in · T are the weights of their corresponding layers, D h is the hidden dimension number.
V lt and V lc will then be used to modulate the ratio of the feature maps to be computed in temporal-wise dynamic convolution and channel-wise dynamic convolution.
During training, we obtain the ﬁnal output of the dynamic convolution by weighted sum of all the feature maps which contains different ratio of fully-computed features as follows:
[Equation]
where f cl ( · , r ) is the channel-wise dynamic convolution with the channel scaling factor r , and f tl ( · , R )
it the temporal-wise dynamic convolution with the temporal stride scaling factor R .
During the inference phase, only the dynamic convolutions whose weights are not zero will be computed.
Shared-weight Training and Inference.
Many works in adaptive computation and neural architecture search suffer from very heavy computational cost and memory usage during training stage due to the large search space.
In our case, under the naive implementation, the training computational cost and parameter size would linearly grow as the search space size increases.
To train our model efﬁciently, we utilize a weight-sharing mechanism to reduce the computational cost and training memory.
To be speciﬁc, we ﬁrst compute all the possible necessary features using a big kernel.
Then, for each dynamic convolution with different scaling factor, we sample its corresponding ratio of necessary features and reconstruct the rest features by cheap operations to get the ﬁnal output.
Though this, we are able to keep the computational cost at a constant value invariant to the search space.
More details on this are included in Section B of the Appendix.
Efﬁciency Loss.
To encourage our network to output a computational efﬁcient subgraph, we introduce the efﬁciency loss L c during the training process, which can be formulated as
[Equation]
where r sl is channel scaling factor of the largest ﬁlter in the series of channel-wise dynamic convolutions, and R sl is stride scaling factor of the largest ﬁlter of temporal-wise dynamic convolutions.
Overall, the loss function of our whole framework can be written as L =
L a + λ e L e , where L a is the accuracy loss of the whole network and λ e is the weight of efﬁciency loss which can be used to balance the importance of the optimization of prediction accuracy and computational cost.
E XPERIMENTS
Datasets.
We conduct our video action recognition experiments on three standard benchmarks:
Mini-Kinetics-200, Kinetics-400, and Moments-In-Time.
Mini-Kinetics-200 (assembled by (Meng et al., 2020)) is a subset of full Kinetics dataset (Carreira & Zisserman, 2017) containing 121k videos for training and 10k videos for testing across 200 action classes.
Moments-In-Time dataset has 802,244 videos in training and 33,900 videos in validation across 339 categories.
To show the generalization ability to different task, we also conduct the video spatio-temporal action localization on J-HMDB-21 (Jhuang et al., 2013).
J-HMDB-21 is a subset of HMDB dataset (Kuehne et al., 2011) which has 928 short videos with 21 action categories.
We report results on the ﬁrst split.
For semantic segmentation experiments, we use ADE20K dataset (Zhou et al., 2017; 2018b), containing 20k images for training and 2k images for validation.
ADE20K is a densely labeled image dataset where objects and object parts are segmented down to pixel level.
We report results on validation set.
Model Architectures.
We evaluate our method on three most widely-used model architectures:
I3D (Carreira & Zisserman, 2017), R(2+1)D
(Tran et al., 2018), and the recent efﬁcient model
X3D (Feichtenhofer, 2020).
We consider I3D-InceptionV2 (denoted as I3D below) and R(2+1)D- (denoted as R(2+1)D below) as our base model.
In our implementation of X3D, we remove all the swish non-linearity (Ramachandran et al., 2017) except those in SE layer (Hu et al., 2018) to save training memory and speed up the inference speed on GPU.
We choose X3D-M (denote as X3D below) as our base model and demonstrate that our method is generally effective across datasets.
Implementation Details.
We train and evaluate our baseline models by mainly following the settings in their original papers (Tran et al., 2018; Xie et al., 2018; Feichtenhofer, 2020).
We train all our base and dynamic models for 120 epochs on mini-Kinetics-200, Kinetics-400, and 60 epochs on
Moments-In-Time dataset.
We use a mini-batch size of 12 clips per GPU and adopt synchronized
SGD with cosine learning rate decaying strategy (Loshchilov & Hutter, 2016) to train all our models.
Dynamic models are ﬁnetuned with efﬁciency loss for 40/20 epochs to reduce density of inference
[Table] [Table]
[Table]
graph while maintaining the accuracy.
During ﬁnetuning
[Table]
we set λ c to 0 .
8 and learning rate to 0 .for R(
+1)D and 0 .
1 for I3D and X3D.
For testing, we adopt K-LeftCenterRight strategy:
K temporal clips are uniformly sampled from the whole video, on which we sample the left, center and right crops along the longer spatial axis, the ﬁnal prediction is obtained by averaging these 3 × K clip predictions.
We set K = 10 on Mini-Kinetics-200 and Kinetics-400 and K = 3 on Moments-In-Time.
More implementation details are included in Section B of Appendix.
For video spatio-temporal action localization, we adopt YOWO architecture in (K ¨ op ¨
ukl ¨ u et al., 2019) and replace 2D branch with 3D backbone to directly compare them.
We freeze the parameters of 3D backbone as suggested in (K ¨ op ¨ ukl ¨ u et al., 2019) due to small number of training video in J-HMDB-21 (Jhuang et al., 2013).
The rest part of the network is optimized by SGD with initial learning rate of 10 − 4 .
Learning rate is reduced with a decaying factor of 0 . 5 at 10k, 20k, 30k and 40k iterations.
For semantic segmentation, we conduct experiments using PSPNet (Zhao et al., 2017), with dilated ResNet-18 (Yu et al., 2017;
He et al., 2016) as our backbone architecture.
As PSPNet is devised for image semantic segmentation, we only apply the channel-wise redundancy reduction to the model and adopt synchronized SGD training for 100k iterations across 4GPUs with 2 images on each GPU.
The learning rate decay follows the cosine learning rate schedule schedule (Loshchilov & Hutter, 2016).
Results on Video Action Recognition.
We ﬁrst evaluate our method by applying it to R(
+1)D
(Tran et al., 2018) with different number of input frames and different size of search space.
Here we use GFLOPs (ﬂoating point operations) to measure the computational cost of the model and report clip-1, video-1 and video-5 metrics to measure the accuracy of our models, where clip-1 is the topaccuracy of model evaluation with only one clip sampled from video, video-1 and video-5 are the top-1 and top-5 accuracy of model evaluated with K-LeftCenterRight strategy.
Note that we report the
FLOPs of a single video clips at the spatial resolution 256 × 256 (for I3D and X3D) or 128 ×(for R(2+1)D).
In addition, we report the speed of each model with the metric of clip/second , which denotes the number of video clips that are processed in one second.
We create the environment with PyTorch 1.6, CUDA 11.0, and a single NVIDIA TITAN RTX (24GB) GPU as our testbed to measure speed of different models.
Table 1 shows the results (In all of the tables, (cid:55) represents the
[Table] [Table]
original ﬁxed model architecture while (cid:51) denote the dynamic model trained using our proposed approach).
Our proposed approach VA-RED 2 signiﬁcantly reduces the computational cost while improving the accuracy.
We observe that dynamic model with the search space size of 2 has the best performance in terms of accuracy, GFLOPS and speed.
We further test our VA-RED 2 with all of the three model architectures: R(2+1)D-18, I3D-InceptionV2, and X3D-M (Table 2) including the very recent temporal pyramid module (Yang et al., 2020) and correlation module (Wang et al., 2020) on Mini-Kinetics-200 dataset.
We choose R(2+1)D-18 with TPN and CorrNet as the backbone architecture and test the performance of our method using a search space of 2 in Table 3 and Table(Left) respectively.
Table 2 shows that method boosts the speed of base I3D-InceptionV2 and R(
+1)D models by 21.7% and 10.6% respectively, showing its advantages not only in terms of GFLOPS but also in actual speed.
Table 4 (Left) shows that our dynamic approach also outperforms the baseline CorrNet by 1.8% in top-1 video accuracy, while reducing the computational cost by 25.2% on Mini-Kinetics-200.
Furthermore, we compare our method with AR-Net (Meng et al., 2020), which is a recent adaptive method that selects optimal input resolutions for video inference.
We conduct our experiments on 16-frame TSN (Wang et al., 2016) with ResNet50 backbone and provide the comparison on FLOPs, parameter size, and accuracy (Table 4 (Right)).
To make a fair comparison, we train AR-Net using the ofﬁcial implementation on the same Mini-Kinetics-200 dataset with Kaiming initialization (He et al., 2015).
Table 4 (Right) shows that our method, VA-RED 2 outperforms AR-Net in both accuracy and GFLOPS, while using about 62% less parameters.
Table 5 and Table 6 show the results of different methods on Kinetics-400 and Moments-In-Time, respectively.
To summarize, we observe that VA-RED 2 consistently improves the performance of all the base models including the recent architectures X3D, TPN, and CorrNet, while offering signiﬁcant reduction in computation.
Moreover, our approach is model-agnostic, which allows this to be served as a plugin operation for a wide range of action recognition architectures.
From the comparison among different models, we ﬁnd that our proposed VA-RED 2 achieves the most computation reduction on I3D-InceptionV2, between 40% and 50% , while reducing less than 20% on X3D-M.
This is because X3D-M is already very efﬁcient both in terms of channel dimension and temporal dimension.
Notice that the frames input to X3D-M are at the temporal stride of 5 , which makes them share less similarity.
Furthermore, we observe that dynamic I3D-InceptionV2 has very little variation of the computation for different input instances.
This could be because of the topology conﬁguration of the InceptionV2, which has lots of parallel structures inside the network architecture.
We also compare VA-RED 2 with a weight-level pruning method (Han et al., 2015b) and a automatic channel pruning method (CGNet) (Hua et al., 2019) on Mini-Kinetics-200.
Table 7 shows that our approach signiﬁcantly outperforms the weight-level pruning method by a margin of about 3%-4% in clip-1 accuracy with similar computation over the original ﬁxed model and consistently outperforms
CGNet while requiring less GFLOPs (maximum 2.8% in 16 frame).
These results well demonstrate the effectiveness of our dynamic video redundancy framework over network pruning methods.
[Table] [Table]
[Table] [Table]
We choose R(2+1)D-18 on Mini-Kinetics-200 and set the search space to 2 in all the dynamic models.
[Table]
Results on Spatio-Temporal Action Localization.
We further extend our method to the spatiotemporal action localization task to demonstrate the generalization ability to different task.
We conduct our method on J-HMDB-21 with two different 3D backbone networks: I3D-InceptionV and X3D-M.
We report frame-mAP at IOU threshold 0 .
5 , recall value at IOU threshold 0 .
5 , and classiﬁcation accuracy of correctly localized detections to measure the performance of the detector.
Table 8 shows that our dynamic approach outperforms the baselines on all three metrics while offering signiﬁcant savings in FLOPs (e.g., more than 50% savings on I3D).
In summary, VA-REDis clearly better than the baseline architectures in terms of both accuracy and computation cost on both recognition and localization tasks, making it suitable for efﬁcient video understanding.
Effect of Efﬁciency Loss.
We conduct an experiment by comparing the model performance before and after being ﬁnetuned with our proposed efﬁciency loss.
Table 9 shows that ﬁnetuning our dynamic model with efﬁciency loss signiﬁcantly reduces the computation without any accuracy loss.
Ablation Experiments on Dynamic Modeling.
We test performance of or approach by turning of dynamic modeling along temporal and channel dimensions on Mini-Kinetics-200.
Table 10 shows that dynamic modeling along both dimensions obtains the best performance while requiring the least computation.
This shows importance of input-dependent policy for deciding how many features need to be computed for both temporal and channel dimensions.
Visualization and Analysis.
To better understand the policy decision process, we dissect the network layers and count the ratio of feature maps that are being computed during each convolution layers for each category.
From Figure 3, we observe that: In X3D, point-wise convolutions which right after the depth-wise convolutions have more variation among classes and network tends to consume more temporal-wise features at the early stage and compute more channel-wise features at the late stage of the architecture.
The channel-wise policy has also more variation than the temporal-wise policy among different categories.
Furthermore, we show few contrasting examples which are in the
[Figure] [Figure]
[Table] [Table]
same category while requiring very different computation in Figure 4.
Video clips which have more complicated scene conﬁguration (e.g. cooking eggs and playing volleyball) and more violent camera motion (e.g. ﬂipping pancake) tend to need more feature maps to do the correct predictions.
More qualitative results can be found in Section E, Section F and Section G of the Appendix.
VA-RED 2 on Dense Visual Tasks.
Our VA-RED 2 framework is also applicable for some dense visual tasks, like semantic segmentation, which requires the pixel-level prediction for the input content.
To prove this, we apply our method to a semantic segmentation model on the ADE-20K dataset (Zhou et al., 2017; 2018b).
We report computational cost of model encoder and the mean IoU (Intersection-Over-Union) in Table 11.
As can be seen from Table 11, our proposed VA-RED 2 has the absolute advantage in terms of efﬁciency while maintaining the precision of segmentation.
This experiment clearly shows that our method is not only effective on the recognition and detection tasks, but also applicable to the dense visual tasks like semantic segmentation.
C ONCLUSION
In this paper, we propose an input-dependent adaptive framework called VA-RED 2 for efﬁcient inference which can be easily plugged into most of existing video understanding models to signiﬁcantly reduce the model computation while maintaining the accuracy.
Extensive experimental results on video action recognition, spatio-temporal localization, and semantic segmentation validate the effectiveness of our framework in multiple standard benchmark datasets.
Acknowledgements.
This work is supported by IARPA via DOI/IBC contract number D17PC00341.
This work is also supported by the MIT-IBM Watson AI Lab.
Disclaimer.
The U
S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon.
The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the ofﬁcial policies or endorsements, either expressed or implied, of IARPA, DOI/IBC, or the U.S. Government
A D ATASET D ETAILS
We evaluate the performance of our approach using three video action recognition datasets, namely
Mini-Kinetics-200 (Meng et al., 2020), Kinetics-400 (Carreira & Zisserman, 2017), and MomentsIn-Time (Monfort et al., 2019) and one spatio-temporal action localization task namely J-HMDB21 (Jhuang et al., 2013).
Kinetics-400 is a large dataset containing 400 action classes and 240K training videos that are collected from YouTube.
The Mini-Kinetics dataset contains 121K videos for training and 10K videos for testing, with each video lasting 6-10 seconds.
The original Kinetics dataset is publicly available to download at https://deepmind.com/research/open-source/ kinetics .
We use the ofﬁcial training/validation/testing splits of Kinetics-400 and the splits released by authors in (Meng et al., 2020) for Mini-Kinetics-200 in our experiments.
Moments-in-time (Monfort et al., 2019) is a recent collection of one million labeled videos, involving actions from people, animals, objects or natural phenomena.
It has 339 classes and each video clip is trimmed to 3 seconds long.
This dataset is designed to have a very large set of both inter-class and intra-class variation that captures a dynamic event at different levels of abstraction (i.e. ”opening” doors, curtains, mouths, even a ﬂower opening its petals).
We use the ofﬁcial splits in our experiments.
The dataset is publicly available to download at http://moments.csail.mit.edu/ .
Joints for the HMDB dataset (J-HMDB-21 (Jhuang et al., 2013)) is based on 928 clips from HMDB comprising 21 action categories.
Each frame has a 2D pose annotation based on a 2D articulated human puppet model that provides scale, pose, segmentation, coarse viewpoint, and dense optical ﬂow for the humans in action.
The 21 categories are brush hair, catch, clap, climb stairs, golf, jump, kick ball, pick, pour, pull-up, push, run, shoot ball, shoot bow, shoot gun, sit, stand, swing baseball, throw, walk, wave.
The dataset is available to download at http://jhmdb.is.tue.mpg.de/ .
B I MPLEMENTATION D ETAILS
Details of Shared-weight Training and Inference.
In this section, we provide more details of the shared-weight mechanism presented in Section 3 of the main paper.
We ﬁrst compute all the possible necessary features using a big kernel and then for each dynamic convolution with different scaling factor, we sample its corresponding ratio of necessary features and reconstruct the rest features by cheap operations to get the ﬁnal output.
For example, the original channel-wise dynamic convolution at ratio r = ( 12 ) ( i − 1) can be analogized to
[Equation]
where [ · : · ] is the index operation along the channel dimension, and i cs is the index of the largest channel-wise ﬁlter, during training phase, we have i cs = 1 , while during inference phase, i cs is the smallest index for V lc , s.t.
V lc
[ i cs ] = 0 .
By utilizing such a share-weight mechanism, the computation of the total channel-wise dynamic convolution is reduce to ( 12 ) i cs − 1 · C ( f l ) .
Further, we have the total computational cost of the adjunct process as
[Equation]
where i ts is the index of largest temporal-wise ﬁlter.
Training and Inference.
We apply our method mainly to 2D convolutions in R(2+1)D since 2D convolution takes the most computational cost compared with 1D convolution.
We train most of our models on 96 NVIDIA Tesla V100-32GB GPUs and perform synchronized BN (Ioffe & Szegedy, 2015) across all the GPUs.
For R(2+1)D
(Tran et al., 2018), the learning rate is initialized as 0 .and
the weight decay is set to be 5 × 10 − 4 .
For I3D (Carreira & Zisserman, 2017; Xie et al., 2018) and X3D (Feichtenhofer, 2020), the learning rates both start from 1 . 8 and weight decay factors are 1 × 10 − 4 and 5 × 10 − 5 respectively.
Cosine learning rate decaying strategy is applied to decrease the total learning rate.
All of the models are trained from scratch and warmed up forepochs on mini-Kinetics/Kinetics, 8 epochs on Moments-In-Time dataset.
We adopt the Nesterov momentum optimizer with an initial weight of 0.01 and a momentum of 0.9.
During training, we follow the data augmentation (location jittering, horizontal ﬂipping, corner cropping, and scale jittering) used in TSN (Wang et al., 2016) to augment the video with different sizes spatially and ﬂip
[Table]
the video horizontally with 50% probability.
We use single-clip, center-crop FLOPs as a basic unit of computational cost.
Inference-time computational cost is roughly proportional to this, if a ﬁxed number of clips and crops is used, as is for our all models.
Note that Kinetics-400 dataset is shrinking in size ( ∼ 15% videos removed from original Kinetics) and the original version used in (Carreira &
Zisserman, 2017) are no longer available from ofﬁcial site, resulting in some difference of results.
C R EDUNDANCY A NALYSIS
To motivate our redundancy reduction approach, we measure and visualize the internal redundancy of well known pretrained networks.
We analyze the internal feature maps of existing pre-trained I3DInceptionV2 and R(2+1)D networks on Moments in Time and Kinetics.
For each model-dataset pair, we extract feature maps for all examples in the validation sets, in both time and channel dimensions, and measure their similarity.
In detail, our method consists of the following steps: (1) For a given input, we ﬁrst extract the output feature maps from all convolutional layers in the network at hand.
In each layer, we measure the similarity of each feature map to each other with Person’s correlation coefﬁcient (CC) and root mean squared error (RMSE).
We additionally ﬂag feature maps that exhibit high similarity as redundant.
(3) After computing this for the validation sets, we average the values over all examples to obtain mean metrics of redundancy per model and per dataset.
We additionally compute the ranges of these values to visualize how much redundancy can vary in a model-dataset pair.
We present quantitative results in Table 12 and show examples of our ﬁndings in Figure 5.
D VA-RED 2 ON L ONGER - TRAINING M ODEL
In our experiments, all of our models are trained under a common evaluation protocol for a fair comparison.
To balance the training cost and model performance, we use a smaller epoch size than the original paper to train our models.
For example, authors in (Tran et al., 2018) and (Feichtenhofer, 2020), train the R(2+1)D models and X3D models for 188 epochs and 256 epochs respectively to pursue the state-of-the art.
However, we only train the models for 120 epochs to largely save the computation resources and training time.
However, to rule out the possibility that our base models (i.e., without using Dynamic Convolution) beneﬁt from longer training epochs while our VA-REDmay not, we conduct an ablation study on the epoch size in Table 13.
We can see that our method still shows superiority over the base model in terms of the computational cost and accuracy on the 256-epoch model.
Thus we conclude that the effectiveness of our method in achieving higher performance with low computation also holds on the longer-training models.
[Table]
E F EATURE M AP V ISUALIZATIONS
To further validate our initial motivation
[Table]
we visualize the feature maps which are fully computed by the original convlution operation and those which are generated by the cheap operations.
We demonstrate those in both temporal dimension (c.f. Figure 6) and channel dimension (c.f. Figure 7).
In both cases we can see that the proposed cheap operation generates meaningful feature maps and some of them looks even no difference from the original feature maps.
F P OLICY V ISUALIZATIONS
To compare with the policy on Mini-Kinetics-200 (Figure 3 of the main paper), we also visualize the ratio of features which are consumed in each layer on Kinetics-400 (c.f. Figure 8) and MomentsIn-Time (c.f. Figure 9).
We can see from these two ﬁgures that the conclusions we draw from
Mini-Kientics-200 still hold.
Speciﬁcally, In X3D, point-wise convolutions which right after the depth-wise convolutions have more variation among classes and network tends to consume more temporal-wise features at the early stage and compute more channel-wise features at the late stage of the architecture.
However, R(2+1)D choose to select fewer features at early stage by both temporal-wise and channel-wise policy.
Furthermore, we count the FLOPs of each instance on
Mini-Kinetics-200, Kinetics-400, and Moments-In-Time and plot pie charts to visualize the the distribution of this instance-level computational cost.
We analyze such distribution with two models:
R(2+1)D-18 and X3D-M.
All of the results are demonstrated in Figure 10.
G Q UALITATIVE R ESULTS
We show additional input examples which consume different levels of computational cost on Kinetics400 dataset (c.f. Figure 11) and Moments-In-Time dataset (c.f. Figure 12).
To be consistent,
we
[Figure]
The feature maps marked with red bounding boxes are cheaply generated.
We do this analysis on 8-frame dynamic R(2+1)D-18 pretrained on Mini-Kinetics-200.
These feature maps are the output of the ﬁrst spatial convolution combined with ReLU non-linearity inside the ResBlock_1 .
We can see that most of the cheaply generated feature maps looks no difference from the original feature maps, which further support our approach.
Best viewed in color.
[Figure] [Figure]
[Figure]
use the 16-frame dynamic R(2+1)D-18 as our pre-trained model.
We can see that the examples consuming less computation tend to have less temporal motion, like the second example in Figure 11, or have a relatively simple scene conﬁguration, like the ﬁrst and second examples in Figure 12.
[Figure]
[Figure] [Figure]