CT-N ET : C HANNEL T ENSORIZATION N ETWORK FOR V IDEO C LASSIFICATION
A BSTRACT
3D convolution is powerful for video classiﬁcation but often computationally expensive, recent studies mainly focus on decomposing it on spatial-temporal and/or channel dimensions.
Unfortunately, most approaches fail to achieve a preferable balance between convolutional efﬁciency and feature-interaction sufﬁciency.
For this reason, we propose a concise and novel Channel Tensorization Network (CTNet), by treating the channel dimension of input feature as a multiplication of
K sub-dimensions.
On one hand, it naturally factorizes convolution in a multiple dimension way, leading to a light computation burden.
On the other hand, it can effectively enhance feature interaction from different channels, and progressively enlarge the 3D receptive ﬁeld of such interaction to boost classiﬁcation accuracy.
Furthermore, we equip our CT-Module with a Tensor Excitation (TE) mechanism.
It can learn to exploit spatial, temporal and channel attention in a high-dimensional manner, to improve the cooperative power of all the feature dimensions in our CT-Module.
Finally, we ﬂexibly adapt ResNet as our CT-Net.
Extensive experiments are conducted on several challenging video benchmarks, e.g., Kinetics-400, Something-Something V1 and V2.
Our CT-Net outperforms a number of recent SOTA approaches, in terms of accuracy and/or efﬁciency.
INTRODUCTION
3D convolution has been widely used to learn spatial-temporal representation for video classiﬁcation (Tran et al., 2015; Carreira & Zisserman, 2017).
However, over parameterization often makes it computationally expensive and hard to train.
To alleviate such difﬁculty, recent studies mainly focus on decomposing 3D convolution (Tran et al., 2018; 2019).
One popular approach is spatial-temporal factorization (Qiu et al., 2017; Tran et al., 2018;
Xie et al., 2018), which can reduce overﬁtting by replacing 3D convolution with 2D spatial convolution and 1D temporal convolution.
But it still introduces unnecessary computation burden, since both spatial convolution and temporal convolution are performed over all the feature channels.
To further decrease such computation cost, channel separation has been recently developed via operating 3D convolution in the depth-wise manner (Tran et al., 2019).
However, it inevitably loses accuracy due to the lack of feature interaction between different channels.
For compensation, it has to introduce point-wise convolution to preserve interaction with extra computation.
So there is a natural question: How to construct effective 3D convolution to achieve a preferable trade-off between efﬁciency and accuracy for video classiﬁcation?
[Figure]
[Table]
[Figure]
This paper attempts to address this question by investigating two design principles.
(1) Convolutional Efﬁciency .
As shown in Table 1, current designs of spatial-temporal convolution mainly focus on decomposition from either spatial-temporal (Tran et al., 2018) or channel dimension (Tran et al., 2019).
To enhance convolutional efﬁciency, we consider decomposing convolution in a higher dimension with a novel representation of feature tensor.
(2) Feature-Interaction Sufﬁciency .
Table 1 clearly shows that, for current decomposition approaches (Tran et al., 2018; 2019), feature interaction only contains one or two of spatial, temporal and channel dimensions at each sub-operation.
Such a partial interaction manner would reduce classiﬁcation accuracy.
On one hand, it decreases the discriminative power of video representation, due to the lack of joint learning on all the dimensions.
On the other hand, it restricts feature interaction in a limited receptive ﬁeld, which ignores rich context from a larger 3D region.
Hence, to boost classiﬁcation accuracy, each sub-operation should achieve feature interaction on all the dimensions, and the receptive ﬁeld of such interaction should be progressively enlarged as the number of sub-operations increases.
Based on these desirable principles, we design a novel and concise Channel Tensorization Module (CT-Module).
Speciﬁcally, we propose to tensorize the channel dimension of input feature as a multiplication of K sub-dimensions, i.e., C = C 1 × C 2 × · · · × C K .
Via performing spatial/temporal separable convolution along each sub-dimension, we can effectively achieve convolutional efﬁciency and feature-interaction sufﬁciency.
For better understanding, we use the case of K = 2 as a simple illustration in Figure 1.
First, we tensorize the input channel into C =
C 1 × C 2 .
Naturally, we separate the convolution into distinct ones along each sub-dimension, e.g., for the 1 st sub-dimension, we apply our spatial-temporal tensor separable convolution with the size C 1 × 1 × t × h × w , which allows us to achieve convolutional efﬁciency on all the spatial, temporal and channel dimensions.
After that, we sequentially perform the tensor separable convolution sub-dimension by sub-dimension.
As a result, we can progressively achieve feature interaction on all the channels, and enlarge the spatialtemporal receptive ﬁeld.
For example, after operating 1 st tensor separable convolution on the 1 st sub-dimension, C 1 channels interact, and 3D receptive ﬁeld of such interaction is 3 × 3 × 3 .
Via further operating 2 nd tensor separable convolution on the 2 nd sub-dimension, all C 1 × C 2 =
C channels have feature interaction, and 3D receptive ﬁeld of such interaction becomes 5 × 5 × 5 .
This clearly satisﬁes our principle of feature-interaction sufﬁciency.
We summarize our contributions in the following.
First, we design a novel Channel Tensorized Module (CT-Module), which can achieve convolutional efﬁciency and feature-interaction sufﬁciency, via progressively performing spatial/temporal tensor separable convolution along each sub-dimension of the tensorized channel.
Second, we equip CT-Module with a distinct Tensor Excitation (TE) mechanism, which can further activate the video features of each sub-operation by spatial, temporal and channel attention in a tensor-wise manner.
Subsequently, we apply this full module in a residual block, and ﬂexibly adopt 2D ResNet as our Channel Tensorized Network (CT-Net).
In this case, we can gradually enhance feature interaction from a broader 3D receptive ﬁeld, and learn the key spatial-temporal representation with light computation.
Finally, we conduct extensive experiments on a number of popular and challenging benchmarks, e.g., Kinetics (Carreira & Zisserman, 2017),
Something-Something V1 and V2 (Goyal et al., 2017b).
Our CT-Net outperforms the state-of-the-art methods in terms of classiﬁcation accuracy and/or computation cost.
R ELATED W ORKS
2D CNN for video classiﬁcation.
2D CNN is a straightforward but useful method for video classiﬁcation (Karpathy et al., 2014; Simonyan & Zisserman, 2014;
Wang et al., 2016; Liu et al., 2020;
Jiang et al., 2019).
For example, Two-stream methods (Simonyan & Zisserman, 2014) learn video representations by fusing the features from RGB and optical ﬂow respectively.
Instead of sampling a single RGB frame, TSN (Wang et al., 2016) proposes a sparse temporal sampling strategy to learn video representations.
To further improve accuracy, TSM (Lin et al., 2019) proposes a zeroparameter temporal shift module to exchange information with adjacent frames.
However, these methods may lack the capacity of learning spatial-temporal interaction comprehensively, which often reduces their discriminative power to recognize complex human actions.
3D CNN for video classiﬁcation.
3D CNN has been widely used to learn a rich spatial-temporal context better (Tran et al., 2015; Carreira & Zisserman, 2017; Feichtenhofer et al., 2019; Sudhakaran et al., 2020; Feichtenhofer, 2020).
However, it introduces a lot of parameters, which leads to a difﬁcult optimization problem and large computational load.
To resolve this issue, I3D (Carreira &
Zisserman, 2017) inﬂates all the 2D convolution kernels pre-trained on ImageNet, which is helpful for optimizing.
Other works also try to factorize 3D convolution kernel to reduce complexity, such as P3D (Qiu et al., 2017) and R(2+1)D
(Tran et al., 2018).
Recently, CSN (Tran et al., 2019) operates 3D convolution in the depth-wise manner.
Nevertheless, all these methods still do not achieve a good trade-off between accuracy and efﬁciency.
To tackle this challenge, we propose CT-Net which learns on spatial-temporal and channel dimensions jointly with lower computation than previous methods.
M ETHODS
In this section, we describe our Channel Tensorization Network (CT-Net) in detail.
First, we formally introduce our CT-Module in a generic manner.
Second, we design a Tensor Excitation (TE) mechanism to enhance CT-Module.
Finally, we ﬂexibly adapt ResNet as our CT-Net to achieve a preferable trade-off between accuracy and efﬁciency for video classiﬁcation.
3. CHANNEL TENSORIZATION MODULE
As discussed in the introduction, the previous approaches have problems in convolutional efﬁciency or feature-interaction sufﬁciency.
To tackle such a problem, we introduce a generic Channel Tensorization Module (CT-Module), by treating the channel dimension of input feature as a multiplication of K sub-dimensions, i.e., C = C 1 × C 2 × · · · × C K .
Naturally, this tensor representation allows to tensorize the kernel size of convolution TConv() as a multiplication of K sub-dimensions, too.
To simplify the notation, the channel dimension of the output is omitted by default.
The output
X out can be calculated as follows:
[Equation]
(1) where X in and W denote the tensorized input and kernel respectively.
However, such an operation requires large computation, so we introduce the tensor separable convolution to alleviate the issue.
Tensor Separable Convolution.
We propose to factorize TConv() along K channel sub-dimensions.
Speciﬁcally, we decompose TConv() as K tensor separable convolutions TSConv() , and apply
TSConv() sub-dimension by sub-dimension as follows:
[Equation]
where X 0 = X in and X out = X K .
On one hand, the kernel size of the k th TSConv() is (1 × · · · × C k × · · · × 1 × t × h × w ) .
It illustrates that only C k channels interact in the k th suboperation, which leads to convolution efﬁciency.
On the other hand, as we stack the TSConv() , each convolution performs on the output features of the previous convolution.
Therefore, the spatialtemporal receptive ﬁeld is enlarged.
Besides, interactions ﬁrst occur in C 1 channels, second in
C 1 × C 2 channels and so on.
Finally, C 1 × C 2 × · · · × C K
=
C channels can progressively interact.
This clearly satisﬁes our principle of feature-interaction sufﬁciency.
Spatial-Temporal Tensor Separable Convolution.
To further improve convolution efﬁciency, we factorize the 3D TSConv() into 2D spatial TSConv() and 1D temporal TSConv() .
Thus, we can obtain the output features X Sk and X Tk as follows:
[Equation]
where S-TSConv() and T-TSConv() represent spatial and temporal tensor separable convolution respectively.
Finally, we attempt to aggregate spatial and temporal convolution.
There are various connection types of spatial and temporal tensor separable convolution, e.g., parallel and serial types.
According to the results of the experiments in Section 4, we utilize the parallel method, which illustrates that we sum the spatial feature X Sk and temporal feature X Tk :
[Equation]
3. TENSOR EXCITATION
Our CT-Module separates feature along spatial, temporal and channel dimensions.
To make full use of their cooperative power to learn distinct video features, we design a concise Tensor Excitation (TE) mechanism for each dimension.
First of all, we attempt to utilize the TE mechanism to enhance spatial and temporal features respectively.
For the spatial feature X Sk obtained by Equation 5, our corresponding spatial TE mechanism can be formulated as:
[Equation]
where T-Pool() represents global temporal pooling, i.e., T × 1 × 1 average pooling.
By performing it on X S k , we obtain the feature with the size ( C 1 × C 2 × · · · × C K × 1 × H × W ) , which gathers spatial contexts along temporal dimension.
Subsequently, the spatial tensor separable convolution STSConv() and the activate function Sigmod() are performed to generate the spatial attention heatmap.
Finally, the element-wise multiplication ⊗ broadcasts the spatial attention along the temporal dimension.
Similarly, we perform the temporal TE mechanism for the temporal feature X
Tk :
[Equation]
(7) where S-Pool() and T-TSConv() are global spatial pooling and temporal tensor separable convolution correspondingly.
At last, after aggregating the spatial and temporal features by addition, i.e., R k =
U k + V k , we perform a channel-wise TE mechanism as follows:
[Equation]
We adopt point-wise tensor separable convolution PW-TSConv () to learn the weights for aggregating distinctive channels.
The rest follows the previous design.
Note that all tensor separable convolutions are performed on the same sub-dimension as the previous convolution, which is essentially different from the SE mechanism (Hu et al., 2020).
Through the cooperation of the TE mechanism along three different dimensions, the spatial-temporal features can be signiﬁcantly enhanced.
3. CHANNEL TENSORIZATIONN NETWORK
We regard ResNet as an exemplar and build up our CT-Net from ResNet50 (or ResNet101).
First, we design a simple CT-Block in Figure 2(a), which adapts the 3 × 3 convolutional layer in Residual
Block (ResBlock) into our CT-Module.
It can achieve both convolutional efﬁciency and featureinteraction sufﬁciency.
Second, we equip our simple CT-Block with the TE mechanism in Figure 2(b), forming a full CT-Block that can improve the cooperative power of all the feature dimensions.
Besides, extra point-wise convolutions are added between different sub-operations, which are beneﬁcial for more sufﬁcient feature interaction.
At last, we build up a novel CT-Net with CT-Block.
As shown in Figure 2(c), we replace one of every two ResBlocks with our CT-Block in every stage.
Such a method guarantees a better balance between efﬁciency and accuracy in our experiments.
Discussion.
In fact, the popular methods in video classiﬁcation like C3D, R(2+1)D and CSN (Tran et al., 2015; 2018; 2019) can be viewed as special cases of our CT-Net.
We can generate different forms by adjusting three hyper-parameters: the number of sub-dimensions ( K ), the corresponding dimension size ( C k ) and the spatial-temporal kernel size ( Kernel k ) .
To degenerate into C3D, we can set K = 1 and Kernel 1 = 3 × 3 × 3 .
When K
= 2 , Kernel 1 = 1 × 3 × 3 , Kernel 2 = 3 × 1 ×and C
=
C 2 = C without channel tensorization, it becomes R(
+1)D.
Unfortunately, because of lacking the decomposition of channels, C3D and R(2+1)D still have a large computational load.
When K = 2 , C = C 1 × C 2 = C × 1 , Kernel 1 = 1 × 1 × 1 , Kernel 2 =
× 3 × 3 , obviously it is equivalent to CSN.
However, CSN has a limited receptive ﬁeld of spatial-temporal interaction.
In our CT-Net, we utilize channel tensorization and perform tensor separable convolution along each sub-dimension in turn.
Such design can not only preserve interaction among spatial, temporal and channel dimensions but also enlarge the receptive ﬁeld of feature interaction progressively.
E XPERIMENTS AND R ESULTS
Datasets and implementation details.
We conduct experiments on three large video benchmarks:
Kinetics-400 (Carreira & Zisserman, 2017), Something-Something V1 and V2 (Goyal et al., 2017b).
We choose ResNet50 and ResNet101 (He et al., 2016) pre-trained on ImageNet as the backbone and the parameters of CT-Module are randomly initialized.
For training, we utilize the dense sampling strategy (Wang et al., 2018) for Kinetics and sparse sampling strategy (Wang et al., 2016) for
Something-Something.
Random scaling and cropping are applied for data argumentation.
Finally, we resize the cropped regions to 256 × 256 .
For testing, we sample multiple clips per video (4 for
Kinetics, 2 for others) for pursuing high accuracy, and average all scores for the ﬁnal prediction.
[Table]
The parallel connection between spatial and temporal convolution is the best choice.
[Table]
and the best
[Table]
is achieved when adopting the rounded middle size (cid:98)√
[Table] [Table]
can bring signiﬁcant
[Table] [Table]
[Table] [Table]
[Table]
Kernel sizes along different dimensions.
The larger ker
[Table]
nel size brings improvement with more calculation.
[Table]
[Table] [Table]
[Table] [Table]
follow the same strategy in Non-local (Wang et al
, 2018) to pre-process the frames and crops of
Because some multi-clip models crops of 256 × 256, we simply multiply the GFLOPs reported in the corresponding papers by (256 / 224)for a fair comparison.
When considering efﬁciency, we use just 1 clip per video and the ﬁnal crop is scaled to 256 × 256 to ensure comparable GFLOPs.
4. A BLATION S TUDIES
Table 2 shows our ablation studies on Something-Something V1, which is a challenging dataset that requires video architecture to have a robust spatial-temporal representation ability and is suitable to verify the effectiveness of our method.
All models use ResNet50 as the backbone.
Effectiveness of CT-Module.
In Table 2a, we replace the 3 × 3 convolutional layer in ResNet50 with different modules in recent methods (Tran et al., 2015; 2018; 2019).
Compared with CSN-Module, our module achieves a better result with similar computation, which reﬂects the importance of sufﬁcient feature interaction.
Besides, it is slightly better than R(2+1)D-Module with much lower calculation, showing the necessity of efﬁcient convolution.
Such results demonstrate the effectiveness of our CT-Module.
It illustrates our two design principles give preferable guidance for designing an efﬁcient module for temporal modeling.
[Table]
Number of sub-dimensions.
Increasing the number of sub-dimensions saves a lot of computation, but the corresponding accuracy ﬁrst increases and then decreases as shown in Table 2b.
Compared with the 1D method, the 4D method signiﬁcantly reduces GFLOPs, achieving comparable accuracy.
As for the decrease of accuracy when K is too large, we argue that the number of channel in the shallow layer is small (64/128), thus there are too few channels in a single dimension, leading to insufﬁcient feature-interaction.
Since the 2D method obtains the best trade-off, we set K = 2 in all the following experiments.
Connection type of spatiotemporal convolution.
The coupling 3 × 3 × 3 convolution can be decomposed into serial or parallel spatial/temporal convolution.
Table 2c reveals that factorizing the 3D kernel can boost results as expected.
Besides, the parallel connection is better, thus we adopt parallel connection as the default.
Dimension size.
As we set K = 2 , it is essential to explore the impact of changing the dimension size C 2 .
We can demonstrate that the computation is the lowest when C 1 = C 2 = √ C .
Since C is not always a perfect square number, we adopt the rounded middle size (cid:98)√ C (cid:99) .
Table 2d shows that when C 2 = (cid:98)√ C (cid:99) , the model not only requires the fewest computation cost but also achieves the best performance.
Hence, we set C 2 = (cid:98)√ C (cid:99) naturally.
Number and location of CT-Blocks.
Table 2e illustrates that simply replacing 1 block in stage can bring signiﬁcant performance improvement (16.9% vs. 42.3%).
As we replace more blocks from the bottom up, the GFLOPs continues to decrease.
Moreover, the bottom blocks seem to be more beneﬁcial to temporal modeling, since replacing the extra 3 blocks in stage2 and stage3 only improve the accuracy by 0.4% (46.9% vs. 47.3%).
Since replacing 7 blocks achieves the highest accuracy, we replace 7 blocks by default.
Kernel sizes along different dimensions.
In Table 2f, we can observe that two concatenated 3convolution kernels are slightly better than those with the same receptive ﬁeld ( 1 3 +5 3 ).
Furthermore, the larger kernel size can bring performance improvement but more calculation.
It reveals that our
CT-Module avoids the limited receptive ﬁeld of feature interaction, and it can progressively enlarge the receptive ﬁeld of such interaction on all the dimensions.
Considering a better trade-off between accuracy and computation, we choose two concatenated 3 3 convolution kernels.
[Table]
Impact of different modules and different spatial resolution.
In Table 2g, our CT-Module can signiﬁcantly boost its baseline (16.9% vs. 47.3%) and the TE mechanism can further improve the accuracy by 2.1% (48.0% vs. 50.1%).
The extra point-wise convolution also boosts performance, which demonstrates that it is beneﬁcial for sufﬁcient feature interaction.
Compared with the SE mechanisms, our TE mechanism focuses more on features in different sub-dimensions individually, thus effectively enhancing spatial-temporal features.
In our experiments, to ensure GFLOPs is comparable with other methods, we crop the input to 256 × 256 during testing.
Table 2h shows both training and testing with a larger spatial resolution of input bring clear performance improvement.
4. C OMPARISONS WITH THE S TATE - OF - THE - ARTS
Something-Something V1&V2.
We make a comprehensive comparison in Table 3.
Compared with NL I3D+GCN 32 f , our CT-Net 8 f gains 4.0% top-1 accuracy improvement with 49 .
1 × fewer
GFLOPs in Something-Something V1.
Besides, our CT-Net 8 f (51.7%) is better than the ir-CSN 32 f (49.3%) which adopts ResNet-152 as the backbone.
Moreover, our CT-Net 16 f outperforms all the single-clip models in Something-Something V1&V2 and even better than most of the multiclip models.
It illustrates that our CT-Net is preferable to capture temporal contextual information efﬁciently.
Surprisingly, with only 280 GFLOPs, our ensemble model CT-Net EN achieves 56.6%(67.8%) top-1 accuracy in Something-Something V1(V2), which outperforms all methods.
Kinetics-400.
Kinetics-400 is a large-scale sence-related dataset, and the lightweight 2D models are usually inferior to the 3D models on it.
Table 4 shows our CT-Net-R50 16 f can surpass all existing lightweight models based on 2D backbone.
Even compared with SlowFast-R50 40 f , our CT-NetR50 16 f also achieves higher accuracy (77.3% vs. 76.4%).
Note that our reproduced SlowFastR50 performs worse than that in the paper (Feichtenhofer et al., 2019), which may result from the missing videos in Kinetics-400.
As for the deeper model, compared with SlowFast-R101 80 f , our
CT-Net-R
16 f requires 3 .
7 × fewer GFLOPs but gains comparable results (78.8% vs. 78.9%).
Besides, it achieves comparable top-1 accuracy with X3D-XL (78.8% vs. 79.1%) under a similar
GFLOPs.
However, X3D requires extensive model searching with an expensive GPU setting, while our CT-Net can be trained traditionally with feasible computation.
We perform score fusion over
CT-Net-R50 16 f and CT-Net-R101 16 f , which mimics two-steam fusion with two temporal rates.
In this setting, our model is 2 .
4 × faster than SlowFast-R101 80 f and shows an 0 .
9% performance gain (79.8% vs. 78.9%) but only uses 32 frames.
4. V ISUALIZATION
We use Saliency Tubes (Stergiou et al., 2019) to generate the visualization, for it can show the most discriminative features that the network locates.
In Figure 3, we sample two videos from SomethingSomething V1 which requires complex temporal modeling.
In the left example, our CT-Net focuses on a larger area around the towel, especially in the fourth and ﬁfth frames, thus predicting that someone is twisting it.
In contrast, R(2+1)D only concentrates on one side of the towel and gives the wrong judgment.
The same situation can be seen in the right example.
We argue that CT-Net can localize the action and object accurately thanks to the larger spatial-temporal receptive ﬁeld.
As for CSN, the regions of interest seem to be scattered, because it lacks sufﬁcient spatial-temporal interaction, thus ignoring the rich context both in space and time.
CONCLUSIONS
In this paper, we construct an efﬁcient tensor separable convolution to learn the discriminative video representation.
We view the channel dimension of the input feature as a multiplication of K subdimensions and stack spatial/temporal tensor separable convolution along each of K sub-dimensions.
Moreover, CT-Module is cooperated with the Tensor Excitation mechanism to further improve performance.
All experiments demonstrate that our concise and novel CT-Net obtains a preferable balance between accuracy and efﬁciency on large-scale video datasets.
Our proposed principles are preferable guidance for designing an efﬁcient module for temporal modeling.
ACKNOWLEDGEMENT
This work is partially supported by National Natural Science Foundation of China (6187617, U1713208), the National Key Research and Development Program of China (No.
2020YFC2004800), Science and Technology Service Network Initiative of Chinese Academy of
Sciences (KFJ-STS-QYZX-092), Shenzhen Institute of Artiﬁcial Intelligence and Robotics for Society.
A A PPENDIX A. M ORE TRAINING DETAILS
We use SGD with momentum 0.9 and cosine learning rate schedule (Loshchilov & Hutter, 2017) to train the entire network.
The ﬁrst 10 epochs are used for warm-up (Goyal et al., 2017a) to overcome early optimization difﬁculty.
For kinetics, the batch, total epochs, initial learning rate, dropout and weight decay are set to 64, 110, 0.01, 0.5 and 1e-4 respectively.
All these hyper-parameters are set to 64, 45, 0.02, 0.3 and 5e-4 respectively for Something-Something.
A. T ENSOR E XCITATION M ECHANISM
The implementation of our Tensor Excitation is shown in Figure 4.
Different from the SE module, we use tensor separable convolution in the TE mechanism.
Moreover, when obtaining the spatial attention, we squeeze the temporal dimension and perform spatial tensor separable convolution, because temporal information is insigniﬁcant for spatial attention and vice versa.
We add the Batch
Normalization (BN) layer for better optimization.
A. R ESULTS ON UCF101 AND HMDB
To verify the generation ability of our CT-Net on smaller datasets, we conduct transfer learning experiments from Kinetics400 to UCF101 (Soomro et al., 2012) and HMDB-51 (Kuehne et al., 2011).
We test CT-Net with 16 input frames and evaluate it over three splits and report the averaged results.
As shown in Table 5, our CT-Net 16 f achieves competitive performance when compared with the recent methods, which demonstrates the generation ability of our CT-Net.
[Table]
Our CT-Net
[Table]
A. M ORE RESULTS ON S OMETHING -S OMETHING V1&V
Table 6 shows more results on Something-Something V1&V2.
We train CT-Net with a different number of input frames and then test these models with different sampling strategies.
We average the prediction scores obtained from the previous models to evaluate the ensemble models.
With more input frames, the corresponding accuracy becomes higher.
As for the reason that CT-Net 24 f is worse than CT-Net 16 f , we argue that is because the model is hard to optimize with too many input frames.
Sampling more clips or more crops also boosts performance.
Moreover, our ensemble models gain the state-of-the-art top-1 accuracy of 56.6%(68.3%) on Something-Something V1(V2).
A. M ORE ABLATION STUDIES ON M INI -K INETICS AND S OMETHING -S OMETHING V
To verify the effectiveness of our module comprehensively, we also conduct experiments in MiniKinetics and Something-Something V2 and report the multi-clip accuracy and single-clip accuracy respectively.
Mini-Kinetics covers 200 action classes and is a subset of Kinetics-400, while
Something-Something V2 covers the same action classes as Something-Something V1 but contains more videos.
As shown in Table 7, the performance trend of different modules is similar to that shown in Table 2a.
Since Mini-Kinetics does not highly depend on temporal modeling, the gap becomes smaller but still demonstrates the effectiveness of our CT-Module.
A. A DAPTING DIFFERENT PRE - TRAINED I MAGE N ET ARCHITECTURES AS CT-N ET
In fact, by directly replacing the 3 × 3 convolution with our CT-Module, we can easily adapt different pre-trained ImageNet architectures as CT-Net.
Table 8 shows that it is also sensible to use
InceptionV3 as the backbone.
We believe that through more elaborate design, our CT-Net based on different backbones can achieve comparable performance.
A. V ALIDATION PLOT
In Figure 5, we plot the accuracy vs per-clip GFLOPs on Kinetics-400.
It reveals that our CT-Net achieves a better trade-off than most of the existing methods on Kinetics-400.
[Table] [Table]
Our
[Table] [Table]
Our CT-Net
[Table]
Baseline
[Table]
Our CT-Net
[Table]