TA DA ! T EMPORALLY -A DAPTIVE C ONVOLUTIONS FOR V IDEO U NDERSTANDING
A BSTRACT
Spatial convolutions are widely used in numerous deep video models.
It fundamentally assumes spatio-temporal invariance, i e
, using shared weights for every location in different frames.
This work presents Temporally-Adaptive Convolutions (TAdaConv) for video understanding 2 , which shows that adaptive weight calibration along the temporal dimension is an efﬁcient way to facilitate modelling complex temporal dynamics in videos.
Speciﬁcally, TAdaConv empowers the spatial convolutions with temporal modelling abilities by calibrating the convolution weights for each frame according to its local and global temporal context.
Compared to previous temporal modelling operations, TAdaConv is more efﬁcient as it operates over the convolution kernels instead of the features, whose dimension is an order of magnitude smaller than the spatial resolutions.
Further, the kernel calibration brings an increased model capacity.
We construct TAda2D and TAdaConvNeXt networks by replacing the 2D convolutions in ResNet and ConvNeXt with TAdaConv, which leads to at least on par or better performance compared to state-of-the-art approaches on multiple video action recognition and localization benchmarks.
We also demonstrate that as a readily plug-in operation with negligible computation overhead, TAdaConv can effectively improve many existing video models with a convincing margin.
I NTRODUCTION
Convolutions are an indispensable operation in modern deep vision models (He et al., 2016; Szegedy et al., 2015; Krizhevsky et al., 2012), whose different variants have driven the state-of-the-art performances of convolutional neural networks (CNNs) in many visual tasks (Xie et al., 2017; Dai et al., 2017;
Zhou et al., 2019) and application scenarios (Howard et al., 2017; Yang et al., 2019).
In the video paradigm, compared to the 3D convolutions (Tran et al., 2015), the combination of 2D spatial convolutions and 1D temporal convolutions are more widely preferred owing to its efﬁciency (Tran et al., 2018; Qiu et al., 2017).
Nevertheless, 1D temporal convolutions still introduce non-negligible computation overhead on top of the spatial convolutions.
Therefore, we seek to directly equip the spatial convolutions with temporal modelling abilities.
One essential property of the convolutions is the translation invariance (Ruderman & Bialek, 1994;
Simoncelli & Olshausen, 2001), resulted from its local connectivity and shared weights.
However, recent works in dynamic ﬁltering have shown that strictly shard weights for all pixels may be suboptimal for modelling various spatial contents (Zhou et al., 2021; Wu et al., 2018).
Given the diverse nature of the temporal dynamics in videos, we hypothesize that the temporal modelling could also beneﬁt from relaxed invariance along the temporal dimension.
This means that the convolution weights for different time steps are no longer strictly shared.
Existing dynamic ﬁlter networks can achieve this, but with two drawbacks.
First, it is difﬁcult for most of them (Zhou et al., io/ .
K K
W t
C
K K
W t
K K
W t
K K
W b
K K
W (a) Spatial convolution in videos (b) TAdaConv in videos
C C
C C
T
H
W
H
W
H
W W
W H
H
H … … … … … …
T C
Convolution ﬁlters
[Figure]
W b α tα tα t T
2021; Yang et al., 2019) to leverage the existing pre-trained models such as ResNet.
This is critical in video applications, since training video models from scratch is highly resource demanding (Feichtenhofer et al., 2019; Feichtenhofer, 2020) and prone to over-ﬁtting on small datasets.
Second, for most dynamic ﬁlters, the weights are generated with respect to its spatial context (Zhou et al., 2021;
Jia et al., 2016) or the global descriptor (Chen et al., 2020a;
Yang et al., 2019), which has difﬁculty in capturing the temporal variations between frames.
In this work, we present the Temporally-Adaptive Convolution (TAdaConv) for video understanding, where the convolution kernel weights are no longer ﬁxed across different frames.
Speciﬁcally, the convolution kernel for the t -th frame W t is factorized to the multiplication of the base weight and a calibration weight:
W t = α t · W b , where the calibration weight α t is adaptively generated from the input data for all channels in the base weight W b .
For each frame, we generate the calibration weight based on the frame descriptors of its adjacent time steps as well as the global descriptor, which effectively encodes the local and global temporal dynamics in videos.
The difference between
TAdaConv and the spatial convolutions is visualized in Fig.
The main advantages of this factorization are three-fold: (i) TAdaConv can be easily plugged into any existing models to enhance temporal modelling, and their pre-trained weights can still be exploited; (ii) the temporal modelling ability can be highly improved with the help of the temporallyadaptive weight; (iii) in comparison with temporal convolutions that often operate on the learned 2D feature maps, TAdaConv is more efﬁcient by directly operating on the convolution kernels.
TAdaConv is proposed as a drop-in replacement for the spatial convolutions in existing models.
It can both serve as a stand-alone temporal modelling module for 2D networks, or be inserted into existing convolutional video models to further enhance the ability to model temporal dynamics.
For efﬁciency, we construct TAda2D by replacing the spatial convolutions in ResNet (He et al., 2016), which leads to at least on par or better performance than the state-of-the-arts.
Used as an enhancement of existing video models, TAdaConv leads to notable improvements on multiple video datasets.
The strong performance and the consistent improvements demonstrate that TAdaConv can be an important operation for modelling complex temporal dynamics in videos.
R ELATED W ORK
ConvNets for temporal modelling.
A fundamental difference between videos and images lies in the temporal dimension, which makes temporal modeling an important research area for understanding videos.
Recent deep CNNs for video understanding can be divided into two types.
The ﬁrst type jointly models spatio-temporal information by 3D convolutions (Carreira & Zisserman, 2017; Tran et al., 2015; Feichtenhofer, 2020; Tran et al., 2019).
The second type builds upon 2D networks, where most approaches employ 2D convolutions that share weights among all the frames for spatial modelling, and design additional operations for temporal modelling, such as temporal shift (Lin et al., 2019a), temporal difference (Wang et al., 2021; Jiang et al., 2019), temporal convolution (Tran et al., 2018; Liu et al., 2021b) and correlation operation (Wang et al., 2020), etc .
Our work directly empowers the spatial convolutions with temporal modelling abilities, which can be further coupled with other temporal modelling operations for stronger video recognition performances.
[Figure]
Corrected
[Figure]
Dynamic networks.
Dynamic networks refer to networks with content-adaptive weights or modules, such as dynamic ﬁlters/convolutions (Jia et al., 2016;
Yang et al., 2019; Li et al., 2021), dynamic activations (Li et al., 2020d; Chen et al., 2020b), and dynamic routing (Wang et al., 2018; Li et al., 2020c), etc .
Dynamic networks have demonstrated their exceeding network capacity and thus performance compared to the static ones.
In video understanding, dynamic ﬁlter (Liu et al., 2021b) or temporal aggregation (Meng et al., 2021) also demonstrate strong capability of temporal modelling.
Recently, some convolutions applies spatially-adaptive weights (Elsayed et al., 2020; Chen et al., 2021), showing the beneﬁt of relaxing the spatial invariance in modelling diverse visual contents.
Similarly, our proposed TAdaConv enhances temporal modelling by relaxing the invariance along the temporal dimension.
TAdaConv has two key differences from previous works: (i) the convolution weight in TAdaConv is factorized into a base weight and a calibration weight, which enables
TAdaConv to exploit pre-trained weights; (ii) the calibration weights are generated according to the temporal contexts.
TA DA C ONV : T EMPORALLY - ADAPTIVE C ONVOLUTIONS
In this work, we seek to empower the spatial convolutions with temporal modelling abilities.
Inspired by the calibration process of temporal convolutions (Sec. 3.1), TAdaConv dynamically calibrates the convolution weights for each frame according to its temporal context (Sec. 3.2).
3. R EVISITING TEMPORAL CONVOLUTIONS
We ﬁrst revisit the temporal convolutions to showcase its underlying process and its relation to dynamic ﬁlters.
We consider depth-wise temporal convolution for simplicity, which is more widely used because of its efﬁciency (Liu et al., 2021b; Jiang et al., 2019).
Formally, for a 3 × 1 × 1 temporal convolution ﬁlter parameterized by β = [ β 1 , β 2 , β 3 ] and placed (ignoring normalizations) after the 2D convolution parameterized by W , the output feature ˜x t of the t -th frame can be obtained by:
[Equation]
where the · indicates the element-wise multiplication, ∗ denotes the convolution over the spatial dimension and δ denotes ReLU activation (Nair & Hinton, 2010).
It can be rewritten as follows:
[Equation]
where W i,j t − 1 =
M i,j t − 1 · β 1 · W , W i,j t =
M i,j t · β 2 · W and W i,j t +1 =
M i,j t +1 · β 3 · W are spatiotemporal location adaptive convolution weights.
M t ∈ R C × H × W is a dynamic tensor, with its value dependent on the result of the spatial convolutions (see Appendix A for details).
Hence, the temporal convolutions in the (2+1)D convolution essentially performs (i) weight calibration on the spatial convolutions and (ii) feature aggregation between adjacent frames.
However, if the temporal modelling is achieved by coupling temporal convolutions to spatial convolutions, a non-negligible computation overhead is still introduced (see Table 2).
3. F ORMULATION OF TA DA C ONV
For efﬁciency, we set out to directly empower the spatial convolutions with temporal modelling abilities.
Inspired by the recent ﬁnding that the relaxation of spatial invariance strengthens spatial modelling (Zhou et al., 2021; Elsayed et al., 2020), we hypothesize that temporally adaptive weights can also help temporal modelling.
Therefore, the convolution weights in a TAdaConv layer are varied on a frame-by-frame basis.
Since we observe that previous dynamic ﬁlters can hardly utilize the pretrained weights, we take inspiration from our observation in the temporal convolutions and factorize the weights for the t -th frame W t into the multiplication of a base weight W b shared for all frames, and a calibration weight α t that are different for each time step:
[Equation]
Calibration weight generation.
To allow for the TAdaConv to model temporal dynamics, it is crucial that the calibration weight α t for the t -th frame takes into account not only the current frame, but more importantly, its temporal context, i.e., α t =
G ( ..., x t − 1 , x t , x t +1 , ... ) .
Otherwise
TAdaConv would degenerate to a set of unrelated spatial convolutions with different weights applied on different frames.
We show an instantiation of the generation function G in Fig. 2(b).
In our design, we aim for efﬁciency and the ability to capture inter-frame temporal dynamics.
For efﬁciency, we operate on the frame description vectors v ∈ R T × C obtained by the global average pooling over the spatial dimension GAP s for each frame, i.e., v t =
GAP s ( x t ) .
For temporal modelling, we apply stacked two-layer 1D convolutions F with a dimension reduction ratio of r on the local temporal context v adjt
= { v t − 1 , v t , v t +1 } obtained from x adjt =
{ x t −
[Equation]
where δ and BN denote the ReLU (Nair & Hinton, 2010) and batchnorm (Ioffe & Szegedy, 2015).
In order for a larger inter-frame ﬁeld of view in complement to the local 1D convolution, we further incorporate global temporal information by adding a global descriptor g to the weight generation process F through a linear mapping function FC:
[Equation]
where g = GAP st ( x ) with GAP st being global average pooling on spatial and temporal dimensions.
Initialization.
The TAdaConv is designed to be readily inserted into existing models by simply replacing the 2D convolutions.
For an effective use of the pre-trained weights, TAdaConv is initialized to behave exactly the same as the standard convolution.
This is achieved by zero-initializing the weight of the last convolution in F and adding a constant vector 1 to the formulation:
[Equation]
In this way, at initial state, W t = 1 · W b = W b , where we load W b with the pre-trained weights.
Calibration dimension.
The base weight W b ∈ R C out × C in × k 2 can be calibrated in different dimensions.
We instantiate the calibration on the C in dimension ( α t ∈ R 1 × C in × 1 ), as the weight generation based on the input features yields a more precise estimation for the relation of the input channels than the output channels or spatial structures (empirical analysis in Table 7).
[Table]
Comparison with other dynamic ﬁlters.
Table 1 compares TAdaConv with existing dynamic ﬁlters.
Mixtue-of-experts based dynamic ﬁlters such as CondConv dynamically aggregates multiple kernels to generate the weights that are shared for all locations.
The weights in most other dynamic ﬁlters are completely generated from the input, such as DynamicFilter (Jia et al., 2016) and
DDF (Zhou et al., 2021) in images and TAM (Liu et al., 2021b) in videos.
Compared to image based ones, TAdaConv achieves temporal modelling by generating weights from the local and global temporal context.
Compared to TANet (Liu et al., 2021b), TAdaConv is better at temopral modellling because of temporally adaptive weights.
Further, TAdaConv can effectively generate weights identical to the pre-trained ones, while it is difﬁcult for previous approaches to exploit pre-trained models.
More detailed comparisons of dynamic ﬁlters are included in Appendix J.
[Table]
Comparison with temporal convolutions.
Table 2 compares the TAdaConv with R(2+1)D in parameters and FLOPs, which shows most of our additional computation overhead on top of the spatial convolution is an order of magnitude less than the temporal convolution.
For detailed computation analysis and comparison with other temporal modelling approaches, please refer to Appendix B
[Table]
TA DA 2D: T EMPORALLY A DAPTIVE 2D N ETWORKS
We construct TAda2D networks by replacing the 2D convolutions in ResNet (R2D, see Appendix D) with our proposed TAdaConv.
Additionally, based on strided average pooling, we propose a temporal feature aggregation module for the 2D networks, corresponding to the second essential step for the temporal convolutions.
As illustrated in Fig. 2(c), the aggregation module is placed after
TAdaConv.
Formally, given the output of TAdaConv ˜x , the aggregated feature can be obtained as follows:
[Equation]
where TempAvgPool k denotes strided temporal average pooling with kernel size of k .
We use different batch normalization parameters for the features extracted by TAdaConv ˜x and aggregated by strided average pooling TempAvgPool k
( ˜x ) , as their distributions are essentially different.
During initialization, we load pre-trained weights to BN 1 , and initialize the parameters of BN 2 to zero.
Coupled with the initialization of TAdaConv, the initial state of the TAda2D is exactly the same as the Temporal Segment Networks (Wang et al., 2016), while the calibration and the aggregation notably increases the model capacity with training (See Appendix I).
In the experiments, we refer to this structure as the shortcut (Sc.) branch and the separate BN (SepBN.) branch.
E XPERIMENTS ON VIDEO CLASSIFICATION
To show the effectiveness and generality of the proposed approach, we present comprehensive evaluation of TAdaConv and TAda2D on two video understanding tasks using four large-scale datasets.
Datasets.
For video classiﬁcation, we use Kinetics-400 (Kay et al., 2017),
Something-SomethingV
(Goyal et al., 2017), and Epic-Kitchens-100 (Damen et al., 2020).
K400 is a widely used action classiﬁcation dataset with 400 categories covered by ∼ 300K videos.
SSV2 includes 220K videos with challenging spatio-temporal interactions in 174 classes.
EK100 includes 90K segments labelled by 97 verb and 300 noun classes with actions deﬁned by the combination of nouns and verbs.
For action localization, we use HACS (Zhao et al., 2019) and Epic-Kitchens-100 (Damen et al., 2020).
Model.
In our experiments, we mainly use ResNet (R2D) as our base model, and construct TAda2D by replacing the spatial convolutions with the TAda-structure in Fig. 2(c).
Alternatively, we also construct TAdaConvNeXt based on the recent ConvNeXt model (Liu et al., 2022).
For TAdaConvNeXt, we use a tubelet embedding stem similar to (Arnab et al., 2021) and only use TAdaConv to replace the depth-wise convolutions in the model.
More details are included in Appendix D.
Training and evaluation.
During training, 8, 16 or 32 frames are sampled with temporal jittering, following convention (Lin et al., 2019a;
Liu et al., 2021b; Feichtenhofer et al., 2019).
We include further training details in the appendix C. For evaluation, we use three spatial crops with 10 orclips (K400&EK100), or 2 clips (SSV2) uniformly sampled along the temporal dimension.
Each crop has the size of 256 × 256, which is obtained from a video with its shorter side resized to 256.
[Table] [Table]
[Table]
5. TA DA C ONV ON EXISTING VIDEO BACKBONES
TAdaConv is designed as a plug-in substitution for the spatial convolutions in the video models.
Hence, we ﬁrst present plug-in evaluations in Table 3.
TAdaConv improves the classiﬁcation performance with negligible computation overhead on a wide range of video models, including SlowFast (Feichtenhofer et al., 2019), R3D
(Hara et al., 2018) and R(2+1)D (Tran et al., 2018), by an average of 1.3% and 2.8% respectively on K400 and SSV2 at an extra computational cost of less than 0.02 GFlops.
Further, not only can TAdaConv improve spatial convolutions, it also notably improve 3D and 1D convolutions.
For fair comparison, all models are trained using the same training strategy.
Further plug-in evaluations for action classiﬁcation is presented in Appendix G.
A BLATION STUDIES
We present thorough ablation studies for the justiﬁcation of our design choices and the effectiveness of our TAdaConv in modelling temporal dynamics.
SSV2 is used as the evaluation benchmark
[Table]
as it is widely acknowledged to have more complex spatio-temporal interactions.
[Table]
Dynamic vs. learnable calibration.
We ﬁrst compare different source of calibration weights (with our initialization strategy) in Table 6.
We compare our calibration with no calibration, calibration using learnable weights, and calibration using dynamic weights generated only from a global descriptor (C × 1).
Compared with the baseline (TSN (Wang et al., 2016)) with no calibration, learnable calibration with shared weights has limited improvement, while temporally varying learnable calibration (different calibration weights for different temporal locations) performs much stronger.
A larger improvement is observed when we use dynamic calibration, where temporally varying calibration further raises the accuracy.
The results also validate our hypothesis that temporal modelling can beneﬁt from temporally adaptive weights.
Further, TAdaConv generates calibration weight from both local (C × T) and global (C × 1) contexts and achieves the highest performance.
[Figure]
[Figure]
Calibration weight initialization.
Next, we show that our initialization strategy for the calibration weight generation plays a critical role for dynamic weight calibration.
As in Table 6, randomly initializing learnable weights slightly degrades the performance, while randomly initializing dynamic calibration weights (by randomly initializing the last layer of the weight generation function) notably degenerates the performance.
It is likely that randomly initialized dynamic calibration weights purturb the pre-trained weights more severely than the learnable weights since it is dependent on the input.
Further comparisons on the initialization are shown in the Table A7 in the Appendix.
Calibration weight generation function.
Having established that the temporally adaptive dynamic calibration with appropriate initialization can be an ideal strategy for temporal modelling, we further ablate different ways for generating the calibration weight in Table 4.
Linear weight generation
function ( Lin. ) applies a single 1D convolution to generate the calibration weight, while non-linear one ( Non-Lin. )
uses two stacked 1D convolutions with batch normalizations and ReLU activation in between.
When no temporal context is considered (K.=1 or (1,1)), TAdaConv can still improve the baseline but with a limited gap.
Enlarging the kernel size to cover the temporal context (K.=3, (1,3), (3,1) or (3,3)) further yields a boost of over 20% on the accuracy, with K.=(3,3) having the strongest performance.
This shows the importance of the local temporal context during calibration weight generation.
Finally, for the scope of temporal context, introducing global context to frame descriptors performs similarly to only generating temporally adaptive calibration weights solely on the global context (in Table 6).
The combination of the global and temporal context yields a better performance for both variants.
We further show in Appendix J that this function in our TAdaConv yields a better calibration on the base weight than other existing dynamic ﬁlters.
Feature aggregation.
The feature aggregation module used in the TAda2D network is ablated in
Table 5.
First, the performance is similar for plain aggregation x = Avg ( x ) and aggregation with a shortcut (Sc.)
branch x = x + Avg ( x ) , with Sc. being slightly better.
Separating the batchnorm (Eq. 7) for the shortcut and the aggregation branch brings notable improvement.
Strided max and mix (avg+max) pooling slightly underperform the average pooling variant.
Overall, the combination of TAdaConv and our feature aggregation scheme has an advantage over the TSN baseline of 31.8%.
[Table]
Calibration dimension.
Multiple dimensions can be calibrated in the base weight.
Table 7 shows that calibrating the channel dimension more suitable than the spatial dimension, which means that the spatial structure of the original convolution kernel should be retained.
Within channels, the calibration works better on C in than C out or both combined.
This is probably because the calibration weight generated by the input feature can better adapt to itself.
Different stages employing TAdaConv.
The solid lines in Fig 3 show the stage by stage replacement of the spatial convolutions in a ResNet model.
It has a minimum improvement of 17.55%, when TAdaConv is employed in Res2 .
Compared to early stages, later stages contribute more to the ﬁnal performance, as later stages provide more accurate calibration because of its high abstraction level.
Overall, TAdaConv is used in all stages for the highest accuracy.
[Table] [Table]
Different proportion of channels calibrated by TAdaConv.
Here, we calibrate only a proportion of channels using TAdaConv and leave the other channels uncalibrated.
The results are presented as dotted lines in Fig 3.
We ﬁnd TAdaConv can improve the baseline by a large margin even if only 1/64 channels are calibrated, with larger proportion yielding further larger improvements.
Visualizations.
We qualitatively evaluate our approach in comparison with the baseline approaches (TSN and R(2+1)D) by presenting the Grad-CAM (Selvaraju et al., 2017) visualizations of the last stage in Fig.
TAda2D can more completely spot the key information in videos, thanks to the temporal reasoning based on global spatial information and the global temporal information.
5. M AIN RESULTS
SSV2.
As shown in Table 8, TAda2D outperforms previous approaches using the same number of frames.
Compared to TDN that uses more frames, TAda2D performs competitively.
Visualization in
Fig. 3(b) also demonstrates the superiority of our performance/efﬁciency trade-off.
An even stronger performance is achieved with a similar amount of computation by TAdaConvNeXt, which provides an accuracy of 67.1% with 94GFLOPs.
Epic-Kitchens-100.
Table 9 lists our results on EK100 in comparison with the previous approaches 3 .
We calculate the ﬁnal action prediction following the strategies in Huang et al.
For fair comparison, we reimplemented our baseline TSN using the same training and evaluation strategies.
TAda2D improves this baseline by 11.46% on the action prediction.
Over previous approaches, TAda2D achieves a higher accuracy with a notable margin.
Kinetics-400.
Comparison with the state-of-the-art models on Kinetics-400 is presented in Table 10, where we show TAda2D performs competitively in comparison with the models using the same
[Table]
[Table]
backbone and the same number of frames.
Compared with models with more frames, e.g., TDN,
TAda2D achieves a similar performance with less frames and computation.
When compared to the more recent Transformer-based models, our TAdaConvNeXt-T provides competitive accuracy with similar or less computation.
E XPERIMENTS ON TEMPORAL ACTION LOCALIZATION
Dataset, pipeline, and evaluation.
Action localization is an essential task for understanding untrimmed videos, whose current pipeline makes it heavily dependent on the quality of the video representations.
We evaluate our TAda2D on two large-scale action localization datasets, HACS (Zhao et al., 2019) and Epic-Kitchens-100 (Damen et al., 2020).
The general pipeline follows (Damen et al., 2020; Qing et al., 2021a;c).
For evaluation, we follow the standard protocol for the respective dataset.
We include the details on the training pipeline and the evaluation protocal in the Appendix C.
Main results.
Table 11 shows that, compared to the baseline, TAda2D provides a stronger feature for temporal action localization, with an average improvement of over 4% on the average mAP on both datasets.
In Appendix H, we further demonstrate the TAdaConv can also improve action localization when used as a plug-in module for existing models.
C ONCLUSIONS
This work proposes Temproally-Adaptive Convolutions (TAdaConv) for video understanding, which dynamically calibrates the convolution weights for each frame based on its local and global temporal context in a video.
TAdaConv shows superior temporal modelling abilities on both action classiﬁcation and localization tasks, both as stand-alone and plug-in modules for existing models.
We hope this work can facilitate further research in video understanding.
Acknowledgement: This research is supported by the Agency for Science, Technology and Research (A*STAR) under its AME Programmatic Funding Scheme (Project #A18A2b0046), by the
RIE2020 Industry Alignment Fund – Industry Collaboration Projects (IAF-ICP)
Funding Initiative, as well as cash and in-kind contribution from the industry partner(s) and by Alibaba Group
through
Alibaba Research Intern Program
A PPENDIX
In the appendix, we provide detailed analysis on the temporal convolutions (Appendix A), computational analysis (Appendix B), further implementation details (Appendix C) on the action classiﬁcation and localization, model structures that we used for evaluation (Appendix D), per-category improvement analysis on Something-Something-V2 (Appendix E), further plug-in evaluations on
Epic-Kitchens classiﬁcation (Appendix G) plug-in evaluations on the temporal action localization task (Appendix H), the visualization of the training procedure of TSN and TAda2D (Appendix I), as well as detailed comparisons between TAdaConv and existing dynamic ﬁlters (Appendix J).
Further, we show additional qualitative analysis in Fig.
A3.
A D ETAILED ANALYSIS ON TEMPORAL CONVOLUTIONS
Here, we provide detailed analysis to showcase the underlying process of temporal modelling by temporal convolutions.
As in Sec. 3.1, we use depth-wise temporal convolutions for simplicity and its wide application.
We ﬁrst analyze the case where temporal convolutions are directly placed after spatial convolutions without non-linear activation in between, before activation functions is inserted in the second part of our analysis.
Without activation.
We ﬁrst consider a simple case with no non-linear activation functions between the temporal convolution and the spatial convolution.
Given a 3 × 1 × 1 depth-wise temporal convolution parameterized by β = [ β 1 , β 2 , β 3 ] , where β 1 , β 2 , β 3 ∈ R C o , a spatial convolution parameterized by W ∈ R C o × C
i × k 2 , the output feature ˜x t of the t -th frame can be obtained by:
[Equation]
where · denotes element-wise multiplication with broadcasting, and ∗ denotes convolution over the spatial dimension.
In this case, β could be grouped with the spatial convolution weight W and the combination of temporal and spatial convolution can be rewritten as Eq.
2 in the manuscript:
[Equation]
where W t − 1 = β 1 · W , W t = β 2 · W and W t +1 = β 3 · W .
This equation share the same form with the Eq.
2 in the manucript.
In this case, the combination of temporal convolution with spatial convolution can be certainly viewed as the temporal convolution is simply performing calibration on spatial convolutions before aggregation, with different weights assigned to different time steps for the calibration.
With activation.
Next, we consider a case where activation is in between the temporal convolution and spatial convolution.
The output feature ˜x t are now obtained by Eq.
1 in the manuscript:
[Equation]
Next, we show that this can be still rewritten in the form of Eq.
Here, we consider the case where
ReLU (Nair & Hinton, 2010) is used as the activation function, denoted as δ :
[Equation]
Hence, the term δ ( W ∗ x t ) can be easily expressed as:
[Equation]
where M t ∈ R C × H × W is a binary map sharing the same shape as x t , indicating whether the corresponding element in W ∗ x t is greater than 0 or not.
That is:
[Equation]
where c, i, j are the location index in the tensor.
Hence, Eq. 1 can be expressed as:
[Equation]
In this case, we can set W ( i,j ) t − 1 = β 1 · M ( i,j ) t − 1 · W , W ( i,j ) t = β 2 · M ( i,j ) t · W , and W ( i,j ) t +1 = β 3 · M ( i,j ) t +1 · W , where ( i, j ) indicate the spatial location index.
In this case, each ﬁlter for a speciﬁc time step t is composed of H × W ﬁlters and Eq.
1 can be rewritten as Eq. 2.
Interestingly, it can be observed that with ReLU activation function, the convolution weights are different for all spatiotemporal locations, since the binary map M depends on the results of the spatial convolutions.
[Table]
B C OMPUTATIONAL ANALYSIS
Consider the input tensor with the shape of C i × T × H × W , where C i denotes the number of input channels, TAdaConv essentially performs 2D convolution, with weights dynamically generated.
Hence
[Table]
a good proportion of the computation is carried out by the 2D convolutions:
[Equation]
where C o denote the number of output channels
[Table]
and k denotes the kernel size of the 2D convolutions.
For the weight generation, the features are ﬁrst aggregated by average pooling over the spatial dimension and the temporal dimension, which contains no parameters:
[Equation]
For the local information, a two layer 1D convolution with kernel size of k w are applied, with reduction ratio of r in between.
For global information, a one layer 1D convolution with kernel size of 1 is applied.
[Equation]
Further, the calibration weight α ∈ R C i × T is multiplied to the kernel weight of 2D convolutions
W
[Equation]
Hence, the overall computation and parameters are:
[Equation]
For the FLOPs, despite the overwhelming number of terms, all the other terms are at least an order of magnitude smaller than the ﬁrst term.
This is in contrast to the (2+1)D convolutions for spatiotemporal modelling, where the FLOPs are C o × C i × ( k 2 + k ) × THW .
The extra computation introduced by the temporal convolutions is only k times smaller than the 2D convolutions.
Table A shows the comparison of computation and parameters with other approaches.
C F URTHER IMPLEMENTATION DETAILS
Here, we further describe the implementation details for the action classiﬁcation and action localization experiments.
For fair comparison, we keep all the training strategies the same for our baseline, the plug-in evaluations as well as our own models.
C. A CTION CLASSIFICATION
Our experiments on the action classiﬁcation are conducted on three large-scale datasets.
For all action classiﬁcation models, we train them with synchronized SGD using 16 GPUs.
The batch size for each GPU is 16 and 8 respectively for 8-frame and 16-frame models.
The weights in TAda2D are initialized using ImageNet (Deng et al., 2009) pre-trained weights (He et al., 2016), except for the calibration function G and the batchnorm statistics (BN 2 ) in the average pooling branch.
In the calibration function, we randomly initialize the ﬁrst convolution layer (for non-linear weight generation) following He et al. (2015), and ﬁll zero to the weight of last convolution layer.
The batchnorm statistics are initialized to be zero so that the initial state behaves the same as without the average pooling branch.
For all models, we use a dropout ratio (Hinton et al., 2012) of 0.
before the classiﬁcation heads.
Spatially, we randomly resize the short side of the video to [256, 320] and crop a region of 224 × 224 to the network in ablation studies, and set the scale to [224, 340] following TANet (Liu et al., 2021b) for comparison against the state-of-the-arts.
Temporally, we perform interval based sampling for Kinetics-400 and Epic-Kitchens-100, with interval of 8 for 8 frames, interval of 5 for 16 frames and interval of 2 for 32 frames.
On Something-Something-V2, we perform segment based sampling.
On Kinetics-400 , a half-period cosine schedule is applied for decaying the learning rate following Feichtenhofer et al.
(2019), with the base learning rate set to 0.24 for ResNet-base models using
SGD.
For TAdaConvNeXt, the base learning rate is set to 0.0001 for the backbone and 0.001 for the head, using adamw (Loshchilov & Hutter, 2017) as the optimizer.
The models are trained for 100 epochs.
In the ﬁrst 8 epochs, we adopt a linear warm-up strategy starting from a learning rate of 0.01.
The weight decay is set to 1e-4.
The frames are sampled based on a ﬁxed interval, which is 8 for 8-frame models, 5 for 16-frame models and 2 for 32-frame models.
Additionally for
TAdaConvNeXt-T, a drop-path rate of 0.4 is employed.
On Epic-Kitchens-100 , the models are initialized with weights pre-trained on Kinetics-400, and are further ﬁne-tuned following a similar strategy as in kinetics.
The training length is reduced toepochs, with 10 epochs for warm-up.
The base learning rate is 0.48.
Following Damen et al.
and Huang et al.
(2021), we connect two separate heads for predicting verbs and nouns.
Action predictions are obtained according to the strategies in Huang et al.
(2021), which is shown to have a higher accuracy over the original one in Damen et al.
For fair comparison, we also trained and evaluated our baseline using the same strategy.
On Something-Something-V2 , we initialize the model with ImageNet pretrained weights for ResNetbased models and Kinetics-400 pre-trained weights for TAdaConvNeXt.
A segment-based sampling strategy is adopted, where for T -frame models, the video is divided into T segments before one frame is sampled from each segment randomly for training or uniformly for evaluation.
The models are trained for 64 epochs, with the ﬁrst 4 being the warm-up epochs.
The base learning rate is set to 0.48 in training TAda2D with SGD, and 0.001/0.0001 respectively for the head and the backbone in training TAdaConvNeXt with adamw.
Following Liu et al.
(2021a), we use stronger augmentations such as mixup (Zhang et al., 2017), cutmix (Yun et al., 2019), random erasing (Zhong et al., 2020) and randaugment (Cubuk et al., 2020) with the same parameters in Liu et al.
It is worth noting that, for SlowFast models (Feichtenhofer et al., 2019) in the plug-in evaluations, we do not apply precise batch normalization statistics in our implementation as in its open-sourced codes, which is possibly the reason why our re-implemented performance is slightly lower than the original published numbers.
C. A CTION L OCALIZATION
We evaluate our model on the action localization task using two large-scale datasets.
The overall pipeline for our action localization evaluation is divided into ﬁnetuning the classiﬁcation models, obtaining action proposals and classifying the proposals.
Finetuning.
On Epic-Kitchens , we simply use the evaluated action classiﬁcation model.
On HACS , following (Qing et al., 2021c), we initialize the model with Kinetics-400 pre-trained weights and train the model with adamW for 30 epochs (8 warmups) using 32 GPUs.
The mini-batch size is 16 videos per GPU.
The base learning rate is set to 0.0002, with cosine learning rate decay as in
Kinetics.
In our case, only the segments with action labels are used for training.
[Table]
Proposal generation.
For the action proposals, a boundary matching network (BMN) (Lin et al., 2019b) is trained over the extracted features on the two datasets.
On Epic-Kitchens , we extract features with the videos uniformly decoded at 60 FPS.
For each clip, we use 8 frames with an interval of 8 to be consistent with ﬁnetuning, which means a feature roughly covers a video clip of one seconds.
The interval between each clip for feature extraction is 8 frames ( i.e., 0.133 sec) as well.
The shorter side of the video is resized to 224 and we feed the whole spatial region into the backbone to retain as much information as possible.
Following Qing et al.
(2021a), we generate proposals using BMN based on sliding windows.
The predictions on the overlapped region of different sliding windows are simply averaged.
On HACS , the videos are decoded at 30 FPS, and extend the interval between clips to be 16 ( i.e., 0.533 sec) because the actions in HACS last much longer than in EpicKitchens.
The shorter side is resized to 128 for efﬁcient processing.
For the settings in generating proposals, we mainly follow Qing et al.
(2021c), except that the temporal resolution is resized to 100 in our case instead of 200.
Classiﬁcation.
On Epic-Kitchens , we classify the proposals with the ﬁne-tuned model using 6 clips.
Spatially, to comply with the feature extraction process, we resize the shorter side to 224 and feed the whole spatial region to the model for classiﬁcation.
On HACS , considering the property of the dataset that only one action category can exist in a video, we obtain the video level classiﬁcation
Evaluation.
For evaluation, we follow the standard evaluation protocol used in the respective datasets, i.e., the average mean Average Precision (average mAP) at IoU threshold [0.5:0.05:0.95] for HACS (Zhao et al., 2019) and [0.1:0.1:0.5] for Epic-Kitchens-100 (Damen et al., 2020).
D M ODEL STRUCTURES
The detailed model structures for R2D, R(2+1)D and R3D is speciﬁed in Table A2.
We highlight the convolutions that are replaced by TAdaConv by default or optionally.
For all of our models, a small modiﬁcation is made in that we remove the max pooling layer after the ﬁrst convolution and set the spatial stride of the second stage to be 2, following Wang et al.
Temporal resolution is kept unchanged following recent works (Feichtenhofer et al., 2019; Li et al., 2020b; Jiang et al., 2019).
Our R3D is obtained by simply expanding the R2D baseline in the temporal dimension by a factor
[Figure] [Figure]
ac y
[Table] [Table]
of three.
We initialize with weights reduced by 3 times, which means the original weight is evenly distributed in adjacent time steps.
We construct the R(2+1)D by adding a temporal convolution operation after the spatial convolution.
The temporal convolution can also be optionally replaced by
TAdaConv, as shown in Table 3 and Table A3.
For its initialization, the temporal convolution weights are randomly initialized, while the others are initialized with the pre-trained weights on ImageNet.
For SlowFast models, we keep all the model structures identical to the original work (Feichtenhofer et al., 2019).
For TAdaConvNeXt, we keep most of the model architectures as in ConvNeXt (Liu et al., 2022), except that we use a tubelet embedding similar to (Arnab et al., 2021), with a size of 3 × 4 × 4 and stride of 2 × 4 × 4.
Center initialization is used as in (Arnab et al., 2021).
Based on this, we simply replace the depth-wise convolutions with TAdaConv to construct TAdaConvNeXt.
E P ER - CATEGORY IMPROVEMENT ANALYSIS ON SSV
This section provides a per-category improvement analysis on the Something-Something-V2 dataset in Fig.
A1.
As shown in Table 5, our TAda2D achieves an overall improvement of 31.7%.
Our per-category analysis shows an mean improvement of 30.35% over all the classes.
The largest improvement is observed in class 0 (78.5%, Approaching something with your camera ), 32 (78.4%,
Moving away from something with your camera ), 30 (74.3%, Lifting up one end of something without letting it drop down ), 44 (66.2%, Moving something towards the camera ) and 41 (66.1%, Moving something away from the camera ).
Most of these categories contain large movements across the whole video, whose improvement beneﬁts from temporal reasoning over the global spatial context.
For class 30, most of its actions lasts a long time (as it needs to be determined whether the end of something is let down or not).
The improvements over the baseline mostly beneﬁts from the global temporal context that are included in the weight generation process.
[Table]
[Table]
F F URTHER ABLATION STUDIES
Here we provide further ablation studies on the kernel size in the calibration weight generation.
As shown in Table A4a and Table A4b, kernel size does not affect the classiﬁcation much, as long as the temporal context is considered.
Further, Table A4c shows the sensitivity analysis on the reduction ratio, which demonstrate the robustness of our approach against different set of hyper-parameters.
G F URTHER PLUG - IN EVALUATION FOR TA DA C ONV ON CLASSIFICATION
In complement to Table 3, we further show in Table A3 the plug-in evaluation on the action classiﬁcation task on the Epic-Kitchens-100 dataset.
As in the plug-in evaluation on Kinetics and
Something-Something-V2, we compare performances with and without TAdaConv over three baseline models, SlowFast (Feichtenhofer et al., 2019), R(2+1)D
(Tran et al., 2018) and R3D (Hara et al., 2018) respectively representing three kinds of temporal modeling techniques.
The results are in line with our observation in Table 3.
Over all three kinds of temporal modelling strategies, adding
TAdaConv further improves the recognition accuracy of the model.
H P LUG - IN EVALUATION FOR TA DA C ONV ON A CTION L OCALIZATION
Here, we show the plug-in evaluation on the temporal action localization task.
Speciﬁcally, we use
SlowFast as our baseline, as it is shown to be superior in the localization performance in Qing et al.
(2021b)
compared to many early backbones.
The result is presented in Table A5.
With TAdaConv, the average mAP on HACS is improved by 1.4%, and the average mAP on Epic-Kitchens-100 action localization is improved by 1.0%.
I C OMPARISON OF TRAINING PROCEDURE
In this section, we compare the training procedure of TSN and TAda2D on Kinetics-400 and
Something-Something-V2.
The results are presented in Fig.
A2.
TAda2D demonstrates a stronger ﬁtting ability and generality even from the early stages of the training,
[Figure]
[Table]
J C OMPARISON WITH EXISTING DYNAMIC FILTERS
In this section
[Table]
we compare our TAdaConv with previous dynamic ﬁlters in two perspectives, respectively the difference in the methodology and in the performance.
J. C OMPARISON IN THE METHODOLOGY LEVEL
For the former comparison, we include Table A6 to show the differences in different approaches, which is a full version of Table 1.
We compare TAdaConv with several representative approaches in image and in videos, respectively CondConv (Yang et al., 2019), DynamicFilter (Jia et al., 2016),
DDF (Zhou et al., 2021) and TAM (Liu et al., 2021b).
The ﬁrst difference in the methodology level lies in the source of weights, where previous approaches obtain weights by mixture of experts or generation completely dependent on the input.
Mixture of experts denotes W = (cid:80) n α n W n , where α n is a scalar obtained by a function f , i.e., W = (cid:80) n f ( x ) n W n .
Completely generated means the weights are only dependent on the input, i.e.,
W = g
(( x )) , where g generates complete kernel for the convolution.
In comparison, the weights in TAdaConv are obtained by calibration , i.e,, W = α W b , where α is a vector calibration weight and
α = h (( x )) where h ( . ) generates the calibration vector for the convolutions.
Hence, this fundamental difference in how to obtain the convolution weights makes the previous approaches difﬁcult to exploit pre-trained weights, while TAdaConv can easily load pre-trained weights in W b .
This ability is essential for video models to speed up the convergence.
The second difference lies in the ability to perform temporal modelling.
The ability to perform temporal modelling does not only mean the ability to generate weights according to the whole sequence in dynamic ﬁlters for videos, but it also requires the model to generate different weights for the same set of frames with different orders.
For example, weights generated by the global descriptor obtained by global average pooling over the whole video GAP st does not have the temporal modelling ability, since they can not generate different weights if the order of the frames in the input sequence are reversed or randomized.
Hence, most image based approaches based on global de-
Calibration Generation
[Table]
scriptor vectors (such as CondConv and DynamicFilter) or based on adjacent spatial contents (DDF) can not achieve temporal modelling.
TAM generates convolution weights for temporal convolutions based on temporally local descriptors obtained by the global average pooling over the spatial dimension GAP s , which yields different weights if the sequence changes.
Hence, in this sense, TAM has the temporal modelling abilities.
In contrast, TAdaConv exploits both temporally local and global descriptors to utilize not only local but also global temporal contexts.
Details on the source of the weight generation process is also shown in Table A7.
The third difference lies in whether the weights generatd are shared for different locations.
For
CondConv, DynamicFilter and TAM, their generated weights are shared for all locations, while for
DDF, the weights are varied according to spatial locations.
In comparison, TAdaConv generate temporally adaptive weights.
J. C OMPARISON IN THE PERFORMANCE LEVEL
Since TAdaConv is fundamentally different from previous approaches in the generation of calibration weights, it is difﬁcult to directly compare the performance on video modelling, especially for those that are not designed for video modelling.
However, since the calibration weight in TAdaConv α is completely generated, i.e., α = f (( x )) , we can use other dynamic ﬁlters to generate the calibration weights for TAdaConv.
Since MoE based approaches such as CondConv were essentially designed for applications with less memory constraint but high computation requirements, it is not suitable for video applications since it would be too memory-heavy for video models.
Hence, we apply approaches that generate complete kernel weights to generate calibration weights, and compare them with TAdaConv.
The performance is listed in Table A7.
It is worth noting that these approaches originally generate weights that are randomly initialized.
However, as is shown in Table 6, our initialization strategy for the calibration weights are essential for yielding reasonable results, we further apply our initialization on these existing approaches to see whether their generation function is better than the one in TAdaConv.
In the following paragraphs, we provide details for applying representative previous dynamic ﬁlters in TAdaConv to generate the calibration weight.
For DynamicFilter (Jia et al., 2016), the calibration weight α is generated using an MLP over the global descriptor that is obtained by performing global average pooling over the whole input GAP st , i.e., α = MLP ( GAP st ( x )) .
In this case, the calibration weights are shared between different time steps.
For DDF (Zhou et al., 2021), we only use the channel branch since it is shown in Table 7 that it is better to leave the spatial structure unchanged for the base kernel.
Similarly, the weights in DDF are also generated by applying an MLP over the global descriptor, i.e.,
α = MLP ( GAP st ( x )) .
The difference between DDF and DynamicFilter is that for different time step, DDF generates a different calibration weight.
The original structure of TAM (Liu et al., 2021b) only generates kernel weights with its global branch, and uses local branch to generate attention maps over different time steps.
In our experiments, we modify the TAM a little bit and further make the local branch to generate kernel calibration weights as well.
Hence, for only-global version of TAM, the calibration weights are calculated as follows: α = G ( GAP s ( x )) , where GAP s denotes global average pooling over the spatial dimension and G denotes the global branch in TAM.
In this case, calibration weights are shared for all temporal locations.
For local+global version of TAM, the calibration weight are calculated by combining the results of the local L and the global branch G , i.e., α = G ( GAP s ( x ))
· L ( GAP s ( x )) , where · denotes element-wise multiplication with broadcasting.
This means in this case, the calibration weights are temporally adaptive.
Note that this is our modiﬁed version of TAM.
The original
TAM does not have a temporally adaptive convolution weights.
The results in Table A7 show that (a) without our initialization strategy, previous approaches that generate random weights at initialization are not suitable for generating the calibration weights in TAdaConv; (b) our initialization strategy can conveniently change this and make previous approaches yield reasonable performance when they are used for generating calibration weights; and (c) the calibration weight generation function in TAdaConv, which combines the local and global context, outperform all previous approaches for calibration.
Further, when we compare TAdaConv without global information with TAM (local*+global branch), it can be seen that although both approach generates temporally varying weights from the frame descriptors GAP s ( x ) with shape C × T , our TAdaConv achieves a notably higer performance.
Adding the global information enables TAdaConv to achieve a more notable lead in the comparison with previous dynamic ﬁlters.
[Figure]