ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning
Abstract
Capitalizing on large pre-trained models for various downstream tasks of interest have recently emerged with promising performance.
Due to the ever-growing model size, the standard full fine-tuning based task adaptation strategy becomes prohibitively costly in terms of model training and storage.
This has led to a new research direction in parameter-efficient transfer learning.
However, existing attempts typically focus on downstream tasks from the same modality ( e.g. , image understanding) of the pre-trained model.
This creates a limit because in some specific modalities, ( e.g. , video understanding) such a strong pre-trained model with sufficient knowledge is less or not available.
In this work, we investigate such a novel cross-modality transfer learning setting, namely parameter-efficient image-to-video transfer learning .
To solve this problem, we propose a new SpatioTemporal Adapter (ST-Adapter) for parameter-efficient fine-tuning per video task.
With a built-in spatio-temporal reasoning capability in a compact design, STAdapter enables a pre-trained image model without temporal knowledge to reason about dynamic video content at a small ( ∼ 8%) per-task parameter cost, requiring approximately 20 times fewer updated parameters compared to previous work.
Extensive experiments on video action recognition tasks show that our ST-Adapter can match or even outperform the strong full fine-tuning strategy and state-of-theart video models, whilst enjoying the advantage of parameter efficiency.
Code and model are available at https://github.com/linziyi96/st-adapter
Introduction
In the NLP field, almost all the state-of-arts across a wide range of downstream tasks have been achieved by adapting from large pretrained models (a.k.a.
foundation models [7]) such as BERT [16] and GPT [52, 8].
The de facto standard approach to adapting a pretrained model to down-stream tasks is fine-tuning either fully or partially ( e.g. , linear probing by training the newly added multi-layer perceptron layers on the top alone), subject to the condition of adopting a similar network architecture as the pretrained model.
Nonetheless, given increasingly larger whilst ever stronger foundation models ( e.g. , GPT-3 with 175B parameters), fully fine-tuning the whole model for every single downstream task would become prohibitively expensive and infeasible in terms of training cost and model storage.
This could significantly restrict their deployment and usability in real-world applications.
In this context, a series of NLP works has been introduced towards efficient transfer learning with better trade-offs between parameter and accuracy [27, 26, 39, 37].
This trend has recently motivated the computer vision community.
For example, the CLIP model [53], trained with 400 million web image-text pairs, achieves promising performances on a variety of
[Figure]
FeedForward
Add
[Figure]
Spatio-Temporal Adapter
[Figure]
Newly added
[Figure]
ViT [
[Figure]
Upsample image recognition and generation tasks.
In the video domain, with significantly more computational cost and resources, Xu et al.
[76] trained a video variant of CLIP but excelled on a smaller number of downstream video tasks.
This is partly attributed to two orders of magnitude more minor training data and limited availability of computing resources, as large video data is notoriously more difficult to collect, manage, and process than image data.
Under these restrictions, large pre-trained image models are arguably still favorable in the selection of model initialization for video tasks.
In this work, we investigate a novel, critical problem of efficiently adapting large pre-trained image models for video downstream tasks , with a focus on the widely influential action recognition task.
Considering that training video models is drastically more expensive in both computing resource and time than image models [22], this problem becomes particularly more useful and valuable in practice.
On the other hand, it is also more challenging and non-trivial due to the extra necessity of overcoming the big gap between image and video in transfer learning.
Especially, pre-trained image models lack the ability to infer temporal structured information, which however is critical in video understanding.
In fact, the key design with state-of-the-art video models [10, 41, 6, 9] is usually about learning the temporal dimension based on contemporary image models.
Although model initialization is still important, they largely go beyond the fine-tuning strategy, as architectural modification is often imposed in addition to full model training/fine-tuning per downstream task.
Given that this is a new problem, we first conduct a comprehensive benchmark using both various fine-tuning methods for image-to-video transfer learning and state-of-the-art video models [6, 9].
Regarding the pretrained image model, we select two Vision Transformer (ViT)
[18] models, with one from CLIP pre-training 53] and the other pre-trained on ImageNet-21K [15].
ViT is representative in terms of network architecture, pre-training algorithm, and training data scale.
Crucially, we further propose an efficient yet effective Space-Time Adapter (ST-Adapter), capable of extracting and leveraging the pre-trained knowledge of a large image model to achieve superior video understanding at a small parameter cost.
Specifically, ST-Adapter is formulated based on a novel parameter-efficient bottleneck with a sequence of operations including feature dimension reduction, spatial-temporal modeling, and feature dimension recovery.
It is easy to implement and scalable for deployment since all the primitive steps are realized with standard operators ( e.g. , fully-connected layer, depth-wise 3D convolution).
With such a lightweight design, our bottleneck can be cheaply integrated throughout the base network for enabling stronger layer-wise spatio-temporal learning.
As a result, our model can be more rapidly optimized using fewer training epochs for significant convergence advantage.
We summarize the contributions as follows.
(1) We investigate a new problem of parameterefficient image-to-video transfer learning .
Our motivation is to advocate the usability and deployment of increasingly larger whilst ever more powerful pre-trained image models in benefiting more challenging video understanding tasks.
(2) We establish a benchmark for action recognition tasks by comprehensively experimenting with a variety of fine-tuning strategies and several state-of-the-art video understanding models.
(3) We introduce a novel parameter-efficient Spatio-Temporal Adapter (ST-Adapter) for more effectively capitalizing a large pre-trained image model in video understanding.
By grounding all the primitives on standard operators, ST-Adaptor is easy to implement and friendly to deployment.
(4) Extensive experiments on action recognition datasets show that our ST-Adapter outperforms not only existing parameter-efficient alternatives and the full fine-tuning strategy, but also state-of-the-art video methods with the same network architecture and model initialization.
Related Work
Parameter-efficient transfer learning Driven by the wider application of large pre-trained language models across a diversity of downstream tasks, the topic of efficient tuning has received increasing attention in NLP.
Existing efficient tuning methods fall broadly into three categories.
The first category is to introduce task-specific adapters [27, 26, 50, 49].
Specifically, an adapter consists of lightweight modules inserted between layers of a pre-trained model.
To be parameter-efficient, only those newly added adapter modules need to be updated during task fine-tuning, whilst all the parameters of the large pre-trained model, which takes the majority proportion of the whole solution, are frozen.
The second category is prompt tuning [39, 51, 33, 59, 42].
Instead of manipulating the network architecture, these methods prepend a set of learnable tokens at the input point of the model or intermediate layers.
Similarly, only these added tokens need to be optimized for each downstream task.
The third category is learning weight approximation [29].
In particular, only the low-rank matrices for approximating the weights need to be updated during training.
Early works for efficient transfer learning in vision focus on parameter sharing in the context of multitask learning [79, 55, 54].
Recently, there are several works for extending the efficient tuning idea from NLP to vision tasks.
CoOp [81] and CoCoOp [82] apply prefix tuning for adapting the CLIP model to various image recognition tasks.
VL-Adapter [64] achieves the performance comparable to full fine-tuning on challenging vision-language tasks.
Commonly, their design focuses are all restricted to the text encoder of the CLIP model.
More recently, [31, 4, 80] introduce the idea of prompt learning to visual backbones.
They obtained favorable results on various image recognition benchmarks.
Moving a step further, in this work, we consider the more challenging adaptation problem from a pre-trained image model without temporal knowledge to video understanding tasks.
Video action recognition Action recognition in the unconstrained video has largely been dominated by deep learning methods, thanks to the availability of large video datasets, e.g. , Kinetics [10–12] and Something-Something [24].
As a key component, the model architectures adopted by existing video methods has expanded from CNNs [34, 67, 22, 21, 74, 69, 70, 46, 41, 45] to Transformers [20, 40, 38, 44, 2, 6].
As temporal information is important for modeling the dynamics, a variety of motion learning techniques has been introduced [72, 32, 48].
Further, different training methods have also been explored, e.g. , unsupervised learning [66, 23, 73], and video-text contrastive learning [63, 76, 75, 65].
New opportunities for stronger video models are created following the introduction of large pretrained foundation models [53, 30, 77].
For example, Wang et al.
[71] equipped the CLIP with temporal modules and good performance can be achieved after the model is fully fine-tuned on video datasets.
Ju et al.
[33] adopted the CLIP model for video recognition tasks by learning videospecific prompts.
In contrast, in this work, we explore the potential of the large pre-trained image models with the parameter-efficient adapter strategy.
Importantly, despite the simplicity, we bring about more significant advantages in performance along with a new benchmark on parameter-efficient image-to-video transfer learning.
Methodology
To capitalize a large pre-trained image model for more challenging video understanding such as action recognition in a cross-modality manner, it is necessary to fill the intrinsic gap between image and video.
For easier understanding, we start with an intuitive baseline based on temporal aggregation.
Temporal aggregation
A straightforward baseline method of exploiting a pre-trained image model for video understanding is to temporally aggregate per-frame feature representations ( e.g. , average pooling).
Concretely, given an input video clip V ∈ R T × H × W , where T, H, W are the number of frames, height and width respectively.
Following [18], we first split each frame into N = H × W/Ppatches of size P × P .
Then, we flatten these patches and project them into a sequence of patch tokens Z t =
[ z 1 , ...
z s , ..., z N ] , z s ∈ R d where d = 3 × P 2 with t = 1 , ..., T .
The sequence of feature vectors is then enhanced with the positional embedding by element-wise addition, along with a trainable class token concatenated.
Subsequently, we feed each sequence with N + 1 tokens to a stack of self-attention based blocks individually.
For each sequence we keep only the classification token
z clst .
We further perform temporal average pooling on the class tokens z final = 1 T (cid:80) t z clst to yield a compact representation for the whole clip.
We obtain the prediction by passing z final through a classifier.
As the sptial information is only naively averaged over time, it is also known as
Space-Only TimeSformer [6].
Spatio-temporal attention For more dedicated structural modeling in the time dimension with
ViTs, a mainstream approach in the video domain is to develop various spatio-temporal attention mechanisms by further imposing temporal attention on top [6, 2, 3, 9, 78, 25].
We choose two representative video ViT models, TimeSformer [6] and XViT [9], in our performance benchmark.
However, state-of-the-art video ViT models often need to fully fine-tuned per task, which is parameterinefficient , given that in this way we have to keep a separate copy of the whole fine-tuned model parameters for every single task.
3.1 Preliminaries
Our method is inspired by the Adapter [27] designed for parameter-efficient transfer learning in
NLP.
Specifically, the adapter module is composed of a down-projection linear layer followed by a non-linear activation function and an up-projection linear layer.
Formally, given an input feature matrix X ∈ R N × d at the i -th layer, the feature adaptation process can be written as:
[Equation]
where W down ∈ R d × r refers to the down projection layer, W up ∈ R
r × d the up-projection layer, and f ( · ) the activation function.
Note, that a residual summation is applied for preserving the information in input as required.
The idea of Adapter has been remarkably successful in NLP due to several advantages: (1) High parameter efficiency across tasks since only a small number of parameters are task-specific; (2) Reaching on-par performance compared to full fine-tuning; (3) Taking significantly small training costs; (4) Avoiding the catastrophic forgetting limitation of full fine-tuning.
We aim to propagate the success of Adapter from NLP to computer vision particularly the imageto-video transfer learning problem as discussed earlier.
To that end, we introduce a novel Adapter tailored specially for spatio-temporal reasoning – a key capability for video understanding which, however, existing NLP Adapter variants lack.
3.2 Spatio-Temporal Adapter (ST-Adapter)
Typically, an image model only considers the ability of spatial modeling.
The objective of our
Spatio-Temporal Adapter (ST-Adapter) is to enable a pre-trained image model to reason about spatial and temporal information of video in a parameter efficient principle.
In design, we consider a couple of practically-crucial criteria: (1) Smaller parameter size : The parameter cost for each downstream task should be small – the essential criterion for parameter efficiency.
(2) Development friendliness : This is critical for real-world development and deployment.
In practice, it is necessary that a model can be easily implemented using the standard highly optimized deep learning toolboxes ( e.g. , PyTorch
This also facilitates the realization of high inference efficiency across a diversity of running platforms due to the best usage of built-in software and hardware resources.
Under these considerations, we formulate the proposed ST-Adapter by sticking to commonly-adopted primitive operators alone .
Starting with the above Adapter (Eq.
(1)) originally developed for
NLP tasks, we further introduce a spatio-temporal operator realized by a standard depth-wise 3Dconvolution layer [21] between the bottlenecks (Figure 1).
In particular, our spatio-temporal operator enables layer-wise temporal inference efficiently, because it only operates in a compressed lowdimensional ( e.g. , 128D) feature space and the depth-wise convolution is highly efficient both in parameter and computation [28].
As a result, this yields an introduction of tiny extra ( ∼ 2%) parameters and ( ∼ 0.3%) computation.
Formally, our ST-Adapter can be expressed as:
[Equation]
where DWConv3D denotes the depth-wise 3D-convolution for spatio-temporal reasoning we introduce.
It is noteworthy that before applying DWConv3D , the down-projected feature representations will be first reshaped from X ′ ∈ R T × N × d to X ′′ ∈ R T × h × w × d (where N = h × w ) to have the spatial and temporal dimensions prepared for reasoning.
With this highly integrated design, our ST-Adapter enjoys the same efficiency and flexibility as the NLP Adapter, while uniquely being able to conduct spatio-temporal modeling. l.
3.3 ST-Adapter Integration
For proper adaptation, the adapter modules are often integrated between layers of a Transformer.
In NLP, a variety of integrating designs have been investigated.
For example, [27] deploys two adapter modules per layer with one following the Multi-Head Self-Attention (MHSA) and the other following the Feed-Forward Networks (FFN) [27].
On the other hand, [62, 5] suggest that adding only one adapter after the FNN suffices.
Similarly, our ST-Adapter can be also integrated generally at distinctive positions.
Empirically, we find that a decent performance can be achieved in case a single
ST-Adapter is placed before the MHSA of each transformer block (Figure 1(a) and Table 5c).
Experiments 4.1 Experiments Setup
Datasets For the benchmark experiments, we use two popular video action recognition datasets.
Kinetics-400 (K400) :
The K400 [35] dataset contains ∼ 240k training videos and 20k validation videos labeled with 400 action categories.
Most videos have a length of 10s or about 300 frames.
While there is a great diversity in these videos, they are largely biased to spatial appearance [58].
Something-Something-v2 (SSv2) :
The SSv2 [24] dataset consists of 220,487 videos coveringhuman actions.
The video length ranges from 2 to 6 seconds.
In contrast to K400, SSv2 presents richer temporal information with much higher significance [58].
Epic-Kitchens-100 (EK100) :
The EK100 [14] dataset consists of 100 hours of video in egocentric perspective recording a person interacting with a variety of objects in the kitchen.
Each video sample is labeled with a verb and a noun.
We report top-1 verb and noun classification accuracy.
Pre-trained models In all experiments, we use the standard ViT [18] as our base backbone model.
We conduct most of our experiments with the ViT-B/16 variant with 12 layers and 86M parameters, taking as input a sequence of patches at size 16 × 16 .
What was learned during pre-training directly decides the knowledge that can be transferred to downstream tasks, thus also the effectiveness upper bound of transfer learning methods.
To this end, we benchmark the same backbone under two different pre-training strategies: pre-training with web-scale raw data that has been recently proposed by CLIP [53] (400M image-text pair) and classical supervised pre-training on annotated data from ImageNet-21K (21k classes and 14M images).
Implementation details.
All details, including training and testing settings and module instantiation details, are provided in the appendix.
Competitors We provide several transfer learning approaches in our benchmark for efficient imageto-video transfer learning.
Note that the parameters of the linear classifier are always updated during training for all approaches.
(1) Full Fine-tuning : Fully updating all the parameters when adapting for a specific target task.
(2) Partial Fine-tuning : Only update the last ViT layer while keeping the rest of the parameter fixed.
(3) Temporal Fine-tuning : We only tune the temporal attention modules ( i.e., TA) in the SA+TA architecture.
(4) Linear Probing :
Freezing all the parameters except those in the linear classification layer.
(5) Adapter [27]: Adding small sub-networks between layers of a pre-trained model.
During fine-tuning, we only update the newly added parameters introduced by the adapters.
(6) Prompt Tuning [31]: Prepending a sequence of learnable prompt tokens to the input visual patch tokens.
During fine-tuning, only these newly added prompts are updated.
(7) Attention Pooling Head : Replacing the original temporal average pooling with a temporal attention pooling layer (similar to the one used in [9]) before the classification head.
These approaches above do not incorporate temporal modeling to the image
ViT.
Hence, we further consider temporally augmented ViT architectures as introduced in state-of-the-art video methods:
(a) Spatial Attention Only (SA) : Space-Only TimeSformer [6].
(b) Spatial Attention + Temporal Attention (SA+TA) :
The default TimeSformer [6] with divided space-time attention (Fig. 1a).
(c) Spatial Attention + Temporal Shift (SA+TS) : XViT
Note that not all fine-tuning protocols are compatible with each of these video ViT variants.
Take
SA+TS for example, the original model behavior is altered with channel shift, as a result, it is not compatible with Linear Probing that requires freezing all the parameters of the backbone.
4.2 Main Results and Analysis
Cross-modality fine-tuning benchmark.
Table 1 presents the results of fine-tuning a ViT-B/ pre-trained with CLIP and ImageNet-21K. All baselines are built by combining existing efficient fine-tuning methods with three state-of-the-art ViT-based action recognition models.
From the results we can see that:
(i) For CLIP pre-trained model, ST-Adapter performs on par with Full Fine-tuning ( 82.0 vs. 81.
for K400 and 66.3 vs. 66.1 for SSv2) while updating far less parameters ( 7.2M vs. 121.57M).
ST-Adapter significantly outperforms all other efficient fine-tuning methods.
We see that baselines like Prompt Tuning and Partial Fine-tuning can provide non-trivial gain in performance compared to Linear Probe, but are still behind our ST-Adapter.
(ii) Our ST-Adapter can generalize across different pre-training datasets and methods.
We can see that CLIP pre-train models dominate over ImageNet-21K pre-train ones.
These results well match the shift of paradigm in current AI research [7], where pre-training no longer needs limiting to curated data and annotations to deliver good performance on downstream tasks, but can take advantage of broader scale web raw data.
Interestingly, we observe that SSv2, a motion-centric dataset in design, also benefits from stronger appearance (image) pre-training.
We think this may attribute to that raw textual description can provide a much richer description ( i.e. , human-object relations) of the image than curated limited categorical labels.
Full fine-tuning on SA+TS (XViT) performs slightly worse with CLIP pretrain than ImageNet-21k pretrain.
We conjecture this is because the channel shift operation breaks the knowledge in the pre-training weights, and thus does not benefit much from stronger pre-training like
CLIP.
Comparison to the state-of-the-art models.
We compare ViT with ST-Adapter to other state-of-thearts methods on both K400 dataset [35] in Table 2, SSv2 dataset [24] in Table 3 and EK100 dataset [14] in Table 4.
We can observe that:
(i) With the proper adaptation method, we can simply turn a large image foundation model into a good video model by only tuning a few parameters.
Our results are comparable to or better than previous
[Table] [Table]
methods tailored for such tasks.
Our largest model with ViT-L backbone set a new state-of-the-art in
K400 by achieving 86.7% top-1 accuracy.
(ii) It is noteworthy that, our method takes significantly fewer frames as input compared to other methods (8 vs. 16, 32, 64, 96).
It is also reflected in terms of GFlops.
Saying that the ViT was not designed for efficiency purposes like [38, 9, 43, 20] but the adapted CLIP ViT has achieved similar accuracy-efficiency trade-offs.
(iii
The paradigm of pre-training and fine-tuning has been widely adopted in most state-of-art methods to achieve good performance.
Between them, most of the approaches start from image pre-trained models, and only a few can afford video pre-training.
Note that for the SomethingSomething dataset, except MViT [20] pre-trained on video data from scratch, the rest of methods are still initialized from image pre-trained weights.
A good image pre-trained model with rich appearance information can facilitate temporal modeling in temporally challenging datasets like SSv2.
iii
It is evident in Table 4 that our ST-Adapter consistently brings a big margin on egocentric videos.
Also, we found that without our ST-Adapter, it is much more difficult to directly adapt
CLIP pre-trained ViT on the domain of egocentric video with high sensitivity to the hyper-parameter setting.
ST-Adapter eases the training process.
It is worthy to note that, all current transformer based approaches need to be pre-trained first on image dataset and then fine-tuned on Kinetics dataset before fine-tuned with egocentric videos.
In contrast, our ST-Adapter can be directly applied to an image model and trained with target egocentric video alone.
4.3 Ablations
Unless otherwise specified, we use ViT-B/16 backbone and 8 input frames in all ablation experiments, and we use one ST-Adapter with bottleneck width 384 before MHSA in each Transformer block.
Where to insert ST-Adapter By default, we insert a ST-Adapter to every Transformer block in the backbone, but we also show the performance impact of using fewer ST-Adapters.
As shown in Table 5b, while more ST-Adapters tend to do better, ST-Adapters at deeper layers boost performance more
[Table]
“GFlops” means 10 9 Flops.
Our fine- a very small
[Table]
Methods
[Table]
Our
[Table]
-L w
[Table]
ST-Adapter
[Table]
than those at shallower layers.
This observation is useful when we insert ST-Adapters into deeper models and having an Adapter for each block might be too expensive.
We also show the performance when inserting ST-Adapters to different positions within a block.
As shown in Table 5c, while the performance is relatively insensitive to the position of the Adapters, using multiple adapters in one block may substantially boost performance on some datasets, like SSv2 in our case.
Training parameter efficiency We experiment with a different number of channels in the middle of the bottleneck design.
As shown in Table 5a and Fig.
2a, our method is effective with a wide range of bottleneck width: even with a channel reduction to 64, our ST-Adapters still obtain relatively good performance, outperforming all baselines in Table 1 except for Full Fine-tuning (SA + TA).
Even with a bottleneck width of 768, our ST-Adapters are still very parameter efficient, introducing only about 1/6 new parameters to a Transformer encoder block.
In contrast to the inverted bottleneck design commonly used with depthwise convolutions [57], ST-Adapters work best with regular bottlenecks.
The success of transfer learning with such low-rank projections again shows the rich knowledge and strong potential of modern foundation models. [Table] [Table] [Figure]
[Table]
Training time efficiency In Fig.
2b we show an enlarged difference between full fine-tuned models and our ST-Adapters with low training budgets.
When we reduce the number of training steps, the accuracy of full fine-tuned models drops significantly faster than models with ST-Adapters.
This shows the advantage of our proposed modules when backbone models are large or computational resources are limited.
We also report the total training GPU-hours and peak memory usage for three models: TimeSformer, ViT-B/16, ViT-B/16 with ST-Adapter (8 input frames, 16 samples per GPU on 8 V100 GPUs) in Table 6.
Training data efficiency Fig.
2c showcases the impact of training data size on action recognition accuracy.
Even with the same pre-trained weights, ST-Adapters tend to obtain higher performance than full fine-tuning especially on smaller datasets: the margin between the two models increases with the shrinkage of data.
This shows that ST-Adapters are powerful tools to transfer to downstream tasks where only a small amount of labeled data is available.
Effects of kernel shape We ablate the effect of kernel size in the depth-wise convolutions inside our proposed ST-Adapter.
It is shown in Table 7 that the temporal span is most sensitive, suggesting the significance of temporal structural modeling as we focus on in this work.
[Table]
Conclusions
[Table]
In this work
[Table]
we have presented a simple yet effective SpatioTemporal Adapter (ST-Adapter) for enabling a less studied parameter-efficient image-to-video transfer learning.
Fully using commonly adopt primitive operators, ST-Adapter is particularly designed to be both lightweight and easy to implement for friendly usability and deployment.
This cross-modality adaptation is a practically critical capability considering that it is dramatically challenging and more costly to build a sufficiently strong large video model in reality.
Encouragingly, extensive experiments on video action recognition show that our STAdapter can match or surpass both the full fine-tuning strategy as well as fully trained state-of-the-art video models, whilst having the benefit of (20 times less updated parameters) parameter-efficiency.
Further, our method is also faster to train and consumes less computing resources with economic and environmental superiority.
We believe this work is inspiring for the research of other video understanding tasks such as action localization and video summarization.
Acknowledgement This work is supported in part by Centre for Perceptual and Interactive Intelligence Limited, in part by the General Research Fund through the Research Grants Council of Hong
Kong under Grants (Nos. 14204021, 14207319).
A Appendix
Implementation details of our method All experiments are implemented in PyTorch [47].
We use the configuration listed in Tab. 8 unless otherwise specified.
In general, we use much simpler data augmentation techniques compared to end-to-end fine-tuning.
Hyper-parameters were briefly tuned to ensure convergence on a 20% held-out validation set.
[Table]
Baseline implementation details The training configuration used for all the baselines is summarized as follows:
• Full Fine-tuning : we largely follow the training configuration provided in their original paper, except that we train all the CLIP initialized layers with 1/100 learning rate and weight decay.
We found these changes are necessary to obtain reasonable results for CLIP pretrained models; Otherwise the accuracy on Kinetics-400 is less than 50%.
We found 1/100 to be the best scaling among { 1 / 10 , 1 / 100 , 1 / 1000 } on Kinetics-400.
• Partial Fine-tuning : we finetune only the last Transformer block and the classifier layer.
For the SA+TA architecture, TA is only added to the last block since the previous blocks need to be frozen in a meaningful state.
We use the identical training configuration as provided in the original paper ( i.e. , without reduction of learning rate or weight decay for any trainable weight) as we found it slightly improves accuracy for this baseline.
• Other baselines use the same training configuration as our proposed method, as stated in the Implementation details section in the main manuscript.
Experiments with other foundation models It is observed that with the same model, CLIP pretraining is superior to ImageNet21K pre-training (not surprising due to the training data scale and richness difference).
However, our main objective is to propose a parameter-efficient fine-tuning alternative to the standard full fine-tuning approach particularly for image-to-video adaptation.
To that end, we have validated the effectiveness and efficiency of turning an image foundation model into strong video action recognition models by tuning only a small fraction of parameters, in comparison to previous state-of-the-art alternatives.
By reporting the results on two different pre-training datasets (i.e., ImageNet21K and CLIP datasets), we would like to demonstrate that our ST-Adapter can generalize across different pre-training datasets and methods.
Moreover, it can shed light on the difference between a foundation model (pretrained with noisy web-scale raw data) and an ImageNet pre-trained model (which has been standard pre-training over the last decade).
To further support our finding, we have also experimented with the latest SWAG [60] foundation model.
As seen in Table 9, our ST-Adapter with a SWAG model can achieve consistent results as with a CLIP model: Reaching similar accuracy in the same tendency whilst outperforming the strong full fine-tuning strategy on both action datasets.
[Table]
Experiments on additional backbone architectures we have additionally provided the results of ST-Adapters on Swin-B models in Table 10.
The results of Swin space only and Swin joint attention are obtained with the training configure of [44] but using (8 frames x 3 views) sampling setting.
Although they are not directly comparable with the results reported in [44] (32 frames × 12 views for K400, 32 frames × 3 views for SSv2), they are highly indicative within reasonable range.
It is expected that on ImageNet-21k pretrained models our ST-Adapters underperforms full fine-tuning, especially when the locality inductive bias of Swin makes tuning on the downstream tasks easier.
However, our ST-Adapter still exhibits strong temporal learning capability, matching the joint-attention Swin and outperforms space-only Swin by a large margin.
Also, we observe higher data efficiency with our ST-Adapter:
The Swin joint attention model on the SSv2 dataset relies on
K400 pretraining (directly fine-tuning from ImageNet-21k results in slightly less than 60% accuracy).
In contrast, Swin w/ ST-Adapter achieves 65.1% even when directly trained from ImageNet-21k weights.
Note, we primarily aim at adapting foundation image models[7] pretrained on larger datasets (e.g., CLIP) other than ImageNet-21k.
[Table]
Inference Speed We provide an inference speed test in Table 11.
We measure the latency at batch size = 1 and throughput at batch size = 32.
It is shown that our model performs slightly lower than
TimeSformer space only, indicating that just a small overhead is introduced in inference speed by
ST-Adapter.
[Table]
UCF-101 and HMDB-51 We verify our method on two additional smaller but also widely studied video recognition datasets, namely UCF-101 [61] and HMDB-51 [36].
For both cases, we finetune from a Kinetics-400 [10] pretrained model, with all CLIP layers fixed and ST-Adapters set to 1/ learning rate and weight decay, and train for 500 steps with a batch size of 128.
Frames are sampled with a temporal stride of 8.
All other training settings are identical to that used for Kinetics-400.
For testing, we use 3 spatial views and 2 temporal views, and report the 3-split mean accuracy for both datasets.
We compare with methods that take only RGB frames as input (without optical flow).
The results are shown in Table 12.
We observe similar top performance by our ST-Adapter in comparison to recent state-of-the-art competitors, including the latest CLIP based VideoPrompt by a large margin.
[Table]
Visualization We provide qualitative results about the attention map change before and after adding the ST-Adapters in Fig.
Videos are sampled from Something-Something-v2 dataset and the attention map of the [CLS] token from the last Transformer block is shown.
The visualization shows that with ST-Adapters, the model attends more to action related regions ( e.g. , hands, fore-ground objects or moving objects), while the CLIP model without adaptation tend to be distracted by the background.
f r a m e .
b e f o r e .
a d a p t .
a f t e r .
a d a p t .
Spinning [something] .
that quickly stops spinning .
f r a m e .
b e f o r e .
a d a p t .
a f t e r .
a d a p t .
Showing [something] .
on top of [something] .
f r a m e .
b e f o r e .
a d a p t .
a f t e r .
a d a p t .
Pushing [something] so that .
it falls oﬀ the table .
f r a m e .
b e f o r e .
a d a p t .
a f t e r .
a d a p t .
Throwing [something] .
f r a m e .
b e f o r e .
a d a p t .
a f t e r .
a d a p t .
[Something] falling like .
a feather or paper .
f r a m e .
b e f o r e .
a d a p t .
a f t e r .
a d a p t .
Bending [something] so that it deforms .
f r a m e . b e f o r e .
a d a p t . a f t e r .
a d a p t . Moving [something] up .
f
r a m e .
b
[Table]
f
[Table]
e .
a
[Table]
t .
a f
[Table]
e r .
a d
[Table]
t .
Folding [something] .
f
[Table]
a m e .
b
[Table]
f
[Table]
e .
a
[Table]
t .
a f
[Table]
r .
a d
[Table]
p t .
Covering [something] with [something] .
f r a m e .
b e f o r e .
a d a p t .
a f t e r .
a d a p t .
Tearing [something] into two pieces .
f r a m e .
b e f o r e .
a d a p t .
a f t e r .
a d a p t .
Holding [something] next to [something] .
f r a m e .
b e f o r e .
a d a p t .
a f t e r .
a d a p t .
Taking [something] from [somewhere]