Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning
Abstract
Existing fine-tuning methods either tune all parameters of the pre-trained model (full fine-tuning), which is not efficient, or only tune the last linear layer (linear probing), which suffers a significant accuracy drop compared to the full fine-tuning.
In this paper, we propose a new parameter-efficient fine-tuning method termed as
SSF, representing that researchers only need to Scale and Shift the deep Features extracted by a pre-trained model to catch up with the performance of full finetuning.
In this way, SSF also surprisingly outperforms other parameter-efficient fine-tuning approaches even with a smaller number of tunable parameters.
Furthermore, different from some existing parameter-efficient fine-tuning methods ( e.g. , Adapter or VPT) that introduce the extra parameters and computational cost in the training and inference stages, SSF only adds learnable parameters during the training stage, and these additional parameters can be merged into the original pre-trained model weights via re-parameterization in the inference phase.
With the proposed SSF, our model obtains 2.46% (90.72% vs. 88.54%) and 11.48% (73.10% vs .
65.57%) performance improvement on FGVC and VTAB-1k in terms of Topaccuracy compared to the full fine-tuning but only fine-tuning about 0.3M parameters.
We also conduct amounts of experiments in various model families (CNNs,
Transformers, and MLPs) and datasets.
Results on 26 image classification datasets in total and 3 robustness & out-of-distribution datasets show the effectiveness of
SSF.
Code is available at https://github.com/dongzelian/SSF .
Introduction
With the popularity of the data-driven methods in the deep learning community, the dataset scale and the model size have both got huge explosions.
There is a tendency to explore large models and then adopt these pre-trained models in downstream tasks to achieve better performance and faster convergence, which gradually becomes a common way.
However, the current procedure depends on full fine-tuning heavily, where all the parameters of the model are updated.
It inevitably causes the model to be over-fitted to the small target dataset and thus cannot be used for other tasks after the fine-tuning.
As a result, the device will need to save a dedicated set of model parameters for each task, which causes a huge amount of storage space, especially for today’s large models ( e.g. , ViT-G/14 [16] 1.8G, CoAtNet [10] 2.4G).
A simple solution for the above problem is linear probing [26], where only the last head layer is fine-tuned.
However, this practice usually yields inferior performance compared to the full fine-tuning proxy.
Motivated by the success of the parameter-efficient fine-tuning strategy with prompt in the field of natural language processing (NLP) [36, 49, 38, 30], the recent work implements a similar proxy on vision tasks [44], termed as Visual Prompt Tuning (VPT).
Specifically, VPT [44] proposes to insert learnable prompts as inputs and append them to the original image tokens.
These prompts
Method
Acc.
Params.(M)
Unified parameterspace
Noextra inferenceparams.
Fullfine-tuning 93.
✓ ✓ Linearprobing 88.
✓ ✓
Adapter[36] 93.
✓ × VPT[44] 93.
× × SSF(ours) 93.
✓ ✓ will interact with the image tokens by performing self-attention and are updated during the fine-tuning process.
In this manner, a significant performance improvement can be achieved in downstream tasks compared to a linear probing proxy.
Nevertheless, compared to the full fine-tuning and linear probing, it additionally raises two issues: i) VPT tunes the number of prompts for different tasks which introduces a task-dependent learnable parameter space.
The fine-tuning performance is sensitive to the number of prompts for each task and needs to be carefully designed.
Too few or too many prompts might either degrade the accuracy of fine-tuning or increase the redundancy of the computation ( e.g. , 200 prompts on Clevr/count vs. 1 prompt on Flowers102); ii) VPT [44], as well as other
Adapter-based methods [36, 60], introduces additional parameters and computational cost in the inference phase compared to the original pre-trained model.
For instance, VPT introduces additional inputs for self-attention with image tokens.
Adapter-based methods insert additional modules into the pre-trained model.
These methods change the specific network architecture or the input of the network, which might result in frequent structure modifications and heavy workload, especially for those models that are already deployed in edge devices ( e.g. , mobile phones).
To cope with the above issues, we attempt to find a general proxy for parameter-efficient finetuning, where the learnable parameter space is unified (task-independent) and no additional inference parameters are introduced.
Inspired by some feature modulation methods [80, 40, 66], we propose a new parameter-efficient fine-tuning method named SSF, where you only need to Scale and Shift your deep Features extracted by a pre-trained model for fine-tuning.
The intuition behind our approach come from the fact that the upstream datasets and downstream datasets have different data distributions [71].
Therefore, it is difficult to apply the model weights trained in the upstream dataset to the downstream dataset.
For instance, a naive linear probing strategy with keeping the weights of backbone frozen will cause performance degradation.
To alleviate the above problem,
SSF introduces scale parameters and shift parameters, which could be considered as variance and mean to modulate the features of the downstream dataset extracted with the pre-trained model on the upstream dataset, such that the modulated feature falls in a discriminative space.
These scale parameters and shift parameters do not depend on any input and have a unified learnable parameter space for different tasks.
Another advantage of SSF is that it only introduces linear transformations because we scale and shift the extracted features.
These linear transformations could be further merged into the original pre-trained weight via model re-parameterization [15] in the inference phase, thus avoiding the extra parameters and FLOPs for downstream tasks.
For a deployed model in edge devices, only the updated weights after fine-tuning need to be uploaded instead of changing the network architecture.
Table 1 shows the specific characteristics comparisons between SSF and other fine-tuning methods.
SSF is simple, effective, and efficient, which also conforms to Occam’s Razor principle.
Therefore, we explore this new baseline and find that it surprisingly outperforms all other parameter-efficient fine-tuning methods.
We evaluate our method on 26 classification datasets in total and SSF obtains state-of-the-art performance compared to other parameter-efficient fine-tuning methods with the trainable parameters and accuracy trade-off (Table 1 and Figure 1).
Compared to the full fine-tuning, our method obtains 2.46% (90.72% vs. 88.54%) and 11.48% (73.10% vs .
65.57%) performance improvement on FGVC and
VTAB-1k in terms of Top-1 accuracy but only with about 0.3M trainable parameters.
Furthermore, our SSF does not require additional parameters during the inference phase.
It is plug-and-play and is very easy to extend to various model families (CNNs, Transformers, and MLPs).
Our SSF establishes a new baseline and we hope that it brings more insight into the field of the efficient model tuning.
Related Work 2.1 Model Families
Convolution has been used for a long time as the main module to extract the image features in computer vision tasks, and CNN-based architectures have been studied [70, 29, 83, 57, 84, 55, 87] with extension on graph-based data [86, 85, 54].
Recently, another architecture family, Transformer, has gained widespread attention owing to its great success in NLP [77, 13, 38].
Following this direction, Dosovitskiy et al.
[16] first employ a transformer in the domain of computer vision and introduce a new architecture paradigm, ViT, which achieves promising results [88, 69].
Subsequently, various transformer-based models, such as DeiT [74] and Swin Transformer [56], are introduced and shown to be effective on a variety of tasks such as object detection, semantic segmentation, action recognition [58], etc .
In another line, Tolstikhin et al.
[73] propose a pure MLP-based architecture, and subsequent papers [35, 50] have interestingly demonstrated that the MLP-based architectures can catch up to transformers.
However, in addition to the well-designed modules, their excellent performance is also attributed to the deployment of large-scale models.
Given a large-scale model pre-trained on a large dataset, how to perform parameter-efficient fine-tuning in downstream tasks is essential but is currently less explored.
In this paper, we propose SSF as a new baseline and show its promising performance with comprehensive validation in a wide variety of tasks.
2.2 Pre-training and Fine-tuning
Early models [29, 39, 37, 83, 72] are usually pre-trained on the ImageNet-1K dataset, and then fine-tuned on downstream tasks to achieve faster convergence [27] or better performance.
Such a procedure is called pre-training and fine-tuning, or transfer learning.
Recent works tend to employ larger models ( e.g. , ViT [16] and Swin Transformer V2 [56]) and train them on larger datasets ( e.g. , ImageNet-21K and JFT-300M) in pursuit of better performance.
Both in the domains of NLP and computer vision, these large models [13, 56, 68, 25, 94, 95] achieve enormous performance improvements compared to the small-scale models and provide pre-trained weights for downstream tasks.
Some other works attempt to explore how to efficiently fine-tune the pre-trained models [23, 96] on the target tasks.
For instance, given a target task, SpotTune [23] investigates which layers need to be fine-tuned.
Touvron et al.
[75] find that fine-tuning the weights of the attention layers and freezing weights of the other parts is sufficient to adapt the vision transformers to other downstream tasks.
Some works also propose to insert adapters into the network to fine-tune in a parameter-efficient way.
These adapters can be a small non-linear network [36], a hyper-network that generates model weights [61], or a compactor [60] which performs a low-rank decomposition to reduce the parameters.
Some works have also tried to only update the bias term [3, 90].
More recently, VPT [44] proposes to insert a small number of learnable parameters (prompts) and optimize them while freezing the backbone, which achieves significant performance improvement compared to the full fine-tuning.
During the submission of this work, some methods [5, 92] are also proposed for parameter-efficient fine-tuning, e.g. , inserting a adapter module or neural prompt search.
Different from all the above works, we propose to scale and shift deep features extracted by a pre-trained model, which is simple but effective and outperforms other parameter-efficient fine-tuning methods.
2.3 Feature Modulation
Many works have attempted to modulate features to obtain better performance.
The most relevant ones to our work are various normalization methods [41, 1, 80].
BN, LN, and GN usually normalize the features and then transform them linearly with scale and shift factors to modulate feature distribution, which has been verified to be effective in amounts of tasks.
STN [43] introduces a learnable module to spatially transform feature maps.
In the field of image generation, AdaIN [40] generates scale and shift factors to characterize specific image styles.
Self-modulation [6] shows GANs benefit from self-modulation layers in the generator.
In vision-language tasks, Conditional BN [11] and
FiLM [66] are often utilized to modulate the features of two modalities.
Unlike some algorithms such as BN, our SSF is not limited to the modulation of normalization layer, and it has a different motivation that is to alleviate the distribution mismatch between upstream tasks and downstream tasks for parameter-efficient fine-tuning.
As a comparison, we also conduct experiments in Sec. 4.3 and show that our SSF is more effective compared to only tuning the normalization layer.
Compared to
STN, AdaIN, FiLM and so on, our method is input-independent and these scale and shift parameters model the distribution of the whole dataset so that they can be absorbed into the original pre-trained model weights in the inference phase.
2.4 Model Re-parameterization
Model re-parameterization has been a common practice to improve inference efficiency.
One of the representative techniques is batch normalization folding used in the model compression algorithms [42].
The parameters introduced by the batch normalization layers [41] are merged into the convolutional layers usually stacked before them.
This technique is further utilized to merge different branches of networks into a new branch [89, 15, 14].
Similarly, our SSF fully adopts linear transformations, which allows the scale and shift parameters in the training phase to be merged into the original pre-trained model weights, thus avoiding the introduction of the extra parameters and computational cost during the inference phase.
Approach 3.1 Preliminaries
Transformers.
In a vision transformer (ViT) [16], an RGB image I ∈ R 3 × H × W is divided into
N × N non-overlapping patches, and then these image patches appended a class token are fed into an embedding layer followed by the L -layer vision transformer blocks with self-attention as the core operation.
The input x ∈ R ( N 2 +1) × d , where d is the embedding dimension, is first transformed to keys K
[Equation]
( N
[Equation]
+
[Equation] [Equation]
d , values
[Equation] [Equation]
+1)
× d , and queries Q ∈ R ( N 2 +1)
× d .
After that, we can calculate a global self-attention by
[Equation] [Equation]
The output of the attention layer will be fed to a two-layer MLP to extract information in the channel dimension.
Adapter.
Adapter [36] is inserted into the transformer layer for efficient fine-tuning.
It is a bottleneck module with a few trainable parameters, which contains a down-projection to reduce the feature dimension, a non-linear activation function, and an up-projection to project back to the original dimension.
Therefore, given the input x ∈ R ( N 2 +1) × d , the output is calculated by
[Equation]
where W down ∈ R d ′ × d (where d ′ ≪ d ), ϕ , and W up ∈ R
d
×
d ′ represent the down-projection matrix, non-linear function, and up-projection matrix, respectively.
VPT.
VPT [44] inserts some learnable parameters ( i.e. , prompts) into the input space after the embedding layer.
These prompts interact with the original image tokens by performing self-attention.
During the fine-tuning, the weights of the backbone network are kept frozen and only the parameters of the prompts are updated.
VPT-Shallow inserts prompts in the first layer while VPT-Deep inserts prompts in all the layers of the transformer.
Assuming that the input is x ∈ R ( N 2 +1)
× d , denote the inserted prompts as p ∈ R n × d , where n is the number of prompts, the combined tokens x ′ is
[Equation]
(3) where x ′ ∈ R ( N 2 + n +1)
× d will be fed into the transformer block for self-attention (Eq.
3.2 Scaling and Shifting Your Features for Fine-tuning
Different from the above methods, we introduce both the scale and shift factors to modulate deep features extracted by a pre-trained model with linear transformation to match the distribution of a target dataset, as mentioned in Sec. 1.
Five main properties are covered in our method: i) SSF achieves on-par performance with the full fine-tuning strategy; ii) all downstream tasks can be inputted to the model independently without relying on any other task; iii) the model only needs to fine-tune very few parameters; iv) unlike VPT [44], which adjusts the number of prompts for each task, the set of parameters for fine-tuning in SSF does not change as the task changes, making it feasible to further fine-tune the parameters later by adding more tasks for multi-task learning or continuous learning 2 ; v) thanks to the linear transformation, SSF avoids the introduction of the extra parameters and computational cost during the inference phase, making our method zero overhead.
[Figure]
The design of SSF.
SSF performs the linear transformation to modulate the features for parameter-efficient fine-tuning as shown in Figure 2.
In Figure 2 (a), given a model pre-trained in the upstream task, we insert SSF-ADA 3 after each operation (OP) of the network to modulate features.
There are K OPs in total and these operations might contain multi-head self-attention (MSA), MLP and layer normalization (LN), etc .
During the fine-tuning, the pre-trained weights in these operations are kept frozen and the SSFADA parameters are kept updated.
The specific SSF-ADA structure is shown in Figure2 (c), where the features output from the previous operation are performed dot product with a scale factor and then summed with a shift factor, which are input-independent.
Formally, given the input x ∈ R ( N 2 +1) × d , the output y ∈ R ( N 2 +1) ×
d (is also the input of the next operation) is calculated by
[Equation]
(4) where γ ∈ R d and β ∈ R
d
are the scale and shift factors, respectively.
⊙ is the dot product.
Re-parameterization.
Since SSF-ADA is a completely linear transformation, we can re-parameterize it by absorbing the scale and shift terms into the previous linear layer as follows
[Equation]
where w and b are the weight and bias terms, respectively.
∗ represents the ‘convolution’ operation in the convolutional layer or the ‘multiplication’ operation in the MLP layer.
t is the input of the previous linear layer.
Since w and b are frozen and γ and β are updated in the fine-tuning, γ and β can be merged into the original parameter space ( w and b ) in the inference stage through the above formulation.
From this perspective, our SSF-ADA makes it possible to perform downstream tasks without adding any extra parameters and computational costs, as shown in Figure2 (b).
Discussion.
The first question is why we want the input γ and β to be input-independent.
As FiLM [66] and AdaIN [40] show, we could obtain γ and β by conditioning an image sample, however, this might cause two shortcomings.
One is that we want γ and β to be input-independent to represent the distribution of the whole downstream dataset so that we can modify the previous weight distribution to fit the downstream dataset by modulating the feature.
Secondly, the conditional input requires the introduction of some additional networks ( e.g. , MLPs) to generate γ and β , which introduces more trainable parameters.
More importantly, to better generate γ and β , a non-linear activation function might be required, which will lead to the intractability of the re-parameterization.
Therefore, we directly perform a fully linear transformation to merge the γ and β factors into the original pre-trained weights, so that weights can be easily uploaded to the edge devices without any modification of the architecture.
The second question is which operations should be followed by SSF-ADA.
Our experience is that you can insert SSF-ADA after each operation with a linear coefficient in ViT. Although we can search for some optimal layers or operations with Neural Architecture Search (NAS) [67, 53, 24, 51], to reduce the number of the trainable parameters, we believe that our method will produce better results (or not worse than NAS) without introducing too many trainable parameters that can be merged for inference, as will be shown in Sec. 4.3.
3.3 Complexity Analysis
We also compare the complexity of Adapter, VPT and our SSF.
Take a ViT as an example, the dimension and number of the tokens are d and N 2 .
Assuming that Adapter projects features from d -dim to d ′ -dim (where d ′ ≪ d ) so that the extra trainable parameters are 2 dd ′ in each layer,
Method
Adapter
VPT-Shallow
VPT-Deep
SSF(ours)
#ExtraParams.
2 Ldd ′ (1) nd (1) nLd (1) mLd (0) #ExtraFLOPs 2 N 2 Ldd ′ (1) 2 n (2 N 2 + n ) d (1) 2 n (2 N 2 + n ) Ld
1) mN 2 Ld
VPT inserts n prompts to obtain nd extra parameters in
and SSF inserts SSF-ADA after each operation with a linear coefficient to obtain md extra parameters in each layer, when the total number of layers is L , the complexity of Adapter, VPT and SSF is shown in Table 2.
The specific number of additional parameters used by
Adapter, VPT and SSF depends on the values of d ′ , n and m .
However, in practice, SSF outperforms
Adapter and VPT-Deep even with slightly fewer parameters in the training stage as we will see in
Sec. 4.
Further, in the inference stage, borrowing the model re-parameterization strategy, the extra parameters and FLOPs of SSF are zero.
However, the complexity of Adapter and VPT remain the same compared to the training, which establishes the strengths of our approach.
Experiments 4.1 Experimental Settings
Datasets .
We mainly conduct our experiments on a series of datasets that can be categorized into three types as detailed below:
FGVC .
Following VPT [ ], we employ five Fine-Grained Visual Classification (FGVC
) datasets to
evaluate the effectiveness of our proposed SSF, which consists of CUB-200-2011 [79], NABirds
, Oxford Flowers [64], Stanford Dogs [46] and Stanford Cars
VTAB-1k .
VTAB-1k benchmark is introduced in [91], which contains 19 tasks from diverse domains
: i) Natural images that are captured by standard cameras; ii) Specialized images that are captured by non-standard cameras, e.g. , remote sensing and medical cameras; iii) Structured images that are synthesized from simulated environments.
This benchmark contains a variety of tasks ( e.g. , object counting, depth estimation) from different image domains and each task only contains 1,000 training samples, thus is extremely challenging.
General Image Classification Datasets .
We also validate the effectiveness of SSF on general image classification tasks.
We choose the CIFAR-100 [47] and ImageNet-1K [12] datasets as evaluation datasets, where CIFAR-100 contains 60,000 images with 100 categories.
ImageNet-1K contains 1.28M training images and 50K validation images with 1,000 categories, which are very large datasets for object recognition.
Models .
For a fair comparison, we follow VPT [44] and mainly select ViT-B/16 [16] model pretrained on ImageNet-21K as the initialization for fine-tuning.
In addition, we also generalize our method to backbones of different model families, including the recent Swin Transformer [56] (SwinB), ConvNext-B [57] and AS-MLP-B [50].
The former builds a hierarchical transformer-based architecture, and the latter two belong to CNN-based architecture and MLP-based architecture respectively.
Baselines .
We first compare our method with the two basic fine-tuning methods: i) full fine-tuning, where all parameters of the models are updated at fine-tuning; ii) linear probing, where only the parameters of the classification head (an MLP layer) are updated.
We also compare our method with recent parameter-efficient fine-tuning methods: iii) Adapter [36], where a new adapter structure with up-projection, non-linear function, and down-projection is inserted into the transformer and only the parameters of this new module are updated; iv) Bias [90], where all the bias terms of parameters are updated; v) VPT [44], where the prompts are inserted into transformers as the input tokens and they are updated at fine-tuning.
Implementation Details.
For the FGVC datasets, we process the image with a randomly resize crop to 224 × 224 and a random horizontal flip for data augmentation.
For VTAB-1k, we directly resize the image to 224 × 224 , following the default settings in VTAB [91].
For CIFAR-100 and ImageNet1K, we follow the fine-tuning setting of ViT-B/16 in [16], where the stronger data augmentation strategies are adopted.
We employ the AdamW [59] optimizer to fine-tune models for 100 epochs for
CIFAR-100, and 30 epochs for ImageNet-1K.
The cosine decay strategy is adopted for the learning rate schedule, and the linear warm-up is used in the first 10 epochs for CIFAR-100 and 5 epochs for
ImageNet-1K.
[Table] [Table]
benchmark with ViT-B/16 models pre-trained on ImageNet-21K
4.2 Performance Comparisons on Image Classification
We compare the performance of our SSF and other baseline methods in 26 image classification tasks and the results on FGVC and VTAB-1k are shown in Table 3 and Table 4 (also see Figure 1), respectively, and the results on CIFAR-100 and ImageNet-1K are shown in Table 5, which are evaluated in Top-1 accuracy (%).
In these three tables, the bold font shows the best accuracy of all methods and the underline font shows the second best accuracy.
We have the following findings by observing them: i)
In Table 3 and Table 4, where the last column is the average of the fine-tuned parameters for each method on the corresponding datasets, our
SSF outperforms VPT [44] and other parameter-efficient fine-tuning methods, and even achieves better performance than full fine-tuning, which is mainly owing to the linear transformation applied on the features.
Specifically, SSF obtains 1.81% (90.72% vs . 89.11%) and 2.46% (90.72% vs .
88.54%) accuracy improvement on five FGVC datasets, and 5.29% (73.10% vs .
69.43%) and 11.48% (73.10% vs . 65.57%) improvement on the VTAB-1k benchmark compared to VPT and full fine-tuning.
Meanwhile, SSF also uses fewer trainable parameters compared to VPT-Deep in both datasets (0.39M vs .
0.85M, 0.24M vs . 0.60M).
SSF maintains a unified learnable parameter space for different tasks with a few parameters while VPT [44] needs to design the different number of prompts for each task, which also shows the conciseness of our approach; ii)
In Table 5, i.e. , in
CIFAR-100 and ImageNet-1K, SSF and other parameter-efficient fine-tuning methods have difficulty in achieving the similar performance to the full fine-tuning, probably because these datasets have sufficient data to prevent over-fitting of the model, especially in ImageNet-1K.
In contrast, in the
VTAB-1k benchmark, the amount of data is not very large ( e.g. , only 1,000 training images), which might cause over-fitting of the model for the full fine-tuning.
Nevertheless, in CIFAR-100 and
ImageNet-1K, our SSF still outperforms previous parameter-efficient fine-tuning methods (Adapter,
Bias, and VPT), which shows the effectiveness of our method; iii)
In Table 5, the results of our SSF with Swin Transformer, ConvNext, and AS-MLP models consistently outperform those of other parameter-efficient fine-tuning methods, which also verifies the effectiveness of SSF on a wide variety of models.
Computational cost.
To validate the efficiency of our method, we show the computational cost of
SSF in Figure 3.
We employ a batch size of 16 for the training stage and inference stage, and use mixed precision training.
All running results in Figure 3 are measured in a single GeForce RTX 2080Ti GPU.
We can see that SSF has similar training time and training memory with VPT but with
[Table]
[Figure] [Figure]
less inference time and inference memory.
Here, we show the computational cost of VPT with 200/ prompts (the same number of prompts to obtain the performance in Table 5) for VPT-Shallow and
VPT-Deep, respectively.
When adding the number of prompts, the time cost and memory will be larger but our SSF achieves zero-overhead inference, which is more advantageous.
4.3 The Impacts of Different Designs
As the core operation of SSF, we thoroughly evaluate how SSF-ADA affects results, e.g. , the insertion locations, the initialization of SSF-ADA and its components.
We conduct experiments to analyze the impacts of different designs for fine-tuning.
All experiments are implemented with pre-trained
ViT-B/16 models on CIFAR-100 and the results are shown in Table 6.
The impact of the number of layers.
We directly insert SSF-ADA into different layers to evaluate the effect of inserting layers, and the results are shown in table 6a.
The values in the #layers column indicate the number of layers with SSF-ADA, where #layers-0 represents linear probing.
From the first and second rows, we find that the results will improve from 88.70% to 92.69% and grow with a small number of trainable parameters (0.08M vs. 0.11M) when only inserting SSF-ADA into the first two layers.
Keep adding SSF-ADA in the subsequent layers will make the results better.
The growth of the results is almost linear with the number of layers of inserted SSF-ADA.
Therefore, we directly choose to insert SSF-ADA into all (12) layers of vision transformer to bring the best results (93.99%) with 0.28M trainable parameters.
The impact of the different insertion locations.
Based on the different operations of ViT, we evaluate the impact of the insertion locations of SSF-ADA.
We separately remove SSF-ADA after these operations and the results are shown in Table 6b.
We find that removing the SSF-ADA in the
MLP operation achieves inferior results than removing those in the Attention operation (93.46% vs .
93.69%) with comparable trainable parameters (0.19M vs. 0.21M), which suggests that performing feature modulation for the MLP operation might be more important.
Although one can use NAS to search for the importance of different operations and thereby insert SSF-ADA in specific locations, the results might not be better than inserting SSF-ADA in all operations.
Therefore, in order to obtain excellent performance, we do not perform NAS but directly insert SSF-ADA into all operations.
[Table]
The impact of initialization.
We also investigate how different ways of initializing the scale and shift factors affect performance in Table 6c.
In our experiments, we first randomly initialize both scale and shift parameters with a mean value of zero, but find that the performance is inferior (90.11%) and cannot converge in some experiments.
After that
[Table]
we randomly initialize the scale factor with a mean value of one and find better performance, which implies that the weights of a pre-trained model should not be completely disrupted in the fine-tuning, instead, we should start from this pre-trained model to optimize our model.
Experiments show that using the normal initialization achieves the best performance, where the mean values of the scale factor and shift factor are one and zero, respectively.
The impact of different components.
We also evaluate the impacts of different components in
SSF-ADA and the results are shown in Table 6d.
We find that removing the scale term yields worse performance than removing the shift term with the same trainable parameters, which shows that the scale term might be more important than the shift term.
Also, note that the difference between ‘w/o. scale’ and the ‘Bias’ method in Table 5 is that we fine-tune the model with an additional shift term in ‘w/o. scale’, while ‘Bias’ fine-tunes the model based on the original biases, suggesting that fine-tuning the model in a res-like manner can obtain slightly better performance (93.49% vs . 93.39%).
We also try to only fine-tune all scale and shift factors in the normalization layer (LN), or fine-tune the model with SSF but set the scale term as a scalar.
These experiments yield inferior performance than SSF (93.26% vs .
93.99%, 93.59% vs . 93.99%), but could probably be considered as an alternative due to the fact that they only use about half of the trainable parameters of SSF
[Table]
4.4 Performance Comparisons on Robustness and OOD Datasets
[Table]
corruption error).
The lower ( ↓ ), the better.
We also conduct experiments to analyze the robustness and Out-Of-Distribution (OOD) ability of our SSF method with the following datasets:
ImageNet-A,
ImageNet-R and ImageNet-C. Please refer to Appendix A.2 for their details.
We perform the robustness and OOD evaluation on these three datasets with the fine-tuned models on ImageNet-1K. All experimental results are listed in Table 7.
From this table, we can see that our SSF obtains better performance than VPT and other parameter-efficient fine-tuning methods on three datasets, which shows our fine-tuning method has stronger robustness and out-of-distribution generalization.
Furthermore, although SSF has lower accuracy than full fine-tuning on ImageNet-1K, the performance on ImageNet-A, ImageNet-R and ImageNet-C is better, which also shows the performance between ImageNet-1K and ImageNet-A/R/C is not absolutely positive relevant.
Such improvements in robustness and OOD datasets might come from the fact that
SSF freezes most of the pre-trained parameters, which maximally preserves the knowledge learned from the large-scale dataset and thus maintains a better generalization ability.
4.5 Visualization and Analysis
Although our goal is to modulate the features extracted by a pre-trained model, the scale and shift parameters are input-independent indeed.
Therefore, these parameters can also be regarded as encoding information of the whole downstream dataset.
After re-parameterization, these scale and
[Figure] [Figure]
shift parameters are absorbed into the original model weights.
To better understand information learned by the SSF, we visualize the distributions of weights and biases before and after finetuning via SSF in Figure 4a.
We can see that the scale and shift parameters adjust the original weights and biases, and change the distribution of weights and biases to fit the downstream task.
[Figure]
As a comparison
[Figure]
we also visualize the original weight distribution and the weight distribution after full fine-tuning in Figure 4b, from which we can find an interesting phenomenon that full fine-tuning does not change the distribution of weights and biases much, but probably only a small portion of the values is changed.
It is worth noting that although SSF does not match the weight distribution of full fine-tuning, it achieves better performance (93.99% vs .
93.82% in Table 5) on CIFAR-100.
To further investigate why SSF can achieve superior performance, beyond weight distribution, we also visualize the feature similarities between full fine-tuning and linear probing, full fine-tuning and VPT-Deep, full fine-tuning and SSF, as shown in Figure 5.
In the last layer, SSF has the most similar feature to full fine-tuning and the accuracy is also the closest.
This shows that even if the weight distribution learned by SSF is different from full fine-tuning, SSF is also able to extract the features of the images in the downstream task very well, which validates the effectiveness of our method.
Conclusion
In this paper, we focus on parameter-efficient fine-tuning and propose an SSF method to scale and shift the features extracted by a pre-trained model.
The intuition behind our method comes from alleviating the distribution mismatch between upstream tasks and downstream tasks by modulating deep features.
SSF surprisingly outperforms other parameter-efficient fine-tuning approaches with a small number of learnable parameters.
Besides, the introduced scale and shift parameters during the fine-tuning can be merged into the original pre-trained model weights via re-parameterization in the inference phase, thereby avoiding extra parameters and FLOPs.
With the proposed SSF method, our model obtains 2.46% (90.72% vs. 88.54%) and 11.48% (73.10% vs .
65.57%) performance improvement on FGVC and VTAB-1k in terms of Top-1 accuracy compared to the full fine-tuning but only fine-tuning about 0.3M parameters.
Experiments on 26 image classification datasets in total and 3 robustness&out-of-distribution datasets with various model families (CNNs, Transformers, and
MLPs) show the effectiveness of SSF, which establishes a new baseline.
Acknowledgement
The authors acknowledge the support from the Singapore National Research Foundation (“CogniVision – Energy-autonomous always-on cognitive and attentive cameras for distributed real-time vision with milliwatt power consumption” grant NRF-CRP20-2017-0003)
– www.green-ic.org/
CogniVision .
Xinchao Wang is the corresponding author.
A Detailed Descriptions for the Evaluation Datasets A.1 Image Classification
We show the detailed descriptions of image classification as follows.
The train/val/test splits and the classes are shown in Table 8.
[Table]
FGVC .
Following VPT [44], we employ five Fine-Grained Visual Classification (FGVC) datasets to evaluate the effectiveness of our proposed SSF, which consists of CUB-200-2011 [79], NABirds [76],
Oxford Flowers [64], Stanford Dogs [46] and Stanford Cars [18].
VTAB-1k .
VTAB-1k benchmark is introduced in [91], which contains 19 tasks from diverse domains: i) Natural images that are captured by standard cameras; ii) Specialized images that are captured by non-standard cameras, e.g. , remote sensing and medical cameras; iii) Structured images that are synthesized from simulated environments.
This benchmark contains a variety of tasks ( e.g. , object counting, depth estimation) from different image domains and each task only contains 1,000 training samples, thus is extremely challenging.
General Image Classification Datasets .
We also validate the effectiveness of SSF on general image classification tasks.
We choose the CIFAR-100 [47] and ImageNet-1K [12] datasets as evaluation datasets, where CIFAR-100 contains 60,000 images with 100 categories.
ImageNet-1K contains 1.28M training images and 50K validation images with 1,000 categories, which are very large datasets for object recognition.
A.2 Robustness and OOD
ImageNet-A is introduced in [34], where 200 classes from 1,000 classes of ImageNet-1K are chosen and the real-world adversarial samples that make the ResNet model mis-classified are collected.
ImageNet-R [32] contains rendition of 200 ImageNet-1K classes and 30,000 images in total.
ImageNet-C [33] consists of the corrupted images, including noise, blur, weather, etc.
The performance of model on ImageNet-C show the robustness of model.
A.3 Detection and Segmentation
We also conduct experiments on downstream tasks beyond image classification, such as object detection, instance segmentation and semantic segmentation.
We employ the COCO dataset [52] for evaluation based on mmdetection [4] framework for the object detection and instance segmentation.
COCO contains 118K training images for training and 5K images for validation, which is one of the most challenging object detection datasets.
We use Mask R-CNN [28] with Swin Transformer backbone to perform our experiments, following the same training strategies as Swin Transformers [56].
For semantic segmentation, we employ the ADE20K dataset [93] for evaluation based on mmsegmentation [9] framework.
ADE20K contains 20,210 training images and 2,000 validation images.
Following Swin Transformer [56], we use UperNet [82] with Swin Transformer backbone .
All models are initialized with weights pre-trained on ImageNet-1K for detection and segmentation.
[Table]
B Experiments on Detection and Segmentation
We also conduct experiments on broader downstream tasks, e.g. , object detection, instance segmentation, and semantic segmentation.
For object detection and instance segmentation, we perform experiments on the COCO dataset with Mask R-CNN [28], where Swin-T pre-trained on ImageNet1K is adopted as the backbone.
The specific hyper-parameter setup and data augmentation refer to
Swin Transformer [56] and mmdetection [4].
We perform i) full fine-tuning; ii) linear probing, where the weights at the backbone layers are frozen and only weights at the neck and head layers are updated; iii) VPT-Deep; iv) SSF.
All models are trained with 1x schedule (12 epochs).
The results are shown in Table 9.
We can see that SSF outperforms linear probing and VPT-Deep [44] on the COCO dataset in terms of object detection and instance segmentation.
For semantic segmentation, we perform experiments on the ADE20K dataset with UperNet [82] and Swin-T pre-trained on ImageNet-1K.
The results in Table 9 show that SSF outperforms linear probing and VPT-Deep [44].
However, for both datasets, SSF still has a large gap compared to the full fine-tuning, which might be due to the fact that detection and segmentation tasks are fundamentally different from classification tasks.
Only fine-tuning a few parameters in the backbone will result in inferior performance.
How to introduce trainable parameters for parameter-efficient fine-tuning in object detection and segmentation will be the future work.
C Visualizations C.1 Feature Distribution
We also visualize the feature distribution of different
The results are shown in Figure 6.
Our SSF achieves better feature clustering results compared to linear probing and VPT-Deep.
Besides, since our method and full fine-tuning have similar accuracy
(93.99% vs. 93.82%), it is difficult to distinguish them in terms of feature distribution, which also shows the effectiveness of our method.
[Figure]
C.2 Attention Map
We also visualize the attention maps of different fine-tuning methods, as shown in Figure 7.
All models are fine-tuned on ImageNet-1K with ViT-B/16 pre-trained on ImageNet-21K.
The specific experimental results refer to Table 5 in the main text.
We find that VPT-Deep has more concentrated attention on the object in some images ( e.g. , the first two lines), but lacks suitable attention on some other images ( e.g. , the last two lines).
In contrast, SSF tends to obtain attention similar to the full fine-tuning but also generates the failure prediction such as the second row.
[Figure]
D Limitations and Societal Impacts
Regarding the limitations of this work, we currently focus on sharing backbone parameters among different tasks while treating each task independently of the rest of the tasks involved.
However, some recent papers ( e.g. , [21, 20]) show that by correlating multiple tasks together during the fine-tuning, the performance for every single task can be further improved.
However, recent works treat this relationship among tasks as a black box that inevitably suffers a huge computational cost.
Thus, we believe an efficient method to find positive task relationships could be a meaningful direction for further exploration.
This work has the following societal impact.
SSF can effectively save parameters compared to the full fine-tuning so that the approach can quickly transfer large models pre-trained on large datasets to downstream tasks, which saves computational resources and carbon emissions.
Thanks to the linear transformation and re-parameterization, we do not need to change the deployed model architecture when the model is transferred to the downstream task.
Only a set of weights need to be replaced, which is also more convenient compared to the methods that introduce additional parameters such as
VPT [44].
However, like other fine-tuning methods, SSF is also based on a pre-trained model, which will probably also cause a violation of the use of fine-tuning methods if this upstream pre-trained model is trained on some illegal data.