FiT: Parameter Efficient Few-shot Transfer Learning
Abstract
Model parameter efficiency is key for enabling few-shot learning, inexpensive model updates for personalization, and communication efficient federated learning.
In this work, we develop FiLM Transfer (F I T) which combines ideas from transfer learning (fixed pretrained backbones and fine-tuned FiLM adapter layers) and metalearning (automatically configured Naive Bayes classifiers and episodic training) to yield parameter efficient models with superior classification accuracy at lowshot.
We experiment with F I T on a range of downstream datasets and show that it achieves better classification accuracy than the leading Big Transfer (BiT) algorithm at low-shot and achieves state-of-the art accuracy on the challenging
VTAB-1k benchmark, with fewer than 1 % of the updateable parameters.
Introduction
Model parameter efficiency is key for enabling few-shot learning, inexpensive model updates for personalization, and communication efficient federated learning.
In order to develop data-efficient and parameter-efficient learning systems, we draw on ideas developed by the few-shot learning community.
Few-shot learning approaches can be characterized in terms of shared and updateable parameters.
From a statistical perspective, shared parameters capture similarities between datasets, while updateable parameters capture the differences.
Updateable parameters are those that are either recomputed or learned as the model is updated or retrained, whereas shared parameters are fixed.
In personalized or federated settings, it is key to minimize the number of updateable parameters while still retaining the capacity to adapt
Broadly, there are two different approaches to few-shot learning: meta-learning [Hospedales et al., 2020] and transfer learning (fine-tuning)
[Yosinski et al., 2014].
Meta-learning approaches provide methods that have a small number of updatable parameters [Requeima et al., 2019].
However, while meta-learners can perform strongly on datasets that are similar to those they are meta-trained on, their accuracy suffers when tested on datasets that are significantly different [Dumoulin et al., 2021].
Transfer learning algorithms often outperform meta-learners, especially on diverse datasets and even at low-shot [Dumoulin et al., 2021, Tian et al., 2020].
However, the leading Big Transfer (BiT)
[Dumoulin et al., 2021, Kolesnikov et al., 2019] algorithm requires every parameter in a large network to be updated.
In summary, performant transfer learners are parameter-inefficient, and parameter-efficient few-shot learners perform relatively poorly.
In this work we propose FiLM Transfer or FiT, a novel method that synthesizes ideas from both the transfer learning and meta-learning communities in order to achieve the best of both worlds parameter efficiency without sacrificing accuracy, even when there are only a small number of training examples available.
From transfer learning, we take advantage of backbones pretrained on large image datasets and the use of fine-tuned parameter efficient adapters.
From meta-learning, we take advantage of metric learning based final layer classifiers trained with episodic protocols that we show are more effective than the conventional linear layer classifier.
Our contributions:
• A parameter and data efficient network architecture for low-shot transfer learning that (i) utilizes frozen backbones pretrained on large image datasets; (ii) augments the backbone with parameter efficient FiLM
[Perez et al., 2018] layers in order to adapt to a new task; and (iii) makes novel use of an automatically configured Naive Bayes final layer classifier instead of the usual linear layer, saving a large number of updateable parameters, yet improving classification performance; • A meta-learning inspired episodic training protocol for low-shot fine-tuning requiring no data augmentation, no regularization, and a minimal set of hyper-parameters; • State-of-the-art results on the challenging VTAB-1k benchmark (74.9% for backbones pretrained on ImageNet-21k) while using ≈ 1% of the updateable parameters when compared to the leading transfer learning method BiT;
FiLM Transfer (F I T)
In this section we detail the F I T algorithm focusing on the few-shot image classification scenario.
Preliminaries We denote input images x ∈ R ch × W × H where W is the width, H the height, ch the number of channels and image labels y ∈ { 1 , . . .
, C } where C is the number of image classes indexed by c .
Assume that we have access to a model f ( x ) =
h ϕ ( b θ ( x )) that outputs class-probabilities for an image p ( y = c
| x , θ , ϕ ) for c = 1 , . . .
, C and is comprised of a feature extractor backbone b θ ( x ) ∈ R d b with parameters θ that has been pretrained on a large upstream dataset such as Imagenet where d b is the output feature dimension and a final layer classifier or head h ϕ ( · ) ∈ R C with weights ϕ .
Let D = { ( x n , y n ) } Nn =1 be the downstream dataset that we wish to fine-tune the model f to.
F I T Backbone For the network backbone, we freeze the parameters θ to the values learned during upstream pretraining.
To enable parameter-efficient and flexible adaptation of the backbone, we add
Feature-wise Linear Modulation (FiLM)
[Perez et al., 2018] layers with parameters ψ at strategic points within b θ .
A FiLM layer scales and shifts the activations a ij arising from the j th channel of a convolutional layer in the i th block of the backbone as FiLM ( a ij , γ ij , β ij ) =
γ ij a ij + β ij , where γ ij and β ij are scalars.
The set of FiLM parameters ψ = { γ ij , β ij } is learned during fine-tuning.
An advantage of FiLM layers is that they enable expressive feature adaptation while adding only a small number of parameters [Perez et al., 2018].
For example, in a ResNet50 with a FiLM layer in every block, the set of FiLM parameters ψ account for only 11648 parameters which is fewer than 0.05% of the parameters in b θ .
F I T Head For the head of the network, we use a specially tailored Gaussian Naive Bayes classifier.
Unlike a linear head, this head can be automatically configured directly from data and has only a small number of free parameters which must be learned, ideal for few-shot, personalization and federated learning.
We will also show that this head is often more accurate than a standard linear head.
The class probability for a test point x ∗ is:
[Equation]
where π c = N c N , µ
[Equation]
=
[Equation]
N
[Equation]
are the maximum likelihood estimates, N c is the number of examples of class c in D , and N ( z | µ, Σ) is a multivariate Gaussian over z with mean µ and covariance Σ .
Estimating the mean µ c for each class c is straightforward and incurs a total storage cost of Cd b .
However, estimating the covariance Σ c for each class c is challenging when the number of examples per class N c is small and the embedding dimension of the backbone d b is large.
In addition, the storage cost for the covariance matrices may be prohibitively high if d b is large.
Here, we use three different approximations to the covariance in place of Σ c in Eq.
[Fisher, 1936, Duda et al., 2012]:
• Quadratic Discriminant Analysis (QDA) : Σ QDA = e 1 Σ class
+ e 2 Σ task + e 3 I • Linear Discriminant Analysis (LDA) :
Σ LDA = e 2 Σ task + e 3 I • ProtoNets
[Snell et al., 2017]: Σ PN
=
I ; i.e. there is no covariance and the class representation is parameterized only by µ c and the classifier logits are formed by computing the squared Euclidean distance between the feature representation of a test point b θ , ψ ( x ∗ ) and each of the class means. In the above, Σ class is the computed covariance of the examples in class c in D , Σ task is the computed covariance of
all the examples in D assuming they arise from a single Gaussian with a single mean, e = { e 1 , e 2 , e 3 } are weights learned during training, and the identity matrix I is used as a regularizer. QDA mainly serves as a baseline since it has a very large set of updateable parameters arising from the fact that it stores a covariance matrix for each class in the dataset. LDA is far more parameter efficient than QDA, sharing a single covariance matrix across all classes. We show that LDA leads to very similar performance to QDA.
The number of model shared and updateable parameters for the three F I T variants as well as the BiT algorithm are detailed in Table 1. [Table] | θ [Equation]
ψ
[Equation] + [Equation] µ
[Equation]
Σ
[Equation]
[Equation]
| e
[Equation]
=
[Equation]
C
[Equation]
d
[Equation]
F I T - ProtoNets
[Equation]
An empirical justification for the use of the F I T-LDA head is shown in Fig.
1a where it outperforms a linear head in the case of FiLM and when all the backbone parameters are learned.
In Fig.
1b, we see for both datasets, F I T-LDA converges faster than BiT, which uses a linear head.
The primary limitation of the Naive Bayes head is the higher (versus linear) computational cost due to having to invert a d b × d b covariance matrix on each training iteration
[Figure] F I T Training We learn the FiLM parameters ψ and the covariance weights e via fine-tuning (parameters θ are fixed from pretraining).
One approach would be to apply standard batch training on the downstream dataset, however it was hard to balance under- and over-fitting using this setup.
Instead, we found that an approach inspired by episodic training [Vinyals et al., 2016] that is often used in meta-learning yielded better performance.
We refer to this approach as episodic fine tuning and it works as follows.
Note that we require ‘training’ data to compute π , µ , Σ to configure the head, and a ‘test’ set to optimize ψ and e via gradient ascent.
Thus, from the downstream dataset
D , we derive two sets – D train and D test .
We randomly split D into D train and D test such that the number of examples or shots in each class c are roughy equal in both partitions and that there is at least one example of each class in both.
Refer to Algorithm A.1 for details.
For each training iteration, we sample a task τ consisting of a support set D τS drawn from D train with S τ examples and a query D τQ set drawn from D test with Q τ examples.
First, D τS is formed by randomly choosing a subset of classes selected from the range of available classes in D train .
Second, the number of shots to use for each selected class is randomly selected from the range of available examples in each class of D train with the goal of keeping the examples per class as equal as possible.
Third, D τQ is formed by using the classes selected for D τS and all available examples from D test in those classes up to a limit of 2000 examples.
See Algorithm A.2 for details.
Episodic fine-tuning is crucial to achieving the best classification accuracy with the Naive Bayes head.
The support set D τS is then used to compute π , µ , and Σ
and we then use D Q = {{ x τ ∗ q , y τ ∗ q } Q τ q =1 } Tτ = to train ψ and e with maximum likelihood. We optimize the following: [Equation]
F I T training hyper-parameters include a learning rate, |D τS | , and the number of training iterations.
For the transfer learning experiments in Section 4 these are set to constant values across all datasets and do not need to be tuned based on a validation set. We do not augment the training data. In the 1-shot case, we do not perform episodic fine-tuning and leave the FiLM parameters at their initial value of γ = 1 , β = 0 and e = (0 . 5 , 0 . 5 , 1 . 0) and predict as described next. F I T Prediction Once the FiLM parameters ψ and covariance weights e have been learned, we use
D for the support set to compute π c , µ c , and Σ c for each class c
and then Eq.
(1) can be used to make a prediction for any unseen test input.  Related Work
We take inspiration from residual adapters [Rebuffi et al., 2017, 2018] where parameter efficient adapters are inserted into a ResNet with frozen pretrained weights.
The adapter parameters and the final layer linear classifier are then learned via fine-tuning.
More recently, a myriad of additional parameter efficient adapters have been proposed including FiLM, Adapter [Houlsby et al., 2019], LoRA [Hu et al., 2021], VPT [Jia et al., 2022], AdaptFormer [Chen et al., 2022], NOAH [Zhang et al., 2022], Convpass [Jie and Deng, 2022], [Mudrakarta et al., 2019], and CaSE [Patacchiola et al., 2022]. For F I T we use FiLM as it is the most parameter efficient adapter, yet it allows for expressive adaptation, and can be used in various backbone architectures including ConvNets and Transformers.
To date, transfer learning systems that employ adapters use a linear head for the final classification layer.
In meta-learning systems it is common to use metric learning heads (e.g. ProtoNets
[Snell et al., 2017]), which have no or few learnable parameters.
Meta-Learning systems that employ a metric learning head are normally trained with an episodic training regime [Vinyals et al., 2016].
Some of these approaches (e.g. TADAM [Oreshkin et al., 2018], FLUTE [Triantafillou et al., 2021], and
Simple CNAPs [Bateni et al., 2020] use both a metric head and FiLM layers to adapt the backbone.
F I T differs from all of the preceding approaches by using a powerful Naive Bayes metric head that uses episodic fine-tuning in the context of transfer learning, as opposed to the usual meta-learning.
We show in Fig.
1a and Section 4 that the episodically fine-tuned Naive Bayes head consistently outperforms a conventional batch trained linear head in the low-shot transfer learning setting.
In this section, we evaluate the classification accuracy and updateable parameter efficiency of F I T on the VTAB-1k [Zhai et al., 2019] benchmark.
The VTAB-1k benchmark [Zhai et al., 2019] is a low to medium-shot transfer learning benchmark that consists of 19 datasets grouped into three distinct categories (natural, specialized, and structured).
From each dataset, 1000 examples are drawn at random from the training split to use for the downstream dataset D .
After fine-tuning, the entire test split is used to evaluate classification performance.
In all experiments, we use Big Transfer (BiT) [Kolesnikov et al., 2019], a leading, scalable, general purpose transfer learning algorithm as a point of comparison.
In addition, we compare F I T to the latest vision transformer based methods that have reported the highest accuracies on VTAB-1k to date.
See Appendix A.2 for experimental details.
Table 2 shows the classification accuracy and updateable parameter count for the three variants of F I T and BiT.
The key observations from our results are: • Both F I T-QDA and F I T-LDA outperform BiT on VTAB-1k.
• The F I T-QDA variant has the best overall performance, showing that the class covariance is important to achieve superior results on datasets that differ from those used in upstream pretraining (e.g. the structured category of datasets).
However, the updateable parameter cost is high.
• F I T-LDA utilizes two orders of magnitude fewer updateable parameters compared to BiT, making it the preferred approach.
[Table] Table 3 shows that F I T-LDA achieves state-of-the-art classification accuracy when compared to leading transfer learning methods pretrained on ImageNet-21k, while requiring the smallest number of updateable parameters and using the smallest backbone.
All competing methods use a linear head
[Table]
[Table]
Discussion
In this work
[Table]
we proposed F I T
[Table]
a parameter and data efficient few-shot transfer learning system that allows image classification models to be updated with only a small subset of the total model parameters. We demonstrated that F I T can outperform BiT using fewer than 1% of the updateable parameters and achieve state-of-the-art accuracy on VTAB-1k.  Acknowledgements This work has been performed using resources provided by the Cambridge Tier-2 system operated by the University of Cambridge Research Computing Service https://www.hpc.cam.ac.uk funded by EPSRC Tier-2 capital grant EP/P020259/1.
Aliaksandra Shysheya, John Bronskill, Massimiliano
Patacchiola and Richard E. Turner are supported by an EPSRC Prosperity Partnership EP/T005386/
A Appendix A. F I T Training Algorithms
Algorithm A.1 and Algorithm A.2 detail how episodic fine-tuning tasks are split and sampled, respectively, for use in the F I T training protocol.
Algorithm
Require:
D = { ( x n , y n ) } Nn =1 = { x , y } : downstream dataset
Require: unique() ≡ function that returns a list of unique classes and list of counts of each class
Require:
select_by_class () ≡ function that extracts samples of a specified class from a dataset 1: procedure SPLIT ( D ) 2:
D train ← [ ] ▷ Create an empty list to hold D train 3:
D test ← [ ] ▷ Create an empty list to hold D test 4: classes, class_counts ← unique ( y ) 5: for all c ∈ classes do 6: assert ( class_counts ( c ) > 1)
▷ Require a minimum of 2 shots per class.
7: train_count ← ceil ( class_counts ( c ) / 2) 8:
D c ← select_by_class ( c ) ▷ Select examples of class c from D 9:
D train ← D train + D c
[: train_count ]
▷ Add train_count examples to D train 10:
D test ← D test + D c
[ train_count :] ▷ Add remaining examples to D test 11: end for 12: return D train , D test 13: end procedure[-Algorithm A.
 Sampling a τ-]
Require: D train = { ( x s , y s ) } S
τ s =1 = { x S , y S } : train portion of downstream dataset
Require: D test = { ( x q , y q ) } Q τ q =1 = { x Q , y Q } : test portion of downstream dataset
Require: support_set_size : size of the support set |D τS |
Require: unique() ≡ function that returns a list of unique classes and list of counts of each class
Require: randint( min, max ) ≡ function that returns a random integer between min and max
Require: choice ( range, count ) ≡ function that returns a random list of count integers from range 1: procedure SAMPLE _ TASK
( D train , D test , support_set_size ) 2:
D τS ← [ ] ▷ Create an empty list to hold D τS 3:
D τQ ← [ ] ▷ Create an empty list to hold D τQ 4: train_classes, train_class_counts ← unique ( y S ) 5: test_classes, test_class_counts ← unique ( y Q ) 6: min_way ← min ( len ( train_classes ) , 5) 7: max_way ← min ( len ( train_classes ) , support_set_size ) 8: way ← randint ( min_way , max_way ) ▷ Classification way to use for this task 9: selected_classes ← choice ( train_classes , way ) ▷ List of classes to use in this task 10: balanced_shots = max ( round ( support_set_size / len ( selected_classes )), 1) 11: max_test_shots ← max (1 , floor (2000 / way )) 12: for all c ∈ selected_classes do 13:
class_shots ← train_class_counts ( c ) 14: shots_to_use ← min ( class_shots , balanced_shots ) 15:
selected_shots ← choice ( class_shots , shots_to_use )
▷ Support shot list 16:
D τS ← D τS + D train [ selected_shots ] ▷ Add examples to D τS 17: class_shots ← test_class_counts ( c ) 18: shots_to_use ← min ( class_shots , max_test_shots ) 19: selected_shots ← choice ( class_shots , shots_to_use )
▷ Query shot list 20:
D τQ ← D τQ + D test [ selected_shots ]
▷ Add examples to D τQ 21: end for 22: return D τS , D τQ 23: end procedure
A.2 Training and Evaluation Details
In this section, we provide implementation details for the experiments in Section 4.
F I T
All of the F I T versus BiT on VTAB-1k transfer learning experiments were carried out on a single NVIDIA A100 GPU with 80GB of memory.
The Adam optimizer [Kingma and Ba, 2015] with a constant learning rate of 0.0035, for 400 iterations, and |D τS | =100 was used throughout.
No data augmentation was used and images were scaled to 384 × 384 pixels unless the image size was 32 × 32 pixels or less, in which case the images were scaled to 224 × 224 pixels.
F I T-QDA, F I T-LDA, and F I T-ProtoNets take approximately 12, 10, and 9 hours, respectively, to fine-tune on all 19 VTAB datasets.
For the F I T-LDA results using the EfficientNetV2-M backbone in Table 3, we used 1000 iterations instead of 400 and ran the experiments on 4 NVIDIA A100 GPUs, each with 80GB of memory.
BiT For the BiT VTAB-1k experiments, we used the three fine-tuned models for each of the datasets that were provided by the authors [Kolesnikov et al., 2020].
We evaluated all of the models on the respective test splits for each dataset and averaged the results of the three models.
The BiT-HyperRule [Kolesnikov et al., 2019] was respected in all runs.
These experiments were executed on a single
NVIDIA GeForce RTX 3090 with 24GB of memory.