R EVISIT F INETUNING STRATEGY FOR F EW -S HOT L EARNING TO T RANSFER THE E MDEDDINGS
Anonymous authors
Paper under double-blind review
A BSTRACT
Few-Shot Learning (FSL) aims to learn a simple and effective bias on limited novel samples.
Recently, many methods have been focused on re-training a randomly initialized linear classiﬁer to adapt it to the novel features extracted by a pre-trained feature extractor (called Linear-Probing-based methods).
These methods typically assumed the pre-trained feature extractor was robust enough, i.e., ﬁnetuning was not needed, and hence the pre-trained feature extractor does not be adapted to the novel samples.
However, the unadapted pre-trained feature extractor distorts the features of novel samples because the robustness assumption may not hold, especially on the out-of-distribution samples.
To extract the undistorted features, we designed Linear-Probing-Finetuning with Firth-Bias (LP-FT-FB) to yield an accurate bias on the limited samples for better ﬁnetuning the pre-trained feature extractor, providing stronger transferring ability.
In
LP-FT-FB, we further proposed inverse Firth Bias Reduction (i-FBR) to regularize the over-parameterized feature extractor on which FBR does not work well.
The proposed i-FBR effectively alleviates the over-ﬁtting problem of the feature extractor in the process of ﬁnetuning and helps extract undistorted novel features.
To show the effectiveness of the designed LP-FT-FB, we conducted comprehensive experiments on the commonly used FSL datasets under different backbones for in-domain and cross-domain FSL tasks.
The experimental results show that the proposed FT-LP-FB outperforms the SOTA FSL methods.
The code is available at https://github.com/whzyf951620/
LinearProbingFinetuningFirthBias .
I NTRODUCTION
Few-shot Learning (FSL) has recently developed quickly in the limited data regime.
FSL aims to learn a suitable inductive bias on the given limited samples of novel classes.
At the very start, the whole model consisting of the feature extractor and the classiﬁer is pre-trained on the samples of base classes, and then is ﬁnetuned on the limited novel samples to obtain an inductive bias.
The performance of the ﬁnetuned model drops signiﬁcantly due to the over-ﬁtting problem of the pre-trained model.
To address the overﬁtting problem, meta-learning-based methods such as Prototypical Networks Snell et al.
(2017) and MAML
Finn et al.
(2017) were proposed to learn the learning strategies for a suitable inductive bias.
Then, Chen et al.
proposed Baseline++ Chen et al.
(2019) to show that a simple Linear Probing (LP) strategy can also get comparable performance to meta-learning-based methods.
LP re-trained a linear classiﬁer to adapt to the novel samples without updating the whole model.
Following Baseline++, many researchers studied the LP-based FSL methods, such as S2M2 Mangla et al.
RFS Tian et al.
(2020), and EMD Zhang et al.
(2020), are proposed to obtain a more powerful fully-trained feature extractor.
LP-based FSL methods assumed that a pre-trained feature extractor is robust enough to novel samples Yang et al.
; Tian et al.
(2020) and hence does not need to be ﬁnetuned.
[Figure]
However, the robustness assumption may not hold, especially on the out-of-distribution (OOD) novel samples, because the existing regularization methods only provided effective robustness to in-distribution (ID) samples.
This leads to “wrong” features of novel samples Kumar et al.
This problem cannot be well addressed by only strengthening the robustness of the pre-trained feature extractor without transferring the extracted features.
Finetuning is a typical and simple technology in the transferring learning on large-scale datasets to quickly transfer the features Kornblith et al.
; He et al.
; Chen et al.
However, the existing FSL ﬁnetuning strategies cannot appropriately ﬁnetune the over-ﬁtted feature extractor Shen et al.
Consequently, designing an appropriate ﬁnetuning strategy for the fully-trained feature extractor on the limited samples is a critical point to transfer the features and hence make the ﬁnetuned feature extractor yield “good” novel features.
To design an appropriate ﬁnetuning strategy, we started by analyzing the existing ﬁnetuning strategies Kumar et al.
; Kanavati & Tsuneki (2021); Levine et al.
; Radford et al.
(2021), especially on the ODD samples Kumar et al.
Linear-Probing-Finetuning (LP-FT)
Kumar et al.
(2022) theoretically proved that the existing ﬁnetuning strategies underperformed due to distorted novel features.
This is because when trying to ﬁt the novel samples with a randomly initialized linear classiﬁer, the extracted ID features and
OOD features change inconsistently.
The inconsistent problem is caused by the unchanged features of the samples in the space orthogonal to the space spanned by the base samples, i.e., out-of-distribution features unchanged (details in Section 2.2).
Kumar et al.
proposed using a linear classiﬁer well-trained on the ﬁtting novel samples to address this distorted problem.
Although LP-FT is effective, it is used to address the ﬁnetuning problem on large-scale out-of-distribution datasets, which contain enough samples to provide an unbiased estimation.
In FSL tasks, limited novel samples lead to a biased estimation of the updated parameters.
Seeing this, we reduce the Firth Bias of the model Ghaffari et al.
; FIRTH (1993) to obtain an unbiased estimation (called Firth Bias Reduction).
However, Firth Bias Reduction (FBR) gives an unbiased estimation only for the linear classiﬁer but not the feature extractor.
Experimental result also shows that FBR decreased the performance of the ﬁnetuned feature extractor.
A deep analysis (Section 2.3) shows that when the scaling factor λ (see Eq. 2) is positive, FBR encourages the distribution of the linear classiﬁer output to be far away from the uniform distribution (See Fig.
1) - FBR strengthens the inﬂuence of novel samples for linear probing.
This accelerates the convergence of the low-parameterized linear classiﬁer but hampers the generalization of the over-parameterized feature extractor.
According to the analysis, we proposed inverse-FBR (i-FRB, negative λ in Eq. 2) to encourage the distribution of the linear classiﬁer output to be close to uniform distribution (See Fig. 1) to address the over-ﬁtting problem M¨uller et al.
Combining LP-FT, FBR, and the proposed i-FBR, we proposed Linear-Probing-Finetuning with Firth Bias (LP-FT-FB) to appropriately ﬁnetune the fully-trained feature extractors to adapt them to the target domain on limited novel samples.
Firstly, we use linear probing to address the distorted feature problem.
The linear probing is regularized with FBR for unbiased estimation.
Then the whole pre-trained model is ﬁnetuned regularized with the proposed i-FBR on the limited novel samples to strengthen the transferring ability of the feature extractor for more separable features.
The proposed LP-FT-FB makes the extracted features of novel samples undistorted and separable.
The whole LP-FT-FB ﬁnetuning strategy is visualized in Fig.
Furthermore, the unbiased ﬁnetuned feature extractor transfers the novel features close to the novel domain.
This is veriﬁed in the cross-domain FSL tasks (Section 3.5).
Our main contributions include (1) studying and ﬁnding the gaps between ﬁnetuning-based and LP-based FSL methods: distorted features and a biased estimation; (2) designing a novel and simple ﬁnetuning strategy, LP-FT-FB, to quickly transfer the extracted features to the novel domain to bridge the gaps; (3) proposing inverse Firth Bias Reduction (i-FBR) to address the biased estimation problem of ﬁnetuning the feature extractor due to limited samples.
M ETHODS
LP-FT is ﬁrst described in Section 2.2.
Then FBR is formulated and analyzed in Section 2.3 to address the unique limited-sample problem in FSL, i.e., the biased estimation problem.
However, FBR is not suitable for the over-parameterized feature extractor.
Based on the analysis of FBR, i-FBR is proposed to ﬁnetune the feature extractor.
In Section 2.4, LP-FT-FB is stated.
2. S ET UP
Following FBR Ghaffari et al.
(2022), we assume a multinomial logitic regression model for the classiﬁer v ( · , β ) .
β = { β c } 1 ≤ c ≤ C and β c denotes the logistic regression weights for class c .
B ( · , θ ) denotes the feature extractor.
The dataset D = { ( x i , y i ) } 1 ≤ i ≤ M totally contains C classes and M samples.
The whole inference process is formulated as: ˆ y i = v ( z i , β ) = v ( B ( x i , θ ) , β ) , where z i =
B
( x i , θ ) .
In the multinomial logistic regression model, the assignment probability of z i to class c is formulated as p ci :=
Pr ( y i =
c | z i ) =
e βTc z
i 1+ (cid:80) Cc (cid:48)
=1 e βTc (cid:48)
z i .
Then the logistic log-likelihood function is formulated as
L logistic := (cid:80) Mi
=1 (cid:80)
Cc =1 1 [ y i
=
c ] · log p ci , where 1 [ · ] denotes the y i in the one-hot manner.
For the linear probing and ﬁnetuning, we used the Cross Entropy Loss L CE = − 1 M (cid:80) Mi
=
[ y i · log( P i )] , where P i = { p ci } Cc =1 .
P i
= ˆ y i denotes the predicted probability vector.
2. L INEAR -P ROBING -F INETUNING
LP-FT explored the reason why LP underperforms FT on in-distribution samples but outperforms FT in outof-distribution samples on large-scale datasets.
LP-FT theoretically proved that when a pre-trained feature extractor tried to ﬁt the ID samples, the transfer of the model will not change the feature of out-of-distribution samples in the space orthogonal to the space spanned by pre-training samples.
To show this, let S denote the subspace spanned the training samples X , and the training loss is
L ( v ( B ( X , θ ) , β ) , Y ) =
(cid:107)
X
θ T β − Y (cid:107) 22 .
B is assumed to be a linear model; Y denotes the one-hot la- bels of X .
The gradients of the training loss with respect to the parameter θ of the feature extractor B is computed as:
[Equation]
With Eq.
1, if u is a sample in the subspace orthogonal to S , the features of u do not change with the ﬁnetuned
B ( · , θ ) : ∆ z = ∇ θ L ( v ( · , β ) , B ( · , θ ))
u = 2 β
( Y − X θ T β ) T ( X · u ) = 0 .
This leads to the distorted features extracted by the ﬁnetuned B ( · , θ ) because the in-distribution and out-of-distribution features inconsistently change.
However, LP-FT explores the feature extractor ﬁnetuning problem on large-scale datasets.
Observing this, we wish to address the feature extractor ﬁnetuning problem with LP-FT in FSL if we can address the limitedsample problem.
Because the novel samples used to ﬁnetune the pre-trained feature extractor are out-ofdistribution in FSL models.
The limited-sample problem is actually a biased estimation problem.
2. INVERSE F IRTH B IAS R EDUCTION
For linear probing, we use Firth Bias Reduction (FRB) to obtain an unbiased estimation.
PMLE FIRTH (1993) added a log-determinant penalty 1 to remove the O ( n − 1 ) term of the asymptotic bias of the maximum likelihood estimate parameters for the unbiased estimation.
FBR Ghaffari et al.
(2022) addressed the situation when det( F ) = 0 and proposed an approximate format for the linear classiﬁer and cosine classiﬁer.
In FBRGhaffari et al.
(2022), the FBR loss is formulated as
[Equation]
where U [0 ,C ] denotes the uniform distribution between 0 and C , and λ denotes the scaling factor of FBR.
From Eq.
2, we concluded that when λ > 0 , the L Firth encourages the distribution of the logits z i to be far away from U [0 ,C ] , which totally differs from Label Smoothing M ¨ uller et al.
When λ < 0 , L Firth encourages the distribution of the logits z i to be close to U [0 ,C ] , and hence avoids too high conﬁdence of z i . This coincides with the effectiveness of Label Smoothing M ¨ uller et al.
; Hein et al.
However, the vanilla FBR is not suitable for ﬁnetuning feature extractor because when λ > 0 (used in FBR
Ghaffari et al.
, the logits z is encouraged to be far away from U [0 ,C ] .
The distribution drag leads to too high similarity between the distribution of the logits z and y , and hence makes the over-parameterized feature extractor over-ﬁtted Hein et al.
Seeing this, according to the analysis of Eq. 2, we proposed inverse
FBR
(i-FBR) to encourage the distribution of the z to be close to U [0 ,C ] with λ < [-
-]0 (we use the λ inv to denote the scaling factor of i-FBR) to address the over-ﬁtting problem.
The proposed i-FBR works similarly to Label Smoothing to a certain extent because both of them encourage the output distribution of the model to be close to the uniform distribution.
Hence they can reduce the inﬂuence of the training novel samples and alleviate the too high conﬁdence problem.
In addition to the above distribution drag, we computed the gradients of L CE and L with respect to r i = β · B ( x i , θ ) as follows to show the gradient variation brought by λ .
[Equation]
where E is a one-full vector with the same size as P i and P i
= ˆy
i is the predict of the whole model.
The detailed derivation is given in Appendix.
According to Eq.
3, we concluded that when λ > 0 , the gradient ∂ L ∂ r ci is increased for the low-parameterized linear classiﬁer to quickly reach the optimal point, and hence the linear probing model performs better Prabhu et al.
The overﬁtting problem does not need to be considered because of the low-parameterized property
However, it is not suitable for ﬁnetuning the over-parameterized feature extractor because of too large similarity between z i and y i .
From Eq.
3, we found that when λ < 0 and the class of x i is c, the gradient ∂ L ∂ r ci is decreased and ∂ L ∂ r ki ( k (cid:54) =
c ) is increased.
This reduces the inﬂuence of x i , which is used to train the over-parameterized feature extractor, and hence alleviates the over-ﬁtting problem.
Furthermore, different from Label Smoothing, the proposed i-FBR decreases the gradients, which is suitable for ﬁnetuning the over-parameterized model, and hence outperforms Label Smoothing (See Table 6).
2. LP-FT-FB
With FBR, LP-FT, and the proposed i-FBR, we proposed LP-FT-FB to quickly transfer the extracted features instead of the robustness assumption to address the over-ﬁtting problem.
Firstly, a randomly initialized linear classiﬁer v ( · , β ) is re-trained on the novel samples and regularized with FBR.
For the C few - way -K- Shot
FSL tasks, the re-trained classiﬁer v (cid:48) ( · , β (cid:48) ) is updated as
[Equation]
where r i = β · B 0 ( x i , θ 0 ) ; ˆy = { ˆy i }
[Equation]
= ;
y = { y
[Equation]
K
[Equation]
=
[Equation]
; α 1 is the LP learning rate.
Then the pre-trained B 0 ( · , θ 0 ) and v (cid:48) ( · , β (cid:48) ) are together ﬁnetuned on the same novel samples and regularized with i-FBR.
FT is stated as follows.
The forward process of FT is formulated as
[Equation]
The loss of FT is computed as:
[Equation]
The feature extractor B 0 ( · , θ 0 ) is updated as
[Equation]
where r (cid:48)
= { r (cid:48) i } K · C few i = and r (cid:48)
i = β (cid:48)
· B 0 ( x i , θ 0 ) .
The linear classiﬁer v (cid:48) ( · , β (cid:48) ) is updated as
[Equation]
where α 2 is the learning rate of FT.
The analytic formula of Eq.
4, Eq. 8, and Eq.
7 are given in Appendix.
With ˆ B ( · , ˆ θ ) and ˆ v ( · , ˆ β ) , the performance of LP-FT-FB is evaluated on the novel query samples.
The whole ﬂow of LP-FT-FB is given in Appendix.
E XPERIMENTS
To comprehensively show the effectiveness of the proposed i-FBR and LP-FT-FB, we conducted many in-domain experiments: 1) 5-way- 1 \ 5 -shot tasks on mini -Imagenet, tiered -Imagenet, and CUB datasets under the typical FSL backbone, WideResNet-28-10; 2) few-shot tasks under multiple scale backbones including ResNet-18 and ResNet-34; 3) N-way-K-shot ( N > 5 , K > 5) tasks on mini -Imagenet and tieredImagenet; 4) few-shot tasks on the extracted features augmented by Distribution Calibration (DC)
Yang et al.
Also, the cross-domain few-shot tasks, mini -Imagenet → CUB and tiered -Imagenet are also evaluated.
3. E XPERIMENTAL S ETUP
1) Datasets.
The experiments are evaluated on three typical FSL datasets, mini -Imagenet Vinyals et al.
(2016), tiered -Imagenet
Ren et al.
(2018), and CUB Wah et al.
mini -Imagenet consists ofclasses from the ImageNet, which are split randomly into 64 base, 16 validation, and 20 novel classes.
Each class has 600 samples of size 84 × 84 .
tiered -Imagenet consists of 608 classes from the ImageNet, which are split randomly into 351 base, 97 validation, and 160 novel classes.
CUB contains 200 classes with a total of 11,788 images of size 84 × 84 .
The base, validation, and novel split contain 100, 50, and 50 classes.
2) Backbones.
To show the effectiveness of LP-FT-FB, we used many backbones.
WideResNet-28Zagoruyko & Komodakis (2016) with Dropout Hinton et al.
(2012) is a typical backbone which used in the
SOTA methods, such as S2M2 Mangla et al.
FBR Ghaffari et al.
(2022), and
DC Yang et al.
Also, multiple scale backbones including ResNet-18 He2 (2016) and ResNet-34, are used.
The ResNets are the same as Mangla et al.
Finetuning hyper-parameters.
The pre-trained hyper-parameters are the same as Mangla et al.
We only give the hyper-parameters in the LP-FT process.
For LP, we used the linear classiﬁer proposed in the Baseline++ Chen et al.
For the optimizer, the SGD is used with the learning rate α 1 = 0 .
01 , momentum 0.9, dampening 0.9, and weight decay 1 e − 3 .
For the FBR of classiﬁer, the factor λ in Eq.
2 is set to 1.
For FT, the feature extractor and the classiﬁer are together manually ﬁnetuned with the learning rate α 2 = 1 e − 3 and the i-FBR factor λ inv in Eq.
7 is set to − 1 e − 3 .
4) Evaluation.
The reported results are averaged in percent with 95% conﬁdence interval on 10000 tasks randomly selected by the Episodic Sampler.
As the settings in Yang et al.
(2021) and Ghaffari et al.
(2022), the query sample number is 15 in each task for evaluating the performance.
3. 5- WAY - 1 \ 5 - SHOT TASKS
In-domain 5-way- 1 \ 5 -shot tasks are the most typical FSL tasks.
We directly used the pre-trained
WideResNet-28-10 provided by S 2 M 2 R Mangla et al.
The evaluated results are reported in Table 1.
On mini -Imagenet, LP-FT-FB outperforms FBR by 1.45% and 0.86% for 1 \ 5 -shot tasks respectively.
With the same pre-trained model, LP-FT-FB outperforms S 2 M 2 R by 2.56% and 1.23% for 1 \ 5 -shot tasks.
Similarly, on the tiered -Imagenet, LP-FT-FB outperforms FBR by 3.04% and 3.75% for 1 \ 5 -shot tasks.
On
CUB, LP-FT-FB outperforms S 2 M 2 R by 1.48% and 1.03% for 1 \ 5 -shot tasks. [Table]
E VALUATION U NDER D IFFERENT B ACKBONES
In addition to WideResNet-28-10, we also evaluated LP-FT-FB under different backbones, ResNet-18 and
ResNet-34.
Because S 2 M 2 R did not provided the pre-trained models of ResNet-18 and ResNet-34, we used the code 3 of S 2 M 2 R to reproduce the pre-trained models.
[Table] As given in Table 2, for ResNet-18, LP-FT-FB outperforms S 2 M 2 R by 2.14% and 2.48% for 1 \ 5 -shot tasks on mini -Imagenet.
LP-FT-FB outperforms S 2 M 2 R by 1.93% and 1.33% for 1 \ 5 -shot tasks on CUB.
For ResNet-34, LP-FT-FB outperforms S 2 M 2 R by 2.42% and 3.51% for 1 \ 5 -shot tasks on mini -Imagenet.
LP-FT-FB outperforms S 2 M 2 R by 3.56% and 3.47% for 1 \ 5 -shot tasks on CUB.
3. E VALUATION ON A UGMENTED S AMPLES
The feature extractor is ﬁnetuned, and the novel features are since equivariant while novel samples totally differ from the base samples.
The equivariant features are since closer to the distribution of the novel sample than the unchanged feature extractor used in DC.
To show this, we used DC to augment the yielded features before evaluation.
The evaluated results are given in Table 3.
The data augmented experiments are conducted on the tiered -Imagenet.
For the 1-shot-10-way task, LPFT-FB outperforms DC and FBR by 0.74% and 0.79%, respectively.
For the 5-shot-10-way task, LP-FTFB outperforms DC and FBR by 1.13% and 1.54%, respectively.
Also, for the 15-way tasks, LP-FT-FB outperforms DC and FBR.
[Table]
3. C ROSS - DOMAIN T ASKS
To show the transferring ability brought by LP-FT-FB, we conducted cross-domain FSL experiments.
LPFT-FB is evaluated on the CUB dataset, but the feature extractor is pre-trained on mini -Imagenet and tieredImagenet, respectively.
In this experiment, we used DC to augment the novel samples, just as the evaluated experiments in FBR Ghaffari et al.
LP-FT-FB + DC denotes the proposed method in Table 4.
As is shown in Table 4, for mini -Imagenet → CUB tasks, LP-FT-FB outperforms DC by 2.07% and 2.29% for 5-shot-10-way and 5-shot-15-way tasks.
For tiered -Imagenet → CUB tasks, LP-FT-FB outperforms DC by [Table] [Table]
A BLATION S TUDY
LP-FT v.s. FBR
v.s. i-FBR.
We give the ablation study to explore the effectiveness of each part.
The study is evaluated on mini -Imagenet and tiered -Imagenet for 5-way- 1 \ 5 -shot tasks.
As given in Table 5, we show the effectiveness of LP-FT-FB’s each part
[Table]
[Table]
i-FBR factor λ inv tuning.
We give the tuning of λ inv as follows to show the pattern of i-FBR.
[Table] [Figure]
As given in Fig.
2, we show the tuning trend of λ inv .
The ablation experiments is evaluated on miniImagenet for the 5-way-1-shot task with λ = 1 .
We obtained the best performance with λ inv = − 7 e − 4 .
While λ inv ≥ 0 , i.e., i-FBR is turned into FBR, the performance is decreased.
i-FBR v.s. smoothing regulariers.
The proposed i-FBR is a sort of regularizer smoothing the output distribution of the model.
We compared the performance of different smoothing regularizers.
As given in
Table 6, the proposed i-FBR outperforms the smoothing regulaizers.
The detailed experiment settings are given in Section Appendix.
[Table]
R ELATED W ORKS
The FSL methods are divided into meta-learning-based methods and ﬁnetuning-based methods.
The ﬁnetuning-based methods are divided into Linear-Probing-based and ﬁnetuning-the-whole methods.
Finetuning-learning-based Methods.
In 2019, Chen et al.
proposed Baseline++ Chen et al.
(2019), which showed that a re-trained linear classiﬁer could get a comparable performance to the meta-learningbased methods when the feature extractor is pre-trained.
The pre-trained feature extractors are easy to be obtained without the need of training starting from scratch (used in meta-learning-based methods).
Following Baseline++, Tian et al.
proposed RFS Tian et al.
(2020) to show that learning a supervised or self-supervised representation on the meta-training set, followed by training a linear classiﬁer on top of this representation, gets a good performance better than most meta-learning-based methods.
Similarly, S2M
Mangla et al.
and SKD Rajasegaran et al.
(2020) proposed that the feature extractor pre-trained with
Rotation Augmentation, Manifold Mixup Verma et al.
(2019), and Knowledge Distillation Hinton et al.
(2015) can make the feature extractor more robust.
These methods are called Linear Probing methods because they only re-trained the linear classiﬁer but ﬁxed the pre-trained feature extractor, and they are based on a strong robustness assumption.
However, the robustness assumption may not hold.
To address the problem, Shen et al.
proposed Shen et al.
(2021) a searching algorithm ﬁnding the layers suitable for ﬁnetuning.
The searching algorithm is time-consuming and performs poorly on a single dataset because it does not address the distorted feature problem.
Additionally, the related meta-learning FSL methods is left to Appendix.
C ONCLUSION
In this paper, we proposed LP-FT-FB ﬁnetuning the pre-trained feature extractor to bring strong transferring ability to the feature extractor instead of a strong robustness assumption.
The transferring ability addressed the distorted feature problem caused by the OOD novel samples.
However, the strong transferring ability is at the cost of the over-parameterized feature extractor ﬁnetuning.
The proposed LP-FT-FB is also timeconsuming, just like the ﬁrst-order gradient computation of Reptile Nichol et al.
In future works, we will try to design a ﬁlter-wise or layer-wise ﬁnetuning method instead of the unit-wise one.