R EVISIT F INETUNING STRATEGY FOR F EW -S HOT L EARNING TO S TRENGTHEN THE E QUIVARIANCE OF E MDEDDINGS
Anonymous authors
Paper under double-blind review
A BSTRACT
Few-Shot Learning (FSL) aims to learn a simple and effective bias on limited novel samples.
Recently, many methods have been focused on re-training a randomly initialized linear classiﬁer to adapt it to the novel features extracted by the pre-trained feature extractor (called Linear-Probing-based methods).
These methods typically assumed the pre-trained feature extractor was robust enough, i.e., ﬁnetuning was not needed, and hence the pretrained feature extractor does not change on the novel samples.
However, the unchanged pre-trained feature extractor will distort the features of novel samples because the robustness assumption may not hold, especially on the out-of-distribution samples.
To extract the undistorted features, we designed Linear-Probing-Finetuning with Firth-Bias (LP-FTFB) to yield an accurate bias on the limited samples for better ﬁnetuning the pre-trained feature extractor, imposing equivariance on the whole model.
In LP-FT-FB, we further proposed inverse Firth Bias Reduction (i-FBR) to regularize the over-parameterized feature extractor on which FBR does not work well.
The proposed i-FBR effectively alleviates the over-ﬁtting problem of the feature extractor in the process of ﬁnetuning and helps extract undistorted novel features.
To show the effectiveness of the designed LP-FT-FB, we conducted a lot of experiments on the commonly used FSL datasets under different backbones, including in-domain and cross-domain FSL tasks.
The experimental results show that the proposed FT-LP-FB outperforms the SOTA FSL methods.
I NTRODUCTION
Few-shot Learning (FSL) has recently developed quickly in the limited data regime.
FSL aims to learn the suitable inductive bias on the given limited samples of novel classes.
At the very start, the whole model consisting of the feature extractor and the classiﬁer is pre-trained on the samples of base classes, and they are ﬁnetuned on the limited novel samples to obtain the inductive bias.
The performance of ﬁnetuned model drops signiﬁcantly due to the over-ﬁtting problem of the pre-trained model.
To address the over-ﬁtting problem, meta-learning-based methods such as Prototypical Networks Snell et al.
and
MAML Finn et al.
(2017) were proposed to learn the learning strategies for suitable inductive bias.
Then, Chen et al.
proposed Baseline++ Chen et al.
(2019) to show that a simple linear classiﬁer re-trained on the given novel samples can also get comparable performance to the meta-learning-based methods. The method of retraining the linear classiﬁer is called Linear Probing (LP).
Following Baseline++, many researchers studied the LP-based FSL methods, such as S2M2 Mangla et al.
RFS Tian et al.
(2020), and EMD Zhang et al.
(2020), are proposed to obtain the fully-trained feature extractor with more powerful expressivity.
LPbased FSL methods assumed that a pre-trained feature extractor is robust enough to novel samples Yang et al.
; Tian et al.
and hence does not need to be ﬁnetuned.
[Figure]
However, the robustness assumption may not hold, especially on the out-of-distribution (OOD) novel samples, because the existing regularization methods only provided effective robustness to the in-distribution (ID) samples.
This leads to the “wrong” features of novel samples Kumar et al.
This problem cannot be well addressed by only strengthening the robustness of the pre-trained feature extractor without strengthening the equivariance.
To strengthen the equivariance, ﬁnetuning is a typical and simple technology in the transferring learning on large-scale datasets Kornblith et al.
; He et al.
; Chen et al.
However, the existing FSL ﬁnetuning strategies cannot appropriately ﬁnetune the over-ﬁtted feature extractor Shen et al.
Consequently, designing an appropriate ﬁnetuning strategy for the fully-trained feature extractor on the limited samples is a critical point to strengthen the equivariance and hence make the ﬁnetuned feature extractor yield “good” novel features.
To design an appropriate ﬁnetuning strategy, we started by analyzing the existing ﬁnetuning strategies Kumar et al.
; Kanavati & Tsuneki (2021); Levine et al.
; Radford et al.
(2021), especially on the ODD samples Kumar et al.
Linear-Probing-Finetuning (LP-FT)
Kumar et al.
(2022) theoretically proved that the existing ﬁnetuning strategies underperformed due to the distorted novel features, which is because when trying to ﬁt the novel samples with a randomly initialized linear classiﬁer, the extracted ID features and
OOD features change inconsistently.
The inconsistent problem is caused by the unchanged features of the samples in the space orthogonal to the space spanned by the base samples, i.e., out-of-distribution features unchanged (Section 2.2).
Kumar et al.
proposed using a linear classiﬁer well-trained on the the ﬁtting novel samples to address this underperforming problem.
Although LP-FT is effective, it is used to address the ﬁnetuning problem on large-scale out-of-distribution datasets, which contain enough samples to provide an unbiased estimation.
In the FSL tasks, limited novel samples lead to a biased estimation of the updated parameters.
Seeing this, we reduce the Firth Bias of the model Ghaffari et al.
; FIRTH (1993) to obtain an unbiased estimation (called Firth Bias Reduction).
However, Firth Bias Reduction (FBR) gives an unbiased estimation only for the linear classiﬁer but not the feature extractor.
Experimental result also shows that FBR decreased the performance of the ﬁnetuned feature extractor.
A deep analysis (Section 2.3) shows that when the scaling factor λ (see Eq. 2) is positive,
FBR encourages the distribution of the linear classiﬁer output to be far away from the uniform distribution (See Fig. 1)
- FBR strengthens the inﬂuence of novel samples for linear probing.
This accelerates the con- vergence of the low-parameterized linear classiﬁer but hampers the generalization of the over-parameterized feature extractor.
Based on the analysis, we proposed inverse-FBR
(i-FRB, negative λ in Eq. 2) to encourages the distribution of the linear classiﬁer output to be close to uniform distribution (See Fig. 1) to address the over-ﬁtting problem M¨uller et al.
Combining LP-FT, FBR, and the proposed i-FBR, we proposed Linear-Probing-Finetuning with Firth Bias (LP-FT-FB) to appropriately ﬁnetune the fully-trained feature extractors to adapt them to the target domain by the given limited novel samples.
Firstly, we use linear probing to address the distorted feature problem.
The linear probing is regularized with FBR for unbiased estimation.
Then the whole pre-trained model is ﬁnetuned regularized with the proposed i-FBR on the limited novel samples to impose the equivariance on the feature extractor for more separable features.
The proposed LP-FT-FB makes the extracted features of novel features undistorted and separable.
The whole LP-FT-FB ﬁnetuning strategy is visualized in Fig.
Furthermore, the equivariance brought by the unbiased ﬁnetuned feature extractor makes the novel features close to the domain of novel samples.
This is showed in the cross-domain FSL tasks (Section 3.5).
Our main contributions include (1) designing a novel ﬁnetuning strategy, LP-FT-FB, to impose the equivariance on the model to address the over-ﬁtting problem caused by the unchanged feature extractor; (2) introducing the ﬁnetuning method (LP-FT) designed for large-scale datasets in the ﬁeld of FSL; (3) proposing inverse Firth Bias Reduction (i-FBR) to address the biased estimation problem of ﬁnetuning the feature extractor due to limited samples.
M ETHODS
LP-FT is ﬁrstly described in Section 2.2.
Then, FBR is formulated and analyzed in Section 2.3 to address the unique limited-sample problem in Few Shot Learning, i.e., the biased estimation problem.
However, FBR is not suitable for the over-parameterized feature extractor.
Based on the analysis of FBR, i-FBR is proposed to ﬁnetune the feature extractor.
In Section 2.4, LP-FT-FB is stated.
2. S ET UP
Following FBR Ghaffari et al.
(2022), we assume a multinomial logitic regression model for the classiﬁer v ( · , β ) .
β = { β c } 1 ≤ c ≤ C and β c denotes the logistic regression weights for class c .
B ( · , θ ) denotes the feature extractor.
The dataset D = { ( x i , y i ) } 1 ≤ i ≤ M totally contains C classes and M samples.
The whole inference process is formulated as: ˆ y i = v ( z i , β ) = v ( B ( x i , θ ) , β ) , where z i =
B
( x i , θ ) .
In the multinomial logistic regression model, the assignment probability of z i to class c is formulated as p ci :=
Pr ( y i =
c | z i ) =
e βTc z
i 1+ (cid:80) Cc (cid:48)
=1 e βTc (cid:48)
z i .
Then the logistic log-likelihood function is formulated as
L logistic := (cid:80) Mi
=1 (cid:80)
Cc =1 1 [ y i
=
c ] · log p ci , where 1 [ · ] denotes the y i in the one-hot manner.
For linear probing and ﬁnetuning, we used Cross Entropy Loss L CE = − 1 M (cid:80) Mi
=
[ y i · log( P i )] , where P i = { p ci } Cc =1 .
P i
= ˆ y i denotes the inference probability vector.
2. L INEAR -P ROBING -F INETUNING
LP-FT explored the reason why LP underperforms FT on in-distribution samples but outperforms FT in out-of-distribution samples on large-scale datasets.
LP-FT theoretically proved that when a pre-trained feature extractor tried to ﬁt the ID samples, the variation of the model will not change the feature of out-ofdistribution samples in the space orthogonal to the space spanned by the pre-training samples.
To understand this, let S denote the subspace spanned the training samples X , and the training loss is
L ( v ( B ( X , θ ) , β ) , Y ) =
(cid:107)
X
θ T β − Y (cid:107) 22 .
B is assumed to be a linear model; Y denotes the one-hot labels of
X .
The gradients of the training loss with respect to the parameter θ of the feature extractor B is computed as:
[Equation]
With Eq.
1, if u is a sample in the subspace orthogonal to S , the features of u do not change with the ﬁnetuned
B ( · , θ ) : ∆ z = ∇ θ L ( v ( · , β ) , B ( · , θ ))
u = 2 β
( Y − X θ T β ) T ( X · u ) = 0 .
This leads to the distorted features extracted by the ﬁnetuned B ( · , θ ) because the in-distribution and out-of-distribution features inconsistently change.
However, LP-FT explores the feature extractor ﬁnetuning problem on large-scale datasets.
Observing this, we wish to address the feature extractor ﬁnetuning problem with LP-FT in Few-Shot Learning if we can address the limited-sample problem.
Because the novel samples used to ﬁnetune the pre-trained feature extractor are out-of-distribution in FSL models.
The limited-sample problem is actually a biased estimation problem.
2. INVERSE F IRTH B IAS R EDUCTION
For linear probing, we use Firth Bias Reduction (FRB) to obtain an unbiased estimation.
PMLE FIRTH (1993) added a log-determinant penalty 1 to remove the O ( n − 1 ) term of the asymptotic bias of the maximum likelihood estimate parameters for unbiased estimation.
FBR Ghaffari et al.
(2022) addressed the situation when det( F ) = 0 and proposed an approximate format for linear classiﬁer and cosine classiﬁer.
In Ghaffari et al.
(2022), the FBR loss is formulated as
[Equation]
where U [0 ,C ] denotes the uniform distribution between 0 and C and λ denotes the scaling factor of FBR.
From Eq.
2, we concluded that when λ > 0 , the L Firth encourages the distribution of the logit z i to be far away from U [0 ,C ] , which totally differs from Label Smoothing M ¨ uller et al.
When λ < 0 , L Firth encourages the distribution of the z i to be close to U [0 ,C ] and avoids too high conﬁdence of z i , which coincides with the effectiveness of Label Smoothing M ¨ uller et al.
; Hein et al.
However, the vanilla FBR is not suitable for ﬁnetuning feature extractor because when λ > 0 (used in FBR
Ghaffari et al.
, the logit z is encouraged to be far away from U [0 ,C ] .
The distribution drag leads to the too high similarity between the distribution of the logit z and y , and hence makes the over-parameterized feature extractor over-ﬁtted Hein et al.
Seeing this, based on the analysis of Eq. 2, we proposed inverse
FBR
(i-FBR) to encourage the distribution of the z close to U [0 ,C ] with λ <
(we use the λ inv to denote the scaling factor of i-FBR) to address the over-ﬁtting problem.
The proposed i-FBR works similarly to Label Smoothing to a certain extent because both of them encourage the output distribution of the model to be close to the uniform distribution.
Hence they can reduce the inﬂuence of the training novel samples and alleviate the too high conﬁdence problem.
In addition to the above distribution drag, we computed the gradients of L CE and L with respect to r i = β · B ( x i , θ ) as follows to show the gradient variation brought by λ .
[Equation]
where E is a one-full vector with the same size as P i and P i
= ˆy
i is the predict of the whole model.
The detailed derivation is given in Appendix.
Based on Eq.
3,we concluded that when λ > 0 , the gradient ∂ L ∂ r ci is increased for the low-parameterized linear classiﬁer to quickly reach the optimal point, and hence the linear
where F
Hess probing model performs better Prabhu et al.
The overﬁtting problem does not need to be considered because of the low-parameterized property.
However, it is not suitable for over-parameterized feature extractor ﬁnetuning because of too large similarity between z i and y i .
From Eq.
3, we found that when λ < 0 and the class of x i is c, the gradient ∂ L ∂ r ci is decreased and ∂ L ∂ r ki ( k (cid:54) =
c ) is increased.
This reduces the inﬂuence of x i , which is used to train the overparameterized feature extractor, and hence alleviates the over-ﬁtting problem.
Furthermore, different from
Label Smoothing, the proposed i-FBR decreases the gradients, which is suitable for the ﬁnetuning of the over-parameterized model and hence outperforms Label Smoothing (See Appendix).
2. LP-FT-FB
With FBR, LP-FT, and the proposed i-FBR, we proposed LP-FT-FB to impose equivariance on the feature extractor instead of the robustness assumption to address the over-ﬁtting problem.
Firstly, a randomly initialized linear classiﬁer v ( · , β ) is re-trained on the novel samples and regularized with FBR.
For the
C few - way -K- Shot 2 FSL tasks, the re-trained classiﬁer is denoted by v (cid:48) ( · , β (cid:48) ) , and the classiﬁer is updated as
[Equation]
where r i = β · B 0 ( x i , θ 0 ) ; ˆy = { ˆy i } C few · K i = ;
y = { y i
[Equation]
; α 1 is the LP learning rate.
Then the pre-trained B 0 ( · , θ 0 ) and v (cid:48) ( · , β (cid:48) ) are together ﬁnetuned on the same novel samples and regularized with i-FBR.
FT is stated as follows.
The forward process of FT is formulated as
[Equation]
The loss of FT is computed as:
[Equation]
The feature extractor B 0 ( · , θ 0 ) is updated as
[Equation]
where r (cid:48)
= { r (cid:48) i } K · C few i = and r (cid:48)
i = β (cid:48)
· B 0 ( x i , θ 0 ) .
The linear classiﬁer v (cid:48) ( · , β (cid:48) ) is updated as
[Equation]
where λ inv denoted the scaling factor of i-FBR, and α 2 is the learning rate of FT.
The analytic formula of
Eq. 4, Eq. 8, and Eq.
7 are given in Appendix.
With ˆ B ( · , ˆ θ ) and ˆ v ( · , ˆ β ) , the performance of LP-FT-FB is evaluated on the novel query samples.
The whole ﬂow of LP-FT-FB is given in Appendix.
E XPERIMENTS
To comprehensively show the effectiveness of the proposed i-FBR and LP-FT-FB, we conducted many in-domain experiments: 1) 5-way- 1 \ 5 -shot tasks on mini -Imagenet, tiered -Imagenet, and CUB datasets under the typical FSL backbone, WideResNet-28-10; 2) few-shot tasks under multiple scale backbones including ResNet-18 and ResNet-34; 3) N-way-K-shot ( N > 5 , K > 5) tasks on mini -Imagenet and tieredImagenet; 4) few-shot tasks on the extracted features augmented by Distribution Calibration (DC)
Yang et al.
Also, the cross-domain few-shot tasks, mini -Imagenet → CUB and tiered -Imagenet → CUB, are also evaluated.
3. E XPERIMENTAL S ETUP
1) Datasets.
The experiments are evaluated on three typical FSL datasets, mini -Imagenet Vinyals et al.
(2016), tiered -Imagenet
Ren et al.
(2018), and CUB Wah et al.
mini -Imagenet consists ofclasses from the ImageNet, which are split randomly into 64 base, 16 validation, and 20 novel classes.
Each class has 600 samples of size 84 × 84 .
tiered -Imagenet consists of 608 classes from the ImageNet, which are split randomly into 351 base, 97 validation, and 160 novel classes.
CUB contains 200 classes with a total of 11,788 images of size 84 × 84 .
The base, validation, and novel split contain 100, 50, and 50 classes.
2) Backbones.
To show the effectiveness of LP-FT-FB, we used many backbones.
WideResNet-28Zagoruyko & Komodakis (2016) with Dropout Hinton et al.
(2012) is a typical backbone which used in the
SOTA methods, such as S2M2 Mangla et al.
FBR Ghaffari et al.
(2022), and
DC Yang et al.
Also, multiple scale backbones including ResNet-18 He2 (2016) and ResNet-34, are used.
The ResNets are the same as Mangla et al.
Finetuning hyper-parameters.
The pre-trained hyper-parameters are the same as Mangla et al.
We only give the hyper-parameters in the LP-FT process.
For LP, we used the linear classiﬁer proposed in the Baseline++ Chen et al.
For the optimizer, the SGD is used with the learning rate α 1 = 0 .
01 , momentum 0.9, dampening 0.9, and weight decay 1 e − 3 .
For the FBR of classiﬁer, the factor λ in Eq.
2 is set to 1.
For FT, the feature extractor and the classiﬁer are together manually ﬁnetuned with the learning rate α 2 = 1 e − 3 and the i-FBR factor λ inv in Eq.
7 is set to − 1 e − 3 .
4) Evaluation.
The reported results are averaged in percent with 95% conﬁdence interval on 10000 tasks randomly selected by the Episodic Sampler.
As the settings in Yang et al.
(2021) and Ghaffari et al.
(2022), the query sample number is 15 in each task for evaluating the performance.
3. 5- WAY - 1 \ 5 - SHOT TASKS
In-domain 5-way- 1 \ 5 -shot tasks are the most typical FSL tasks.
We directly used the pre-trained
WideResNet-28-10 provided by S 2 M
R Mangla et al.
The evaluated results are reported in Table 1.
On mini -Imagenet, LP-FT-FB outperforms FBR by 1.45% and 0.86% for 1 \ 5 -shot tasks respectively.
With the same pre-trained model, LP-FT-FB outperforms S 2 M 2 R by 2.56% and 1.23% for 1 \ 5 -shot tasks.
Similarly, on the tiered -Imagenet, LP-FT-FB outperforms FBR by 3.04% and 3.75% for 1 \ 5 -shot tasks.
On
CUB, LP-FT-FB outperforms S 2 M 2 R by 1.48% and 1.03% for 1 \ 5 -shot tasks.
3. E VALUATION U NDER D IFFERENT B ACKBONES
In addition to WideResNet-28-10, we also evaluated LP-FT-FB under different backbones, ResNet-18 and
ResNet-34.
Because S 2 M 2 R did not provided the pre-trained models of ResNet-18 and ResNet-34, we used the code 3 of S 2 M 2 R to reproduce the pre-trained models.
As given in Table 2, for ResNet-18, LP-FT-FB outperforms S 2 M 2 R by 2.14% and 2.48% for 1 \ 5 -shot tasks on mini -Imagenet.
LP-FT-FB outperforms S 2 M 2 R by 1.93% and 1.33% for 1 \ 5 -shot tasks on CUB.
[Table] [Table]
LP-FT-FB 73. For ResNet-34, LP-FT-FB outperforms S 2 M 2 R by 2.42% and 3.51% for 1 \ 5 -shot tasks on mini -Imagenet.
LP-FT-FB outperforms S 2 M 2 R by 3.56% and 3.47% for 1 \ 5 -shot tasks on CUB.
3. E VALUATION ON A UGMENTED S AMPLES
The feature extractor is ﬁnetuned, and the novel features are since equivariant while novel samples totally differ from the base samples.
The equivariant features are since closer to the distribution of the novel sample than the unchanged feature extractor used in DC.
To show this, we used DC to augment the yielded features before evaluation.
The evaluated results are given in Table 3.
The data augmented experiments are conducted on the tiered -Imagenet.
For the 1-shot-10-way task, LPFT-FB outperforms DC and FBR by 0.74% and 0.79%, respectively.
For the 5-shot-10-way task, LP-FTFB outperforms DC and FBR by 1.13% and 1.54%, respectively.
Also, for the 15-way tasks, LP-FT-FB outperforms DC and FBR.
[Table]
3. C ROSS - DOMAIN T ASKS
To show the equivariance brought by LP-FT-FB, we conducted cross-domain FSL experiments.
LP-FT-FB is evaluated on the CUB dataset, but the feature extractor is pre-trained on mini -Imagenet and tiered -Imagenet, respectively.
In this experiment, we used DC to augment the novel samples, just as the evaluated experiments in FBR Ghaffari et al.
LP-FT-FB + DC denotes the proposed method in Table 4.
[Table]
As is shown in Table 4, for mini -Imagenet → CUB tasks, LP-FT-FB outperforms DC by 2.07% and 2.29% for 5-shot-10-way and 5-shot-15-way tasks.
For tiered -Imagenet → CUB tasks
DC by 1.65% and 2.09%
5-shot
10-way
R ELATED W ORKS
The FSL methods are divided into meta-learning-based methods and ﬁnetuning-based methods.
The ﬁnetuning-based methods are divided into Linear-Probing-based and ﬁnetuning-the-whole methods.
Finetuning-learning-based Methods.
In 2019, Chen et al.
proposed Baseline++ Chen et al.
(2019), which showed that a re-trained linear classiﬁer could get a comparable performance to the meta-learningbased methods when the feature extractor is pre-trained.
The pre-trained feature extractors are easy to be obtained without the need of training starting from scratch (used in meta-learning-based methods).
Following Baseline++, Tian et al.
proposed RFS Tian et al.
(2020) to show that learning a supervised or self-supervised representation on the meta-training set, followed by training a linear classiﬁer on top of this representation, gets a good performance better than most meta-learning-based methods.
Similarly, S2M
Mangla et al.
and SKD Rajasegaran et al.
(2020) proposed that the feature extractor pre-trained with
Rotation Augmentation, Manifold Mixup Verma et al.
(2019), and Knowledge Distillation Hinton et al.
(2015) can make the feature extractor.
These methods are called Linear Probing methods because they only re-trained the linear classiﬁer but ﬁxed the pre-trained feature extractor, and they are based on a strong robustness assumption.
However, the robustness assumption may not hold.
To address the problem, Shen et al.
proposed Shen et al.
(2021) a searching algorithm ﬁnding the layers suitable for ﬁnetuning.
The searching algorithm is time-consuming and performs poorly on a single dataset because it does not address the distorted feature problem.
Additionally, the related meta-learning FSL methods is left to Appendix.
C ONCLUSION
In this paper, we proposed LP-FT-FB ﬁnetuning the pre-trained feature extractor to impose strong equivariance on the feature extractor instead of a strong robustness assumption.
The equivariance addressed the distorted feature problem caused by the OOD novel samples.
However, the strong equivariance is at the cost of the over-parameterized feature extractor ﬁnetuning.
The proposed LP-FT-FB is also time-consuming, just like the ﬁrst-order gradient computation of Reptile Nichol et al.
In future works, we will try to design a ﬁlter-wise or layer-wise ﬁnetuning method instead of the unit-wise one.