M ETA - FREE FEW - SHOT LEARNING VIA REPRESENTATION LEARNING WITH WEIGHT AVERAGING
Anonymous authors
Paper under double-blind review
A BSTRACT
Recent studies on few-shot classiﬁcation using transfer learning pose challenges to the effectiveness and efﬁciency of episodic meta-learning algorithms.
Transfer learning approaches are a natural alternative, but they are restricted to few-shot classiﬁcation.
Moreover, little attention has been on the development of probabilistic models with well-calibrated uncertainty from few-shot samples, except for some Bayesian episodic learning algorithms.
To tackle the aforementioned issues, we propose a new transfer learning method to obtain accurate and reliable models for few-shot regression and classiﬁcation.
The resulting method does not require episodic meta-learning and is called meta-free representation learning (MFRL).
MFRL ﬁrst ﬁnds low-rank representation generalizing well on meta-test tasks.
Given the learned representation, probabilistic linear models are ﬁne-tuned with few-shot samples to obtain models with well-calibrated uncertainty.
The proposed method not only achieves the highest accuracy on a wide range of few-shot learning benchmark datasets but also correctly quantiﬁes the prediction uncertainty.
In addition, weight averaging and temperature scaling are effective in improving the accuracy and reliability of few-shot learning in existing meta-learning algorithms with a wide range of learning paradigms and model architectures.
I NTRODUCTION
Currently, the vast majority of few-shot learning methods are within the general paradigm of metalearning (a.k.a.
learning to learn) (Bengio et al., 1991; Schmidhuber, 1987; Thrun & Pratt, 1998), which learns multiple tasks in an episodic manner to distill transferrable knowledge (Vinyals et al., 2016; Finn et al., 2017; Snell et al., 2017).
Although many episodic meta-learning methods report state-of-the-art (SOTA) performance, recent studies show that simple transfer learning methods with ﬁxed embeddings (Chen et al., 2019; Tian et al., 2020) can achieve similar or better performance in few-shot learning.
It is found that the effectiveness of optimization-based meta-learning algorithms is due to reusing high-quality representation, instead of rapid learning of task-speciﬁc representation (Raghu et al., 2020).
The quality of the presentation is not quantitatively deﬁned, except for some empirical case studies (Goldblum et al., 2020).
Recent machine learning theories (Saunshi et al., 2021) indicate that low-rank representation leads to better sample efﬁciency in learning a new task.
However, those theoretical studies are within the paradigm of meta-learning and do not reveal how to obtain low-rank representation for few-shot learning outside the realm of meta-learning.
This motivates us to investigate ways to improve the representation for adapting to new few-shot tasks in a meta-free manner by taking the advantage of simplicity and robustness in transfer learning.
In parallel, existing transfer learning methods also have limitations.
That is, the existing transfer learning methods may not ﬁnd representation generalizing well to unseen few-shot tasks (Chen et al., 2019; Dhillon et al., 2020) , compared with state-of-the-art meta-learning methods (Ye et al., 2020; Zhang et al., 2020).
Although some transfer learning methods utilize knowledge distillation and self-supervised training to achieve strong performance in few-shot classiﬁcation, they are restricted to few-shot classiﬁcation problems (Mangla et al., 2020; Tian et al., 2020).
To the best of our knowledge, no transfer learning method is developed to achieve similar performance to metalearning in few-shot regression.
As such, it is desirable to have a transfer learning method that ﬁnds high-quality representation generalizing well to unseen classiﬁcation and regression problems.
The last limitation of the existing transfer learning methods in few-shot learning is the lack of uncertainty calibration.
Uncertainty quantiﬁcation is concerned with the quantiﬁcation of how likely certain outcomes are.
Despite a plethora of few-shot learning methods (in fact, machine learning in general) to improve the point estimation accuracy, few methods are developed to get probabilistic models with improved uncertainty calibration by integrating Bayesian learning into episodic metatraining (Grant et al., 2018; Finn et al., 2018; Yoon et al., 2018; Snell & Zemel, 2021).
Few-shot learning models can be used in risk-averse applications such as medical diagnosis (Prabhu et al., 2019).
The diagnosis decision is made on not only point estimation but also probabilities associated with the prediction.
The risk of making wrong decisions is signiﬁcant when using uncalibrated models (Begoli et al., 2019).
Thus, the development of proper ﬁne-tuning steps to achieve well-calibrated models is the key towards practical applications of transfer learning in few-shot learning.
In this paper, we develop a simple transfer learning method as our own baseline to allow easy regularization towards more generalizable representation and calibration of prediction uncertainty.
The regularization in the proposed transfer learning method works for regression and classiﬁcation problems so that we can handle both problems within a common architecture.
The calibration procedure is easily integrated into the developed transfer learning method to obtain few-shot learning models with good uncertainty quantiﬁcation. Therefore, the resulting method, called Meta-Free Representation Learning (MFRL), overcomes the aforementioned limitations in existing transfer learning methods for few-shot learning.
Our empirical studies demonstrate that the relatively overlooked transfer learning method can achieve high accuracy and well-calibrated uncertainty in few-shot learning when it is combined with the proper regularization and calibration. Those two tools are also portable to meta-learning methods to improve accuracy and calibration, but the improvement is less signiﬁcant compared with that of transfer learning.
We use stochastic weight averaging (SWA) (Izmailov et al., 2018), which is agnostic to loss function types, as implicit regularization to improve the generalization capability of the representation. We also shed light on that the effectiveness of SWA is due to its bias towards low-rank representation.
To address the issue of uncertainty quantiﬁcation, we ﬁne-tune appropriate linear layers during the meta-test phase to get models with well-calibrated uncertainty.
In MFRL, hierarchical Bayesian linear models are used to properly capture the uncertainty from very limited training samples in few-shot regression, whereas the softmax output is scaled with a temperature parameter to make the few-shot classiﬁcation model well-calibrated.
Our method is the ﬁrst one to achieve well-calibrated few-shot models by only ﬁne-tuning probabilistic linear models in the meta-test phase, without any learning mechanisms related to the meta-training or representation learning phase.
Our contributions in this work are summarized as follows:
• We propose a transfer learning method that can handle both few-shot regression and classiﬁcation problems with performance exceeding SOTA.
• For the ﬁrst time, we empirically ﬁnd the implicit regularization of SWA towards low-rank representation, which is a useful property in transferring to few-shot tasks.
• The proposed method results in well-calibrated uncertainty in few-shot learning models while preserving SOTA accuracy.
• The implicit regularization of SWA and temperature scaling factor can be applied to existing meta-learning methods to improve their accuracy and reliability in few-shot learning.
R ELATED W ORK
Episodic meta-learning approaches can be categorized into metric-based and optimization-based methods.
Metric-based methods project input data to feature vectors through nonlinear embeddings and compare their similarity to make the prediction.
Examples of similarity metrics include the weighted L 1 metric (Koch et al., 2015), cosine similarity (Qi et al., 2018; Vinyals et al., 2016), and
Euclidean distance to class-mean representation (Snell et al., 2017).
Instead of relying on predeﬁned metrics, learnable similarity metrics are introduced to improve the few-shot classiﬁcation performance (Oreshkin et al., 2018; Sung et al., 2018).
Recent metric-based approaches focus on developing task-adaptive embeddings to improve few-shot classiﬁcation accuracy.
Those task-adaptive embeddings include attention mechanisms for feature transformation (Fei et al., 2021;
Gidaris &
Komodakis, 2018; Ye et al., 2020; Zhang et al., 2021), graph neural networks (Garcia & Estrach, 2018), implicit class representation (Ravichandran et al., 2019), and task-dependent conditioning (Oreshkin et al., 2018; Yoon et al., 2020; 2019).
Although metric-based approaches achieve strong performance in few-shot classiﬁcation, they cannot be directly applied to regression problems.
Optimization-based meta-learning approaches try to ﬁnd transferrable knowledge and adapt to new tasks quickly.
An elegant and powerful meta-learning approach, termed model-agnostic metalearning (MAML), solves a bi-level optimization problem to ﬁnd good initialization of model parameters (Finn et al., 2017).
However, MAML has a variety of issues, such as sensitivity to neural network architectures, instability during training, arduous hyperparameter tuning, and high computational cost.
On this basis, some follow-up methods have been developed to simplify, stabilize and improve the training process of MAML (Antoniou et al., 2018; Flennerhag et al., 2020; Lee &
Choi, 2018; Nichol et al., 2018; Park & Oliva, 2019).
In practice, it is very challenging to learn high-dimensional model parameters in a low-data regime.
Latent embedding optimization (LEO) attempts to learn low-dimensional representation to generate high-dimensional model parameters (Rusu et al., 2019).
Meanwhile, R2-D2 (Bertinetto et al., 2019) and MetaOptNet (Lee et al., 2019) reduce the dimensionality of trainable model parameters by freezing feature extraction layers during inner loop optimization.
Note that the proposed method is fundamentally different from R2-D2 and
MetaOptNet because our method requires neither episodic meta-learning nor bi-level optimization.
Transfer learning approaches ﬁrst learn a feature extractor on all training data through standard supervised learning, and then ﬁne-tune a linear predictor on top of the learned feature extractor in a new task (Chen et al., 2019).
However, vanilla transfer learning methods for few-shot learning do not take extra steps to make the learned representation generalizing well to unseen meta-test tasks.
Some approaches in this paradigm are developed to improve the quality of representation and boost the accuracy of few-shot classiﬁcation, including cooperative ensembles (Dvornik et al., 2019), knowledge distillation (Tian et al., 2020), and auxiliary self-supervised learning (Mangla et al., 2020).
Nevertheless, those transfer learning methods are restricted to few-shot classiﬁcation.
MFRL aims to ﬁnd representation generalizing well from the perspective of low-rank representation learning, which is supported by recent theoretical studies (Saunshi et al., 2021).
Furthermore, MFLR is the ﬁrst transfer learning method that can handle both few-shot regression and classiﬁcation problems and make predictions with well-calibrated uncertainty.
B ACKGROUND 3. E PISODIC META - LEARNING
In episodic meta-learning, the meta-training data contains T episodes or tasks, where the τ th episode consists of data D τ = { ( x τ,j , y τ,j ) } N τ j =1 with N τ samples.
Tasks and episodes are used interchangeably in the rest of the paper.
Episodic meta-learning algorithms aim to ﬁnd common model parameters θ which can be quickly adapted to task-speciﬁc parameters
φ τ ( τ = 1 , ..., T ).
For example,
MAML-type algorithms assume φ τ is one or a few gradient steps away from θ
(Finn et al., 2017; 2018; Grant et al., 2018; Yoon et al., 2018), while other meta-learning approaches assume that φ τ and θ share the parameters in the feature extractor and only differ in the top layer (Bertinetto et al., 2019; Lee et al., 2019;
Snell et al., 2017).
3. S TOCHASTIC WEIGHT AVERAGING
The idea of stochastic weight averaging (SWA) along the trajectory of SGD goes back to
Polyak–Ruppert averaging (Polyak & Juditsky, 1992).
Theoretically, weight averaging results in faster convergence for linear models in supervised learning and reinforcement learning(Bach &
Moulines, 2013; Lakshminarayanan & Szepesvari, 2018).
In deep learning, we are more interested in tail stochastic weight averaging (Jain et al., 2018), which averages the weights after T training epochs.
The averaged model parameters θ SWA can be computed by running s additional training epochs using SGD
[Equation]
where θ i denotes the model parameters at the end of the i -th epoch.
SWA has been applied to supervised learning of deep neural neural networks to achieve higher test accuracy (Izmailov et al., 2018).
M ETHODOLOGY
The proposed method is a two-step learning algorithm: meta-free representation learning followed by ﬁne-tuning.
We employ SWA to make the learned representation low-rank and better generalize to meta-test data.
Given a meta-test task, a new top layer is ﬁne-tuned with few-shot samples to obtain probabilistic models with well-calibrated uncertainty.
Note that MFRL can be used for both regression and classiﬁcation depending on the loss function.
The pseudocode of MFRL is presented in Appendix A.1.
4. R EPRESENTATION L EARNING
Common representation can be learned via maximization of the likelihood of all training data with respect to θ rather than following episodic meta-learning.
To do so, we group the data
D τ = { ( x τ,j , y τ,j ) } N τ j =1 from all meta-training tasks into a single dataset D tr .
Given aggregated training data D tr = { X , Y } , representation can be learned by maximizing the likelihood p ( D tr | θ ) with respect to θ .
Let θ =
[ θ f , W ] , where θ f represents parameters in the feature extractor and W denotes the parameters in the top linear layer.
The feature extractor h ( x ) ∈ R p is a neural network parameterized by θ f and outputs a feature vector of dimension p .
The speciﬁc form of the loss function depends on whether the task is regression or classiﬁcation and can be given as follows:
[Equation]
classiﬁcation where
[Equation]
For regression problems, the model learns T regression tasks ( W = [ w 1 , ..., w T ] ∈ R ( p +1) ×T ) simultaneously using the loss function L MSE given in Eq. 2, whereas the model learns a C -class classiﬁcation model 1 ( W = [ w 1 , ..., w C ] ∈ R ( p +1) ×C ) for classiﬁcation problems using the loss function L CE in Eq.
The loss function - either Eq. 2 or 3 - can be minimized through standard stochastic gradient descent, where N (cid:48)
=
(cid:80) T τ =1 N τ is the total number of training samples.
Post-processing via SWA Minimizing the loss functions in Eq.
2 and 3 by SGD may not necessarily result in representation that generalizes well to few-shot learning tasks in the meta-test set.
The last hidden layer of a modern deep neural network is high-dimensional and may contain spurious features that over-ﬁt the meta-training data.
Recent meta-learning theories indicate that better sample complexity in learning a new task can be achieved via low-rank representation, whose singular values decay faster (Saunshi et al., 2021).
We aim to ﬁnd low-rank representation Φ = h ( X ) without episodic meta-learning, which is equivalent to ﬁnding the conjugate kernel K C
=
ΦΦ (cid:62) with fast decaying eigenvalues.
To link the representation with the parameter space, we can linearize the neural network by the ﬁrst-order Taylor expansion at θ T and get the ﬁnite width neural tangent kernel (
[Equation]
) J ( X ) (cid:62) , where J ( X ) = ∇ θ f θ ( X ) ∈ R N (cid:48)
×| θ | is the Jacobian matrix, and K NTK is a composite kernel containing K C (Fan & Wang, 2020).
The distributions of eigenvalues for K NTK and K C are empirically similar.
Analyzing K NTK could shed light on the properties of K C .
In parallel, K NTK shares the same eigenvalues of the Gauss-Newton matrix
G = 1 N (cid:48)
J ( X ) (cid:62) J ( X ) .
For linearized networks with squared loss, the Gauss-Newton matrix G well approximates the Hessian matrix H when y is well-described by f θ ( x ) (Martens, 2020).
This is the case when SGD converges to θ T within a local minimum basin.
A Hessian matrix with a lot of small eigenvalues corresponds to a ﬂat minimum, where the loss function is less sensitive to the perturbation of model parameters (Keskar et al., 2017).
It is known that averaging the weights after
SGD convergence in a local minimum basin pushes θ T towards the ﬂat side of the loss valley (He et al., 2019).
As a result, SWA could result in a faster decay of eigenvalues in the kernel matrix, and thus low-rank representation.
Our conjecture about SWA as implicit regularization towards low-rank representation is empirically veriﬁed in Section 5.
4. F INE -T UNING
After representation learning is complete, W is discarded and θ f is frozen in a new few-shot task.
Given the learned representation, we train a new probabilistic top layer in a meta-test task using fewshot samples.
The new top layer will be conﬁgured differently depending on whether the few-shot task is a regression or a classiﬁcation problem.
In a few-shot regression task , we learn a new linear regression model y = w (cid:62) h
( x ) + (cid:15) on a ﬁxed feature extractor h ( x ) ∈ R p with few-shot training data D
= { ( x i , y i ) } ni =1 , where w denotes the model parameters and (cid:15) is Gaussian noise with zero mean and variance σ 2 .
To avoid interpolation on few-shot training data ( n (cid:28) p ), a Gaussian prior p ( w | λ ) = (cid:81) pi =
0 N ( w i | 0 , λ ) is placed over w , where λ is the precision in the Gaussian prior.
However, it is difﬁcult to obtain an appropriate value for λ in a few-shot regression task because no validation data is available in D .
Hierarchical Bayesian linear models can be used to obtain optimal regularization strength and grounded uncertainty estimation using few-shot training data only.
To complete the speciﬁcation of the hierarchical Bayesian model, the hyperpriors on λ and σ 2 are deﬁned as p ( λ ) =
Gamma ( λ | a, b ) and p ( σ − 2 ) =
Gamma( σ − 2 | c, d ) , respectively.
The hyper-priors become very ﬂat and non-informative when a , b , c and d are set to very small values.
The posterior over all latent variables given the data is p (cid:0) w , λ, σ 2 | X , y (cid:1) , where X = { x } ni =1 and y =
{ y i } ni
=1 .
However, the posterior distribution p (cid:0) w , λ, σ 2 | X , y (cid:1) is intractable.
The iterative optimization based approximate inference (Tipping, 2001) is chosen because it is highly efﬁcient.
The point estimation for λ and σ 2 is obtained by maximizing the marginal likelihood function p (cid:0) y | X , λ, σ 2 (cid:1) .
The posterior of model parameters p (cid:0) w | X , y , λ, σ 2 (cid:1) is calculated using the estimated λ and σ 2 .
Previous two steps are repeated alternately until convergence.
The predictive distribution for a new sample x ∗ is
[Equation]
which can be computed analytically because both distributions on the right hand side of Eq.are Gaussian.
Consequently, hierarchical Bayesian linear models avoids over-ﬁtting on few-shot training data and quantiﬁes predictive uncertainty.
In a few-shot classiﬁcation task , a new logistic regression model is learned with the post-processed representation.
A typical K -way n -shot classiﬁcation task D
=
( x i , y i ) } nKi =1 consists of K classes (different from meta-training classes) and n training samples per class.
Minimizing an un-regularized cross-entropy loss results in a signiﬁcantly over-conﬁdent classiﬁcation model because the norm of logistic regression model parameters W ∈ R ( p +1)
× K becomes very large when few-shot training samples can be perfectly separated in the setting nK (cid:28) p .
A weighted L 2 regularization term is added to the cross-entropy loss to mitigate the issue
[Equation]
where λ is the regularization coefﬁcient, which affects the prediction accuracy and uncertainty.
It is difﬁcult to select an appropriate value of λ in each of the meta-test tasks due to the lack of validation data in D .
We instead treat λ as a global hyper-parameter so that the value of λ should be determined based on the accuracy on meta-validation data.
Note that the selected λ with high validation accuracy does not necessarily lead to well calibrated classiﬁcation models.
As such, we introduce the temperature scaling factor (Guo et al., 2017) as another global hyper-parameter to scale the softmax output.
Given a test sample x ∗ , the predicted probability for class c becomes
[Equation]
where T is the temperature scaling factor.
In practice, we select the L2 regularization coefﬁcient λ and the temperature scaling factor T as follows. At ﬁrst, we set T to 1, and do grid search on the meta-validation data to ﬁnd the λ resulting in the highest meta-validation accuracy. However, ﬁnetuning λ does not ensure good calibration. It is the temperature scaling factor that ensures the good uncertainty calibration.
Similarly, we do grid search of T on the meta-validation set, and choose the temperature scaling factor resulting in the lowest expected calibration error (Guo et al., 2017).
Note that different values of T do not affect the classiﬁcation accuracy because temperature scaling is accuracy preserving.
E XPERIMENTS
We follow the standard setup in few-shot learning literature.
The model is trained on a meta-training dataset and hyper-parameters are selected based on the performance on a meta-validation dataset.
The ﬁnal performance of the model is evaluated on a meta-test dataset.
The proposed method is applied to few-shot regression and classiﬁcation problems and compared against a wide range of alternative methods.
5. F EW - SHOT REGRESSION RESULTS
Sine waves (Finn et al., 2017) and head pose estimation (Patacchiola et al., 2020) datasets are used to evaluate the performance of MFRL in few-shot regression.
We use the same backbones in literature (Patacchiola et al., 2020) to make fair comparisons.
Details of the few-shot regression experiments can be found in Appendix A.2.
The results for few-shot regression are summarized in Table 1.
In the sine wave few-shot regression,
MFRL outperforms all meta-learning methods, demonstrating that high-quality representation can be learned in supervised learning, without episodic meta-learning.
Although DKT with a spectral mixture (SM) kernel achieves high accuracy, the good performance should be attributed to the strong inductive bias to periodic functions in the SM kernel (Wilson & Adams, 2013).
Additional results for
MFRL with different activation functions are reported in Appendix A.3.
In the head pose estimation experiment, MFRL also achieves the best accuracy.
In both few-shot regression problems, SWA results in improved accuracy, suggesting that SWA can improve the quality of features and facilitate the learning of downstream tasks.
In Fig.
1, uncertainty is correctly estimated by the hierarchical
Bayesian linear model with learned features using just 10 training samples.
[Figure]
[Figure]
et al.,
[Figure]
MAML (
[Figure]
et al., 2018)
[Figure]
ALPaCA (Harrison et al., 2018)
[Figure]
R2D2 (Bertinetto et al., 2019)
[Figure]
[Table]
5. F EW - SHOT CLASSIFICATION RESULTS
We conduct few-shot classiﬁcation experiments on four widely used few-shot image recognition benchmarks: miniImageNet (Ravi & Larochelle, 2017), tieredImageNet (Ren et al., 2018), CIFARFS (Bertinetto et al., 2019)
[Table]
and FC100 (Oreshkin et al., 2018).
In addition, we test our approach on a cross-domain few-shot classiﬁcation task from the miniImageNet to CUB.
The experiment details about the few-shot classiﬁcation datasets can be found in Appendix A.2.
The proposed method is applied to three widely used network architectures: ResNet-12 (Lee et al., 2019; Ravichandran et al., 2019), wide ResNet (WRN-28-10) (Dhillon et al., 2020; Rusu et al., 2019) [Table] and a 4-layer convolutional neural network with 64 channels (Chen et al., 2019; Patacchiola et al., 2020) (in Appendix A.3)
[Table]
[Table]
[Table]
The results of the proposed method and previous SOTA methods using similar backbones are reported in Table 2 and 3.
The proposed method achieves the best performance in most of the experiments when compared with previous SOTA methods.
Our method is closely related to Baseline++ (Chen et al., 2019) and ﬁne-tuning on logits (Dhillon et al., 2020).
Baseline++ normalizes both classiﬁcation weights and features, while the proposed method only normalizes features.
It allows our method to ﬁnd a more accurate model in a more ﬂexible hypothesis space, given high-quality representation.
Compared with ﬁne-tuning on logits, our method obtains better results by learning a new logistic regression model on features, which store richer information about the data.
Some approaches pretrain a C -class classiﬁcation model on all training data and then apply highly sophisticated meta-learning techniques to the pretrained model to achieve SOTA performance (Rusu et al., 2019; Sun et al., 2019).
Our approach with SWA outperforms those pretrained-then-meta-learned models, which demonstrates that SWA obtains high-quality representation that generalizes well to unseen tasks.
Compared with improving representation quality for few-shot classiﬁcation via selfdistillation (Tian et al., 2020), the computational cost of SWA is signiﬁcantly smaller because it does not require training models from scratch multiple times.
Moreover, SWA can be applied to ﬁnd good representation for both few-shot regression and classiﬁcation, while previous transfer learning approaches can only handle few-shot classiﬁcation problems (Mangla et al., 2020; Tian et al., 2020).
MFRL is also applied to the cross-domain few-shot classiﬁcation task as summarized in Table 4.
MFRL outperforms other methods in this challenging task, indicating that the learned representation has strong generalization capability. We use the same hyperparameters (training epochs, learning rate, learning rate in SWA, SWA epoch, etc.) as in Table 2. The strong results indicate that MFRL is robust to hyperparameter choice.
Surprisingly, meta-learning methods with adaptive embeddings do not outperform simple transfer learning methods like Baseline++ when the domain gap between base classes and novel classes is large.
We notice that Tian et al.
(2020) also reports similar results that transfer learning methods show superior performance on a large-scale cross-domain few-shot classiﬁcation dataset.
We still believe that adaptive embeddings should be helpful when the domain gap between base and novel classes is large.
Nevertheless, how to properly train a model to obtain useful adaptive embeddings in novel tasks is an open question.
[Table]
5. E FFECTIVE RANK OF THE REPRESENTATION
The rank of representation deﬁnes the number of independent bases.
For deep learning, noise in gradients and numerical imprecision can cause the resulting matrix to be full-rank.
Therefore, simply counting the number of non-zero singular values may not be an effective way to measure the rank of the representation.
To compare the effective ranks
[Table]
we plot the normalized singular values of the representation of meta-test data in Fig.
[Table]
where the representation with SWA has a faster decay in singular values [Table] thus indicating the lower effective rank of the presentation with SWA. The results empirically verify our conjecture that SWA is an implicit regularizer towards low-rank representation.
[Figure]
i n g u l a r v a l u e i / m a x
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
1.0 Confidence 0.
A cc u r a c y
ECE: 0.0080 MCE: 0.0437 BRI: 0.
[Figure]
Net Calibrated 0.
1.0 Confidence 0.
A cc u r a c y
ECE: 0.0040 MCE: 0.0135 BRI: 0.
MFRL
5. F EW - SHOT CLASSIFICATION RELIABILITY
The proposed method not only achieves high accuracy in few-shot classiﬁcation but also makes the classiﬁcation uncertainty well-calibrated.
A reliability diagram can be used to check model calibration visually, which plots an identity function between prediction accuracy and conﬁdence when the model is perfectly calibrated (DeGroot & Fienberg, 1983).
Fig.
3 shows the classiﬁcation reliability diagrams along with widely used metrics for uncertainty calibration, including expected calibration error (ECE) (Guo et al., 2017), maximum calibration error (MCE)
(Naeini et al., 2015), and Brier score (BRI) (Brier, 1950).
ECE measures the average binned difference between conﬁdence and accuracy, while MCE measures the maximum difference.
BRI is the squared error between the predicted probabilities and one-hot labels.
MAML is over-conﬁdent because tuning a deep neural network on few-shot data is prone to over-ﬁtting.
Meanwhile, Proto Net and Matching Net are better calibrated than MAML because they do not ﬁne-tune the entire network during testing.
Nevertheless, they are still slightly over-conﬁdent.
The results indicate that MFRL with a global temperature scaling factor can learn well-calibrated models from very limited training samples.
5. A PPLICATION IN META - LEARNING
Meanwhile, we also apply SWA to episodic meta-learning methods, such Proto Net, MAML and
Matching Net, to improve their classiﬁcation accuracy.
The results in Table 5 indicate that SWA can improve the few-shot classiﬁcation accuracy in both transfer learning and episodic meta-learning.
SWA is orthogonal to the learning paradigm and model architecture.
Thus, SWA can be applied to a wide range of few-shot learning methods to improve accuracy.
[Table]
Furthermore, the temperature scaling factor can be applied to calibrate meta-learning methods, including MAML, Proto Net, and Matching Net.
The reliability diagrams in Fig.
3 indicate that the temperature scaling factor not only calibrates classiﬁcation uncertainty of transfer learning approaches, such as the proposed MFRL, but also makes the classiﬁcation uncertainty well-calibrated in episodic meta-learning methods.
Therefore, the temperature scaling factor can be applied to a wide range of few-shot classiﬁcation methods to get well-calibrated uncertainty, while preserving the classiﬁcation accuracy.
D ISCUSSION
SWA has been applied to supervised learning of deep neural networks (Izmailov et al., 2018; Athiwaratkun et al., 2019) and its effectiveness was attributed to convergence to a solution on the ﬂat side of an asymmetric loss valley (He et al., 2019).
However, it does not explain the effectiveness of SWA in few-shot learning because the meta-training and meta-testing losses are not comparable after the top layer is retrained by the few-shot support data in a meta-test task.
The effectiveness of SWA in few-shot learning must be related to the property of the representation.
Although our results empirically demonstrate that SWA results in low-rank representation, further research about their connection is needed.
Explicit regularizers can also be used to obtain simple input-output functions in deep neural networks and low-rank representation, including L1 regularization, nuclear norm, spectral norm, and
Frobenius norm (Bartlett et al., 2017; Neyshabur et al., 2018; Sanyal et al., 2020).
However, some of those explicit regularizers are not compatible with standard SGD training or are computationally expensive.
In addition, it is difﬁcult to choose the appropriate strength of explicit regularization.
Too strong explicit regularization can bias towards simple solutions that do not ﬁt the data.
In comparison, SWA is an implicit regularizer that is completely compatible with the standard SGD training without much extra computational cost.
Thus, it can be easily combined with transfer learning and meta-learning to obtain more accurate few-shot learning models.
In parallel, SWA is also robust to the choice of the hyperparameters - the learning rate and training epochs in the SWA stage (see details in Appendix A.4).
C ONCLUSIONS
In this article, we propose MFRL to obtain accurate and reliable few-shot learning models.
SWA is an implicit regularizer towards low-rank representation, which generalizes well to unseen meta-test tasks.
The proposed method can be applied to both classiﬁcation and regression tasks.
Extensive experiments show that our method not only outperforms other SOTA methods on various datasets but also correctly quantiﬁes the uncertainty in prediction
A A PPENDIX A. P SEUDO CODE FOR MFRL
Merge all training tasks D tr = {D τ } T τ =
Initialize model parameters θ =
[ θ f , W ]
Maximize the likelihood on all training data p ( D tr | θ ) using SGD
Minimize the squared loss for regression problems Minimize the cross-entropy loss for classiﬁcation problems Run SWA to obtain θ SWA Discard W and freeze θ f
Learn a new top layer using support data D in a test task:
Learn a hierarchical Bayesian linear model for a regression task
Learn a logistic regression model with the temperature scaling factor for a classiﬁcation task
A. E XPERIMENT DETAILS
Sine waves are generated by y =
A sin( x − ϕ )
+ (cid:15) , where amplitude
A ∈ [0 . 1 , 5 . 0] , phase ϕ ∈ [0 , π ] and (cid:15) is white noise with standard deviation of 0.1 (Finn et al., 2017).
Each sine wave containssamples by sampling x uniformly from [ − 5 . 0 , 5 . 0] .
We generate 500 waves for training, validation and testing, respectively.
All sine waves are different from each other.
We use the same backbone network described in MAML (Finn et al., 2017): a two-layer MLP with 40 hidden units in each layer.
We use the SGD optimizer with a learning rate of 10 − 3 over 8 × 10 4 training iterations and run SWA over 2 × 10 4 training iterations with a learning rate of 0.05.
Head pose regression data is derived from the Queen Mary University of London
multi-view face dataset (Gong et al., 1996).
It contains images from 37 people and 133 facial images per person.
Facial images cover a view sphere of 90 ◦ in yaw and 120 ◦ in tilt.
The dataset is divided intotraining samples (24 people), 1064 validation samples (8 people), and 665 test samples (5 people).
We use the same feature extractor described in literature (Patacchiola et al., 2020): a three-layer convolutional neural network, each with 36 output channels, stride 2, and dilation 2.
We train the model on the training people set for 300 epochs using the SGD optimizer with a learning rate of 0.01 and run 25 epochs of SWA with a learning rate of 0.01.
miniImageNet is a 100-class subset of the original ImageNet dataset (Deng et al., 2009) for fewshot learning (Vinyals et al., 2016).
Each class contains 600 images in RGB format of the size× 84.
miniImageNet is split into 64 training classes, 16 validation classes, and 20 testing classes, following the widely used data splitting protocol (Ravi & Larochelle, 2017).
tieredImageNet is another subset of the ImageNet dataset for few-shot learning (Ren et al., 2018).
It contains 608 classes grouped into 34 categories, which are split into 20 training categories ( classes), 6 validation categories (97 classes), and 8 testing categories (160 classes).
Compared with miniImageNet, training classes in tieredImageNet are sufﬁciently distinct from test classes, making few-shot classiﬁcation more difﬁcult.
CIFAR-FS is a derivative of the original CIFAR-100 dataset by randomly splitting 100 classes into 64, 16, and 20 classes for training, validation, and testing, respectively (Bertinetto et al., 2019).
FC100 is another derivative of CIFAR-100 with minimized overlapped information between train classes and test classes by grouping the 100 classes into 20 superclasses (Oreshkin et al., 2018).
They are further split into 60 training classes (12 superclasses), 20 validation classes (4 superclasses), and 20 test classes (4 superclasses).
miniImageNet to CUB is a cross-domain few-shot classiﬁcation task, where the models are trained on miniImageNet and tested on CUB (Welinder et al., 2010).
Cross-domain few-shot classiﬁcation is more challenging due to the big domain gap between two datasets.
We can better evaluate the generalization capability in different algorithms.
We follow the experiment setup in Yue et al.
and use the WRN2-28-10 as the backbone. The backbone model is trained on all training classes using C -class cross-entropy loss by the SGD optimizer (momentum of 0.9 and weight decay of 1e-4) with a mini-batch size of 64.
The learning rate is initialized as 0.05 and is decayed by 0.1 after 60, 80, and 90 epochs (100 epochs in total).
After the SGD training converges, we run 100 epochs of SWA with a learning rate of 0.02.
Note that MFRL is not sensitive to training epochs and learning rates in SWA (see Appendix A.4).
The training images are augmented with random crop, random horizontal ﬂip, and color jitter.
During testing, we conduct 5 independent runs of 600 randomly sampled few-shot classiﬁcation tasks from test classes and calculate the average accuracy.
Each task contains 5 classes, 1 × 5 or 5 × 5 support samples, and 75 query samples.
A logistic regression model is learned using only the support samples.
The classiﬁcation accuracy is evaluated on the query samples.
A. A DDITIONAL RESULTS ON FEW - SHOT REGRESSION AND CLASSIFICATION
The additional results on few-shot regression using different activation functions are reported in
Table 6.
MFRL achieves high accuracy with different activation functions.
[Table]
The few-shot classiﬁcation results using a 4-layer convolutional neural network (or similar architectures) are reported in Table 7 and 8.
Similar to the results using ResNet-12 and WRN-28-10, the proposed method outperforms a wide range of meta-learning approaches.
Our method is only
[Figure]
[Figure]
[Table]
[Table]
second to few-shot embedding adaptation with transformer (FEAT) (Ye et al., 2020) on miniImageNet dataset.
Recently, meta-learned attention modules are built on top of the convolutional neural network to get improved few-shot classiﬁcation accuracy.
Direct comparison to those methods with attention modules (Ye et al., 2020; Fei et al., 2021; Zhang et al., 2021) may not be fair because recent studies show that transformer itself can achieve better results than convolutional neural networks in image classiﬁcation (Dosovitskiy et al., 2021).
It is difﬁcult to determine whether the performance improvement is due to the meta-learning algorithm or the attention modules.
To make a fair comparison, we add convolutional block attention modules (Woo et al., 2018) on top of ResNet12 features (before global average pooling).
As shown in Fig.
9, MFRL with attention modules achieves comparable results with MELR and IEPT.
The uncertainty calibration results of MFRL with the temperature scaling factor are presented in Fig.
The prediction conﬁdence aligns well with the prediction accuracy.
It demonstrates that MFRL with the temperature scaling factor results in well calibrated models.
A. S ENSITIVITY OF MFRL
The performance of MFRL is not sensitive to learning rates in SWA.
As shown in Fig.
6, the representation learned by SWA generalizes better than the one from standard SGD, as long as the learning rate in SWA is in a reasonable range.
In addition, the prediction accuracy on meta-test tasks keeps stable even after running SWA for many epochs on the training data.
Therefore, MFRL is not sensitive to training epochs.
This desirable property makes the proposed method easy to use when solving few-shot learning problems in practice.
[Table]
[Figure]
A. C OMPARISON WITH EXPONENTIAL MOVING AVERAGING
Exponential moving average (EMA) decays the importance of model weights from early training epochs exponentially.
Let θ avg ← aθ avg + (1 − a ) θ new .
We try EMA with different values of a .
In Table
[Table]
EMA improves the performance when a is within a reasonable range.
Note that EMA introduces one extra hyperparameter, the forgetting factor.
It makes EMA less desirable in practice.
[Table]
A. H IERARCHICAL B AYESIAN LINEAR CLASSIFICATION MODEL
Similar to the hierarchical Bayesian linear regression model [Table] the prior distribution over w is p ( w | λ ) =
(cid:81) pi =
0 N ( w i | 0 , λ ) , where λ is the precision in the Gaussian prior.
The hyperprior on λ is deﬁned as p ( λ ) =
Gamma ( λ | a, b ) .
The posterior over all latent variables given the data is p ( w , λ | X , y ) = p ( y | X , w , λ ) p ( w | λ ) p
[Equation]
1.0 Confidence 0.
A cc u r a c y
ECE: 0.1009 MCE: 0.1934 BRI: 0.
miniImageNet 5-shot 0.
1.0 Confidence 0.
A cc u r a c y
ECE: 0.0913 MCE: 0.2537 BRI: 0.
tieredImageNet 5-shot 0.
1.0 Confidence 0.
A cc u r a c y
ECE: 0.0876 MCE: 0.2819 BRI: 0.
CIFAR-FS 5-shot 0.
1.0 Confidence 0.
A cc u r a c y
ECE: 0.2533 MCE: 0.3552 BRI: 0.
FC100 5-shot
MCMC sampling (Hoffman & Gelman, 2014) is used to avoid potential deterioration in predictive performance due to approximated inference.
A ﬂat and non-informative hyperprior ( a = b = 10 − 6 ) is used because no prior knowledge is available.
In Table 11, the hierarchical Bayesian linear classiﬁcation model achieves slightly worse performance than the logistic regression model.
However, the classiﬁcation model is not well calibrated, as shown in Fig.
[Table]
After ﬁne-tuning a and b using the meta-validation data, it is possible to get better calibrated classiﬁcation models on test tasks.
Besides, the classiﬁcation accuracy is still slightly worse than logistic regression after hyperparameter tuning.
Our observations align with a recent study, which shows that the Bayesian classiﬁcation model cannot achieve similar performance to the non-Bayesian counterpart without tempering the posterior (Wenzel et al., 2020).
We do not further experiment tempered posterior in the hierarchical Bayesian linear classiﬁcation model because it introduces an extra temperature hyperparameter that requires tuning.
The original purpose of introducing the hierarchical
Bayesian model is to get an accurate and well calibrated classiﬁcation model without hyperparameter tuning.
Consequently, the hierarchical Bayesian model is not used in few-shot classiﬁcation in that hierarchical Bayesian linear classiﬁcation models cannot achieve high accuracy and good uncertainty calibration from a non-informative hyperprior.
If hyperparameter tuning is inevitable, it is much easier to tune a logistic regression model with a temperature scaling factor, compared with tuning a hierarchical Bayesian model.
Furthermore, the computational cost of learning a hierarchical Bayesian linear classiﬁcation model via MCMC sampling is much larger than that of learning a logistic regression model.