FeLMi : Few shot Learning with hard Mixup
Abstract
Learning from few examples is a challenging computer vision task.
Traditionally, meta-learning based methods have been used to solve this problem.
However recent approaches show improvements by learning a feature extractor on the abundant base examples and transferring these to the fewer novel examples.
However, the finetuning stage is often prone to overfitting due to the small size of the novel dataset.
To this end, we propose Fe w shot L earning with hard Mi xup ( FeLMi ) using manifold mixup to synthetically generate samples which helps in mitigating the data scarcity issue.
Different from naive mixup, our approach selects the hard mixup samples using an uncertainty based criteria.
To the best of our knowledge, we are the first to use hard-mixup for few-shot learning problem.
Our approach allows to better exploit the pseudo-labeled base examples through base-novel mixup and entropy based filtering.
We evaluate our approach on common fewshot benchmarks, e.g., FC-100, CIFAR-FS, miniImageNet and tieredImageNet and obtain improvement in both 1-shot and 5-shot settings.
Additionally, we experimented on cross-domain few-shot setting (miniImageNet -> CUB) and obtain significant improvements in both 1-shot and 5-shot settings.
Code: https: //github.com/aniket004/Felmi.git
Introduction
Supervised deep learning has revolutionized data-driven computer vision tasks, e g., object detection, segmentation [5] etc.
However, it is expensive to collect and annotate large amounts of data, especially in some fields like medical image diagnosis where one needs to rely on highly-trained experts for annotating data.
Despite recent progress in learning from large data, learning robust models using few labeled examples still remains quite challenging.
Few-shot learning (FSL) [31, 28, 33] deals with this problem, where there exists base classes with plenty of training examples, and novel classes with few training samples.
The goal is to learn a representation from the base classes, then quickly and efficiently adapt to the novel classes from few examples.
There are two paradigms for training few-shot learner - (1) episodic training and (2) transfer learning based approach.
During episodic training, in each episode, a few examples from the training data are sampled to form the “support set”, on which training is performed and evaluated on the “query set” drawn from the test dataset.
Generally, the model is trained for a large number of episodes to get better generalization.
However, this approach is both time consuming and also incurs inductive (prior knowledge driven bias) and sampling bias.
On the other hand, recent approaches use simple transfer learning to the novel classes and obtained comparative or even better performance.
Tian et al.
[31] show that pretraining on the large scale base dataset to learn a good representation, followed by finetuning on the few novel class examples using a simple linear classifier significantly outperforms the more complicated meta-learning approaches in most of the FSL benchmarks.
The performance is further boosted using self-distillation and distillation across base and novel class representations [21].
Data scarcity is a fundamental issue in few-shot learning problem.
To mitigate that, AssoAlign [1] has proposed to use the nearest neighbor base examples in addition to the novel examples with adversarial augmentation.
Jian et al.
[11] generates pseudolabels for the base class samples and retrain on the entire dataset by “hallucinating” the labels of the base class samples.
However, literature in semi-supervised learning has shown that poorly labeled psedolabeled samples hurt performance [21].
We overcome issues in these prior approaches by entropy filtering the noisy pseudolabels.
This step helps improve the training by discarding noisy pseudolabeled base samples during training.
Next, we propose a solution to the data scarcity issue by synthetically generating training examples with both the novel and base examples using mixup.
These mixup samples are generated with minimal overhead while adding the much needed regularization by synthesizing new points close to the novel distribution.
Motivated by the efficacy of hard example mining in computer vision [26], we propose to improve naive mixup with uncertainty guided hard-mixup.
Further, we propose a technique to make use of base samples for mixup in addition to the novel samples.
Note that Mangla et.
al [16] has also used Mixup for few-shot learning, but the motivation is completely different in our setting. [16] used Mixup as a regularizer across layers, therefore, not adding any more samples for training.
To the contrary, we are the first to propose generating hard mixup samples to tackle the data scarcity issue in FSL.
Another contribution of our work is that we redefine hard-mixup for few-shot learning setup.
Note that hard-mixup has been used in contrastive learning, where positive and negative classes are already defined [12].
In contrast, for N-way K-shot problem, where there exists N classes where the notion of hard examples is not straight forward.
While hard mixup similar to these approaches could be extended in our setup by mixing and assigning a hard label, we instead work with soft labels and figure out the optimal examples to mix and the right lambda.
To this end, we define hard example mixing based on margin based uncertainty measure, inspired from active learning.
We use margin to select hardness of a sample which measures uncertainty in predicting a particular class with high probability.
In summary, the contributions in this work are as follows:
To handle the data scarcity problem in few-shot learning, we use mixup-based data augmentation to improve the performance.
We perform hard mixup based on margin based uncertainty measure, which further elevates performance.
To the best of our knowledge, we are the first to use hard-mixup for few-shot learning.
We propose a way to make use of both base and novel examples for generating more samples for novel training.
We validate our approach using extensive experimentation of standard few-shot benchmarks, e.g., FC100, CIFAR-FS, miniImageNet and tieredImageNet.
Experiments on cross-domain benchmark (miniImageNet -> CUB) also provide significant improvements.
Related Work
Few-shot learning methods can be divided into two broad categories, (1) Meta-learning, (2) Transfer learning based methods.
Meta Learning Traditionally meta-learning based methods have been effective for few-shot learning setting, which consists of meta-training and meta-testing phase.
In each of the meta-training and testing phase, multiple episodes are sampled from a task distribution and the model is trained on the support samples of the episode and evaluated on the query set.
Thus the model is trained on unseen classes by learning to learn.
Meta-learning methods can also be subdivided into two categories, i.e., metric-based meta learning and optimization based meta-learning.
Metric based meta-learning methods predicts the label of the query as a weighted sum of the labels over the support samples.
Popular metric based meta-learning methods include Prototypical
Networks [28], Relation Network [30], Matching network [33], TADAM [17] etc.
Optimization based meta-learning methods try to adapt to the model parameters using small numbers of gradient steps, e.g., MAML [9], LEO [23] etc.
MetaOptNet [19] uses a differentiable convex optimization problem for few-shot learning problem with better generalization.
There are other meta-learning based methods using earth-mover’s distance [38], set-to-set functions [35], human interpretable concepts [5] etc.
Transfer learning based methods Recent methods have shown that contrastive pretraining on large base example dataset and simple finetuning on the scarce novel examples performs surprisingly better than the complex meta learning based methods.
E.g., RFS [31] outperforms all the meta learning baselines by learning a self-supervised representation followed by simple logistic classifier.
Following the similar trend, SKD [18] uses rotational self-supervised distillation to further improve the performance.
Rizve et al.
[22] explored complementary strength of invariant and equivariant representation and self distillation for better pretraining and performed significantly better than previous methods.
Partner assisted learning [15], which also uses feature level knowledge distillation has also been proven to be effective.
Base examples for few-shot learning AssoAlign [1] makes use of large base dataset by searching nearest neighbors of the novel examples in the pool of base examples along with adversarial associative alignmentment.
Pseudolabeling is a well-known technique for semi-supervised learning and successfully used in several vision tasks, e.g., image and video classification [21].
Jian et al.
[11] used pseudolabeling of the entire base examples and obtained significant performance boost.
Afrasiyabi et al.
[2] also used mixture based feature space learning [2] and matching feature sets [3] to further improve the performance.
Mixup Mixup is a simple yet effective regularization technique for training deep networks.
Several variants of mixup have been proposed in the literature, viz.
, input mixup [39], cutmix [37], manifoldmixup [32] etc.
Manifold mixup uses feature level mixup, which provides smoother decision boundary and flattened class representations.
Mangla et al.
[16] used manifold mixup for few-shot learning from a regularization perspective.
However, instead we are using manifold mixup to tackle the data scarcity problem in few-shot learning by generating more samples.
Problem Statement
We first define the few-shot classification problem addressed in this work.
Typically, there exists a large scale labeled base dataset and comparatively less number of novel classes with few samples per class.
The task is to learn a feature representation on the base class with abundant samples and then discriminatively learn a representation to recognize subsequent novel classes.
Let the base dataset be denoted as D base = { x base t , y base t } N base t =1 , where x base t is the base class sample and the corresponding label is y base t ∈ C base .
Similarly, the novel dataset is denoted by D novel = { x novel t , y novel t } N novel t =1 , where x novel t is the novel class sample and the corresponding label is y novel t ∈ C novel .
The base and novel classes are disjoint, i.e., C base ∩ C novel = ∅ , and | C base | ≥ | C novel | .
The training and testing is performed in episodes on the novel class samples.
In each episode i of a N-way K-shot problem, the few-shot learner is trained on a support set, i.e., D support i = { x support i,t , y support i,t } NKt =1 for N novel classes containing K samples per class.
Then the few-shot learner is evaluated on the query set D query i
= { x query i,t , y query i,t } N ′ K t =1 on the same N classes of D support i .
Proposed Methodology
Inspired by the efficacy of pseudolabeling, we further increase sample size by mixup and choose more informative samples by hard mixup based sample selection.
Our proposed method which is outlined in Fig.
1 consists of six stages as follows:
Learning embedding on the base dataset ( D base ).
Pseudo labeling of base dataset. 3.
Entropy based filtering of base dataset. 4.
Mixup sample generation using both base and novel examples.
Uncertainty based hard-mixup sample selection.
Finetune on the entire dataset using base example, novel example and hard-mixed examples.
The detailed methodology is discussed as follows: [Figure]
𝑓
[Figure] 𝑓 ! [Figure] 4.1 Learning embedding on the base dataset
We first learn a discriminative representation from the base dataset D base .
The convolutional neural network consists of a backbone f θ and a final classification layer h ϕ and the parameters ( θ, ϕ ) are jointly learned using cross-entropy loss on the base dataset D base .
We use the self-supervised
Invariant and Equivariant Represenatation learning (IER) framework by Rizve et al.
[22] framework which makes use of self-supervised losses in addition to standard cross-entropy loss to learn good representations.
[Equation]
Large size of base dataset ensures that we have a good pretrained model to use for meta-testing stage.
4.2 Pseudo labeling of base dataset using classifier trained on novel dataset
In this step, we train a linear logistic regression classifier using the novel examples.
Then using the trained classifier we generate the pseudolabels of the entire base dataset as performed by Jian et al.
This pseudolabeling step partially mitigate the data scarcity problem prevalent in few-shot learner training.
For each episode i , we learn a linear classifier ϕ
i using the support set novel examples
D support i , such that
[Equation]
Then the classifier g ϕ i is used to generate pseudolabels for the entire base dataset, i.e., ˆ y base t =
g ϕ
i ( f θ base ( x t )) for t = 1 , .., N base .
4.3 Entropy based Pseudolabel filtering
Given that base classes are disjoint from novel classes, not all classes will provide information useful for the task at hand.
Specifically, base samples with high entropy pseudolabels might lead to confusing samples.
Therefore, we perform a simple pseudolabel filtering based on entropy.
[Equation]
t =
[Equation]
[Equation]
We empirically select the entropy threshold τ .
After refinement we have |Y filt | base examples.
H ( . )
here denotes the entropy.
We call these filtered pseudolabeled base sample set be D base_filt
4.4 Mixup sample generation
Handling the data scarcity problem using pseudolabels motivate us to generate even more samples using simple yet effective mixup strategy.
Particularly, we are using feature level manifold mixup to generate more synthetic samples.
Since we have both the base and novel examples, we have the freedom of mixing up both base-novel and novel-novel pairs.
Note that, since the mixed samples should be close to novel examples, we are only performing novel-novel and base-novel mixup.
A straightforward way of generating synthetic examples through mixup is to mix two pure novel examples.
We call this strategy ‘Novel-Novel’ mixup.
4.4.1 Novel-Novel Mixup
We perform manifold mixup on novel-novel samples to create a pool of mixed up samples P N,N .
We select { ( x novel , y novel ) , (¯ x novel , ¯ y novel ) } ∈ D support , pass through the feature extractor f θ base and mixup with λ n ∼ Beta ( α, α ) to generate ( x mix , y mix ) such that,
[Equation]
The pool of novel-novel mixup samples of size l are generated through randomly sampling λ n , i.e.,
P N,N = { ( x N − N mix , y N − N mix ) i } li =1 .
4.4.2 Base-Novel Mixup
Access to the base examples during training and their pseudolabels allows us to make use of these for mixup.
The motivation behind using mixup is to generate and augment the small novel set.
Since the base classes are disjoint from novel classes and we have a weak mapping only through the pseudolabels we employ the following tricks while using base examples with mixup.
1) We refrain from mixing two base examples, always mixing a base example with a novel example.
We only mix with base examples that are close to novel examples.
We ensure this by choosing base examples which have a low pseudo-label entropy.
We choose a small mixup lambda corresponding to the base examples when mixing them which ensures that we do not stray too far away from the novel distribution.
Specifically, we sample λ b ∼ Uniform (0 , 0 . 2) .
For example, from ( x base , ˆ y base ) ∈ D base , we select k-lowest entropy base examples, i.e., ( x base sel , ˆ y base sel ) , such that,
[Equation]
Novel examples are ( x novel
, y novel ) ∈ D support mixed with these selected base examples to generate ( x B − N mix , y B − N mix ) as follows.
[Equation]
The pool of base-novel mixup samples of size l are generated through randomly sampling λ b , i.e.,
[Equation]
4.5 Hard mixup sample generation
In the section above, we have generated a pool of base-novel and novel-novel mixed up samples denoted by P B,N and P N,N respectively.
Now, we concatenate those to generate a larger pool of mixed-up samples, i.e., P mix =
P B,N ∪ P N,N and then choose hardest N samples based on a uncertainty measure.
To the best of our knowledge, we are the first to use hard-mixup for few-shot learning problem.
More specifically, to get an estimate of hardness of a mixed example chosen from P mix , we first obtain a (potentially noisy) estimate of the class probability p ( = g ϕ i ( x ) ) using the classifier trained in step(2).
We then estimate margin (difference in top-2 probabilities), a measure of uncertainty that is commonly used in Active learning [24] and pick mixed samples which have a low margin.
Intuitively, samples with a lower margin lie close to the class boundary thus providing a stronger learning signal.
We choose k-smallest margin values as hard examples (denoted as P hard _ mix ) chosen from both the base and novel mixed up samples are as follows.
[Equation]
4.6 Finetune on the entire dataset
In the last stage, the model is finetuned on combined loss computed using the filtered base examples ( D base_filt ), the novel examples ( D novel ) and the hard mixup samples ( P hard_mix ).
For the base examples, KL divergence loss is used, whereas Cross-entropy loss has been used for novel and mixup samples.
The final loss is denoted as,
[Equation]
β and γ are scaling parameters for losses corresponding to novel and mixed up samples respectively.
Finally, the trained model is evaluated on the query set of the episode i
( D query i ) and averaged accuracy over all the episodes are reported.
Experimental Results 5.1 Experimental Setup
Datasets: We experiment on four widely used few-shot benchmarks: FC-100 [17], CIFAR-FS [4] and miniImageNet [33] and tieredImageNet [20].
FC-100 is a subset of CIFAR-100, containingclasses for meta-training, 20 classes for meta-validation and 20 classes for meta-testing.
CIFAR-FS is also obtained from CIFAR-100, containing 64 classes for meta-training, 16 classes for meta-validation and 20 classes for meta-testing.
miniImageNet is derived from ImageNet with images downsampled to a resolution of 84 × 84 pixels.
It has 64 classes for meta-training, 16 classes for meta-validation and 20 classes for meta-testing.
tieredImageNet [20] is also another subset of ImageNet, containing total of 608 classes, from which 351 classes used for meta-training, 97 classes for meta-validation and 160 classes for meta-testing.
Implementation details: For fair comparison, we have used the ResNet-12 architecture as the backbone network.
On top of that, we train a two layer MLP, which outputs N-way logits.
We used
SGD optimizer with momentum 0.9, learning rate of backbone and classifier be 0.025 and 0. respectively with weight decay of 5e-4.
Following [11], we use the temperature coefficient as of 4.0 for our KD loss.
We use a minibatch size of 250 for training.
Using standard setting, we also perform data augmentation using color jittering, random crop and horizontal flip.
Further details of the network architecture and training will be provided in the supplementary material.
5.2 Results on benchmark datasets
We have compared our proposed method (FeLMi) with state-of-the-art techniques on the benchmark datasets: FC-100, CIFAR-FS, miniImageNet and tieredImageNet.
We show consistent improvements against state-of-the-art showing efficacy of our approach on the all the aforementioned fewshot benchmarks across both the 1-shot and 5-shot settings in Table 1, Table 2 and Table 3. 5.3 Results on cross-domain benchmark
We tested our approach for more challenging cross-domain setting, where the base-pretraining is done on miniImageNet and the novel class stems from samples from another domain, e.g., CUB [34].
Our approach achieves significant performance improvements for both the 5-shot and 1-shot setting in this setting as shown in Table 4.
[Table] [Table]
[Table] [Table] [Table] [Figure]
[Table] [Figure] [Table]
[Figure] [Table] [Figure]
[Table]
Ablation of each individual contribution: In this section, we investigate the effect on each stage on the overall performance.
We start with the base pretraining following IER [22] and following [Table] [Table] [Figure]
Jian et al.
[11] method of pseudolabeling the entire base dataset.
Entropy based filtering of the pseudolabels provides an improvement of 0.04%.
Using both base and novel sample mixup, we get an improvement of 0.53%.
Margin-based hard mixup sample selection additionally provide a performance boost of 0.19% on FC-100 dataset in 5-way 5-shot setting as shown in Table.
Effect of mixup strategies:
In this section, we evaluate different mixup strategies, as shown in
Tab. 6 on both FC-100 and CIFAR-FS in 5-way 5-shot settings.
In both the cases, we observe that best results are obtained when incorporating base-novel mixup with λ b ∼ Uniform(0,0.2) in addition to novel-novel mixup with λ n ∼ Beta(1,1).
This aforementioned setting ensures the mixed up base-novel samples (denoted by yellow points in Fig.
4 (a)) lie close to the novel examples (labeled 0-4 in the Fig.
4 (a)).
Effect of mixup sample size: We observe the variation of 5-way 5-shot classification accuracy vs number of mixup sample size for FC-100, CIFAR-FS and miniImageNet in Fig.
2a, Fig. 2b and
Fig.
2c respectively.
We notice that in all the settings, mixup sample size of 1000 performs the best.
Increasing the number of mixed up samples seems to hurt the performance a bit.
We hypothesize this might be due to the increasing imbalance between ‘real’ and ‘generated’ points thus hurting the optimization.
Effect of mixup parameters: The variation of 5-way 5-shot classification accuracy with respect to mixup parameter α for FC-100, CIFAR-FS and miniImageNet used for performing novel-novel sample mixup is shown in Fig.
In CIFAR-FS and miniImageNet, α = 0 .
5 seems to perform better.
However, in FC-100, we observe the opposite trend. tSNE visualizations: Next, we analyse the tSNE visualizations of the representations learned using our approach.
The visualizations (Figure 4) show these for representations at the first step of training and after the training is completed for one episode.
We observe that the representations for each class get more clustered as the training progresses.
Further, representations of the query samples get closer to the corresponding class support samples.
Our generated hard mixup samples are close to the query samples as seen in the figure, which provide the necessary training signal to improve the model.  Limitations
Like most prior approaches, we make use of the base examples when adapting the classifier for novel examples.
This requires storing of the base example features which can be difficult for very large scale datasets.
We would like to fix this requirement in future work.
Conclusion
Few-shot learning is an important computer vision problem.
The inherent data scarcity issue makes this problem more challenging.
To address this issue, we propose to use hard mixup to synthetically generate mixup samples based on lowest margin based uncertainty measure.
To the best of our knowledge, we are the first to propose hard mixup for few-shot learning.
Moreover, we also performed hard mixup on the novel-novel and base-novel samples and further improve performance on the benchmark datasets, e.g., FC-100, CIFAR-FS, miniImageNet and tieredImageNet in both the 1-shot and 5-shot settings.
Our approach also significantly outperform the state-of-the-art in cross-domain few-shot (miniImageNet -> CUB) settings.
Acknowledgement
The authors would like to thank ONR MURI grant N00014-20-1-2787 for funding the research.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[No]
(c) Did you discuss any potential negative societal impacts of your work?
[No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] Code will be released later.
To ease reproducibility, we include pseudocode in Appendix.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See 5 and Supplementary material.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] We report error bars with respect to episodes.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Supplementary material 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[No] We will mention along with code release.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]