FeLMi : Few shot Learning with hard Mixup
Abstract
Learning from a few examples is a challenging computer vision task.
Traditionally, meta-learning-based methods have shown promise towards solving this problem.
Recent approaches show benefits by learning a feature extractor on the abundant base examples and transferring these to the fewer novel examples.
However, the finetuning stage is often prone to overfitting due to the small size of the novel dataset.
To this end, we propose Fe w shot L earning with hard Mi xup ( FeLMi ) using manifold mixup to synthetically generate samples that helps in mitigating the data scarcity issue.
Different from a naïve mixup, our approach selects the hard mixup samples using an uncertainty-based criteria.
To the best of our knowledge, we are the first to use hard-mixup for the few-shot learning problem.
Our approach allows better use of the pseudo-labeled base examples through base-novel mixup and entropy-based filtering.
We evaluate our approach on several common few-shot benchmarks - FC-100, CIFAR-FS, miniImageNet and tieredImageNet and obtain improvements in both 1-shot and 5-shot settings.
Additionally, we experimented on the cross-domain few-shot setting (miniImageNet → CUB) and obtain significant improvements.
Code:
https://github.com/aniket004/Felmi
Introduction
Supervised deep learning has revolutionized data-driven computer vision tasks, e g., object detection, segmentation [5] etc.
However, it is expensive to collect and annotate large amounts of data, especially in some fields like medical image diagnosis where one needs to rely on experts for annotating the data.
Despite recent progress in learning from large data, learning robust models using a few labeled examples still remains quite challenging.
Few-shot learning (FSL) [31, 28, 33] deals with this problem, where there exists base classes with plenty of training examples, and novel classes with only a few training samples.
The goal is to learn a representation from the base classes, then quickly and efficiently adapt to the novel classes from few examples.
There are two paradigms for training few-shot learner - (1) episodic training and (2) transfer learning.
During episodic training, in each episode, a few examples from the training data are sampled to form the “support set”, on which training is performed and evaluated on the “query set” drawn from the test dataset.
Generally, the model is trained for a large number of episodes to get better generalization.
On the one hand, this approach is both time consuming and also incurs inductive (prior knowledge driven bias) and sampling bias.
On the other hand, recent approaches use simple transfer learning to the novel classes and obtained comparative performance.
Tian et al.
[31] show that pretraining on the large scale base dataset to learn a good representation, followed by finetuning on the few novel class examples using a simple linear classifier significantly outperforms the more complicated meta-learning approaches in most of the FSL benchmarks.
The performance is further boosted using self-distillation and distillation across base and novel class representations [21].
Data scarcity is a fundamental issue in few-shot learning.
To circumvent this, AssoAlign [1] uses the nearest neighbor base examples in addition to the novel examples with adversarial augmentation.
Jian et al.
[11] generates pseudo-labels for the base class samples and retrains on the entire dataset by “hallucinating” the labels of the base class samples.
However, literature in semi-supervised learning has shown that poorly pseudo-labeled samples degrade the performance [21].
We overcome the above issues in these prior approaches using entropy filtering on the noisy pseudolabels.
This step helps improve the training by discarding noisy pseudo-labeled base samples during training.
Next, we propose a solution to the data scarcity issue by synthetically generating training examples with both novel and base examples using a mixup strategy.
The mixup samples are generated with minimal overhead while adding the much needed regularization by synthesizing new points close to the novel distribution.
Motivated by the efficacy of hard example mining in computer vision [26], we propose to improve the naïve mixup step with uncertainty guided hard-mixup.
Further, we propose a technique to make use of base samples for mixup in addition to the novel samples.
Note that Mangla et.
al [16] has also used mixup for few-shot learning, but the motivation is completely different in our setting.
Their approach does not add any more samples to training and use it as a regularizer across layers.
On the contrary, we are the first to propose generating hard mixup samples to tackle the data scarcity issue in FSL.
Another contribution of our work is that we redefine hard-mixup for the few-shot learning setup.
Note that hard-mixup has been used in contrastive learning, where positive and negative classes are already defined [12].
In contrast, for N-way K-shot problem, where there exists N classes, the notion of hard examples is not straightforward.
While, hard mixup similar to these approaches could be extended in our setup by mixing and assigning a hard label, we instead work with soft labels and select the optimal examples to mix with the appropriate lambda values.
To this end, we define hard example mixing based on a margin-based uncertainty measure, inspired by the concept of active learning.
We use the margin to estimate the hardness of a sample, thereby measuring the uncertainty in predicting a particular class with high probability.
In summary, the contributions of this work are as follows:
To handle the data scarcity problem in few-shot learning, we use mixup-based data augmentation.
We perform hard mixup using a novel margin-based uncertainty measure, which further elevates performance.
To the best of our knowledge, we are the first to use hard-mixup for few-shot learning.
We propose an approach (FeLMi) to make use of both base and novel examples for generating more samples for novel training.
We validate our approach through extensive experiments on standard few-shot benchmarksFC100, CIFAR-FS, miniImageNet and tieredImageNet demonstrating state-of-the-art results.
Our approach also demonstrates promising improvements on the standard miniImageNet → CUB cross domain
benchmark.
Related Work
Few-shot learning methods can be divided into two broad categories, (1) Meta-learning, (2) Transfer learning methods.
Meta Learning.
Traditionally, meta-learning based methods have been effective for few-shot learning tasks and usually consists of a meta-training and a meta-testing phase.
In both meta-training and testing phases, multiple episodes are sampled from a task distribution and the model is trained on the support samples of the episode and evaluated on the query set.
Thus, the model is trained on unseen classes via a learning to learn approach.
Meta-learning methods can also be subdivided into two categories, viz.
, metric-based meta learning and optimization-based meta-learning.
Metric-based meta-learning methods predict the label of the query as a weighted sum of the labels over the support samples.
Popular metric-based meta-learning methods include Prototypical Networks [28],
Relation Networks [30], Matching networks [33], TADAM [17], etc.
Optimization-based metalearning methods adapt the model parameters using a small number of gradient steps, e.g., MAML [9],
LEO [23], etc.
MetaOptNet [19] solves a differentiable convex optimization problem for few-shot learning with better generalization.
Other meta-learning methods use earth-mover’s distance [38], set-to-set functions [35], human interpretable concepts [5], etc.
Transfer learning methods. Recent methods have shown that contrastive pretraining on large base example dataset and simple finetuning on the scarce novel examples performs surprisingly better than the complex meta-learning-based methods.
As an example, RFS [31] outperforms all the meta learning baselines by learning a self-supervised representation followed by a simple logistic classifier.
Following a similar trend, SKD [18] uses rotational self-supervised distillation to further improve the performance.
Rizve et al.
[22] explores the complementary strengths of invariant and equivariant representations and self-distillation for better pretraining and performs significantly better than previous methods.
Partner-assisted learning [15], which also uses feature level knowledge distillation has proven to be effective.
Base examples for few-shot learning. AssoAlign [1] makes use of large base dataset by searching nearest neighbors of the novel examples in the pool of base examples along with adversarial associative alignment.
pseudo-labeling is a well-known technique for semi-supervised learning and have been successfully used in several vision tasks, e.g., image and video classification [21].
Jian et al.
[11] used pseudo-labeling of the entire base examples and obtained significant performance boost.
Afrasiyabi et al.
[2] also used mixture-based feature space learning [2] and matching feature sets [3] to further improve the performance.
Mixup Approaches. These are simple yet effective regularization techniques for training deep networks.
Several variants of mixup have been proposed in the literature, viz.
, input mixup [39], cutmix [37], manifold-mixup [32], etc.
Manifold mixup uses feature level mixup, which provides smoother decision boundary and flattened class representations.
Mangla et al.
[16] use manifold mixup for few-shot learning from a regularization perspective.
In contrast to the above methods, we are using manifold mixup to tackle the data scarcity problem in few-shot learning by generating more samples.
Problem Statement
We first define the few-shot classification problem addressed in this work.
Typically, there exists a large-scale labeled base dataset and novel classes with few samples per class.
The task is to learn a feature representation on the base class with abundant samples and then discriminatively learn a representation to recognize subsequent novel classes.
Let the base dataset be denoted as
D base = { x base t , y base t } N base t =1 , where x base t is the base class sample and the corresponding label is y base t ∈ C base .
Similarly, the novel dataset is denoted by D novel = { x novel t , y novel t } N novel t =1 , where x novel t is the novel class sample and the corresponding label is y novel t ∈ C novel .
The base and novel classes are disjoint, i.e., C base ∩ C novel = ∅ , and | C base | ≥ | C novel | .
The training and testing is performed in episodes on the novel class samples.
In each episode i of a N-way K-shot problem, the few-shot learner is trained on a support set, i.e., D support i = { x support i,t , y support i,t } NKt =1 for N novel classes containing K samples per class.
Then, the few-shot learner is evaluated on the query set D query i
= { x query i,t , y query i,t } NKt =1 on the same N classes of D support i .
Proposed Methodology
Inspired by the efficacy of pseudo-labeling, we further increase sample size by mixup and choose more informative samples by hard mixup-based sample selection.
Our proposed method (FeLMi), outlined in Fig. 1 consists of the following six stages: (i) Learning an embedding on the base dataset ( D base ), (ii)
Pseudo labeling of the base dataset, (iii) Entropy-based filtering of the pseudo-labeled base dataset, (iv) Mixup sample generation using both base and novel examples, (v) Uncertainty-based hard-mixup sample selection, and (vi) Finetune on the entire dataset using filtered base examples, novel examples and hard-mixed examples.
We next describe each of these steps in detail.
[Figure]
𝑓
[Figure]
[Equation]
[Figure]
4.1 Learning embedding on the base dataset
We first learn a discriminative representation from the base dataset D base .
The convolutional neural network consists of a backbone f θ and a final classification layer h ϕ .
The parameters ( θ, ϕ ) are jointly learned using the cross-entropy loss on the base dataset D base .
We use the self-supervised Invariant and Equivariant Represenatation learning (IER) framework by Rizve et al.
[22] which makes use of self-supervised losses in addition to standard cross-entropy loss to learn good representations.
[Equation]
Large base datasets ensure that we have a good pretrained model to use for meta-testing stage.
4.2 pseudo-labeling of base dataset using classifier trained on novel dataset
In this step, we train a logistic regression classifier using the novel examples.
Then, using the trained classifier, we generate the pseudo-labels on the entire base dataset following Jian et al.
This pseudo-labeling step partially mitigates the data scarcity problem prevalent in few-shot training.
For each episode i , we learn a linear classifier ϕ
i using the support set of novel examples D support i , such that
[Equation]
Then, the classifier g ϕ i is used to generate pseudo-labels for the entire base dataset, i.e., ˆ y base t =
g ϕ i ( f
[Equation] base ( x t )) for t = 1 , .., N base .
4.3 Entropy based pseudo-label filtering
Given that base classes are disjoint from novel classes, not all classes will provide useful information for the task at hand.
Specifically, base samples with high entropy pseudo-labels might lead to confusing samples.
Therefore, we perform a simple pseudo-label filtering based on entropy.
[Equation]
t = 1 , · · · , N base } [Equation]
We empirically select the entropy threshold τ .
After refinement we have |D base_filt | base examples.
H ( . )
here denotes the entropy.
4.4 Mixup sample generation
Handling the data scarcity problem using pseudo-labels motivates us to generate even more samples using a simple yet an effective mixup strategy.
Particularly, we use a feature level manifold mixup step to generate more synthetic samples.
Since we have both the base and novel examples, we mix up both base-novel and novel-novel pairs.
Note that, since the mixed samples should be close to novel examples, we are not performing base-base mixup.
A straightforward way of generating synthetic examples through mixup is to mix two pure novel examples.
We call this strategy ‘Novel-Novel’ mixup.
4.4.1 Novel-Novel Mixup
We perform manifold mixup on novel-novel samples to create a pool of mixed up samples P N,N .
We select { ( x novel , y novel ) , (¯ x novel , ¯ y novel ) } ∈ D support , pass them through the feature extractor f θ base and mixup with λ n ∼ Beta ( α, α ) to generate ( x mix , y mix ) where:
[Equation]
The pool of novel-novel mixup samples of size l are generated through randomly sampling λ n , i.e.,
P N,N = { ( x N − N mix , y N − N mix ) i } li = [Equation]
4.4.2 Base-Novel Mixup
Access to the base examples during training and their pseudo-labels allows us to make use of these for mixup.
The motivation behind using mixup is to generate and augment the small novel set.
Since the base classes are disjoint from novel classes and we only have a weak mapping through the pseudo-labels, we employ the following steps while using base examples with mixup.
1) We refrain from mixing two base examples, instead always mixes a base example with a novel example.
We only mix with base examples that are close to novel examples.
We ensure this by choosing base examples which have a low pseudo-label entropy.
We choose a small mixup lambda corresponding to the base examples when mixing them ensuring they remain proximal to the distribution of novel samples.
Specifically, we sample λ b ∼ Uniform (0 , 0 . 2) .
From ( x base , ˆ y base ) ∈ D base_filt , we select k-lowest entropy base examples, i.e., ( x base sel , ˆ y base sel ) , such that,
[Equation]
Novel examples are ( x novel
, y novel ) ∈ D support mixed with these selected base examples to generate ( x B − N mix , y B − N mix ) as follows.
[Equation]
The pool of base-novel mixup samples of size l are generated through randomly sampling λ b , i.e.,
P B,N = { ( x B − N mix , y B − N mix ) i } li =
4.5 Hard mixup sample generation
In the section above, we have generated a pool of base-novel and novel-novel mixed up samples denoted by P B,N and P N,N respectively.
Now, we concatenate the generated mixed-up samples, i.e.,
P mix =
P B,N ∪ P N,N and then choose hardest N samples based on a uncertainty measure.
More specifically, to get an estimate of hardness of a mixed example chosen from P mix , we first obtain a (potentially noisy) estimate of the class probability p ( = g ϕ i ( x ) ) using the classifier trained in stage 2.
We then estimate margin (difference in top-2 probabilities), a measure of uncertainty that is commonly used in active learning [24] and pick mixed samples which have a low margin.
Intuitively, samples with a lower margin lie close to the class boundary thus providing a stronger learning signal.
We choose k-smallest margin values as hard examples (denoted as P hard _ mix ) chosen from both the base and novel mixed up samples are as follows.
[Equation]
4.6 Finetune on the entire dataset
In the last stage, the model is finetuned on a combined loss computed using the filtered base examples ( D base_filt ), the novel examples ( D novel ), and the hard mixup samples ( P hard _ mix ).
For the base examples, KL divergence loss is used, whereas cross-entropy loss has been used for novel and mixup samples.
The final loss is denoted as,
[Equation]
β and γ are scaling parameters for losses corresponding to novel and mixed up samples, respectively.
Finally, the trained model is evaluated on the query set of the episode i
( D query i ) and averaged accuracy over all the episodes are reported.
Experimental Results 5.1 Experimental Setup
Datasets: We experiment on four widely used few-shot benchmarks: FC-100 [17], CIFAR-FS [4], miniImageNet [33] and tieredImageNet [20].
FC-100 is a subset of CIFAR-100, containing 60 classes for meta-training, 20 classes for meta-validation and 20 classes for meta-testing.
CIFAR-FS is also obtained from CIFAR-100, containing 64 classes for meta-training, 16 classes for meta-validation and 20 classes for meta-testing.
miniImageNet is derived from ImageNet with images downsampled to a resolution of 84 × 84 pixels.
It has 64 classes for meta-training, 16 classes for meta-validation and 20 classes for meta-testing.
tieredImageNet [20] is another subset of ImageNet, containing total of 608 classes, from which 351 classes are used for meta-training, 97 classes for meta-validation and 160 classes for meta-testing.
Implementation details: For a fair comparison with prior works, we use the ResNet-12 architecture as the backbone network.
We attach a two-layer MLP on top of the feature extractor which outputs
N-way logits.
We use the SGD optimizer with momentum 0.9.
Learning rate of backbone and classifier are set to 0.025 and 0.05 respectively with weight decay of 5e-4.
Following [11], we use the temperature coefficient of 4.0 for our KD loss.
We use a minibatch size of 250 for training.
Using standard setting, we also perform data augmentation using color jittering, random crop and horizontal flip.
Further details of the hyperparameters and training are provided in the supplementary material.
5.2 Results on benchmark datasets
We have compared our proposed method (FeLMi) with state-of-the-art techniques on the benchmark datasets: FC-100, CIFAR-FS, miniImageNet and tieredImageNet.
We obtain consistent improvements over state-of-the-art showing the efficacy of our approach on all the aforementioned few-shot benchmarks for both 1-shot and 5-shot settings in Table 1, Table 2 and Table 3.
5.3 Results on cross-domain benchmark
We test our approach for more challenging cross-domain settings, where the base-pretraining is done on miniImageNet and the novel classes stem from samples from another domain, e.g., CUB [34].
Our approach achieves significant performance improvements for both the 5-shot and 1-shot setting in this setting as shown in Table 4.
Since CUB performs fine-grained classification, we hypothesize that hard mixup samples are even more useful in this case.
[Table] [Table]
[Table] [Table]
[Figure]
[Table] [Table]
[Figure]
5.4 Ablation Studies
In this section we investigate the effect of each stage of our method on the overall performance.
Ablation of each individual contribution: We start with the base pretraining similar to IER [22] followed by pseudo-labeling of the entire base dataset akin to Jian et al.
Entropy-based filtering of the pseudo-labels provides an improvement of 0.04%.
Using both base and novel sample mixup, we get an improvement of 0.53%.
Margin-based hard mixup sample selection additionally provide a performance boost of 0.19% on FC-100 dataset in 5-way 5-shot setting as shown in Table.
Effect of mixup strategies: Next, we evaluate different mixup strategies, results are provided in
Table. 6 on both FC-100 and CIFAR-FS in 5-way 5-shot settings.
In both the cases, we observe that best results are obtained when incorporating base-novel mixup with λ b ∼ Uniform(0,0.2) in addition to novel-novel mixup with λ n ∼ Beta(1,1).
This aforementioned setting ensures the mixed up base-novel samples (denoted by yellow points in Fig.
4 (a)) lie close to the novel examples (labeled 0-4 in Fig.
4 (a)).
Effect of mixup sample size: We observe the variation of 5-way 5-shot classification accuracy vs. number of mixup sample size for FC-100, CIFAR-FS and miniImageNet in Fig.
2a, Fig. 2b and
Fig.
2c, respectively.
We notice that in all the settings, mixup sample size of 1000 performs the best.
Increasing the number of mixed up samples seems to lower the performance a bit.
We hypothesize this might be due to the increasing imbalance between ‘real’ and ‘generated’ points, thus hurting the learning.
Effect of mixup parameters: The variation of 5-way 5-shot classification accuracy with respect to mixup parameter α for FC-100, CIFAR-FS and miniImageNet used for performing novel-novel sample mixup is shown in Fig.
In CIFAR-FS and miniImageNet, α = 0 .
5 seems to perform better.
However, in FC-100, we observe the opposite trend.
tSNE visualizations: Next, we analyse the tSNE visualizations of the representations learned using our approach.
The visualizations (Figure 4) show these for representations at the first step of training (Figure 4(a)) and after the training is completed for one episode (Figure 4 (b)).
We observe that the representations for each class get more clustered as the training progresses.
Further, representations of the query samples get closer to the corresponding class support samples.
Our generated hard mixup samples are close to the query samples as seen in the figure, which provide the necessary training signal to improve the model.
Limitations
Like most prior approaches, we make use of the base examples when adapting the classifier for novel examples.
This requires access to base examples during meta-testing which can be difficult for very large scale datasets.
We would like to fix this requirement in future work.
Conclusion
Few-shot learning is an important computer vision problem.
The inherent data scarcity issue makes this problem more challenging.
To address this issue, we propose to use hard mixup to synthetically generate mixup samples based on lowest margin based uncertainty measure.
To the best of our knowledge, we are the first to propose hard mixup for few-shot learning.
Moreover, we also perform hard mixup on the novel-novel and base-novel samples and further improve performance on the benchmark datasets, e.g., FC-100, CIFAR-FS, miniImageNet and tieredImageNet in both the 1-shot and 5-shot settings.
Our approach also significantly outperforms the state-of-the-art in cross-domain few-shot (miniImageNet → CUB) settings.
Acknowledgement
This work was partially supported by an ONR MURI grant N00014-20-1-2787.