C LASS I MBALANCE IN F EW -S HOT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Few-shot learning aims to train models on a limited number of labeled samples from a support set in order to generalize to unseen samples from a query set.
In the standard setup, the support set contains an equal amount of data points for each class.
This assumption overlooks many practical considerations arising from the dynamic nature of the real world, such as class-imbalance.
In this paper, we present a detailed study of few-shot class-imbalance along three axes: dataset vs. support set imbalance, effect of different imbalance distributions (linear, step, random), and effect of rebalancing techniques.
We extensively compare overstate-of-the-art few-shot learning methods using backbones of different depths on multiple datasets.
Our analysis reveals that 1) compared to the balanced task, the performances of their class-imbalance counterparts always drop, by up to 18 . 0% for optimization-based methods, although feature-transfer and metric-based methods generally suffer less, 2) strategies used to mitigate imbalance in supervised learning can be adapted to the few-shot case resulting in better performances, 3) the effects of imbalance at the dataset level are less signiﬁcant than the effects at the support set level.
The code to reproduce the experiments is released under an open-source license.
I NTRODUCTION
Deep learning methods are well known for their state-of-the-art performances on a variety of tasks (LeCun et al., 2015; Russakovsky et al., 2015; Schmidhuber, 2015).
However, they often require to be trained on large labeled datasets to acquire robust and generalizable features.
Few-Shot Learning (FSL) (Chen et al., 2019; Wang et al., 2019b; Bendre et al., 2020) aims at reducing this burden by deﬁning a distribution over tasks , with each task containing a few labeled data points ( support set ) and a set of target data ( query set ) belonging to the same set of classes.
A common way to train
FSL methods is through episodic meta-training (Vinyals et al., 2017) with the model repeatedly exposed to batches of tasks sampled from a task-distribution and then tested on a different but similar distribution in the meta-testing phase.
The preﬁx “meta” is commonly used to distinguish the highlevel training and evaluation routines of meta-learning (outer loop), from the training and evaluation routines at the single-task level (inner loop).
Limitations.
Standard meta-training overlooks many challenges stemming from real-world dynamics, such as class-imbalance (CI).
The standard setting assumes that all classes in the support set contain the same number of data points, whereas in many practical applications, the number of samples for each class may vary (Buda et al., 2018; Leevy et al., 2018).
Given the limited amount of data used in FSL, a small difference in the number of samples between classes could already introduce signiﬁcant levels of imbalance.
Most FSL methods are not designed to cope with these more challenging settings.
Figure 1 exempliﬁes these considerations by showing that several state-of-the-art
FSL methods underperform when tested under three CI regimes (linear, step, random).
Previous work.
Previous work mainly focuses on the single imbalance case or grouping several settings into one task, offering limited insights into the effects of CI on FSL and making it challenging to quantify its effects (Guan et al., 2020;
Triantaﬁllou et al., 2020; Lee et al., 2019; Chen et al., 2020).
A common approach to mitigate imbalance is Random-Shot meta-training (Triantaﬁllou et al., 2020), which exposes the model to imbalanced tasks during meta-training.
However, previous work provides little insight into the effectiveness of this procedure on the imbalanced FSL evaluation task.
Furthermore, minimal work exists that investigates meta-training outcomes under an imbalanced distribution of classes at the (meta-)dataset level, while this case is common in recent FSL applications (Ochal et al., 2020; Guan et al., 2020) and meta-learning benchmarks (Triantaﬁllou et al., 2020).
The CI problem is well-known within the supervised learning community,
B a s e li n e ( [Figure] N N ) B a s e li n e ( f i n e - t u n e ) B a s e li n e + +
M a t c h i n g N e t
P r o t o N e t P r o t o N e t ( 2 0 - w a y )
R e l a t i o n N e t ( C E )
D K T
S i m p l e S h
o t M A M L P r o t o M A M L B M A M L
B M A M L ( c h a s e r )
[Figure]
which has systematically produced strategies to deal with the problem, such as the popular Random Over-Sampling (Japkowicz & Stephen, 2002) that aims at rebalancing minority classes by uniform sampling.
While such strategies have been extensively studied on many supervised learning problems, there is little understanding of how they behave with the recently proposed FSL methods in the low-data regime.
Our work and main contributions.
In this paper, we provide, for the ﬁrst time, a detailed analysis of the CI problem within the FSL framework.
Our results show that even small CI levels can introduce a signiﬁcant performance drop for all the methods considered.
Moreover, we ﬁnd that only a few models beneﬁt from Random-Shot meta-training (Triantaﬁllou et al., 2020; Lee et al., 2019;
Chen et al., 2020) over the classical (balanced) episodic meta-training (Vinyals et al., 2017); while pairing the meta-training procedures with Random Over-Sampling offers a substantial advantage.
The experimental results show that imbalance severity at the dataset level depends on the size of the dataset.
Our contributions can be summarized as follows:
A systematic, comprehensive and in-depth study of the effects of CI within the FSL framework along three axes: (i) dataset vs. support set imbalance, (ii) effect of different imbalance distributions (linear, step, random), (iii) effect of rebalancing techniques, such as random over-sampling and the recently proposed Random-Shot meta-training (Triantaﬁllou et al., 2020).
We reveal novel insights into the meta-learning and support set adaptation capabilities to the CI regime, supported by extensive results on over 10 FSL methods with different imbalance settings, backbones, support set sizes, and datasets.
We provide insight into the previously unaddressed CI problem in the (meta-)training dataset, showing that the effects of imbalance at the dataset level are less signiﬁcant than the effects at the support set level.
R ELATED W ORK 2. C LASS I MBALANCE
In classiﬁcation, imbalance occurs when at least one class (the majority class) contains a higher number of samples than the others.
The classes with the lowest number of samples are called minority classes.
If uncorrected, conventional supervised loss functions, such as (multi-class) cross-entropy, skew the learning process in favor of the majority class, introducing bias and poor generalization toward the minority class samples (Buda et al., 2018; Leevy et al., 2018).
Imbalance approaches are categorized into three groups: data-level, algorithm-level, and hybrid.
Data-level strategies manipulate and create new data points to equalize data sampling.
Popular data-level methods include
Random Over-Sampling (ROS) and Random Under-Sampling (RUS)
(Japkowicz & Stephen, 2002).
ROS randomly resamples data points from the minority classes, while RUS randomly leaves out a randomly selected portion of the majority classes to decrease imbalance levels.
Algorithm-level strategies use regularization or minimization of loss/cost functions.
Weighted loss is a common approach where each sample’s loss is weighted by the inverse frequency of that sample’s class.
Focal loss (Lin et al., 2017) is another type of cost function that has seen wide success.
Hybrid methods combine one or more types of strategies (e.g. Two-Phase Training Havaei et al. Modeling Imbalance. The object recognition community studies class imbalance using realworld datasets or distributions that approximate real-world imbalance (Buda et al., 2018; Johnson & Khoshgoftaar, 2019; Liu et al., 2019).
Buda et al.
(2018) note that two distributions can be used: linear and step imbalance (deﬁned in our methodology Section 3). At large-scale, datasets with many samples and classes tend to follow a long-tail distribution (Liu et al., 2019; Salakhutdinov et al., 2011; Reed, 2001), with most of the classes occurring with small frequency and a few classes occurring with high frequency. Our work primarily focuses on the tail-end of the distribution and does not consider the case of large sample size.
Therefore, we do not examine the long-tail mechanisms.
2. F EW -S HOT L EARNING
FSL methods can be broadly categorized into metric-learning, optimization-based, hallucination, data-adaptation, and probabilistic approaches (Chen et al., 2019).
Metric-learning approaches such as Prototypical Networks (Snell et al., 2017), Relation Networks (Sung et al., 2017), the Neural
Statistician (Edwards & Storkey, 2017) and Matching Networks (Vinyals et al., 2017), learn a feature extractor capable of parameterizing images into embeddings, and then use distance metrics to classify mapped query samples based on their distance to support points.
Optimization-based approaches such as MAML (Finn et al., 2017) and Meta-Learner LSTM (Ravi & Larochelle, 2016)), are meta-trained to use guided optimization steps on the support set for quick adaptation.
Hallucination or data augmentation techniques perform afﬁne and color transformations on the support set to create additional data points (Zhang et al., 2018).
Probabilistic methods use Bayesian inference to learn and classify samples, for example, the recently proposed Deep Kernel Transfer (DKT)
(Patacchiola et al., 2020), which uses Gaussian Process at inference time.
We use the term domain adaptation to represent those approaches using standard transfer-learning with a pre-training stage on a large set of classes and a ﬁne-tune stage on the support set – examples are Baseline and Baseline++ from Chen et al.
(2019), and the recently proposed Transductive Fine-Tuning from Dhillon et al.
The details of the methods used in our experiments are reported in Appendix A.
For completeness, it is worth mentioning Incremental Few-Shot Learning (Ren et al., 2018; Gidaris & Komodakis, 2018; Hariharan & Girshick, 2017) which is an extension of FSL.
It considers maintaining performance on base classes (meta-training dataset) while incrementally learning about novel classes using limited data, typically without re-training from scratch on all data.
Here, we focus on studying how imbalance affects the learning on novel classes, therefore we will not consider incremental FSL any further.
2. I MBALANCE IN F EW -S HOT AND M ETA L EARNING
Class Imbalance in the low-data regime has received some attention, although the current work is not comprehensive (Guan et al., 2020;
Triantaﬁllou et al., 2020; Lee et al., 2019; Chen et al., 2020).
We identify that in FSL, class-imbalance occurs at two levels: the task-level and the meta-dataset level.
At the task level, class-imbalance occurs in the support set or the query-set, directly affecting learning and evaluation procedures.
Class imbalance at the meta-dataset level is caused by imbalanced dataset classes in one (or more) of the three data splits: meta-training, meta-validation, meta-testing.
This disproportion affects the distribution of tasks that a model is exposed to during meta-training, affecting their ability to generalize to new tasks.
In Figure 2, we highlight the differences between imbalanced task and imbalanced meta-dataset.
Related to, but distinct from, these two class-imbalance types is task-distribution imbalance (Lee et al., 2019); skewed task-distribution can occur as a result of meta-dataset level class-imbalance or as a result of the task-sampling procedure.
In extreme cases, task-distribution imbalance can lead to out-of-distribution tasks during meta-evaluation.
Task-distribution imbalance has already received some attention (Lee et al., 2019; Cao et al., 2020); therefore, it will not be considered in this work.
Class Imbalance in Tasks.
Triantaﬁllou et al.
(2020) uses imbalanced support sets to create a more realistic and challenging benchmark for meta-learning.
The authors use random-shot tasks with randomly selected classes (way) and samples (shot), which replace the balanced task during meta-training and meta-evaluation.
A similar idea is explored by Lee et al.
, Chen et al.
(2020), and Guan et al.
(2020) with the last two using a ﬁxed number of classes (way).
However, none of these works quantify the impact of class-imbalance on the FSL task nor the advantages of Random-Shot meta-training.
Lee et al.
(2019) explores a small range of class-imbalance in the support set.
However, details into the effects of class-imbalance are lost when combined with taskdistribution imbalance, making it challenging to attribute any performance changes caused by classimbalance.
Chen et al.
(2020) explore a pure class-imbalance problem on the support set, but their
Task
C l a ss e s
Samples
S a m p li n g
T a s k
Dataset
Imbalanced Support Set
Task
Task 2 C l a ss e s
Samples
S a m p li n g
T a s k
Dataset
Imbalanced Meta-Training Dataset
Task Task
Task
Following standard practice in the literature, query sets are kept balanced analysis is limited to just two methods (their proposal and MAML). In Guan et al.
(2020), metalearning is applied on aerial imagery, exploring step imbalance ranging from 5 to 140 samples per class (shot); however, only two FSL methods are compared (Prototypical Networks and their RFMML method).
Previous work provides limited insight into class-imbalance at the task level.
Class Imbalance in the Meta-Dataset.
Standard meta-datasets (e.g. Mini-ImageNet) can be swapped for other domain-speciﬁc datasets, such as CUB (Wah et al., 2011), VGG Flowers (Nilsback & Zisserman, 2008), and others (Triantaﬁllou et al., 2020).
These datasets sometimes contain an unequal number of class samples, but previous work has never reported the effects of classimbalance in the meta-training dataset (Guan et al., 2020; Triantaﬁllou et al., 2020; Lee et al., 2019; Chen et al., 2020). We emphasize that studying the impact of imbalance at this level is important since imbalanced domain-speciﬁc meta-datasets are common in real-world applications (Guan et al., 2020; Ochal et al., 2020) and recent benchmarks (Triantaﬁllou et al., 2020).
Our work is the ﬁrst to provide quantitative insights into this setting.
M ETHODOLOGY 3. S TANDARD FSL
A standard K -shot N -way FSL classiﬁcation task is deﬁned by a small support set ,
S = { ( x 1 , y 1 ) , ..., ( x s , y s ) } ∼ D , containing N × K image-label pairs drawn from N unique classes with K samples per class ( |S| = K × N ).
The goal is to correctly predict labels for a query set ,
Q = { ( x 1 , y 1 ) , ..., ( x t , y t ) } ∼ D , containing a different set of M samples drawn from the same N classes (i.e. Q ( x )
∩ S ( x ) = ∅ and Q ( y ) ≡ S ( y ) ).
The support set can also be referred to as sample set and the query set as target set .
3. C LASS -I MBALANCED FSL
We deﬁne a class-imbalanced FSL task as a K min - K max -shot N -way
I -distribution task.
Similarly to the standard FSL task, a model is given a small support set , S ∼ D and a query set , Q ∼ D , containing a different set of samples drawn from the same N classes.
However, in the imbalance case, the support set contains between K min to K max (inclusive) number of samples per class distributed according to the imbalance I -distribution, where I ∈ { linear, step, random } (Buda et al., 2018).
Similarly, the query set can contain M min to M max samples per class distributed according to the I -distribution.
In our experiments, we keep a balanced query set ( M = M min =
M max ) for fair evaluation.
1 For brevity, but without loss of generality, we deﬁne imbalance Idistribution in relation to the support set (see Figure 2) as:
• Linear imbalance .
The number of class samples, K i , for classes i ∈ { 1 ..N } is deﬁned by: K i = round
[Equation]
[Equation]
where c = 0 .
499 for rounding purposes.
For example, this means that for linear 1-9-shot 5-way task, K i ∈ { 1 , 3 , 5 , 7 , 9 } , and for linear 4-6-shot 5-way task K i ∈ { 4 , 4 , 5 , 6 , 6 } .
• Step imbalance .
The number of class samples, K i , is determined by an additional variable N min specifying the number of minority classes.
Speciﬁcally, for classes i ∈ { 1 ..N } :
[Equation]
For example, in a step 1-9-shot 5-way task with 1 minority class K i ∈ { 1 , 9 , 9 , 9 , 9 } .
• Random imbalance .
The number of class samples, K i , is sampled from a uniform distribution, i e. K i ∼ Unif ( K min , K max ) , with K min and K max inclusive.
This is appropriate for the problem at hand (small number of classes), but it could be replaced by a Zipf/Power Law (Reed, 2001) for a more appropriate class imbalance in problems with a large number of classes.
We also report the imbalance ratio ρ , which is a scalar identifying the level of class-imbalance; this is often reported in the CI literature for the supervised case (Buda et al., 2018).
We deﬁne ρ to be the ratio between the number of samples in the majority and minority classes in the support set:
[Equation]
3. C LASS -I MBALANCED M ETA -D ATASET
Training FSL methods involves three phases: meta-training, meta-validation, and meta-testing.
Each phase samples tasks from a different dataset, D train , D val , and D test , respectively.
A balanced dataset contains D N ∗ classes with D K ∗ samples per class, where ∗ ∈ { train, val, test } .
However, in the real-world, datasets can contain any number of samples with imbalance.
For fair evaluation, we control dataset imbalance according to the I -distribution described in Section 3.2 but with K min ,
K max , N , N min changed for D K min ∗ , D K max ∗ , D N ∗ , D N min ∗ .
Similarly, we report the imbalance ratio ρ .
In our experiments, we apply imbalance only at the meta-training stage to limit the factors of interest, but a similar procedure could be used at the meta-testing and meta-validation stages.
3. R EBALANCING T ECHNIQUES AND S TRATEGIES
Random Over-Sampling.
We apply Random Over-Sampling (ROS) and for each classimbalanced task, we match the number of support samples in the non-majority classes to the number of support samples in the majority class, K i = max i ( K i ) .
This means that for I ∈ { linear, step } , the number of samples in each class is equal to K max .
We match K i to max i ( K i ) by resampling uniformly at random the remaining max
i ( K i ) − K i support samples belonging to class i , and then appending them to the support set.
When applying ROS with augmentation ( ROS+ ), we perform further data transformation on the resampled supports.
A visual representation of a class imbalanced task after applying ROS and ROS+ is presented in the Appendix A (Figure 7).
Random-Shot Meta-Training.
We apply Random-Shot meta-training similarly to the Standard episodic (meta-)training (Vinyals et al., 2017) but with the balanced tasks exchanged with K minK max -shot random -distribution tasks, as deﬁned above.
We use random-distribution following previous work (Triantaﬁllou et al., 2020; Lee et al., 2019), since in real-world applications, the actual imbalance distribution is likely to be unknown at (meta-)evaluation time. Rebalancing Loss Functions.
We apply two rebalancing loss functions:
Weighted Loss (Buda et al., 2018) and Focal Loss (Lin et al., 2017).
Both of them have been applied to the inner loop of optimization-based methods.
Full details are reported in the supplementary material (Appendix A).
E XPERIMENTS 4. S ETUP
Class Imbalance Scenarios and Tasks.
We address two class-imbalance scenarios within the FSL framework: 1) imbalanced support set, and 2) imbalanced meta-training dataset.
For the imbalanced support set scenario, we ﬁrst focus on the very low-data range with an average support set size ofsamples (5 avr.
shot).
We train FSL models using Standard (episodic) meta-training (Vinyals et al., 2017) using 5-shot 5-way tasks, as well as Random-Shot meta-training
(Triantaﬁllou et al., 2020;
Lee et al., 2019; Chen et al., 2020) using 1-9shot 5-way random-distribution tasks (as described in
Section 3.2).
We pre-train baselines (i.e., Fine-Tune, 1-NN, Baseline++) using mini-batch gradient descent, and then ﬁne-tune on the support or perform a 1-NN classiﬁcation.
We evaluate all baselines and models using a wide range of imbalanced meta-testing tasks.
In contrast to previous work, we evaluate models using two additional imbalance distributions, linear and step ; this allows us to control the imbalance level deterministically and provide insights from multiple angles.
For the imbalanced meta-dataset scenario, we vary the class distributions of the meta-training datasets.
We isolate this level of imbalance by meta-training and meta-evaluating on balanced FSL tasks.
All main experiments are repeated three times with different initialization seeds.
Each data point represents the average performance of over 600 meta-testing tasks per run.
Additional details.
We adapted a range of 11 unique baselines and FSL methods: Fine-tune baseline (Pan & Yang, 2010), 1-NN baseline, Baseline++ (Chen et al., 2019), SimpleShot (Wang et al., 2019a), Prototypical Networks (Snell et al., 2017), Matching Networks (Vinyals et al., 2017), Relation Networks (Sung et al., 2017), MAML (Finn et al., 2017), ProtoMAML (Triantaﬁllou et al., 2020), DKT (Patacchiola et al., 2020), and Bayesian MAML (BMAML) (Yoon et al., 2018).
Implementation details of these algorithms are supplied in Appendix A. We used a 4 layer convolutional network as backbone for each model, following common practice (Chen et al., 2019).
We train and evaluate all methods on MiniImageNet (Ravi & Larochelle, 2016; Vinyals et al., 2017), containing 64 classes with 600 image samples each.
In the imbalanced meta-dataset setting, we half the MiniImageNet dataset to contain 300 samples per class on average, and control imbalance as described in section 3.3.
For full implementation details, see Appendix A.
4. C LASS I MBALANCED S UPPORT S ET
Effect of Class Imbalance with Standard Meta-Training.
Figure 1 highlights the crux of the class-imbalance problem at the support set level.
Speciﬁcally, the ﬁgure shows standard metatrained FSL models (Vinyals et al., 2017) and pre-trained baselines, evaluated on the balanced 5shot 5-way task and three imbalanced tasks.
We observe that introducing even a small level of imbalance (linear 4-6-shot 5-way, ρ = 1 . 5 ) produces a signiﬁcant 2 performance difference for 6 out of 13 algorithms, compared with the balanced 5-shot task.
The average accuracy drop is − 1 .
5% for metric-based models and − 8 .
2% for optimization-based models.
On tasks with a larger imbalance (1-9shot random, ρ = 9 . 0 ), the performance drops by an average − 8 .
4% for metric-based models and − 18 .
0% for optimization-based models compared to the balanced task.
Interestingly, despite the additional 12 samples in the support set in 1-9shot step tasks with 1 minority class ( ρ = 9 . 0 ), the performance drops by − 5 .
0% on the balanced task with 25 support samples in total.
[Table]
[Figure]
Standard vs. Random-Shot Meta-Training.
In Figure 3, we show the accuracy for increasing imbalance levels ( ρ ) using evaluation tasks with a linear distribution and ﬁxed support set size ( K i ≈ 5 ) for a fair comparison.
Comparing Standard and Random-Shot meta-training (solid black and solid red lines) reveals that only a few methods beneﬁt from Random-Shot meta-training.
On the balanced 5-shot task, we observe a − 6 .
0% decrease in accuracy, caused by Random-Shot over Standard meta-training.
On the imbalanced 1-9shot random task, Random-Shot offers a limited improvement over the Standard, with a signiﬁcant increase in performance for only 3 out of 10 models.
Those improvements include +2 .
5% for Relation Net, and +6 .
6% for BMAMLs.
These results suggest that exposing FSL methods to imbalanced tasks during meta-training does not automatically lead to improved performance at meta-test time.
Interestingly, in an extreme imbalance case (1-21shot step 4minor, Appendix E) only ProtoNet and RelationNet obtained a signiﬁcantly higher performance with Random-Shot ( +18% compared to Standard).
This suggests that the advantage may emerge from coupling Random Shot with the prototype calculation mechanism unique to those methods.
The results also suggest that some models have natural robustness to imbalance: Relation Net,
MatchingNet, and DKT only drop slightly compared to other methods.
Random-Shot with Random Over-Sampling.
In Figure 3, we observe that the performances of optimization-based methods such as MAML and BMAML signiﬁcantly improve by applying random over-sampling with augmentation (ROS+) and without augmentation (ROS).
In the largest imbalance case in the graph ( ρ =9), we observe that models using ROS+ at inference (dotted and yellow lines) improve over the Standard (solid black) by +6 .
7% ; in particular, optimization-based methods improve by +12 .
2% , ﬁne-tune baselines by +7 .
4% , and metric-based by +2 .
In the imbalanced task, the least affected model is MatchingNet only dropping − 1 .
9% compared to the balanced task; we provide a list of top-50 performing models in Table 3 (Appendix C.1).
Standard (ROS+ at inference) achieves the highest average performance gains ( +8 . 5% ); tieing for second best is Random-Shot with ROS (ROS+ at inference) with +6 .
9% and Random-Shot with ROS+ ( +6 . 4% ).
We breakdown the results by type in Appendix C.2 (Figure 9).
Imbalance with More Shots.
We explored additional settings with a higher number of shots, see
Figure 4.
Speciﬁcally, we train models using Random-Shot meta-training with 1-29 shot and 1random episodes.
We then evaluate those models on imbalanced tasks with an average number ofshots and 25 shots, respectively.
The bottom row of Figure 4 shows the difference in performance between the imbalance and balanced tasks.
We observe that for the high-shot condition (right column), the general model performance increases while the models are less affected by imbalance; however, the gap with respect to the balanced condition remains signiﬁcant.
Models achieve 5560% of their performance on the balanced task within ﬁrst 5 avr.
shots; increasing the number of shots to 15 only boosts their performance by +7% .
This may explain why imbalance will have an inevitable impact on small classiﬁcation tasks: better performance achieved via a higher numbers of support samples in the majority classes, does not offset the performance lost due to lack of samples in the minority classes.
In Appendix C.2, we breakdown the results for each model type.
[Figure] [Figure]
Backbones.
In Figure 5, we report the combined average accuracy of all models and imbalance strategies against different backbones (Conv4, Conv6, ResNet10, ResNet34).
Overall, deeper backbones seem to perform slightly better on the imbalanced tasks, suggesting a higher tolerance for imbalance.
For instance, using Conv4 gave − 8 .
6% difference between the balanced and the 1-9shot random task, while using ResNet10 the gap is smaller ( − 6 . 8% ).
The performance degradation observed with ResNet34 is similar to that reported by Chen et al.
and is most likely caused by the intrinsic instability of meta-training routines on larger backbones.
In Appendix C.3, we breakdown the results across different models and training strategies.
Precision and Recall.
Looking at the precision and recall tables in Appendix C.4, provides additional insights about each algorithm.
For instance, DKT (Patacchiola et al., 2020) shows very strong performance in classes with a small number of shots and well-balanced performances for higher shots.
This may be due to the partitioned Bayesian one-vs-rest scheme used for classiﬁcation by DKT, with a separate Gaussian Process for each class; this could be more robust to imbalance.
BMAML, on the other hand, fails to correctly classify samples with K =and K =
3 samples, showing that the method has a strong bias towards the majority classes. [Figure]
balanced task. Rebalancing Cost Functions We applied Focal Loss (Lin et al., 2017) and Weighted Loss (Buda et al., 2018; Japkowicz & Stephen, 2002) to the inner-loop of optimization-based methods at inference time only.
Results in Figure 6 and Appendix D.1 show that overall, Focal Loss and Weighted Loss are not as effective as ROS+ techniques. However, ROS+ and Weighted Loss perform very similarly, suggesting a similar effect on the imbalanced task. 4. C LASS I MBALANCED M ETA -D ATASET Imbalanced
Mini-ImageNet. To induce dataset imbalance, we half the number of samples per class in Mini-ImageNet.
In table(left), we show the accuracy after training via standard episodic (meta-)training (Vinyals et al., 2017) with (balanced) 5-shot 5-way tasks.
In this particular scenario, we use signiﬁcantly higher imbalance levels ( ρ = 19 ) compared to those in the previous section; despite this, we observe small, insigniﬁcant performance differences between balanced and imbalanced conditions.
In additional experiments, we further reduced the dataset size to contain a total of 4800 images and 32 randomly selected classes.
In Figure 10 (Appendix C.5), we observe a more signiﬁcant performance drop as we increase the number of minority classes.
Meta-evaluating on CUB showed a similar trend, with an average drop of − 1 .
6% on the most extreme imbalance setting: 30-510 step with 24 minority classes and ρ = 17 .
(Appendix C.5).
When we breakdown the results by model in Appendix C.5, we observe that optimization-based approaches and ﬁne-tune baselines have a slight advantage over the metric-based, most likely due to the ability to adapt during inference.
Interestingly, in this setting RelationNet performs the worse with a drop of − 4 .
3% w. r. t. the balanced task in the most extreme setting (24 minority, Mini-ImageNet).
Additional results.
To evaluate the performance under a strong dataset shift, we evaluated MiniImageNet trained models on tasks sampled from CUB-200-2011.
In Table 1 (right), we observe that models are not affected at all by the imbalanced setting despite the harder scenario.
In Appendix D, we provide additional experiments with BTAML (Lee et al., 2019), and an analysis of the correlation between meta-dataset size and performance  D ISCUSSION FSL robustness to class imbalance.
All examined FSL methods are susceptible to class imbalance, although some show more robustness (e.g. Matching Net, Relation Net and DKT).
Optimization-based methods and ﬁne-tune baselines suffer more as they use conventional supervised loss functions in the inner-loop which are known to be particularly susceptible to imbalance (Buda et al., 2018; Johnson & Khoshgoftaar, 2019; Japkowicz & Stephen, 2002).
Moreover, the problem
[Table] of class imbalance persists as the backbone complexity and support set size increase. Those results suggest that current solutions will offer sub-optimal performance in real-world few-shot problems
[Table]
Effectiveness of Random-Shot meta-training.
Our experiments test a simple solution to class imbalance that has been popular in the meta-learning community – Random-Shot meta-training
(Triantaﬁllou et al., 2017; Lee et al., 2019; Chen et al., 2020).
Contrarily to popular belief, our ﬁnding reveals that this method is scarcely effective.
Extensive analysis and validation performance through epochs (see Appendix B) suggests that this results is genuine and unlikely to be the result of inappropriate parameter tuning.
Our ﬁnding have important consequence, suggesting that robustness to imbalance cannot be obtained by the simple exposure to imbalance tasks.
Effectiveness of re-balancing procedures.
The results suggest that a simple procedure, Random over sampling (ROS), is quite effective in tackling class imbalance issues.
Therefore, we encourage the community to include it in their evaluation as ROS is simple to implement and it can be applied to almost any algorithm.
However, ROS does not provide any particular advantage to methods in the highest levels of the performance ranking, like MatchingNet and DKL.
This could be due to diminishing returns and should be investigated on a case-by-case basis.
Effect of imbalance at the meta-dataset level.
Imbalance in the (meta-)dataset has minimal effect on the meta-learning procedure.
This could be the result of standard episodic (meta-)training which samples classes with equal probability and causes natural re-sampling.
Likely, datasets with lower intra-class variation and larger imbalance (Liu et al., 2019; Wang et al., 2017; Salakhutdinov et al., 2011) could produce more dramatic performance changes.
C ONCLUSION
In this work, we have provided a detailed analysis of class-imbalance in FSL, showing that classimbalance at the support set level is problematic for many methods.
We found that most metric-based models present a built-in robustness to support-set imbalance, while in optimization-based models imbalance issues can be alleviated using oversampling.
In our experiments, Random-Shot metatraining provided minimal beneﬁts suggesting that meta-learning methods do not learn to balance from random-shot episodes alone.
Results on meta-dataset imbalance showed just a small negative effect, but this effect is not as dramatic as with the task-level imbalance.
In future work, the insights gained with our investigation could be used to design novel few-shot methods that can guarantee a stable performance under the imbalance condition.
A I MPLEMENTATION D ETAILS A. D ATASETS
For the imbalanced support set experiments, we used MiniImageNet (Vinyals et al., 2017; Ravi & Larochelle, 2016) following the same version popular version as (Ravi & Larochelle, 2016;
Cheng et al., 2018).
All meta-learning models used 64, 16, 20 classes for the meta-training D train , meta-validation D val , and meta-testing D test datasets, respectively, with each class containing 600 samples.
All images are resized to 84 by 84px.
For the feature-transfer baselines (1-NN, ﬁne-tune, and Baseline ++ ), we used a conventionally partitioned training and validation datasets.
Speciﬁcally, we combined D train and D val classes (i.e., 64 + 16 = 80 classes), then partitioned the samples of each class into 80% - 20% split for pre-training - validation, forming D (cid:48) train and D (cid:48) val (where D (cid:48) ( y ) train ≡ D (cid:48)
( y ) val ).
Thus, the baseline were trained on the same number of training samples as the meta-learning methods, albeit with more classes and less samples per class.
All models were evaluated on FSL tasks sampled from D test .
For the imbalanced meta-dataset experiments, we used two variants of MiniImageNet.
In the ﬁrst, referring to Table 13, we halved the average number of samples per class in the meta-training dataset D train to allow us to introduce imbalance artiﬁcially into the dataset.
In the second scenario, referring to Figure 10, we reduced the size of the meta-training by a more considerable degree.
Speciﬁcally, the size of the meta-training dataset was controlled to contain a total of 4800 images distributed among 32 randomly selected classes from the meta-training dataset of Mini-ImageNet.
For meta-learning methods, we kept the original 16 and 20 classes for meta-validation and meta-testing with 600 samples each.
To allow as fair comparison as possible, we used the same meta-training datasets for the baselines and meta-learning models.
However, the baselines used a balanced validation set created from the leftover samples from the original meta-training dataset.
For the imbalanced meta-dataset experiments, we also evaluated on tasks sampled fromrandomly selected classes from CUB-200-2011 (Wah et al., 2011), following the same line of work as Chen et al.
A. T RAINING P ROCEDURE .
All methods follow a similar three-phase learning procedure: meta-training, meta-validation, and meta-testing.
During meta-training, an FSL model was exposed to 100k tasks sampled from D train .
After every 500 tasks, the model was validated on tasks from D val and the best performing model was updated.
At the end of the meta-training phase, the best model was evaluated on tasks sampled from D test .
The baselines (i.e., ﬁne-tune, 1-NN, Baseline ++ ) follow a similar three-phase procedure but with the meta-training / meta-validation phases exchanged for conventional pre-training / validation on mini-batches (of size 128) sampled from D (cid:48) train and D (cid:48) val as outlined above.
In all three meta- phase’s tasks, we used 16 query samples per class, except for the 20-way Prototypical Network, where we used 5 query samples per class during meta-training to allow for a higher number of samples in the support set.
Meta-/Pre- Training Details.
In the imbalanced support set setting, we meta-train FSL methods using standard episodic meta-training (Vinyals et al., 2017) using 5-shot 5-way tasks.
We also explore random-shot episodic training (Lee et al., 2019) using 1-9shot 5-way random-distribution tasks (as described in section 3).
We meta-/pre- trained on 100k tasks/mini-batches, using a learning rate of 10 − 3 for the ﬁrst 50k episodes/mini-batches, and 10 − 4 for the second half.
The baselines and SimpleShot are trained using 100k balanced mini-batches with a batch size of 128.
All methods were meta-validated on 200 tasks/mini-batches every 500 meta-training tasks/mini-batches to select the best performing model.
Meta-Testing.
The ﬁnal test performances were measured on a random sample of 600 tasks.
We report the average 95% mean conﬁdence interval in brackets/errorbars.
In the imbalanced support set experiments, we evaluate tasks with various imbalance levels and distributions, as speciﬁed in ﬁgures and tables.
In the imbalanced meta-dataset experiments, we evaluate using regular, balanced 5-shot 5-way tasks.
Data Augmentation.
During the meta-/pre-training phases, we apply standard data augmentation techniques, following a similar setup to Chen et al.
(2019), with a random rotation of 10 degrees, scaling, random color/contrast/brightness jitter.
Meta-validation and meta-testing had no augmentation apart from in the Random-Shot (ROS+) setting where the same augmentations were applied on the oversampled support images.
All images are resized to 84 by 84 pixels.
A. B ACKBONE A RCHITECTURES
All methods shared the same backbone architecture.
For the core contribution of our work, we used
Conv4 architecture consisting of 4 convolutional layers with 64 channels (padding 1), interleaved by batch normalization (Ioffe & Szegedy, 2015), ReLU activation function, and max-pooling (kernel size 2, and stride 2) (Chen et al., 2019).
Relation Network used max-pooling only for the lastlayers of the backbone to account for the Relation Module.
The Relation Module consisted of two additional convolutional layers, each followed by batch norm, ReLU, max-pooling).
For experiments with different backbones: Conv6, ResNet10, and ResNet34 (Chen et al., 2019).
Conv6 extended the Conv4 backbone to 6 convolutional layers, and max-pooling applied after each ﬁrst 4 layers.
ResNet models (He et al., 2016) followed the same setup as Chen et al.
For imbalanced meta-dataset and imbalanced reduced meta-dataset, we used the Conv4 model, with 32 channels instead of 64, due to less training data.
A. FSL M ETHODS AND B ASELINES
In our experiments, we used a wide range of FSL methods (full details can be found in our source code):
Baseline (ﬁne-tune) (Pan & Yang, 2010) represents a classical way of applying transfer learning, where a neural network is pre-trained on a large dataset, then ﬁne-tuned on a smaller domain-speciﬁc dataset.
The baseline’s backbone followed a single linear classiﬁcation layer with a single output for each meta-training dataset class.
The whole network was trained during pre-training.
During meta-testing, the baseline’s pre-trained linear layer was exchanged for another randomly initialized classiﬁcation layer with outputs matching the task’s number of classes ( N -way).
Fine-tuning was performed on the new randomly initialized classiﬁcation layer using the support set S . 2.
Baseline (1-NN) is another classical method of applying transfer learning but using a knearest neighbor classiﬁer instead of the classiﬁcation layer during meta-validation.
Pretraining was performed in the same was the ﬁne-tune baselines.
During the meta-testing time, instead of ﬁne-tuning, the model matched query samples to the nearest support sample’s class based on Euclidian distance.
Baseline ++ (Chen et al., 2019) augments the ﬁne-tune baseline by using Cosine Similarity on the last layer.
Matching Network (Matching Net) (Vinyals et al., 2017) uses context embeddings with an LSTM to effectively perform k-nearest neighbor in embedding space using cosine similarity to classify the query set.
Prototypical Networks (ProtoNet) (Snell et al., 2017) maps images into a feature space and calculates class means (called prototypes).
The query samples are then classiﬁed based on the closest Euclidian distance to a classes’ prototype.
We evaluate two models, one meta-trained like the others on 5-way episodes, and another variation trained on 20-way episodes.
During 20-way meta-training, we set the class’ query size to 5. 6.
Relation Networks (Relation Net) (Sung et al., 2017) augment the classical Prototypical Networks by introducing a relation module (another neural network) that compares the distance instead of using Euclidian.
The original method uses Mean Squared Error to minimize the relation score between samples of the same type.
However, we follow work by Chen et al.
(2019), use cross-entropy loss that expediates meta-training.
The structure of the relation module is described in section A.3.
DKT (formally called GPShot) proposed by Patacchiola et al.
(2020) is a probabilistic approach that utilizes the Gaussian Processes with a deep neural network as a kernel function.
We used Batch Norm Cosine distance for the kernel type.
SimpleShot (Wang et al., 2019a) augments the 1-NN baseline model by normalizing and centering the feature vector using the dataset’s mean feature vector.
The query samples are assigned to the nearest prototype’s class according to Euclidian distance.
In contrast to the baseline models, pre-training is performed on the meta-training dataset like other meta-learning algorithms, and meta-validation is used to select the best model based on performance on tasks sampled from D val .
MAML (Finn et al., 2017) is a meta-learning technique that learns a common initialization of weights that can be quickly adapted for task using ﬁne-tuning on the support set.
The task adaptation process uses a standard gradient descent algorithm minimizing Cross-Entropy loss on the support set.
The original method uses second-order derivates; however, due to more efﬁcient calculation, we use the ﬁrst-order MAML, which has been shown to work just as well.
We set the inner-learning rate to 0.1 with 10 iteration steps.
We optimize the meta-learner model on batches of 4 meta-training tasks.
These hyperparameters were selected based on our hyperparameter ﬁne-tuning.
ProtoMAML (Triantaﬁllou et al., 2020) augments traditional ﬁrst-order MAML by reinitializing the last classiﬁcation layer between tasks.
Speciﬁcally, the weights of the layer are assigned to the prototype for each class’s corresponding output.
This extra step combines the ﬁne-tuning ability of MAML and the class regularisation ability of Prototypical Networks.
We set the inner-loop learning rate to 0.1 with 10 iterations.
Unlike for MAML, we found that updating the meta-learner after a single meta-training task gave the best performance.
Bayesian-MAML (BMAML) (Yoon et al., 2018) augments the MAML method by replacing the inner loop’s standard stochastic gradient descent with Bayesian gradient-based updates.
BMAML uses Stein Variational Gradient Descent that is a non-parametric variational inference method combining strengths of Monte Carlo approximation and variational inference.
The algorithm learns to approximate a posterior over the initialization parameters conditioned on the task support set.
Yoon et al.
(2018) also adds a chaser loss, which utilizes the samples in the query set during meta-training to approximate the true task posterior.
Minimization of the KL divergence between the true task posterior and the estimated task parameter posterior can be used to drive the meta-training process.
We set the innerloop learning rate to 0.1 with 1 inner-loop step.
We found that using a higher inner-loop step number destabilized performance.
We used 20 particles.
The chaser loss variation used a learning rate of 0.5.
Again, we found these combinations of hyperparameters to give the best results.
Bayesian-TAML (BTAML) (Lee et al., 2019)
[Left out of the main paper body.
This method augments the MAML method with four main changes: 1) task-dependent parameter initialization z , 2) task-dependent per-layer learning rate multipliers γ , 3) taskdependent per-class learning rate multiplier ω , 4) meta-learned per-parameter learning rate α (similar to Meta-SGD, Li et al.
However, our results for BTAML were unstable, suggesting a fault in our implementation.
Unfortunately, we did not identify it in time for the submission
and we decided to move the method’s results to the appendix.
In our experiments, we explored several variations to this method with various parameters ( z, ω, γ, α ) turned on and off, as well as different hyperparameters.
We found that using a meta-learning rate of 10 − 4 performed better than 10 − 3 in contrast to the other models.
We set the inner-learning rate to 0.1 with 10 iteration steps, and optimize the meta-learner model on batches of 4 meta-training tasks.
Again, we found this set up worked best in our experiments.
A. C LASS I MBALANCE T ECHNIQUES AND S TRATEGIES
We pair FSL methods with popular data-level class-imbalance strategies: 1.
Random Over-Sampling (ROS) (Japkowicz & Stephen, 2002) without and with data augmentation ( ROS and ROS+ ).
For the augmentations we used: random sized cropping between 0.15 and 1.1 scale of the image, random horizontal ﬂip, and random color/brightness/contrast jitter.
A visualization of ROS and ROS+ is presented in Figure 7.
During meta-training, ROS+ augmentations were applied twice: once when sampling from the meta-training dataset, and the second time during the support set resampling.
This may have slightly destabilized meta-training, which would explain why sometimes Random-
Shot with ROS (ROS+ at inference only) achieved better performance than Random-Shot with ROS+ in Figures 3 and 4. 2.
Random-Shot Meta-Training (Triantaﬁllou et al., 2020; Lee et al., 2019; Chen et al., 2020) was applied as speciﬁed in the main body of the paper (Section 3.4).
Focal Loss (Lin et al., 2017).
Focal Loss has been found to be very effective in combating the class-imbalance problem on the one-stage object detectors.
We exchanged the innerloop cross-entropy loss of optimization-based algorithms and ﬁne-tune baselines with the focal loss with γ = 2 and α = 1 . Results are presented in Figure 6 in the main paper body and in Appendix D.1.
Weighted Loss . Weighted loss is also commonly used to rebalance the effects of classimbalance (Buda et al., 2018; Leevy et al., 2018). We weight the inner-loop cross-entropy loss of optimization-based algorithms and ﬁne-tune baselines by inverse class frequency of support set samples.
Results are presented in Figure 6 in the main paper body and in Appendix D.1.
B V ERIFICATION OF I MPLEMENTATION
We implement the FSL methods in PyTorch, adapting the implementation of (Chen et al., 2019) but also borrowing from other implementations online (see individual method ﬁles in the source code for individual attribution).
However, we have heavily modiﬁed these implementations to ﬁt our imbalanced FSL framework, which also offers standard and continual FSL compatibility (Antoniou et al., 2020).
We provide our implementations for ProtoMAML and BTAML for which no opensource implementation in PyTorch existed as of writing.
To verify our implementations, we compare methods on the standard balanced 5-shot 5-way task with reported accuracy.
Results are presented in Table 2.
We see that algorithms achieve very similar performance with no less than 3% accuracy points compared to the reported performance.
The discrepancies can be accounted for due to smaller training batch for SimpleShot, different augmentation strategies for the other methods
[Table]
and natural variance stemming from random initialization.
We show the validation performance over epochs for each method in Figure 8 on the next page
[Table]
[Table]
[Figure]
different seeds.
C B REAKDOWN OF R ESULTS
In this section
[Table]
we breakdown the results from the main body of the paper.
Speciﬁcally, we provide the top-50 performing models on the imbalaned 1-9shot linear task in subsection C.1.
The breakdown of higher shots experiment (Figure 4) is provided in subsection C.2.
In subsection C.3, we include the breakdown of the backbone experiment (Figure 5) showing the performance by algorithm type and training procedure.
We provide precision and recall tables of linear 1-9shot 5-way tasks for each meta-training procedure in subsection C.4.
In section C.5, we provide the results for the imbalanced reduced meta-dataset (Figure 10)
[Table]
C. T OP -50 PERFORMING MODELS ON 1-9 SHOT LINEAR
[Table]
C. R ANDOM 1-29 SHOT AND 1-49 SHOT TASKS
[Figure]
C. B ACKBONE E XPERIMENTS
We run additional experiments with different backbone models.
In Tables 4 and 5, we show the random 1-9 shot 5-way performance on Conv6, ResNet
[Table]
and ResNet34.
We can observe that for many of the methods, ROS still beneﬁts the models.
However, the reader should exercise caution since we used the same hyper-parameterization as for the four-layered CNN.
We did not perform any hyperparameter ﬁne-tuning on these backbones; the results would likely be higher if we allowed for longer meta-training.
Some results are missing due to destabilization in meta-training caused by deeper backbones.
[Table] [Table]
C. P RECISION AND R ECALL T ABLES
[Table] [Table]
[Table] [Table]
C. I MBALANCED R EDUCED M ETA -T RAINING D ATASET
In this section, we present the results for the reduced meta-training MiniImageNet dataset (withclasses).
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Table]
D A DDITIONAL E XPERIMENTS D. A DDITIONAL I MBALANCE S TRATEGIES
In Figure 11, we compare Standard meta-training and Random-Shot meta-training with focal loss.
We observe no signiﬁcant advantage of using Focal Loss Random-Shot meta-training over the Standard meta-training experiments.
[Figure]
D. BTAML
We implemented and trained Bayesian TAML (Lee et al., 2019); however, the training performance graphs suggest a mistake in our implementation, which we did not manage to identify in time for the submission.
For this reason, we have left out these results from the main paper.
The BTAML ( α, ω, γ, z ) corresponds to full BTAML, while others indicate variants with the corresponding components turn off.
We provide models’ performances with the full performance in Appendix
[Figure]
[Equation]
[Table]
[Equation]
[Table]
[Figure]
[Table]
[Figure]
D. A NALYSIS OF S AMPLES P ER C LASS IN M ETA -T RAINING D ATASET
[Table]
E F ULL R ESULTS FOR M AIN E XPERIMENTS
Figures 14 and 13 show accuracy and F1 scores, respectively, on imbalanced tasks of Standard (5shot) and Random Shot (1-9shot) meta-training.
Their corresponding result tables are in Tablesto 21.
Table 22 provides the full results for the main imbalanced meta-dataset experiments.
[Table]
li
[Table]
e ( 1 - NN )
B
[Table]
li n e ( f i n e - t un e )
B a s
[Table]
li n e + +
M a t c h i n g
N e t
P r o t o
N e t
P r o t o
N e t ( 20 - w a y )
R e l a t i o n
N e t (
C E )
D K T
S i m p l e S h
o t
M A M
L
P r o t o
M A M
L
B M A
M L
B M A
M L ( c h a s e r )
B T A M
L ( , , , z )
B T A M
L ( , , z )
B T A M
L ( )
[Table]
A cc u r a c
y , %
A cc u r a c
y , %
A cc u r a c
y , %
A cc u r a c
y , %
A cc u r a c
y , %
A cc u r a c
y , %
A cc u r a c
y , %
A cc u r a c
y , %
A cc u r a c
y , %
A cc u r a c
y , %
[Table]
s e li n e ( 1 - NN )
B a s e li n e ( f i n e - t un e )
B a s e li n e + +
M a t c h i n g
N e t
P r o t o
N e t
P r o t o
N e t ( 20 - w a y )
R e l a t i o n
N e t (
C E )
D K T
S i m p l e S h
o t
M A M
L
P r o t o
M A M
L
B M A
M L
B M A
M L ( c h a s e r )
B T A M
L ( , , , z )
B T A M
L ( , , z )
B T A M
L ( )
[Table]
1 S c o r e s , %
F 1 S c o r e s , % F 1 S c o r e s , %
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
T a b l e 22 :
F u ll r e s u lt s f o r T a b l e 1 .
I m b a l a n ce d
M i n iI m a g e N e t
I m b a l a n c e d
M i n iI m a g e N e t →
[Table]
U
[Table]
- 200I -
[Table]
i s t
r
i bu ti on
[Table]
l a n
[Table]
li
[Table]
ea r
[Table]
a ndo m
[Table]
e p
[Table]
ea r r a ndo m
[Table]
m
[Table]
l
[Table] [Table]
i
[Table] [Table]
M i
[Table]
# s a m p l e
[Table] [Table]
i n
[Table]
[Table] [Table]
i
[Table] [Table]
n
[Table] [Table]
[Table] [Table]
[Table] [Table]