H OW S ENSITIVE ARE M ETA -L EARNERS TO D ATASET I MBALANCE ?
A BSTRACT
Meta-Learning (ML) has proven to be a useful tool for training Few-Shot Learning (FSL) algorithms by exposure to batches of tasks sampled from a meta-dataset
However, the standard training procedure overlooks the dynamic nature of the real-world where object classes are likely to occur at different frequencies.
While it is generally understood that imbalanced tasks harm the performance of supervised methods, there is no signiﬁcant research examining the impact of imbalanced meta-datasets on the FSL evaluation task.
This study exposes the magnitude and extent of this problem.
Our results show that ML methods are more robust against meta-dataset imbalance than imbalance at the task-level with a similar imbalance ratio ( ρ < 20 ), with the effect holding even in long-tail datasets under a larger imbalance ( ρ = 65 ).
Overall, these results highlight an implicit strength of ML algorithms, capable of learning generalizable features under dataset imbalance and domain-shift.
The code to reproduce the experiments is released under an open-source license
I NTRODUCTION
Few-Shot Learning (FSL) aims at reducing the burden of training machine-learning models on a large number of labeled data points.
A common way of training FSL models is through MetaLearning (ML) (Hospedales et al., 2020; Vinyals et al., 2017) with the model repeatedly exposed to batches of FSL tasks/episodes sampled from a (meta-)training dataset that is different but similar to the one seen during the (meta-)testing phase.
We will use the preﬁx “meta” to distinguish the highlevel training and evaluation routines of ML (outer loop) from the training and evaluation routines at the single-task level (inner loop).
Motivation.
Standard FSL benchmarks (eg.
Omniglot, Mini-ImageNet) assume an equal number of data points for each class in the meta-dataset.
However, in real-world applications, it is common to encounter datasets with varying numbers of data points per class (Guan et al., 2020; Ochal et al., 2020; Massiceti et al., 2021).
In some cases, the dataset distribution can be immeasurably large or even unknown, like in online class imbalance (Wang et al., 2014).
Although meta-dataset imbalance is present in recent FSL benchmarks, such as Meta-Dataset (Triantaﬁllou et al., 2020) or meta-iNat (Wertheimer & Hariharan, 2019), its impact on ML has remained unexplored.
Contributions.
We provide quantitative insights into the dataset imbalance problem and show that meta-learners are robust to meta-dataset imbalance under 1) various imbalance distributions, 2) dataset sizes, and 3) moderate cross-domain-shift.
The impact of imbalance at the task level is much more signiﬁcant in comparison.
R ELATED W ORK
Class Imbalance.
In classiﬁcation, imbalance occurs when at least one class (the majority class) contains a higher number of samples than the others.
The classes with the lowest number of samples are called minority classes.
If uncorrected, conventional supervised loss functions, such as (multiclass) cross-entropy, skew the learning process in favor of the majority class, introducing bias and poor generalization toward the minority class samples (Buda et al., 2018; Leevy et al., 2018).
The object recognition community studies class imbalance using real-world datasets or distributions that approximate real-world imbalance (Buda et al., 2018; Johnson & Khoshgoftaar, 2019; Liu et al., 2019).
Buda et al.
(2018) state that two distributions can be used: linear and step imbalance.
At large-scale, datasets with many samples and classes tend to follow a long-tail distribution (Liu et al., 2019;
Salakhutdinov et al., 2011; Reed, 2001), with most of the classes occurring with small frequency and a few classes occurring with high frequency.
Meta-Dataset FSL Benchmarks.
FSL methods (Snell et al., 2017; Sung et al., 2017; Edwards & Storkey, 2017; Vinyals et al., 2017; Finn et al., 2017; Ravi & Larochelle, 2016; Chen et al., 2019; Dhillon et al., 2020; Patacchiola et al., 2020; Zhang et al., 2021) are typically compared against balanced benchmark (eg.
Omniglot, MiniImageNet).
More recent benchmarks (Triantaﬁllou et al., 2020; Wertheimer & Hariharan, 2019) contain some levels of imbalance in the metadataset.
Speciﬁcally, Meta-Dataset (Triantaﬁllou et al., 2020) combines datasets of different sizes (e.g., Omniglot, CUB, Fungi, Aircraft, ImageNet, etc.) into a single corpus.
A different dataset, meta-iNat (Wertheimer & Hariharan, 2019), models imbalance according to a long-tail distribution.
While imbalance at the task-level has received some attention (Triantaﬁllou et al., 2020; Chen et al., 2020; Lee et al., 2019; Guan et al., 2020), limited research quantiﬁes the impact of imbalance at the dataset-level .
It can be cumbersome to evaluate the imbalance of Meta-Dataset and meta-iNat speciﬁcally, as it requires access to a balanced version of the datasets with an equal total number of samples.
Therefore, in our analysis, we artiﬁcially induce imbalance into datasets and model imbalance according to various distributions approximating many real-world scenarios.
P ROBLEM D EFINITION
Standard Task/Meta-Training.
Benchmarking FSL methods typically involves three phases: meta-training/pre-training, meta-validation, and meta-testing.
Each phase samples batches of data points or tasks from a separate dataset: D train , D val , and D test , respectively, such that the classes and samples between each of the datasets are non-overlapping.
We assume that a dataset D is balanced when it contains N D classes and K D samples for each class.
Similarly, a standard K S -shot N S -way FSL classiﬁcation task is deﬁned by a small support set ,
S = { ( x 1 , y 1 ) , . . .
, ( x s , y s ) } ∼ D , containing N S × K S image-label pairs drawn from N S unique classes with K S samples per class ( |S| = K S × N S and K S (cid:28) K D and N S (cid:28) N D ).
The goal is to minimize some loss over a query set , Q = { ( x 1 , y 1 ) , . . .
, ( x q , y q ) } ∼ D , containing a different set of K Q samples drawn from the same N S classes (i.e. N Q = N S , Q ( x ) ∩ S ( x ) = ∅ and
Q ( y ) ≡ S ( y ) ).
Imbalanced Tasks/Meta-Dataset.
For brevity, but without loss of generality, we deﬁne a distribution for a set of data points ∗ ∈
{D , S , Q} as a tuple ( K ∗ min , K ∗ max , N ∗ , M ∗ ) for a distribution
I ∈ { linear, step, long - tail } (Buda et al., 2018; Wertheimer & Hariharan, 2019), where K ∗ min is the minimum number of samples per class, K ∗ max is the maximum number of samples per class, N ∗ is the number of classes, and M ∗ is an additional parameter used for step and long - tail imbalance.
We induce imbalance in our experiments using one of I distributions deﬁned as:
• Linear imbalance .
The K ∗ i number of samples for each class i ∈ { 1 ..N ∗ } is deﬁned by
[Equation]
where c = 0 .
499 for rounding purposes.
For example, this means that for linear (1,9,5,-) set, K ∗ i ∈ { 1 , 3 , 5 , 7 , 9 } , and for linear (4,6,5,-) set, K ∗ i ∈ { 4 , 4 , 5 , 6 , 6 } .
• Step imbalance .
The number of class samples, K ∗ i , is determined by an additional variable M ∗ specifying the number of minority classes.
Speciﬁcally, for classes i ∈ [Equation] 1 ..N ∗ } :
[Equation]
For example, in a step (1,9,5,1) set, there is 1 minority class
and K ∗ i ∈ { 1 , 9 , 9 , 9 , 9 } .
• Long-Tail imbalance .
Imbalance could be modeled by a Zipf’s/Power Law (Reed, 2001) for a more realistic imbalance distribution.
[Figure]
i
[Figure]
t o b a l a n c e d
[Figure]
We also report the imbalance ratio ρ , which is a scalar identifying the level of class imbalance; this is often reported in the CI literature for the supervised case (Buda et al., 2018).
We deﬁne ρ to be the ratio between the number of samples in the majority and minority classes in a set of data points:
[Equation]
E XPERIMENTS
Setup.
We meta-trained a range of FSL and ML methods on multiple datasets and imbalance distributions.
Speciﬁcally, we investigate Prototypical Networks (Snell et al., 2017), Matching Networks (Vinyals et al., 2017), Relation Networks (Sung et al., 2017), MAML (Finn et al., 2017),
ProtoMAML (Triantaﬁllou et al., 2020), DKT (Patacchiola et al., 2020), SimpleShot (Wang et al., 2019), and supervised pre-training methods – Baseline and Baseline++ from Chen et al. (2019).
Each model was meta-trained/pre-trained on 100k tasks/mini-batches sampled from variations of the meta-training dataset of Mini-ImageNet (Ravi & Larochelle, 2016)
( D train ), originally containing 64 classes and 600 samples per class.
To get enough samples for the majority classes, we halved the total number of samples from the original D train of Mini-ImageNet, and denote this dataset as D (cid:48) train , where |D (cid:48) train | ≈ 64 × 300 = 19200 .
We induced imbalance into the dataset according to one of the I -distributions outlined in Section 3.
For simplicity, we did not modify the meta-validation and meta-testing datasets ( D val and D test ), keeping them like in the original MinImageNet.
To emulate stronger domain-shift, we also evaluated models on 50 randomly selected classes from CUB-2011 (Wah et al., 2011).
More details and experiments are in Appendix A.
Meta-learners are robust to dataset imbalance.
We compare dataset imbalance to task imbalance using a similar distribution.
In the dataset imbalance condition, tasks (5-shot, 5-way) are sampled from a a linearly-imbalanced dataset D (cid:48) train (30,570,64,-).
In the task imbalance condition, tasks are sampled from a linearly-imbalanced distribution (1,9,5,-) – also called 1-9shot 5-way – and from a balanced dataset D (cid:48) train (300,300,64,-).
Finally, in the combined imbalance condition imbalanced tasks (1,9,5,-) are sampled from the imbalanced dataset D (cid:48) train (30,570,64,-).
Figure 1 shows the difference in D test accuracy between dataset imbalance (light red) and task imbalance (dark red) compared against the score of standard training ( balanced at both dataset and task levels).
Dataset imbalance causes an insigniﬁcant drop in performance, being mostly within error bars with respect to balanced .
On the other hand, the imbalanced task condition causes a signiﬁcant drop in performance, up to − 20% for some methods despite the slightly smaller imbalance magnitude ( ρ = 9 c.f. ρ = 19 ).
This result suggests that meta-learners are quite robust to dataset imbalance.
Interestingly, the combined task and dataset imbalance has a slight compounding effect on performance, yielding − 15% on average and performing worse than the combined sum of the two imbalance conditions taken individually ( − 12% ).
Simpleshot and MAML are the only methods where the difference between combined imbalance and task imbalance is not signiﬁcant, suggesting that they could be particularly resistant to this compound effect.
Robustness against domain-shift.
For the remaining sections, we kept the tasks balanced (5-shot 5-way), digging deeper into dataset imbalance.
In this paragraph, we examined various step imbalance distributions and evaluated performance with domain-shift.
Table 1 shows the meta-testing
[Table]
to balanced
[Table]
[Figure]
The results show a small negative difference ( − 3 . 1% ) between the balanced dataset and the step scenario with 32 minority classes ( step -32). Evaluation of models trained under the step -32 distribution on CUB, suggests only a small difference ( − 1 . 6 ) under the domain-shift condition.
Note that D (cid:48) train contains 22 animal classes, including 3 classes of birds, and this might not represent a very large domain-shift.
In Figureand Appendix C.1, we examine more step imbalance distributions on a smaller dataset containing only 1/8 th of total samples of D train from Mini-ImageNet.
Overall, the performance drops as the number of minority classes increases, but the effect still remains quite low ( < 3% absolute difference).
Robustness against larger domain-shift.
Next, we examined a stronger domain-shift scenario.
We set all 22 animal classes in D (cid:48) train to contain K D min = 25 samples each, and we set the otherclasses to have K D max = 444 samples each.
We call this setting “ step -animal” because it reduces the diversity of animal samples seen during meta-training.
As a control variable, we examine a similar step imbalance with 22 minority classes picked uniformly at random ( step -22).
Results in
Table 1 suggest that step -animal performs slightly worse ( 1 . 6% ) compared to step -22 on average.
SimpleShot and the two Baselines are particularly affected by the larger domain-shift on CUB, getting − 4 .
0% performance drop on step -animal compared to step -22.
Perhaps, this drop is due to the particular training procedure used for these methods, which are pre-trained on mini-batches instead of tasks.
This suggests an implicit strength of ML algorithms against larger domain-shift.
Robustness against larger imbalance.
In Appendix C.2, we examined long - tail distributions with an imbalance ratio ρ = 65 on a larger dataset ( 6 . 5 × larger than D (cid:48) train , and derived directly from ImageNet).
We observed a more signiﬁcant drop in accuracy w.r.t.
the balanced dataset for
ProtoNet and MAML ( − 6 .
3% and − 4 .
1% , respectively).
This suggests that a more natural imbalance distribution and a higher imbalance ratio could cause a larger performance drop.
However, the performance drop still remained smaller compared to task-level imbalance in Figure 1.
D ISCUSSION AND C ONCLUSION
In this work, we have provided insights into the meta-dataset imbalance problem in meta-learning, showing that models are quite robust to meta-dataset imbalance – meaning that they would likely experience only small drops in accuracy points when exposed to dataset imbalance in the real-world.
In contrast, the support-set imbalance yields a signiﬁcantly larger (an order of magnitude) performance drop.
Overall, our results suggest that dataset imbalance has a small negative effect on the ML procedure when tasks are balanced.
This seems to point to an implicit strength of ML algorithms, which can learn generalizable features even when exposed to imbalanced meta-datesets.
Higher levels of step and long-tail imbalance will likely produce more dramatic performance differences.
In this study, we evaluated generalization to novel tasks and classes only - however, it is likely that imbalance in D train would impact performance on base tasks and classes sampled from D train .
Moreover, in this study, we were constrained by the size of meta-training dataset of Mini-ImageNet and ImageNet as we had to control the size of the dataset.
In the real-world, meta-datasets can be very large with very high imbalance ratios ( ρ (cid:29) 100 ) and following long-tail distributions (Liu et al., 2019;
Salakhutdinov et al., 2011).
Future work should investigate how these larger imbalance levels could affect meta-learning and cross-domain generalization.
A CKNOWLEDGMENT
We want to thank Eleni Triantaﬁllou, Hae Beom Lee, Hayeon Lee, and the members of the Bayesian and Neural Systems group at the University of Edinburgh for valuable comments, suggestions, and discussions offered at various stages of this work.
This work was supported by the EPSRC Centre for Doctoral Training in Robotics and Autonomous Systems, funded by the UK Engineering and
Physical Sciences Research Council (Grant No. EP/S515061/1) and SeeByte Ltd, Edinburgh, UK.
A I MPLEMENTATION D ETAILS A. FSL M ETHODS AND B ASELINES
In our experiments we examined a wide range of meta-learning and few-shot learning models and baselines.
Hyper-parameters are located in our source code.
Baseline (ﬁne-tune) (Pan & Yang, 2010; Chen et al., 2019) represents a classical way of applying transfer learning, where a neural network is pre-trained on a large dataset, then ﬁne-tuned on a smaller domain-speciﬁc dataset.
The backbone of the Baseline has a single linear classiﬁcation layer with an output for each meta-training class.
The network was trained during pre-training.
During meta-testing, the pre-trained linear layer was exchanged for another randomly initialized layer with outputs matching the number of classes in the tasks ( N -way).
Fine-tuning was performed on the new randomly initialized classiﬁcation layer using the support set S . 2.
Baseline ++ (Chen et al., 2019) augments the ﬁne-tune baseline by using Cosine Similarity on the last layer.
Matching Network (Matching Net) (Vinyals et al., 2017) uses context embeddings with an LSTM to effectively perform k-nearest neighbor in embedding space using cosine similarity to classify the query set.
Prototypical Networks (ProtoNet) (Snell et al., 2017) maps images into a feature space and calculates class means (called prototypes).
The query samples are then classiﬁed based on the closest Euclidian distance to class prototypes.
We evaluate two models, the ﬁrst meta-trained like the others on 5-way episodes, and the second trained on 20-way episodes.
During 20-way meta-training, we set the query size to 5. 5.
Relation Networks (Relation Net) (Sung et al., 2017) augment the classical Prototypical Networks by introducing a relation module (another neural network) that compares the distance using a learnable score.
The original method uses Mean Squared Error to minimize the relation score between samples of the same type.
However, we follow Chen et al.
and use cross-entropy (CE) loss to improve performances.
The structure of the relation module is described in section A.2.
DKT (formally called GPShot) proposed by Patacchiola et al.
(2020) is a probabilistic approach that utilizes the Gaussian Processes with a deep neural network as a kernel function.
We used Batch Norm Cosine distance for the kernel type.
SimpleShot (Wang et al., 2019) augments the 1-NN baseline model by normalizing and centering the feature vector using the mean feature vector of the dataset.
The query samples are assigned to the nearest prototype according to the Euclidian distance.
In contrast to the baseline models, pre-training is performed on the meta-training dataset like other metalearning algorithms, and meta-validation is used to select the best model based on tasks sampled from D val .
MAML (Finn et al., 2017) is a meta-learning technique that learns a common initialization of weights that can be quickly adapted for various tasks using ﬁne-tuning on the support set.
The task adaptation uses a standard gradient descent algorithm minimizing CrossEntropy loss on the support set.
The original method uses second-order derivates; however, due to more efﬁcient calculation, we use the ﬁrst-order MAML, which has been shown to work just as well.
We set the inner-learning rate to 0.1 with 10 iteration steps based on our hyperparameter ﬁne-tuning.
We optimize the meta-learner model on batches ofmeta-training tasks.
ProtoMAML (Triantaﬁllou et al., 2020) augments traditional ﬁrst-order MAML by reinitializing the last classiﬁcation layer between tasks.
Speciﬁcally, the weights of the layer are assigned to the prototype for each corresponding output.
This extra step combines the ﬁne-tuning ability of MAML and the class regularization ability of Prototypical Networks.
We set the inner-loop learning rate to 0.01 with 5 iterations.
Unlike MAML, we found that updating the meta-learner after a single meta-training task gave the best performance. A.
B ACKBONE A RCHITECTURES
All methods shared the same backbone architecture.
For the core contribution of our work, we used
Conv4 architecture consisting of 4 convolutional layers with 64 channels (padding 1), interleaved by batch normalization (Ioffe & Szegedy, 2015), ReLU activation function, and max-pooling (kernel size 2, and stride 2) (Chen et al., 2019).
Relation Network used max-pooling only for the lastlayers of the backbone to account for the relation module.
The relation module consisted of two additional convolutional layers, each followed by batch norm, ReLU, and max-pooling.
A. D ATASETS AND D ISTRIBUTIONS
We meta-trained methods on a variety of meta-training datasets and distributions.
Speciﬁcally, we used Mini-ImageNet (Ravi & Larochelle, 2016; Vinyals et al., 2017),
CUB-2011 (Wah et al., 2011), and ImageNet Russakovsky et al.
Our main set of experiments controls imbalance of the meta-training set of Mini-ImageNet (Ravi & Larochelle, 2016)
( D train ).
To estimate how signiﬁcant dataset imbalance can be, we have to eliminate other factors that could inﬂuence the meta-learning performance.
For this reason, and to have enough samples for the majority classes, we halve the total number of samples from the original D train of Mini-ImageNet.
We denote this halved dataset as D (cid:48) train , where |D (cid:48) train | ≈ 64 × 300 = 19200 .
We induce imbalance into the dataset according to one of the
I -distributions as outlined in Section 3.
For simplicity, we do not modify the distribution in the meta-validation and meta-testing datasets ( D val and D test ), and we keep them the same as in the original Min-ImageNet.
To emulate a stronger domain-shift scenario, we also evaluate trained models on 50 randomly selected classes from CUB-2011 (Wah et al., 2011).
In Appendix C.1 we explore reducing the size of the meta-training dataset to contain |D (cid:48)(cid:48) train | ≈ 32 × 150 = 4800 .
In
Appendix C.2, we induce long-tail imbalance in ImageNet (Russakovsky et al., 2015).
More details can be found in the appropriate sections.
A. M ETA -T RAINING / P RE -T RAINING
All methods follow a similar three-phase learning procedure: meta-training, meta-validation, and meta-testing.
During meta-training, an FSL model was exposed to 50k tasks sampled from D train or imbalanced variants.
After every 500 tasks, the model was validated on tasks from D val and the best performing model was updated.
At the end of the meta-training phase, the best model was evaluated on tasks sampled from D test .
SimpleShot follows a similar three-phase procedure but with the meta-training phases exchanged for conventional pre-training on mini-batches of size 128.
In all three meta-phases, we used 15 query samples per class, except for the 20-way Prototypical
Network, where we used 5 query samples per class during meta-training to allow for a higher number of samples in the support set.
All methods were meta-validated on 200 tasks/mini-batches everymeta-training tasks/mini-batches to select the best performing model.
We used a learning rate of 1 × 10 − 3 for the ﬁrst 12.5k tasks, then reduced it to 1 × 10 − 4 for the remaining tasks.
Data Augmentation.
During the meta-/pre-training phases, we apply standard data augmentation techniques, following a similar setup to Chen et al.
(2019), with a random rotation of 10 degrees, scaling, random color/contrast/brightness jitter.
Meta-validation and meta-testing had no augmentation.
All images are resized to 84 by 84 pixels.
A. M ETA -T ESTING
The ﬁnal test performances were measured on a random sample of 600 tasks.
For all evaluations, we used standard, balanced 5-shot 5-way tasks.
We report the average one standard deviation in brackets and error-bars.
B V ERIFICATION OF I MPLEMENTATION
We implement the FSL methods in PyTorch, adapting the implementation of (Chen et al., 2019) but also borrowing from other implementations online (see individual method ﬁles in the source code for individual attribution).
However, we have heavily modiﬁed these implementations to ﬁt our imbalanced FSL framework, which also offers standard and continual FSL compatibility (Antoniou et al., 2020).
We provide our implementations for ProtoMAML for which no open-source implementation in PyTorch existed.
To verify our implementations, we compare methods on the standard balanced 5-shot 5-way task with reported accuracy.
Results are presented in Table 2.
We see that algorithms achieve very similar performance with no less than 3% accuracy points compared to the reported performance
[Table] [Table]
C A DDITIONAL E XPERIMENTS . C. S MALL AND I MBALANCED M ETA -T RAINING D ATASET
In this section, we examine whether the effect of imbalance could be inﬂuenced by a smaller dataset size.
Speciﬁcally, we construct a new set of datasets denoted by D (cid:48)(cid:48) train containing 1/8 th of samples in
D train of Mini-ImageNet, and 32 classes selected uniformly at random, |D (cid:48)(cid:48) train | = 4800 .
Tableare a break-down of Figure 2.
Overall, the performance drops as the number of minority classes increases, but the effect still remains quite low ( < 3% absolute difference)
[Table] [Table]
53.16 ± 0.
53.20 ± 0.
[Table]
C. L ONG -T AIL I MBALANCE
In this section, we explore the performance under a larger imbalance setting with long-tail distribution (Salakhutdinov et al., 2011).
We induce a long-tail distribution on classes from ImageNet (Russakovsky et al., 2015).
We used ResNet-10 as the backbone.
We partitioned ImageNet to contain 900 classes for D train distributed according to balanced or long - tail , while D val and D test contained 50 classes each with 500 randomly selected samples per class.
The set of classes in each dataset is kept the same for all experiment repeats.
We induce long - tail imbalance in D train using the Power-Law distribution (Reed, 2001) with a power of 10, a minimum of 20 samples per class (to allow 5-shot 15-query task), and a maximum of 1300 samples per class possible for ImageNet.
This means that ρ = 65 and the top 20% majority classes in distribution account for 80% of all data points in D train .
We shufﬂed the classes within D train such that the number of samples for a particular class could vary between repeats.
The Balanced dataset contained samples distributed uniformly among the 900 classes (i.e. 137 samples per class) such that |D train |
≈ 123300 for both distributions (within 500 samples difference between them).
Table 4 shows the accuracy performance after training on 1800 balanced tasks across 3 seeds.
The results show that meta-learner models –
ProtoNet, MAML, ProtoMAML– are particularly susceptible to the larger imbalance, experiencing -6.3%, -4.1%, -4.5% drop in accuracy, respectively.
Interestingly, the Baseline seems to be the least affected by the imbalance with a non-signiﬁcant drop in performance (-0.1%).
[Figure] [Table]