  Few-shot learning aims to train models on a limited number of labeled samples
given in a support set in order to generalize to unseen samples from a query
set. In the standard setup, the support set contains an equal amount of data
points for each class. However, this assumption overlooks many practical
considerations arising from the dynamic nature of the real world, such as
class-imbalance. In this paper, we present a detailed study of few-shot
class-imbalance along three axes: meta-dataset vs. task imbalance, effect of
different imbalance distributions (linear, step, random), and effect of
rebalancing techniques. We extensively compare over 10 state-of-the-art
few-shot learning and meta-learning methods using unbalanced tasks and
meta-datasets. Our analysis using Mini-ImageNet reveals that 1) compared to the
balanced task, the performances on class-imbalance tasks counterparts always
drop, by up to $18.0\%$ for optimization-based methods, and up to $8.4$ for
metric-based methods, 2) contrary to popular belief, meta-learning algorithms,
such as MAML, do not automatically learn to balance by being exposed to
imbalanced tasks during (meta-)training time, 3) strategies used to mitigate
imbalance in supervised learning, such as oversampling, can offer a stronger
solution to the class imbalance problem, 4) the effect of imbalance at the
meta-dataset level is less significant than the effect at the task level with
similar imbalance magnitude. The code to reproduce the experiments is released
under an open-source license.
