M ETA -L EARNING BY H ALLUCINATING U SEFUL E XAMPLES
Anonymous authors
Paper under double-blind review
A BSTRACT
Learning to hallucinate additional examples has recently been shown as a promising direction to address few-shot learning tasks, which aim to learn novel concepts from very few examples.
The hallucination process, however, is still far from generating effective samples for learning.
In this work, we investigate two important requirements for the hallucinator — (i) precision : the generated examples should lead to good classiﬁer performance, and (ii) collaboration : both the hallucinator and the classiﬁcation component need to be trained jointly.
By integrating these requirements as novel loss functions into a general meta-learning with hallucination framework, our model-agnostic PrecisE Collaborative hAlluciNator (PECAN) facilitates data hallucination to improve the performance of new classiﬁcation tasks.
Extensive experiments demonstrate state-of-the-art performance on competitive mini ImageNet and ImageNet based few-shot benchmarks in various scenarios.
I NTRODUCTION
Modern deep learning models rely heavily on large amounts of annotated examples (Deng et al., 2009).
Their data-hungry nature limits their applicability to real-world scenarios, where the cost of annotating examples is prohibitive, or they involve rare concepts (Zhu et al., 2014; Fink, 2011).
In contrast, humans can grasp a new concept rapidly and make meaningful generalizations, even from a single example (Schmidt, 2009).
To bridge this gap, there has been a recent resurgence of interest in few-shot learning that aims to learn novel concepts from very few labeled examples (Fei-Fei et al., 2006; Vinyals et al., 2016; Wang & Hebert, 2016;
Snell et al., 2017; Finn et al., 2017).
Existing work tries to solve this problem from the perspective of meta-learning (Thrun, 1998;
Schmidhuber, 1987), which is motivated by the human ability to leverage prior experiences when tackling a new task.
Unlike the standard machine learning paradigm, where a model is trained on a set of exemplars, meta-learning is performed on a set of tasks, each consisting of its own training and test sets (Vinyals et al., 2016).
By sampling small training and test sets from a large collection of labeled examples of base classes, meta-learning based few-shot classiﬁcation approaches learn to extract task-agnostic knowledge, and apply it to a new few-shot learning task of novel classes.
One notable type of task-agnostic (or meta) knowledge comes from the shared mechanism of data augmentation or hallucination across categories (Wang et al., 2018; Gao et al., 2018; Schwartz et al., 2018; Zhang et al., 2018a).
Hallucinating additional training data by generating images may seem like an easy solution for few-shot learning, but it is often challenging.
In fact, the success of this paradigm is usually restricted to certain domains like handwritten characters (Lake et al., 2013), or requires additional supervision (Dixit et al., 2017; Zhang et al., 2018b) or sophisticated heuristics (Hariharan & Girshick, 2017).
An alternative to generating raw data in the form of visually realistic images is to hallucinate examples in a learned feature space (Wang et al., 2018; Gao et al., 2018; Schwartz et al., 2018; Zhang et al., 2018a; Xian et al., 2019).
This can be achieved by, for example, integrating a “hallucinator” module into a meta-learning framework, where it generates hallucinated examples, guided by real examples (Wang et al., 2018).
The learner then uses an augmented training set which includes both the real and the hallucinated examples to learn classiﬁers.
While the existing approaches showed that it is possible to adjust the hallucinator to generate examples that are helpful for classiﬁcation, the generation process is still far from producing effective samples in the few-shot regime.
Our key insight is that, to facilitate data hallucination to improve the performance of new classiﬁcation tasks, two important requirements should be satisﬁed: (i) precision : the generated
[Figure]
examples should lead to good classiﬁer performance, and (ii) collaboration : all the components including the hallucinator and the learner need to be trained jointly.
In this work, we propose PrecisE Collaborative hAlluciNator (PECAN) , which integrates these requirements into a general meta-learning with hallucination framework, as shown in Figure 1.
Assume that we have a hallucinator to generate additional examples from the original small training set.
A precise hallucinator indicates that a classiﬁer trained on both the hallucinated and the few real examples should produce superior validation accuracy.
This can be achieved by training the hallucinator end-to-end with the learner, and back-propagating a classiﬁcation loss based on groundtruth labels of validation data (Wang et al., 2018).
Since this precision is measured using ground-truth labels, we term it as hard precision .
And more importantly, if the hallucinator perfectly captures the target distribution, a classiﬁer trained on a set of hallucinated examples, despite being generated from a small set of real examples, should produce roughly the same validation accuracy as a classiﬁer trained on a large set of real examples, when these two sets are of the same sample size (Shmelkov et al., 2018).
This indicates similar level of realism and diversity between the generated and the real examples, as shown in Figure 1a.
Motivated by this observation, we introduce an additional precision-inducing loss function, which explicitly encourages the hallucinator to generate examples so that a classiﬁer trained on them makes predictions similar to the one trained on a large amount of real examples.
Given that this precision is measured based on classiﬁer predictions, we term it as soft precision .
This precision, which is complementary to hard precision and effective, as shown in our experiment, is lacking in current approaches (Wang et al., 2018).
Satisfying the precision requirement alone is not sufﬁcient, since the classiﬁcation objective is still directly associated with the learner, and thus the hallucinator continues to rely on the back-propagated signal to update its parameters.
This leads to a potential undesirable effect of imbalanced training between the hallucinator and the learner: the learner tends to be stronger and makes allowances for errors in the hallucination, whereas the hallucinator becomes “lazy” and does not make its best effort to capture the data distributions, which is empirically observed in our experiments (See Figure 3).
To address this issue, our key insight is to enforce direct and early supervision for the hallucinator, and make its contribution to the overall classiﬁcation transparent, as shown in Figure 1b.
Hence, we introduce a collaborative objective for the hallucinator, which allows us to directly inﬂuence the generation process to favor highly discriminative examples right after hallucination , and to strengthen the cooperation between the hallucinator and the learner.
Our contributions are three-fold.
(1) We propose a novel loss that helps produce precise hallucinated examples, by using the classiﬁer trained on real examples as a guidance, and encouraging the classiﬁer trained on hallucinated examples to mimic its behavior.
(2) We introduce a collaborative objective for the hallucinator as early supervision, which directly facilitates the generation process and improves the cooperation between the hallucinator and the learner.
By integrating these properties, we develop a general meta-learning with hallucination framework, which is model-agnostic and can be combined with any meta-learning models to consistently boost their few-shot learning performance.
Here we mainly focus on few-shot classiﬁcation tasks, and we show that our approach applies to few-shot regression tasks as well in the appendix A.7.
R ELATED WORK
As one of the unsolved problems in machine learning and computer vision, few-shot learning is attracting growing interest in the deep learning era (Miller et al., 2000; Fei-Fei et al., 2006;
Lake et al., 2015; Santoro et al., 2016; Wang & Hebert, 2016; Vinyals et al., 2016; Snell et al., 2017; Finn et al., 2017; Hariharan & Girshick, 2017; George et al., 2017; Triantaﬁllou et al., 2017; Edwards &
Storkey, 2017; Mishra et al., 2018; Douze et al., 2018; Wang et al., 2018; Chen et al., 2019a; Dvornik et al., 2019).
Successful generalization from few training samples requires appropriate “inductive biases” or shared knowledge from related tasks (Baxter, 1997), which is commonly acquired through transfer learning and more recently meta-learning (Thrun, 1998; Schmidhuber, 1987; Schmidhuber et al., 1997; Bengio et al., 1992).
By explicitly “learning-to-learn” over a series of few-shot learning tasks (i.e., episodes), which are simulated from base classes, meta-learning exploits accumulated task-agnostic knowledge to target few-shot learning problems of novel classes.
Within this paradigm of approaches, various types of meta-knowledge has been recently explored, including (1) a generic feature embedding or metric space, in which images are easy to classify using a distance-based classiﬁer such as cosine similarity or nearest neighbor (Koch et al., 2015; Vinyals et al., 2016;
Snell et al., 2017; Sung et al., 2018; Ren et al., 2018; Oreshkin et al., 2018); (2) a common initialization of network parameters (Finn et al., 2017; Nichol & Schulman, 2018;
Finn et al., 2018) or learned update rules (Andrychowicz et al., 2016; Ravi & Larochelle, 2017; Munkhdalai & Yu, 2017; Li et al., 2017;
Rusu et al., 2019); (3) a transferable strategy to estimate model parameters based on few novel class examples (Bertinetto et al., 2016; Qiao et al., 2018;
Qi et al., 2018; Gidaris & Komodakis, 2018), or from an initial small dataset model (Wang & Hebert, 2016; Wang et al., 2017).
Complementary to these discriminative approaches, our work focuses on synthesizing samples to deal with data scarcity.
There has been progress in this direction of data hallucination, either in pixel or feature spaces (Salakhutdinov et al., 2012;
George et al., 2017; Lake et al., 2013; 2015;
Wong & Yuille, 2015; Rezende et al., 2014; Goodfellow et al., 2014; Radford et al., 2016; Dixit et al., 2017; Hariharan & Girshick, 2017; Wang et al., 2018; Gao et al., 2018; Schwartz et al., 2018;
Zhang et al., 2018a).
However, it is still challenging for modern generative models to capture the entirety of data distribution (Salimans et al., 2016) and produce useful examples that maximally boost the recognition performance (Wang et al., 2018), especially in the small sample-size regime.
In the context of generative adversarial networks (GANs), Shmelkov et al.
(2018) show that images synthesized by state-of-the-art approaches, despite their impressive visual quality, are insufﬁcient to tackle recognition tasks, and encourage the use of quantitative measures based on classiﬁcation results to evaluate GAN models.
Rather than using classiﬁcation results as a performance measure, we go a step further in this paper by leveraging classiﬁcation objectives to guide the generation process.
Other related work such as Wang et al.
(2018) proposed a general data hallucination framework based on meta-learning, which is a special case of our approach.
A GAN-like hallucinator takes a seed example and a random noise vector as input to generate a new sample.
This hallucinator is trained jointly with the classiﬁer in an end-to-end manner.
Delta-encoder (Schwartz et al., 2018) is a variant of Wang et al.
(2018), where instead of using noise vectors, it modiﬁes an auto-encoder to extract transferable intra-class deformations, i.e., “deltas”, and applies them to novel samples to generate new instances.
Unlike the above approaches that directly use the produced samples to train the classiﬁer,
MetaGAN (Zhang et al., 2018a) trains the classiﬁer in an adversarial manner to augment the classiﬁer with the ability to discriminate between real and synthesized data.
Another variant (Gao et al., 2018) explicitly preserves covariance information to enable better augmentation.
Our work investigates critical yet unexplored properties in this paradigm that the data hallucinator should satisfy.
These properties are general and can be ﬂexibly incorporated into existing meta-learning approaches and hallucination methods, providing signiﬁcant gains irrespective of these choices.
M ETA - LEARNING WITH HALLUCINATION
We begin by presenting the general meta-learning mechanism (Vinyals et al., 2016;
Snell et al., 2017; Finn et al., 2017) and our meta-learning with hallucination framework for the task of Noise " # $%&'( ℎ %*&+ ℎ , - , ℎ .+/ # $%&'(∗ # $%&'(, # $%&'(& # $*/$
Sample
S a m p l e - %
*&+ - ℓ 4%* ( ̂- %*&+ , ̂- 8 ) ℓ .+/ (-,:) few-shot image classiﬁcation.
Let I be the space of images.
We are given two disjoint sets of classes: a base class set C base and an unseen novel class set C novel .
The corresponding base dataset
D base = { ( I i , y i ) , I i ∈ I , y i ∈ C base } contains a large number of labeled examples per class, while the novel dataset D novel =
{ ( I i , y i ) , I i ∈ I ,
y i ∈ C novel } consists of only a small number n of labeled examples per class.
The goal is to learn a classiﬁer h cls θ h parametrized by θ h on D base that can cross-generalize (Bart & Ullman, 2005) to C novel even when n is as few as one.
Meta-learning aims to achieve such generalization through episodic meta-training that explicitly mimics the few-shot learning scenario on D base (Vinyals et al., 2016).
Speciﬁcally, in each episode of the meta-training stage, the meta-learner simulates a few-shot classiﬁcation task out of D base .
This task is constructed by ﬁrst randomly sampling a subset of m classes from C base , and then randomly sampling a small “training” set S train (also called the support set) and a small “test” set S test (also called the query set).
The learner, i.e., the classiﬁer h cls θ h , outputs estimated conditional probabilities p for each example ( x, y ) in S test based on S train .
That is, p ( x ) =
h cls θ h ( x, S train ) .
The meta-learner back-propagates the gradient of the total classiﬁcation loss (cid:96) cls =
(cid:80) ( x,y ) ∈ S test loss( h cls θ h ( x, S train ) , y ) in S test to update the learner parameters θ h .
During the meta-testing stage, the resulting h cls θ h is used to address the few-shot classiﬁcation task on D novel , which predicts class probabilities of unlabeled test examples conditioned on the given small labeled training set S train of C novel .
Our meta-learning with hallucination framework introduces an additional “hallucinator” module
G θ G with parameters θ G to augment the small training set S train .
To facilitate training, we follow recent work (Hariharan & Girshick, 2017; Wang et al., 2018) and ﬁrst pre-train a deep convolutional network on D base using a standard cross-entropy loss.
We use it to extract the feature representation x ∈ X for an input image I .
Meta-learning is then performed over the pre-trained features { x i } .
As shown in the shaded region in Figure 2, given an initial S train , the hallucinator G θ G generates additional examples for each class.
Our framework applies to various types of hallucinators, and here we consider a powerful GAN-like hallucinator in Wang et al.
Each hallucinated example is of the form ( G θ G ( x, z ) , y ) , where ( x, y ) is a sampled seed example from S train , and z is a sampled noise vector.
The set of generated examples S G train is added to S train to create an augmented training set S augtrain .
In the next section, we show how to meta-train G θ G on C base , so that it can hallucinate new examples to augment S train of C novel during meta-testing.
P RECISE COLLABORATIVE HALLUCINATOR
We now present our PrecisE Collaborative hAlluciNator (PECAN) shown in Figure 2, which exploits two important criteria for useful hallucination: precision and collaboration .
As important constraints and guidance, these criteria facilitate hallucination to improve the classiﬁcation performance.
Basic hallucinator with hard precision.
At ﬁrst, a precise hallucinator indicates that a classiﬁer trained on S augtrain should produce superior validation accuracy.
We achieve this by training the hallucinator end-to-end with the learner (Wang et al., 2018).
As shown in the shaded region in Figure 2, during each episode of meta-training, the learner module h cls θ h uses S augtrain to produce conditional probabilities h cls θ h ( x, S augtrain ) for each example ( x, y ) in the test set S test .
The meta-learner then backpropagates the gradient of the total classiﬁcation loss (cid:96) cls =
(cid:80) ( x,y ) ∈ S test loss( h cls θ h ( x, S augtrain ) , y ) to update both the learner parameters θ h and the hallucinator parameters θ G .
Soft precision-inducing hallucinator.
One of the important characteristics of an optimal generative model is that the generated examples should be indistinguishable from real ones (Goodfellow et al., 2014).
We argue that, in terms of our recognition task oriented hallucinator, this means that the classiﬁer trained on hallucinated examples needs to be similar to the classiﬁer trained on real examples.
As shown in Figure 2, given an initial relatively large training set S ∗ train , which contains n ∗ examples for each of the m classes, we randomly sample n ( n (cid:28) n ∗ ) examples per class, and obtain a subset
S train .
From S train , the hallucinator G θ G generates n ∗ examples per class as S G train .
This produces two training sets: S ∗ train with real examples and S G train with hallucinated examples, where both contain the same number of examples.
Importantly , note that S G train is hallucinated from the subset
S train instead of the initial large set S ∗ train , and because n (cid:28) n ∗ , we rule out the trivial identity hallucinator or memorization .
We train two additional classiﬁcation networks: h real based on S ∗ train and h G based on S G train , both of which have the same architecture as h cls .
When evaluated on the same test set S test composed of real examples, a comparable performance between h real and h G shows that the hallucinated samples are sufﬁciently precise, and as diverse as the real training set.
Otherwise, when the hallucinator is imperfect, the accuracy of h G will be lower than that of h real .
This similarity of classiﬁcation accuracy essentially measures the difference between the learned (i.e., hallucinated) and the target (i.e., real) distributions, which could serve as an additional supervisory signal for training a better hallucinator.
Since quantifying the similarity of accuracy directly would be difﬁcult (Hinton et al., 2015), we instead introduce a loss function that acts on the network predictions.
For an example ( x, y ) in S test , the two networks produce conditional probabilities
[Equation]
respectively.
While only the largest entry in p real ( x ) or p G ( x ) is used to make predictions associated with the ground-truth label y , other entries still carry rich information about the recognition task and the network, as observed in (Hinton et al., 2015; Dvornik et al., 2019).
We thus leverage the probabilities (cid:98) p real and (cid:98) p G in the absence of the ground-truth label and measure their similarity using the negative cosine distance:
[Equation]
where (cid:98) p real and (cid:98) p G are obtained by removing the logit for y in p real and p G , and re-normalizing the remaining logits using softmax with a learnable temperature.
We treat the classiﬁcation networks h cls , h real , and h G as the new learner h and use shared parameters for them.
Their difference thus lies in different conditional training sets.
We obtain the soft precision-inducing loss (cid:96) pre by summing the loss (2) in S test and then combine it with the hard precision (i.e., the classiﬁcation) loss as the classiﬁcation objective .
p G is now encouraged to not only make the right prediction according to the ground-truth label, but also make similar second-best, third-best, etc., choice predictions as p real .
Collaboration between hallucinator and learner.
We now consider the interaction between the hallucinator G and the learner h .
While hallucination is conducted in the pre-trained feature space X , the ﬁnal classiﬁcation is performed in a new embedding space Φ learned by the learner.
Since the classiﬁcation objective is directly imposed on the learner h , the hallucinator G continues to rely on the back-propagated signal to update its parameters.
We may end up with a good embedding space Φ but a poor hallucinator G in the original space X .
This undesired effect implies a potential imbalance between the hallucinator and the learner — a stronger learner that is able to make allowances for errors in hallucination, but a “lazy” hallucinator that does not make its best effort to capture the data distributions.
Indeed, as is empirically validated in the experimental section (see Figure 3), despite being able to match the class distributions in the embedding space Φ , the hallucinated examples are initially pulled away from the class distributions in the feature space X .
To mitigate this issue, we introduce a simple collaborative objective to the hallucinator, which provides an additional constraint or regularization on the hallucination process.
This collaborative objective is the same as the above classiﬁcation objective (i.e., a combination of the classiﬁcation loss and the precision-inducing loss), but enforces direct and early supervision for the hallucinator in the pre-trained feature space X .
By doing so, we directly inﬂuence the update process of the hallucinator parameters, and generate much more discriminative examples right after hallucination than would be the case if we had to rely on gradual back-propagation from the learner alone.
Our objective thus strengthens the cooperation between the hallucinator and the learner for the ﬁnal classiﬁcation performance, which can be viewed as a source of deep supervision that introduces auxiliary losses to intermediate layers when training deep neural networks (Simonyan & Zisserman, 2015; Lee et al., 2015).
The overall objective combines the classiﬁcation objective L learner (on the learner) and the collaborative objective L hal (on the hallucinator), each of which consists of a classiﬁcation loss (cid:96) cls (hard precision as cross-entropy with respect to ground-truth) and a soft precision-inducing loss (cid:96) pre :
[Equation]
where λ , λ 1 , λ 2 , and λ 3 are scalar hyper-parameters.
Our hallucinator is general and applies to different types of h (i.e., meta-learning algorithms).
Here we focus on the widely used and powerful prototypical networks (PN)
(Snell et al., 2017), prototype matching networks (PMN) (Vinyals et al., 2016; Wang et al., 2018), and cosine classiﬁers (Cos) (Gidaris & Komodakis, 2018; Chen et al., 2019a).
Without loss of generality, we take PN as an example to explain the overall meta-training and meta-testing process.
PN learns an embedding space Φ and uses a non-parametric nearest centroid classiﬁer to assign class probabilities for a test example based on its distances from class means in Φ .
As before, in each meta-training episode, after sampling S ∗ train , S train , and S test and hallucinating S G train in the pre-trained feature space X , we perform nearest centroid classiﬁcation and produce the collaborative objective L hal on S test .
We then feed the examples to the PN learner, obtain their embedded features in Φ , perform nearest centroid classiﬁcation, and produce the classiﬁcation objective L learner on S test .
The ﬁnal loss is back-propagated to update both the PN learner parameters θ h and the hallucinator parameters θ G .
Figure 2 shows a schematic of the entire process.
During meta-testing, we use the resulting
G θ G to hallucinate new examples to augment S train of C novel , and we combine the predicted class probabilities in X and Φ as the ﬁnal predictions.
E XPERIMENTAL EVALUATION
We explore the use of our meta-learning with hallucination framework for few-shot visual classiﬁcation tasks.
We focus the evaluation on the ImageNet based few-shot benchmark (Hariharan &
Girshick, 2017; Wang et al., 2018).
This is one of the largest datasets by far used for few-shot classiﬁcation and it captures more realistic scenarios than others based on handwritten characters (Lake et al., 2015) or low-resolution images (Vinyals et al., 2016).
The benchmark divides the 1 , 000 ImageNet categories (Russakovsky et al., 2015) into 389 base classes C base , with thousands of training images per class, and 611 novel classes C novel , with a small number n of training images per class.
Following Hariharan & Girshick (2017), we use C base to train a convolutional network (ConvNet) based feature extractor and to conduct meta-training.
Meta-testing is performed on C novel , and the performance is evaluated on a held-out test set, i.e., the original validation set of ImageNet.
In addition, to avoid over-ﬁtting, both C base and C novel are further split into two disjoint subsets.
193 of the base classes C cvbase and 300 of the novel classes C cvnovel are used for cross-validating hyper-parameters, and the remaining 196 base classes C ﬁnbase and 311 novel classes C ﬁnnovel are used for the ﬁnal evaluation.
Here we focus on hallucinating novel instances and thus evaluate the performance primarily on the novel classes C ﬁnnovel , which is also consistent with most of the contemporary work (Vinyals et al., 2016;
Snell et al., 2017; Finn et al., 2017).
We report the mean top-1 and top-5 accuracies for 311 -way, n = 1 , 2 , 5 , 10 , 20 -shot classiﬁcation, with each of them averaged over 5 trials.
In addition to this challenging version of ImageNet, we also evaluate on the widely used mini ImageNet (Vinyals et al., 2016) dataset to show the generality of our approach.
mini ImageNet is a subset of 100 classes selected randomly from ImageNet with 600 images sampled from each class.
Following the data split in Ravi & Larochelle (2017), we use 64 base, 16 validation, and 20 novel classes.
We evaluate in the standard 5 -way, 1 -shot and 5 -way, 5 -shot settings (Vinyals et al., 2016). [Table]
different meta-learning models to improve their performance.
[Table]
5. R ESULTS ON I MAGE N ET
Implementation details.
We mainly use a ResNet-10 architecture (He et al., 2016) as the feature extractor, following Hariharan & Girshick (2017);
Wang et al.
Additionally, we provide results using a deeper ResNet-50 architecture in Section A.3.
We extract and record the features and perform meta- learning by using these pre
-computed features.
We consider three widely-used powerful meta -learning approaches: prototypical networks PN ) (Snell et al., 2017), prototype matching networks (PMN
(Vinyals et al Wang et al
, and cosine classiﬁers (Cos-Cls) used in Gidaris Komodakis
Chen et al. 2019a
More implementation details are included in Section A.1.
Baselines.
First we compare with the state-of-the-art meta-learning with hallucination method (Wang et al., 2018), which is a special case of our approach learned with only the hard precision loss.
While Wang et al.
(2018) focused on ‘PN w/ G’ and ‘PMN w/ G’, here we consider an additional type of classier with hallucination, ‘Cos-Cls w/ G’, to show the generality of our work.
In addition, we compare with a variety of baselines, including (1) these meta-learning approaches with standard data augmentation techniques (Chen et al., 2019a); (2) data hallucination approaches which are not meta-learned: logistic regression with analogies hallucination (Hariharan & Girshick, 2017) and
Gaussian hallucinator (Wang et al., 2018); (3) other recent meta-learning approaches: matching networks (MN) (Vinyals et al., 2016), model-agnostic meta-learning (MAML) (Finn et al., 2017), and ‘cosine classiﬁer & attentive weight generators (Cos & Att)’
(Gidaris & Komodakis, 2018); (4) classical few-shot learning approaches: Siamese networks (SN) (Koch et al., 2015); and (5) simple baselines which are not meta-learned: logistic regression (Hariharan & Girshick, 2017).
For fair comparison, all these baselines and our approach use the same pre-trained ConvNet backbone.
Comparisons with the state of the art.
Table 1 shows that our PECAN consistently outperforms all the baselines by large margins across different scenarios .
For this challenging 311-way classiﬁcation, our improvements are of the order of 1% to 2% , while standard deviations for accuracy are of the order of 0 .
For example, in the case of top-5 accuracy, our ‘PN w/ G + PECAN’ outperforms ‘PN w/ G’ by 1.5 points for n = 1 and 1.6 points for n = 20 .
Similar trends can be observed for ‘PMN w/
G + PECAN’ and ‘Cos-Cls w/ G + PECAN’, and also in the top-1 accuracy regime.
This indicates that our approach is general and can work with different meta-learners.
Ablation studies.
We conduct a series of ablations to evaluate the contribution of each component and different design choices.
We use the prototypical network (PN) here due to its fast training speed.
[Table] [Table]
Variants of PECAN.
PECAN leverages two requirements for the meta-learned hallucinator: precision and collaboration.
‘ (cid:96) clslearner ’ is the basic hallucinator with only the hard precision based on the classiﬁcation loss.
Table 2 shows that each requirement by itself yields performance superior to the basic hallucinator.
The soft precision-inducing loss (cid:96) pre consistently helps when combined with the hard precision (cid:96) cls : ‘ (cid:96) clslearner + (cid:96) prelearner ’ outperforms ‘ (cid:96) clslearner ’ and ‘ (cid:96) clshal + (cid:96) prehal ’ outperforms ‘ (cid:96) clshal ’.
The collaboration objective integrates (cid:96) learner and (cid:96) hal to boost the performance: ‘ (cid:96) clslearner + (cid:96) clshal ’ outperforms ‘ (cid:96) clslearner ’.
Each component is thus essential and complementary to each other, enabling our full PECAN to outperform its variants.
Choice of similarity measure in soft precision-inducing loss.
Our precision-inducing loss measures the similarity between classiﬁer predictions p real and p G .
We used negative cosine distance between the probabilities (cid:98) p real and (cid:98) p G in the absence of ground-truth labels.
Table 3 compares with other types of similarity: variant of negative cosine distance, cross-entropy as in knowledge distillation (Hinton et al., 2015)
et al., 2019)
Impact of collaborative objective.
Our collaborative objective introduces additional direct and early supervision to train the hallucinator.
Table 2 shows quantitatively its contribution to the overall accuracy.
Here, we further qualitatively understand its impact though t-SNE visualizations (van der
Maaten & Hinton, 2008) of the hallucinated examples for novel classes.
For ease of analysis, we do not use the precision-inducing loss.
Without the collaborative objective, despite being able to match the class distributions in the embedding space Φ (Figure 3b), the hallucinated examples are initially pulled away from the class distributions in the pre-trained feature space X (Figure 3a), indicating a “lazy” hallucinator.
In contrast, the collaborative objective enforces the hallucinator to generate more discriminative examples right after hallucination (Figure 3c), leading to improved performance.
Qualitative visualizations.
To better understand the hallucination process, Figure 4 shows some examples of classiﬁcation results for our PECAN and the state-of-the-art baseline (Wang et al., 2018).
[Figure]
5. R ESULTS ON mini I MAGE N ET
To show the generality of our approach, we further evaluate on mini ImageNet.
We use a ResNetarchitecture and focus on incorporating our hallucinator into the metric-learning-based meta-learning approach, prototype matching networks (PMN) (Wang et al., 2018), and the optimization-based meta-learning approach, MAML (Finn et al., 2017). For MAMl, in each meta-training episode, we sample a batch of few-shot classiﬁcation tasks. For each of the tasks, we sample training sets S ∗ train and S train , sample a test test S test , hallucinate S G train , and obtain an augmented training set S augtrain . In the MAML inner loop, for each task, conditioning on its S augtrain ( S G train , or S ∗ train ), we adapt the parameters of h cls ( h G , or h real ) using few gradient updates.
For each task, the adapted h cls ( h G , or h real ) is evaluated on the corresponding S test . In the MAML outer loop, we average the classiﬁcation objective on S test across the batch of tasks. In a similar way, we compute the collaborative objective in the pre-trained feature space. The ﬁnal loss is used to update the initial MAML model and the hallucinator.
From Table 4, our PECAN signiﬁcantly outperforms all these state-of-the-art competitors, including other hallucination based approaches such as MetaGAN (Zhang et al., 2018a), delta-encoder (Schwartz et al., 2018), IDeMe-Net (Chen et al., 2019b), and SalNet (Zhang et al., 2019).
Our superior performance over MetaGAN, a GAN-based approach to hallucinate data, shows that directly matching the classiﬁcation performance is more desirable than matching the data distribution between hallucinated and real examples for recognition tasks.
Our generic framework can be combined with more recent meta-learning methods, such as LEO (Rusu et al., 2019) and MetaOptNet-SVM (Lee et al., 2019), for further improvement.
C ONCLUSION
We have presented an approach to few-shot classiﬁcation that uses a precise collaborative hallucinator to generate additional examples.
Our hallucinator integrates two important requirements that facilitate data hallucination in a way that most improves the classiﬁcation performance, and is trained end-toend through meta-learning.
The extensive experiments demonstrate our state-of-the-art performance on the challenging ImageNet and mini ImageNet based few-shot benchmark in various scenarios.
A A PPENDIX A. A DDITIONAL IMPLEMENTATION DETAILS
The embedding architecture of PN is composed of two-layer MLPs with leaky ReLU nonlinearity of slope 0 .
and a Euclidean distance similar to Snell et al.
The embedding architecture of
PMN is composed of a one-layer bi-directional LSTM and attention LSTM and a cosine distance as in (Vinyals et al., 2016;
Wang et al., 2018).
The embedding architecture of Cos-Cls is composed of two-layer MLPs with leaky ReLU nonlinearity of slope 0.01 and an additional one-layer MLP without nonlinearity and a cosine distance with a learnable temperature similar to Gidaris & Komodakis (2018).
The initial value of the temperature is 100.
The classiﬁcation weight vector is estimated by averaging the feature vectors of the training examples for each class.
The hallucinator G is a three-layer MLP with leaky ReLU nonlinearity of slope 0 .
01 , with its parameters initialized to block diagonal identity matrices (Wang et al., 2018).
The dimensionality of the hidden layers is 512 for
ResNet-10 features and 2 , 048 for ResNet-50 features
[Figure]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Table] Both the baselines (PN/PMN/Cos-Cls and PN/PMN/Cos-Cls with hallucination) and our approach are meta-trained on ImageNet for 60 , 000 episodes by SGD with an initial learning rate 0 .
for
PN/PMN and 0 .
005 for Cos-Cls, decayed by a factor of 10 every 20 , 000 episodes.
In each episode of the meta-training stage, we sample all the base classes following Wang et al.
(2018), which found that it is advantageous to use more classes rather than fewer.
We sample n ∗ =20 examples per class from the base dataset D base , leading to an initial training set S ∗ train .
Consistent with Wang et al.
(2018), to make a single hallucinator robust to different sample sizes, we randomly sample different sized examples per class from S ∗ train , from 1 to 15 examples per class, to obtain S train .
One random seed example is sampled from S train , and fed into the hallucinator G with different noise vectors to generate 20 examples per class as S G train .
Hallucinated examples are sampled from S G train and added to S train until there are exactly 20 examples per class in S augtrain .
For the test set S test , we have 5 random examples per class for prototypical networks (PN)
(Snell et al., 2017) and cosine classiﬁers(Cos-Cls) (Gidaris & Komodakis, 2018), and 1 random example per class for prototype matching networks (PMN) (Vinyals et al., 2016; Wang et al., 2018).
For the soft precision-inducing loss, we use softmax with temperature to produce conditional class probabilities (cid:98) p
real and (cid:98) p G .
This temperature is a learnable parameter.
It is shared between (cid:98) p real and (cid:98) p G , but is not shared between L learner and L hal . [Figure]
The hyper-parameters obtained by cross-validation on ImageNet are: λ 1 =0.5, λ 2 =0.02, and λ 3 =0.1 for
PN with ResNet-10; λ 1 =0.2, λ 2 =0.0001, and λ 3 =0.05 for PN with ResNet-50; λ 1 =0.4, λ 2 =2.0, and λ 3 =0.1 for PMN with ResNet-10; and λ 1 =0.5, λ 2 =0.000001, and λ 3 =0.005 for the cosine classiﬁer with ResNet-10.
During the meta-testing stage, following Wang et al.
we use n =1, 2, 5, 10 or 20 examples per class from the novel dataset D novel , and then hallucinate a ﬁxed number of additional examples for each novel class.
By cross-validation, the number of hallucinated examples per class is set to 10 for
PN, Cos-Cls and Cos-Cls w/ G + PECAN with ResNet-10, 8 for Cos-Cls w/ G with ResNet-10, 5 for
PN with ResNet-50, and 20 for PMN with ResNet-10.
We combine the classiﬁer prediction results in the pre-trained feature space and the learned embedding space using a scalar hyper-parameter.
By cross-validation, this hyper-parameter is set to 0.05 for PN with ResNet-10, 0.07 for PN with
ResNet-50, 1 for PMN with ResNet-10, and 0.00002 for the cosine classiﬁer with ResNet-10.
For the test set S test , following Wang et al.
we have 50 real examples per class, and we average the top-1 or top-5 accuracy of them over the novel classes.
A. P ERFORMANCE ON BASE CLASSES
While we signiﬁcantly improve the classiﬁcation performance on novel classes, our approach remains accurate on base classes.
For example, for prototypical networks (PN)
(Snell et al., 2017), both our ‘PN w/ G + PECAN’ and the baseline ‘PN w/ G’ achieve the same top-5 accuracy 92 .
4% on
ImageNet.
A. I MPACT OF DEEPER REPRESENTATION MODELS
Figure A.1 shows the results on ImageNet using features from a ResNet-50 architecture.
As expected, deeper networks result in better performance for all the approaches, but our PECAN hallucination strategy still provides large gains across the board over the state-of-the-art meta-learned hallucinator in (Wang et al., 2018).
A. A NALYSIS OF HYPER - PARAMETER SENSITIVITY
Hyper-parameters in the overall objective.
We conduct sensitivity experiments for the hyperparameters λ 1 , λ 2 , and λ 3 , which trade off different loss components in the overall objective of our PECAN.
We vary one of the three hyper-parameters while ﬁxing the remaining two to their cross-validated values.
Figures A.2a, A.2b, and A.2c show the top-5 accuracy of ‘PN w/ G + PECAN’ on the novel classes for the ImageNet based n -shot classiﬁcation benchmark.
We can see that the top-5 accuracy is stable over a wide range of hyper-parameter values, for example when the value of λ 1 becomes 50 times larger or 100 times smaller than λ 1 used in the main paper.
Across the board, our PECAN consistently and signiﬁcantly outperforms the baselines shown in the main paper.
Number of hallucinated examples.
We also show how the top-5 accuracy changes for n = 1 -shot classiﬁcation with respect to the number of hallucinated images in Figure A.2d.
We can see that when the number of hallucinated examples is changed from 0 to 10, the performance of our PECAN gradually improves, and then saturates and drops slightly with more than 10 images generated.
A. A DDITIONAL ANALYSIS OF SOFT PRECISION - INDUCING LOSS
Our soft precision-inducing loss measures the similarity between classiﬁer predictions p real and p G .
This is a general similarity measure which applies to various types of classiﬁers, including parametric and non-parametric classiﬁers.
For prototypical networks (PN) (Snell et al., 2017), a non-parametric nearest centroid classiﬁer is used to assign class probabilities for a test example based on its distances from class means.
Hence, in this special case, to measure the similarity between classiﬁer predictions, we can directly calculate the distance between the mean of real examples m real and the mean of hallucinated examples m G .
Table A.1 compares our generic similarity with this speciﬁc similarity for PN on the ImageNet based n -shot classiﬁcation benchmark.
The result shows that our similarity generalizes signiﬁcantly better for novel classes.
A. A DDITIONAL VISUALIZATIONS OF CLASSIFICATION RESULTS
Similar to Figure 5 in the main paper, here we provide more examples of classiﬁcation results for our
PECAN and the state-of-the-art meta-learned hallucinator (Wang et al., 2018) in Figure A.3.
A. E XTENSION TO FEW - SHOT REGRESSION
In the main paper, we focus on few-shot classiﬁcation tasks.
However, our approach is general and applies to few-shot regression tasks as well.
When extending our approach to address regression tasks, we need to slightly modify the design of the hallucinator G .
Speciﬁcally, in classiﬁcation tasks, the hallucinator G takes as input a seed example ( x, y ) and outputs a hallucinated example ( x (cid:48) , y (cid:48) )
=
( G ( x, z ) , y ) , where the class label y (cid:48)
=
y , indicating that the hallucinated example belongs to the same class as the seed example.
In contrast, in regression tasks, y becomes a continuous quantity.
We thus cannot enforce the constraint y (cid:48)
=
y .
To address this issue, we modify the hallucinator G to directly generate the tuple ( x (cid:48) , y (cid:48) )
=
G ( x, y, z ) .
This is achieved by the concatenation of x and y as input to G .
We incorporate our hallucinator with MAML (Finn et al., 2017) and evaluate on the n = 5 -shot sinusoidal regression task proposed in Finn et al.
Each task regresses from the input to the output of a sine curve, and different tasks differ in the amplitude and phase of the sinusoid.
The amplitude and the phase are uniformly distributed within [0 . 1 , 5 . 0] and [0 , π ] , respectively.
During meta-training and meta-testing, the input datapoints x of each task are uniformly sampled from [ − 5 . 0 , 5 . 0] .
The prediction loss is measured by the mean squared error (MSE) between the prediction and true value.
The regressor is a feedforward neural network with 2 hidden layers of size 40 with
ReLU nonlinearity.
We use MAML (Finn et al., 2017) with one gradient update based on n =examples with a ﬁxed step size α = 0 .
01 , and use Adam as the meta-optimizer (Kingma & Welling, 2014).
[Table]
We compare with two baselines: (1) standard MAML
[Table]
and (2) ‘MAML w/ G’, which is MAML with the hallucinator in (Wang et al., 2018).
While Wang et al.
(2018) focus on classiﬁcation tasks, here we extend the use of its hallucinator in the similar way as we discussed before.
Table A.2 shows that our PECAN consistently outperforms the baselines for the regression task as well, indicating the generality of our approach.
A. V ISUALIZATIONS OF HALLUCINATED EXAMPLES
Our hallucination is performed in the pre-trained feature space, and visualizing them directly is not intuitive.
In addition to t-SNE visualization of hallucinated samples in Figure 3, here in Figure A. we include an additional visualization of hallucinated examples in the pixel space, using the nearest neighbor real image in the feature space, corresponding to each hallucinated feature sample.