M ETA -L EARNING FOR S EMI -S UPERVISED F EW -S HOT C LASSIFICATION
Anonymous authors
Paper under double-blind review
A BSTRACT
In few-shot classiﬁcation, we are interested in learning algorithms that train a classiﬁer from only a handful of labeled examples.
Recent progress in few-shot classiﬁcation has featured meta-learning, in which a parameterized model for a learning algorithm is deﬁned and trained on episodes representing different classiﬁcation problems, each with a small labeled training set and its corresponding test set.
In this work, we advance this few-shot classiﬁcation paradigm towards a scenario where unlabeled examples are also available within each episode.
We consider two situations: one where all unlabeled examples are assumed to belong to the same set of classes as the labeled examples of the episode, as well as the more challenging situation where examples from other distractor classes are also provided.
To address this paradigm, we propose novel extensions of Prototypical
Networks (Snell et al., 2017) that are augmented with the ability to use unlabeled examples when producing prototypes.
These models are trained in an end-to-end way on episodes, to learn to leverage the unlabeled examples successfully.
We evaluate these methods on versions of the Omniglot and mini ImageNet benchmarks, adapted to this new framework augmented with unlabeled examples.
We also propose a new split of ImageNet, consisting of a large set of classes, with a hierarchical structure.
Our experiments conﬁrm that our Prototypical Networks can learn to improve their predictions due to unlabeled examples, much like a semi-supervised algorithm would.
I NTRODUCTION
The availability of large quantities of labeled data has enabled deep learning methods to achieve impressive breakthroughs in several tasks related to artiﬁcial intelligence, such as speech recognition, object recognition and machine translation.
However, current deep learning approaches struggle in tackling problems for which labeled data are scarce.
Speciﬁcally, while current methods excel at tackling a single problem with lots of labeled data, methods that can simultaneously solve a large variety of problems that each have only a few labels are lacking.
Humans on the other hand are readily able to rapidly learn new classes, such as new types of fruit when we visit a tropical country.
This signiﬁcant gap between human and machine learning provides fertile ground for deep learning developments.
For this reason, recently there has been an increasing body of work on few-shot learning, which considers the design of learning algorithms that speciﬁcally allow for better generalization on problems with small labeled training sets.
Here we focus on the case of few-shot classiﬁcation, where the given classiﬁcation problem is assumed to contain only a handful of labeled examples per class.
One approach to few-shot learning follows a form of meta-learning 1 (Thrun, 1998; Hochreiter et al., 2001), which performs transfer learning from a pool of various classiﬁcation problems generated from large quantities of available labeled data, to new classiﬁcation problems from classes unseen at training time.
Meta-learning may take the form of learning a shared metric (Vinyals et al., 2016;
Snell et al., 2017), a common initialization for few-shot classiﬁers (Ravi & Larochelle, 2017;
Finn et al., 2017) or a generic inference network (Santoro et al., 2016;
Mishra et al., 2017).
[Figure]
These various meta-learning formulations have led to signiﬁcant progress recently in few-shot classiﬁcation.
However, this progress has been in a limited scenario, which differs in many dimensions from how humans learn new concepts.
In this paper we aim to generalize the few-shot setting in two ways.
First we consider a scenario in which the new classes are learned in the presence of additional unlabeled data.
While there has been many successful applications of semi-supervised learning to the regular setting of a single classiﬁcation task (Chapelle et al., 2010) where classes at training and test time are the same, such work has not addressed the challenge of performing transfer to new classes never seen at training time, as we consider here.
Second, we consider the situation where the new classes to be learned are not viewed in isolation.
Instead, many of the unlabeled examples are from different classes; the presence of such distractor classes introduces an additional and more realistic level of difﬁculty to the few-shot problem.
This work is a ﬁrst study of this challenging semi-supervised form of few-shot learning.
First, we deﬁne the problem and propose benchmarks for evaluation that are adapted from the Omniglot and mini ImageNet benchmarks used in ordinary few-shot learning.
We perform an extensive empirical investigation of the two settings mentioned above, with and without distractor classes.
Second, we propose and study three novel extensions of Prototypical Networks (Snell et al., 2017), a state-ofthe-art approach to few-shot learning, to the semi-supervised setting.
Finally, we demonstrate in our experiments that our semi-supervised variants successfully learn to leverage unlabeled examples and outperform purely supervised Prototypical Networks.
B ACKGROUND
We start by deﬁning precisely the current paradigm for few-shot learning and the Prototypical Network approach to this problem.
2. F EW - SHOT LEARNING
Recent progress on few-shot learning has been made possible by following an episodic paradigm for few-shot learning.
Consider a situation where we have a large labeled dataset for a set of classes
C train .
However, after training on examples from C train , our ultimate goal is to produce classiﬁers for a disjoint set of new classes C test , for which only a few labeled examples will be available.
The idea behind the episodic paradigm is to simulate the types of few-shot problems that will be encountered at test using the large quantities of available labeled data for classes C train .
Speciﬁcally, models are trained on K -shot, N -way episodes constructed by ﬁrst sampling a small subset of N classes from C train and then generating: 1) a training (support) set S
= { ( x 1 , y 1 ) , ( x 2 , y 2 ) , . . .
, ( x N × K , y N × K ) } containing K examples from each of the N classes and 2) a test (query) set Q = { ( x ∗ 1 , y ∗ 1 ) , ( x ∗ 2 , y ∗ 2 ) , . .
, ( x ∗ T , y ∗ T ) } of different examples from the same
N classes.
Each x i ∈
R D is an input vector of dimension D and y i ∈ { 1 , 2 , . . .
, N } is a class label (similarly for x ∗ i and y ∗ i ).
Training on such episodes is done by feeding the support set S to the model and updating its parameters to minimize the loss of its predictions for the examples in the query set Q .
One way to think of this approach is that our model effectively trains to be a good learning algorithm.
Indeed, much like a learning algorithm, the model must take in a set of labeled examples and produce a predictor that can be applied to new examples.
Moreover, training directly encourages the classiﬁer produced by the model to have good generalization on the new examples of the query set.
Due to this analogy, training under this paradigm is often referred to as learning to learn or meta-learning.
On the other hand, referring to the content of episodes as training and test sets and to the process of learning on these episodes as meta-learning or meta-training (as is sometimes done in the literature) can be confusing.
So for the sake of clarity, we will refer to the content of episodes as support and query sets, and to the process of iterating over the training episodes simply as training.
2. P ROTOTYPICAL N ETWORKS
Prototypical Network (Snell et al., 2017) is a few-shot learning model that has the virtue both of being simple while obtaining state-of-the-art performance.
At a high-level, it uses the support set S to extract a prototype vector from each class, and classiﬁes the inputs in the query set based on their distance to the prototype of each class.
More precisely, Prototypical Networks learn an embedding function h ( x ) , parameterized as a neural network, that maps examples into a space where examples from the same class are close and those from different classes are far.
All parameters of Prototypical Networks lie in the embedding function.
To compute the prototype p c of each class c , a per-class average of the embedded examples is performed:
[Equation]
, where z i,c = 1 [ y
[Equation]
= c ]
[Equation]
These prototypes deﬁne a predictor for the class of any new (query) example x ∗ , which assigns a probability over any class c based on the distances between x ∗ and each prototype, as follows:
[Equation]
The loss function used to update Prototypical Networks for a given training episode is then simply the average negative log-probability of the correct class assignments, for all query examples:
[Equation]
Training proceeds by minimizing the average loss, iterating over training episodes and performing a gradient descent update for each.
Generalization performance is measured on test set episodes, which contain images from classes in C test instead of C train .
For each test episode, we use the predictor produced by the Prototypical
Network for the provided support set S to classify each of query input x ∗ into the most likely class
ˆ y = argmax c
p ( c | x ∗ , { p c } ) .
S EMI -S UPERVISED F EW -S HOT L EARNING
We now move to deﬁning the semi-supervised setting considered in this work for few-shot learning.
We denote our training set as a tuple of labeled and unlabeled examples: ( S , R ) .
The labeled portion is the usual support set S of the few-shot learning literature, containing a list of tuples of inputs and targets.
In addition to classic few-shot learning, we introduce an unlabeled set R containing only inputs: R = { ˜ x 1 , ˜ x 2 , . . . ,
˜ x M } .
As in the purely supervised setting, our models are trained to perform well when predicting the labels for the examples in the episode’s query set Q .
Figureshows a visualization of training and test episodes.
3. S EMI -S UPERVISED P ROTOTYPICAL N ETWORKS
In their original formulation, Prototypical Networks do not specify a way to leverage the unlabeled set R .
In what follows, we now propose various extensions that start from the basic deﬁnition of prototypes p c and provide a procedure for producing reﬁned prototypes ˜
p c using the unlabeled examples in R .
Training
Testing
Query Set Query Set
Support Set Support Set
Unlabeled Set
Unlabeled Set ? ?
[Figure]
After the reﬁned prototypes are obtained, each of these models is trained with the same loss function for ordinary Prototypical Networks of Equation 3, but replacing p c with ˜ p c .
That is, each query example is classiﬁed into one of the N classes based on the closeness of its embedded position with the corresponding reﬁned prototypes, and the average negative log-probability of the correct classiﬁcation is used for training.
3.1. P ROTOTYPICAL N ETWORKS WITH S OFT k -M EANS
We ﬁrst consider a simple way of leveraging unlabeled examples for reﬁning prototypes, by taking inspiration from semi-supervised clustering.
Viewing each prototype as a cluster center, the reﬁnement process could attempt to adjust the cluster locations to better ﬁt the examples in both the support and unlabeled sets.
Under this view, cluster assignments of the labeled examples in the support set are considered known and ﬁxed to each example’s label.
The reﬁnement process must instead estimate the cluster assignments of the unlabeled example and adjust the cluster locations (the prototypes) accordingly.
One natural choice would be to borrow from the inference performed by soft k -means.
We prefer this version of k -means over hard assignments since hard assignments would make the inference non-differentiable.
We start from the regular Prototypical Network’s prototypes p c (as speciﬁed in
Equation 1) as the cluster locations.
Then, the unlabeled examples get a partial assignment ( ˜ z j,c ) to each cluster based on their Euclidean distance to the cluster locations.
Finally, reﬁned prototypes are obtained by incorporating these unlabeled examples.
This process can be summarized as follows:
[Equation]
, where ˜ z j,
[Equation]
Predictions of each query input’s class is then modeled as in Equation 2, but using the reﬁned prototypes ˜ p c .
We could perform several iterations of reﬁnement, as is usual in k -means.
However, we have experimented with various number of iterations and found results to not improve beyond a single reﬁnement step.
3.1. P ROTOTYPICAL N ETWORKS WITH S OFT k -M EANS WITH A D ISTRACTOR C LUSTER
The soft k -means approach described above implicitly assumes that each unlabeled example belongs to either one of the N classes in the episode.
However, it would be much more general to not make that assumption and have a model robust to the existence of examples from other classes, which we refer to as distractor classes.
For example, such a situation would arise if we wanted to classify between pictures of unicycles and scooters, and decided to add an unlabeled set by downloading images from the web.
It then would not be realistic to assume that all these images are of unicycles or scooters.
Even with a focused search, some may be from similar categories, such as bicycle.
Since soft k -means distributes its soft assignments across all classes, distractor items could be harmful and interfere with the reﬁnement process, as prototypes would be adjusted to also partially account for these distractors.
A simple way to address this is to add an additional cluster whose purpose is to capture the distractors, thus preventing them from polluting the clusters of the classes of interest:
[Equation]
Here we take the simplifying assumption that the distractor cluster has a prototype centered at the origin.
We also consider introducing length-scales r c to represent variations in the within-cluster distances, speciﬁcally for the distractor cluster:
[Equation]
[Equation]
For simplicity, we set r 1 ...N to 1 in our experiments, and only learn the length-scale of the distractor cluster r N +1 .
3.1. P ROTOTYPICAL N ETWORKS WITH S OFT k -M EANS AND M ASKING
Modeling distractor unlabeled examples with a single cluster is likely too simplistic.
Indeed, it is inconsistent with our assumption that each cluster corresponds to one class, since distractor examples may very well cover more than a single natural object category.
Continuing with our unicycles and bicycles example, our web search for unlabeled images could accidentally include not only bicycles, but other related objects such as tricycles or cars.
This was also reﬂected in our experiments, where we constructed the episode generating process so that it would sample distractor examples from multiple classes.
To address this problem, we propose an improved variant: instead of capturing distractors with a high-variance catch-all cluster, we model distractors as examples that are not within some area of any of the legitimate class prototypes.
This is done by incorporating a soft-masking mechanism on the contribution of unlabeled examples.
At a high level, we want unlabeled examples that are closer to a prototype to be masked less than those that are farther.
More speciﬁcally, we modify the soft k -means reﬁnement as follows.
We start by computing normalized distances ˜
d j,c between examples ˜ x j and prototypes p c :
[Equation]
Then, soft thresholds β c and slopes γ c are predicted for each prototype, by feeding to a small neural network various statistics of the normalized distances for the prototype:
[Equation]
This allows each threshold to use information on the amount of intra-cluster variation to determine how aggressively it should cut out unlabeled examples.
Then, soft masks m j,c for the contribution of each example to each prototype are computed, by comparing to the threshold the normalized distances, as follows:
[Equation]
where σ ( · ) is the sigmoid function.
When training with this reﬁnement process, the model can now use its MLP in Equation 8 to learn to include or ignore entirely certain unlabeled examples.
The use of soft masks makes this process entirely differentiable 2 .
Finally, much like for regular soft k -means (with or without a distractor cluster), while we could recursively repeat the reﬁnement for multiple steps, we’ve found a single step to perform well enough.
R ELATED W ORK
We summarize here the most relevant work from the literature on few-shot learning, semi-supervised learning and clustering.
The best performing methods for few-shot learning use the episodic training framework prescribed by meta-learning.
The approach within which our work falls is that of metric learning methods.
Previous work in metric-learning for few-shot-classiﬁcation includes Deep Siamese Networks (Koch et al., 2015), Matching Networks (Vinyals et al., 2016), and Prototypical Networks (Snell et al., 2017), which is the model we extend to the semi-supervised setting in our work.
The general idea here is to learn an embedding function that embeds examples belonging to the same class close together while keeping embeddings from separate classes far apart.
Distances between embeddings of items from the support set and query set are then used as a notion of similarity to do classiﬁcation.
Lastly, closely related to our work with regard to extending the few-shot learning setting, Bachman et al.
(2017) employ Matching Networks in an active learning framework where the model has a choice of which unlabeled item to add to the support set over a certain number of time steps before classifying the query set.
Unlike our setting, their meta-learning agent can acquire ground-truth labels from the unlabeled set, and they do not use distractor examples.
Other meta-learning approaches to few-shot learning include learning how to use the support set to update a learner model so as to generalize to the query set.
Recent work has involved learning either the weight initialization and/or update step that is used by a learner neural network (Ravi & Larochelle, 2017; Finn et al., 2017).
Another approach is to train a generic neural architecture such as a memory-augmented recurrent network (Santoro et al., 2016) or a temporal convolutional network (Mishra et al., 2017) to sequentially process the support set and perform accurate predictions of the labels of the query set examples.
These other methods are also competitive for few-shot learning, but we chose to extend Prototypical Networks in this work for its simplicity and efﬁciency.
As for the literature on semi-supervised learning, while it is quite vast (Zhu, 2005; Chapelle et al., 2010), the most relevant category to our work is related to self-training (Yarowsky, 1995; Rosenberg et al., 2005).
Here, a classiﬁer is ﬁrst trained on the initial training set.
The classiﬁer is then used to classify unlabeled items, and the most conﬁdently predicted unlabeled items are added to the training set with the prediction of the classiﬁer as the assumed label.
This is similar to our soft k -means extension to Prototypical Networks.
Indeed, since the soft assignments (Equation 4) match the regular Prototypical Network’s classiﬁer output for new inputs (Equation 2), then the reﬁnement
In addition to the original k -means method (Lloyd, 1982), the most related work to our setup involving clustering algorithms deals with using k -means in the presence of outliers (Hautamäki et al., 2005; Chawla & Gionis, 2013; Gupta et al., 2017).
The goal here is to correctly discover and ignore the outliers so that they do not wrongly shift the cluster locations to form a bad partition of the true data.
This objective is also important in our setup as not ignoring outliers (or distractors) will wrongly shift the prototypes and negatively inﬂuence classiﬁcation performance.
Our contribution to the semi-supervised learning and clustering literature is to go beyond the classical setting of training and evaluating within a single dataset, and consider the setting where we must learn to transfer from a set of training classes C train to a new set of test classes C test .
E XPERIMENTS 5. D ATASETS
We evaluate the performance of our model on three datasets: two benchmark few-shot classiﬁcation datasets and a novel large-scale dataset that we hope will be useful for future few-shot learning work.
Omniglot (Lake et al., 2011) is a dataset of 1,623 handwritten characters from 50 alphabets.
Each character was drawn by 20 human subjects.
We follow the few-shot setting proposed by Vinyals et al.
(2016), in which the images are resized to 28 × 28 pixels and rotations in multiples of 90 ◦ are applied, yielding 6,492 classes in total.
These are split into 4,800 training classes and 1,692 classes for test.
mini ImageNet (Vinyals et al., 2016) is a modiﬁed version of the ILSVRC-12 dataset (Russakovsky et al., 2015), in which 600 images for each of 100 classes were randomly chosen to be part of the dataset.
We rely on the class split used by Ravi & Larochelle (2017).
These splits use 64 classes as training, 16 for validation, and 20 for test.
All images are of size 84 × 84 pixels.
tiered ImageNet is our proposed dataset for few-shot classiﬁcation.
Like mini Imagenet, it is a subset of ILSVRC-12.
However, tiered ImageNet represents a larger subset of ILSVRC-12 (608 classes rather than 100 for mini ImageNet).
Analogous to Omniglot, in which characters are grouped into alphabets, tiered ImageNet groups classes into broader categories corresponding to higher-level nodes in the ImageNet (Deng et al., 2009) hierarchy.
There are 34 categories in total, with each category containing between 10 and 30 classes.
These are split into 26 training categories and 8 testing categories (details of the dataset can be found in the supplementary material).
This ensures that all of the training classes are sufﬁciently distinct from the testing classes, unlike mini ImageNet and other alternatives such as rand ImageNet proposed by Vinyals et al.
For example, “pipe organ” is a training class and “electric guitar” is a test class in the Ravi & Larochelle (2017) split of mini Imagenet, even though they are both musical instruments.
This scenario would not occur in tiered ImageNet since “musical instrument” is a high-level category and as such is not split between training and test classes.
This represents a more realistic few-shot learning scenario since in general we cannot assume that test classes will be similar to those seen in training.
Additionally, the tiered structure of tiered ImageNet may be useful for few-shot learning approaches that can take advantage of hierarchical relationships between classes.
We leave such interesting extensions for future work.
5. E XPERIMENTAL S ETUP
For each dataset, we ﬁrst create an additional split to separate the images of each class into disjoint labeled and unlabeled sets.
For Omniglot and tiered ImageNet we sampled 10% of the images of each class to form the labeled split.
The remaining 90% can only be used in the unlabeled portion of episodes.
For mini ImageNet we instead used 40% of the data for the labeled split and the remaining 60% for the unlabeled, since we noticed that 10% was too small to achieve reasonable performance and avoid overﬁtting.
We would like to emphasize that due to this labeled/unlabeled split, we are using strictly less label information than in the previously-published work on these datasets.
Because of this, we do not
[Table] [Table]
expect our results to match the published numbers, which should instead be interpreted as an upperbound for the performance of the semi-supervised models deﬁned in this work.
Episode construction then is performed as follows.
For a given dataset, we create a training episode by ﬁrst sampling N classes uniformly at random from the set of training classes C train .
We then sample K images from the labeled split of each of these classes to form the support set, and M images from the unlabeled split of each of these classes to form the unlabeled set.
Optionally, when including distractors, we additionally sample H other classes from the set of training classes and
M images from the unlabeled split of each to act as the distractors.
These distractor images are added to the unlabeled set along with the unlabeled images of the N classes of interest (for a total of MN + MH images).
The query portion of the episode is comprised of a ﬁxed number of images from the labeled split of each of the N chosen classes.
Test episodes are created analogously, but with the N classes (and optionally the H distractor classes) sampled from C test .
Note that we used
M = 5 for training and M = 20 for testing, thus also measuring the ability of the models to generalize to a larger unlabeled set size.
We also used H = N = 5 , i.e. used 5 classes for both the labeled classes and the disctractor classes.
In each dataset we compare our three semi-supervised models with two baselines.
The ﬁrst baseline, referred to as Supervised in our tables, is an ordinary Prototypical Network that is trained in a purely supervised way on the labeled split of each dataset.
The second baseline, referred to as SemiSupervised Inference, uses the embedding function learned by this supervised Prototypical Network, but performs semi-supervised reﬁnement of the prototypes at inference time using a step of Soft k -Means reﬁnement.
This is to be contrasted with our semi-supervised models that perform this reﬁnement both at training time and at test time, therefore learning a different embedding function.
We evaluate each model in two settings: one where all unlabeled examples belong to the classes of interest, and a more challenging one that includes distractors.
Details of the model hyperparameters can be found in the appendix.
5. R ESULTS
Results for Omniglot, mini ImageNet and tiered ImageNet are given in Tables 1, 2 and 3, respectively, while Figure 4 shows the performance of our models on tiered ImageNet (our largest dataset) using different values for M (number of items in the unlabeled set per class).
Across all three benchmarks, at least one of our proposed models outperform the baselines, demonstrating the effectiveness of our semi-supervised meta-learning procedure.
In particular, Soft kmeans+Cluster performs the best on 1-shot non-distractor settings, as the extra cluster seems to provide a form of regularization that pushes the clusters farther apart.
Soft k -Means performs well on 5-shot non-distractor settings, as it considers the most unlabeled examples.
Masked Soft k -Means shows the most robust performance in distractors settings, in both 1-shot and 5-shot tasks.
For 5-
ProtoNet Model
[Table]
Supervised
[Table]
Semi-Supervised Inference
[Table]
Soft k -Means
[Table]
Soft k -Means+Cluster
[Table]
Masked Soft k -Means
[Table]
[Figure]
shot, Masked soft k -Means reaches comparable performance compared to the upper bound of the best non-distractor performance.
From Figure 4, we observe clear improvement in test accuracy when the number grows from 0 to 25.
Note that our models were trained with M = 5 and thus are showing an ability to extrapolate in generalization.
This conﬁrms that, through meta-training, the models learned to acquire a better representation that will be more helpful after semi-supervised reﬁnement.
C ONCLUSION
In this work, we propose a novel semi-supervised few-shot learning paradigm, where an unlabeled set is added to each episode.
We also extend the setup to more realistic situations where the unlabeled set has classes not belonging to the labeled classes.
To address the problem that current few-shot classiﬁcation dataset is too small for a labeled vs. unlabeled split and does not have hierarchical levels of labels, we also introduce a new dataset, tiered ImageNet.
We propose several novel extensions of Prototypical Networks, and they show consistent improvements under semi-supervised settings compared to our baselines.
As future work, we are working on incorporating fast weights (Ba et al., 2016; Finn et al., 2017) into our framework so that examples can have different embedding representation given the contents in the episode.
A tiered I MAGENET D ATASET D ETAILS
Each high-level category in tiered ImageNet contains between 10 and 30 ILSVRC-12 classes (17.
on average).
In the ImageNet hierarchy, some classes have multiple parent nodes.
Therefore, classes belonging to more than one category were removed from the dataset to ensure separation between training and test categories.
Test categories were chosen to reﬂect various levels of separation between training and test classes.
Some test categories (such as “working dog”) are fairly similar to training categories, whereas others (such as “geological formation”) are quite different.
The list of categories is shown below and statistics of the dataset can be found in Table 4.
A visualization of the categories according to the ImageNet hierarchy is shown in Figure 5.
The full list of classes per category will also be made public, however for the sake of brevity we do not include it here.
[Table]
Train Categories :
n02087551 (hound, hound dog), n02098550 (sporting dog, gun dog), n02092468 (terrier), n02120997 (feline, felid), n02370806 (ungulate, hoofed mammal), n02469914 (primate), n01726692 (snake, serpent, ophidian), n01674216 (saurian), n01524359 (passerine, passeriform bird), n01844917 (aquatic bird), n03405265 (furnishing), n04081844 (restraint, constraint), n03574816 (instrument), n03738472 (mechanism), n03800933 (musical instrument, instrument), n03699975 (machine), n03125870 (craft), n03791235 (motor vehicle, automotive vehicle), n04451818 (tool), n03414162 (game equipment), n03278248 (electronic equipment), n03257877 (durables, durable goods, consumer durables), n03419014 (garment), n03297735 (establishment), n02913152 (building, ediﬁce), n04014297 (protective covering, protective cover, protection).
Test Categories : n02103406 (working dog), n01473806 (aquatic vertebrate), n02159955 (insect), n04531098 (vessel), n03839993 (obstruction, obstructor, obstructer, impediment, impedimenta), n09287968 (geological formation, formation), n00020090 (substance), n (solid).
B E XTRA E XPERIMENTAL R ESULTS
Figure 6 shows test accuracy values with different number of unlabeled items during test time.
Figure 7 shows our mask output value distribution of the masked soft k-means model on Omniglot.
The mask values have a bi-modal distribution, corresponding to distractor and non-distractor items.
C H YPERPARAMETER D ETAILS
For Omniglot, we adopted the best hyperparameter settings found for ordinary Prototypical Networks in Snell et al.
In these settings, the learning rate was set to 1e-3, and cut in half every 2 K updates starting at update 2K.
We trained for a total of 20K updates.
For mini Imagenet and tiered ImageNet, we trained with a starting learning rate of 1e-3, which we also decayed.
We started the decay after 25K updates, and every 25K updates thereafter we cut it in half.
We trained for a total of 200K updates.
We used ADAM (Kingma & Ba, 2014) for the optimization of our models.
For the MLP used in the Masked Soft k -Means model, we use a single hidden layer with 20 hidden units with a tanh non-linearity for all 3 datasets.
We did not tune the hyparameters of this MLP so better performance may be attained with a more rigorous hyperparameter search.
[Table]
i c l e
[Table]
t i v e v
[Table]
c l
[Table]
c an i ne , c an i d n dog , do m
e s t i c dog , C an
i s f a m ili a r
i s n03122748 c o v e r
i ng n c l o t h
i ng , a r t i c
l e o f c l o t h i n g , v e s t u r e , w ea r , n p r o t e c t
i v e
c
o v e r
i ng , p r o t e c t i v e
c
o v e r , p r o t e c t [ 24 c l a ss e s ] n m a mm a l , m a mm a li an n aqua
t
i c v e r t eb r a t e [ 16 c l a ss e s ] n r ep t il e , r ep t ili an n01503061b i r d n hound , ho u nd dog [ 19 c l a s s e s ] ns t r u c t u r e , c on s t r u c t i on n ob s t r u c t i on , ob s t r u c t o r , ob s t r u c t e r , i m ped i m en [ 10 c l a ss e s ] n029 1bu il d i ng , ed i ﬁc e [ 14 c l a ss e s ] n03297735e s t ab li s h m en t [ 10 c l a ss e s ] n03294048equ i p m en t n ga m e equ i p m en t [ 12 c l a ss e s ] n geo l
og i c a l f o r m a t
i on , f o r m a t
i on [ 10 c l a ss e s ]
n03125870 c r a ft [ 20 c l a ss e s ] n03738472 m e c han
i s m [ 12 c l a ss e s ] n du r ab l e s , du r ab l e good s , c on s u m e
r du r ab l e s [ 12 c l a ss e s ] n p l a c en t a l , p l a c en t a l m a mm a l , eu t he r i an , eu t he r n ungu l a t e , hoo f ed m a mm a l [ 17 c l a ss e s ]
n02075296 c a r n i v o r e n02469914p r
i m a t e [ 20 c l a ss e s ] n02159955 i n s e c t [ 27 c l a ss e s ] n o r gan i s m , be i ng n an
i m a l , an i m a t
e
be i ng , bea s t, b r u t e , c r ea t u r e , n00001930ph ys i c a l en t i t y n ob j e c t, ph ys i c a l ob j e c t
n00020827 m a tt e r n03419014ga r m en t [ 25 c l a ss e s ] n do m
e s t
i c an
i m a l , do m e s t i c a t ed an
i m a l n01767661a r t h r opod n c o mm od
i t y , t r ade good , good n c on s u m e r good
s n04531098 v e ss e l [ 23 c l a
[Table]
s
[Table]
F i g u re 5 :
H i e r a r c hy o f ti e
r e d I m a g e n e t ca t e go r i e s .
T r a i n i ng ca t e go r i e s a r e h
i gh li
gh t e d i n r e d a nd t e s t ca t e go r
i e
s i n b l u e .
E ac h ca t e go r y i nd
i ca t e s t
h e
nu m b e r
o
f a ss o c
i a t e d c l a ss e s fr o m
I L S
V RC - 12 .
B e s t
v i e w e
d
z
oo m e d - i n on e l ec t r on i c v e r s i on .
[Table] [Table]
[Figure]
[Table]
T e s t
A cc
[Table] [Table]
[Figure]
shot w/o Distractors
[Figure]
[Figure] [Figure]