U NSUPERVISED L EARNING VIA M ETA -L EARNING
A BSTRACT
A central goal of unsupervised learning is to acquire representations from unlabeled data or experience that can be used for more effective learning of downstream tasks from modest amounts of labeled data.
Many prior unsupervised learning works aim to do so by developing proxy objectives based on reconstruction, disentanglement, prediction, and other metrics.
Instead, we develop an unsupervised meta-learning method that explicitly optimizes for the ability to learn a variety of tasks from small amounts of data.
To do so, we construct tasks from unlabeled data in an automatic way and run meta-learning over the constructed tasks.
Surprisingly, we ﬁnd that, when integrated with meta-learning, relatively simple task construction mechanisms, such as clustering embeddings, lead to good performance on a variety of downstream, human-speciﬁed tasks.
Our experiments across four image datasets indicate that our unsupervised meta-learning approach acquires a learning algorithm without any labeled data that is applicable to a wide range of downstream classiﬁcation tasks, improving upon the embedding learned by four prior unsupervised learning methods.
I NTRODUCTION
Unsupervised learning is a fundamental, unsolved problem (Hastie et al., 2009) and has seen promising results in domains such as image recognition (Le et al., 2013) and natural language understanding (Ramachandran et al., 2017).
A central use case of unsupervised learning methods is enabling better or more efﬁcient learning of downstream tasks by training on top of unsupervised representations (Reed et al., 2014; Cheung et al., 2015; Chen et al., 2016) or ﬁne-tuning a learned model (Erhan et al., 2010).
However, since the downstream objective requires access to supervision, the objectives used for unsupervised learning are only a rough proxy for downstream performance.
If a central goal of unsupervised learning is to learn useful representations, can we derive an unsupervised learning objective that explicitly takes into account how the representation will be used?
The use of unsupervised representations for downstream tasks is closely related to the objective of meta-learning techniques: ﬁnding a learning procedure that is more efﬁcient and effective than learning from scratch.
However, unlike unsupervised learning methods, meta-learning methods require large, labeled datasets and hand-speciﬁed task distributions.
These dependencies are major obstacles to widespread use of these methods for few-shot classiﬁcation.
To begin addressing these problems, we propose an unsupervised meta-learning method: one which aims to learn a learning procedure, without supervision, that is useful for solving a wide range of new, human-speciﬁed tasks.
With only raw, unlabeled observations, our model’s goal is to learn a useful prior such that, after meta-training, when presented with a modestly-sized dataset for a human-speciﬁed task, the model can transfer its prior experience to efﬁciently learn to perform the new task.
If we can build such an algorithm, we can enable few-shot learning of new tasks without needing any labeled data nor any pre-deﬁned tasks.
To perform unsupervised meta-learning, we need to automatically construct tasks from unlabeled data.
We study several options for how this can be done.
We ﬁnd that a good task distribution should be diverse, but also not too difﬁcult: na¨ıve random approaches for task generation produce tasks that contain insufﬁcient regularity to enable useful meta-learning.
To that end, our method proposes tasks by ﬁrst leveraging prior unsupervised learning algorithms to learn an embedding of the input data, and then performing an overcomplete partitioning of the dataset to construct numerous categorizations of the data.
We show how we can derive classiﬁcation tasks from these categorizations for use with meta-learning algorithms.
Surprisingly, even with simple mechanisms for partitioning the embedding space, such as k -means clustering, we ﬁnd that meta-learning acquires priors that, when used to learn new, human-designed tasks, learn those tasks more effectively than methods that directly learn on the embedding.
That is, the learning algorithm acquired through unsupervised meta-learning achieves better downstream performance than the original representation used to derive meta-training tasks, without introducing any additional assumptions or supervision.
See Figure 1 for an illustration of the complete approach.
The core idea in this paper is that we can leverage unsupervised embeddings to propose tasks for a meta-learning algorithm, leading to an unsupervised meta-learning algorithm that is particularly effective as pre-training for human-speciﬁed downstream tasks.
In the following sections, we formalize our problem assumptions and goal, which match those of unsupervised learning, and discuss several options for automatically deriving tasks from embeddings.
We instantiate our method with two meta-learning algorithms and compare to prior state-of-the-art unsupervised learning methods.
Across four image datasets (MNIST, Omniglot, miniImageNet, and CelebA), we ﬁnd that our method consistently leads to effective downstream learning of a variety of human-speciﬁed tasks, including character recognition tasks, object classiﬁcation tasks, and facial attribute discrimination tasks, without requiring any labels or hand-designed tasks during meta-learning and where key hyperparameters of our method are held constant across all domains.
We show that, even though our unsupervised meta-learning algorithm trains for one-shot generalization, one instantiation of our approach performs well not only on few-shot learning, but also when learning downstream tasks with up to 50 training examples per class.
In fact, some of our results begin to approach the performance of fully-supervised meta-learning techniques trained with fully-speciﬁed task distributions.
[Figure]
[Figure]
[Figure]
U NSUPERVISED M ETA -L EARNING
In this section, we describe our problem setting in relation to that of unsupervised and semisupervised learning, provide necessary preliminaries, and present our approach.
2. P ROBLEM S TATEMENT
Our goal is to leverage unlabeled data for the efﬁcient learning of a range of human-speciﬁed downstream tasks.
We only assume access to an unlabeled dataset D
= { x i } during meta-training.
After learning from the unlabeled data, which we will refer to as unsupervised meta-training, we want to apply what was learned towards learning a variety of downstream, human-speciﬁed tasks from a modest amount of labeled data, potentially as few as a single example per class.
These downstream tasks may, in general, have different underlying classes or attributes (in contrast to typical semi-supervised problem assumptions), but are assumed to have inputs from the same distribution as the one from which datapoints in D are drawn.
Concretely, we assume that downstream tasks are M -way classiﬁcation tasks, and that the goal is to learn an accurate classiﬁer using K labeled datapoints ( x k , y k ) from each of the M classes, where K is relatively small (i.e. between 1 and 50 ).
The unsupervised meta-training phase aligns with the unsupervised learning problem in that it involves no access to information about the downstream tasks, other than the fact that they are M -way classiﬁcation tasks, for variable M upper-bounded by N .
The upper bound N is assumed to be known during unsupervised meta-training, but otherwise, the values of M and K are not known a priori .
As a result, the unsupervised meta-training phase needs to acquire a sufﬁciently general prior for applicability to a range of classiﬁcation tasks with variable quantities of data and classes.
This problem deﬁnition is our prototype for a practical use-case in which a user would like to train an application-speciﬁc image classiﬁer, but does not have an abundance of labeled data.
2. P RELIMINARIES
Unsupervised embedding learning .
An unsupervised embedding learning algorithm E is a procedure that takes as input an unlabeled dataset D
= { x i } and outputs a mapping from { x i } to embeddings { z i } .
These embedded points are typically lower-dimensional and arranged such that distances correspond to meaningful differences between inputs, in contrast to distances between the original inputs, such as image pixels, which are not meaningful measures of image similarity.
Task .
An M -way K -shot classiﬁcation task T consists of K training datapoints and labels { ( x k , (cid:96) k ) } per class, which are used for learning a classiﬁer, and Q query datapoints and labels per class, on which the learned classiﬁer is evaluated.
That is, in a task there are K + Q = R datapoints and labels for each of the M classes.
Meta-learning .
A supervised meta-learning algorithm M ( · ) takes as input a set of supervised metatraining tasks {T t } .
It produces a learning procedure F ( · ) , which, in turn, ingests the supervised training data of a task to produce a classiﬁer f ( · ) .
The goal of M is to learn F such that, when faced with a meta-test time task T t (cid:48) held-out from {T t } , F can learn a f t (cid:48) that accomplishes T t (cid:48) .
At a high level, the quintessential meta-learning strategy is to have M iterate over {T t } , cycling between applying the current form of F t on training data from T t to learn f t , assessing its performance by calculating some meta-loss L on held-out data from the task, and optimizing L to improve the learning procedure.
We build upon two meta-learning algorithms: model agnostic meta-learning (MAML)
(Finn et al., 2017) and prototypical networks (ProtoNets)
(Snell et al., 2017).
MAML aims to learn the initial parameters of a deep network such that one or a few gradient steps leads to effective generalization; it speciﬁes F as gradient descent starting from the meta-learned parameters.
ProtoNets aim to metalearn a representation in which a class is effectively identiﬁed by its prototype, deﬁned to be the mean of the class’ training examples in the meta-learned space; F is the computation of these class prototypes, and f is a linear classiﬁer that predicts the class whose prototype is closest in Euclidean distance to the query’s representation.
Task generation for meta-learning .
We brieﬂy summarize how tasks are typically generated from labeled datasets { ( x i , y i ) } for supervised meta-learning, as introduced by Santoro et al.
For simplicity, consider the case where the labels are discrete scalar values y i .
To construct an N -way classiﬁcation task T (assuming N is not greater than the number of unique y i ), we can sample
N classes, sample R datapoints { x r } n for each of the N classes, and sample a permutation of N distinct one-hot vectors ( (cid:96) n ) to serve as task-speciﬁc labels of the N sampled classes.
The task is then deﬁned as T = { ( x n,r , (cid:96) n ) | x n,r ∈ { x r } n } .
Of course, this procedure is only possible with labeled data; in the next section, we discuss how we can construct tasks without ground-truth labels.
2. U NSUPERVISED M ETA -L EARNING WITH A UTOMATICALLY C ONSTRUCTED T ASKS
We approach our problem from a meta-learning perspective, framing the problem as the acquisition, from unlabeled data, of an efﬁcient learning procedure that is transferable to human-designed tasks.
In particular, we aim to construct classiﬁcation tasks from the unlabeled data and then learn how to efﬁciently learn these tasks.
If such tasks are adequately diverse and structured , then metalearning these tasks should enable fast learning of new, human-provided tasks.
A key question, then, is how to automatically construct such tasks from unlabeled data D
= { x i } .
Notice that in the supervised meta-learning task generation procedure detailed in Section 2.2, the labels
y
i induce a partition P
= {C c } over { x i } by assigning all datapoints with label y c to subset C c .
Once a partition is obtained, task generation is simple; we can reduce the problem of constructing tasks to that of
A na¨ıve approach is to randomly partition the data D .
While such a scheme introduces diverse tasks, there is no structure; that is, there is no consistency between a task’s training data and query data, and hence nothing to be learned during each task, let alone across tasks.
As seen in Table 3, providing a meta-learner with purely random tasks results in failed meta-learning.
To construct tasks with structure that resembles that of human-speciﬁed labels, we need to group datapoints into consistent and distinct subsets based on salient features.
With this motivation in mind, we propose to use k -means clustering.
Consider the partition P
=
{C c } learned by k -means as a simpliﬁcation of a Gaussian mixture model p ( x | c ) p ( c ) .
If the clusters can recover a semblance of the true class-conditional generative distributions p ( x | c ) , creating tasks based on treating these clusters as classes should result in useful unsupervised meta-training.
However, the result of k -means is critically dependent on the metric space on which its objective is deﬁned.
Clustering in pixel-space is unappealing for two reasons: (1) distance in pixel-space correlates poorly with semantic meaning, and (2) the high dimensionality of raw images renders clustering difﬁcult in practice.
We empirically show in Table 3 that meta-learning with tasks deﬁned by pixel-space clusters, with preprocessing as directed by Coates & Ng (2012), also fails.
We are now motivated to cluster in spaces in which common distance functions correlate to semantic meaning.
However, we must satisfy the constraints of our problem statement in the process of learning such spaces.
To these ends, we use state-of-the-art unsupervised learning methods to produce useful embedding spaces.
For qualitative evidence in the unsupervised learning literature that such embedding spaces exhibit semantic meaning, see Cheung et al.
; Bojanowski & Joulin (2017);
Donahue et al.
We note that while a given embedding space may not be directly suitable for highly-efﬁcient learning of new tasks (which would require the embedding space to be precisely aligned or adaptable to the classes of those tasks), we can still leverage it for the construction of structured tasks, a process for which requirements are less strict.
Thus, we ﬁrst run an out-of-the-box unsupervised embedding learning algorithm E on D , then map the data { x i } into the embedding space Z , producing { z i } .
To produce a diverse task set, we generate P partitions {P p } by running clustering P times, applying random scaling to the dimensions of
Z to induce a different metric, represented by diagonal matrix A , for each run of clustering.
With µ c denoting the learned centroid of cluster C c , a single run of clustering can be summarized with
[Equation]
We derive tasks for meta-learning from the partitions using the procedure detailed in Section 2.2, except we begin the construction of each task by sampling a partition from the uniform distribution
U ( P ) , and for x i ∈ C c , specify y i
= c .
To avoid imbalanced clusters dominating the meta-training tasks, we opt not to sample from p ( c )
∝ |C c | , but instead sample N clusters uniformly without replacement for each task.
We note that Caron et al.
(2018) are similarly motivated in their design decision of sampling data from a uniform distribution over clusters.
With the partitions being constructed over { z i } , we have one more design decision to make: should we perform meta-learning on embeddings or images?
We consider that, to successfully solve new tasks at meta-test time, a learning procedure F that takes embeddings as input would depend on the embedding function’s ability to generalize to out-of-distribution observations.
On the other hand, by meta-learning on images, F can separately adapt f to each evaluation task from the rawest level of representation.
Thus, we choose to meta-learn on images.
We call our method clustering to automatically construct tasks for unsupervised meta-learning (CACTUs).
We detail the task construction algorithm in Algorithm 1, and provide an illustration of the complete unsupervised meta-learning approach for classiﬁcation in Figure 1.
R ELATED W ORK
The method we propose aims to address the unsupervised learning problem (Hastie et al., 2009; Le et al., 2013), namely acquiring a transferable learning procedure without labels.
We show that our method is complementary to a number of unsupervised learning methods, including ACAI (Berthelot et al., 2018), BiGAN (Donahue et al., 2017; Dumoulin et al., 2017), DeepCluster (Caron et al., 2018), and InfoGAN (Chen et al., 2016): we leverage these prior methods to learn embeddings used for constructing meta-learning tasks, and demonstrate that our method learns a more useful representation than the embeddings.
The ability to use what was learned during unsupervised pretraining to better or more efﬁciently learn a variety of downstream tasks is arguably one of the most practical applications of unsupervised learning methods, which has a long history in neural network training (Hinton et al., 2006; Bengio et al., 2007; Ranzato et al., 2006; Vincent et al., 2008; Erhan et al., 2010).
Unsupervised pre-training has demonstrated success in a number of domains, including speech recognition (Yu et al., 2010), image classiﬁcation (Zhang et al., 2017), machine translation (Ramachandran et al., 2017), and text classiﬁcation (Dai & Le, 2015; Howard & Ruder, 2018;
Radford et al., 2018).
Our approach, unsupervised meta-learning, can be viewed as an unsupervised learning algorithm that explicitly optimizes for few-shot transferability.
As a result, we can expect it to better learn human-speciﬁed downstream tasks, compared to unsupervised learning methods that optimize for other metrics, such as reconstruction (Vincent et al., 2010;
Higgins et al., 2017), ﬁdelity of constructed images (Radford et al., 2016; Salimans et al., 2016; Donahue et al., 2017; Dumoulin et al., 2017), representation interpolation (Berthelot et al., 2018), disentanglement (Bengio et al., 2013; Reed et al., 2014; Cheung et al., 2015; Chen et al., 2016; Mathieu et al., 2016; Denton &
Birodkar, 2017), and clustering (Coates & Ng, 2012; Kr ¨ ahenb ¨ uhl et al., 2016; Bojanowski & Joulin, 2017; Caron et al., 2018).
We empirically evaluate this hypothesis in the next section.
In contrast to many previous evaluations of unsupervised pre-training, we focus on settings in which only a small amount of data for the downstream tasks is available, since this is where the unlabeled data can be maximally useful.
Unsupervised pre-training followed by supervised learning can be viewed as a special case of the semi-supervised learning problem (Zhu, 2011; Kingma et al., 2014;
Rasmus et al., 2015; Oliver et al., 2018).
However, in contrast to our problem statement, semi-supervised learning methods assume that a signiﬁcant proportion of the unlabeled data, if not all of it, shares underlying labels with the labeled data.
Additionally, our approach and other unsupervised learning methods are wellsuited for transferring their learned representation to many possible downstream tasks or labelings, whereas semi-supervised learning methods typically optimize for performance on a single task, with respect to a single labeling of the data.
Our method builds upon the ideas of meta-learning (Schmidhuber, 1987; Bengio et al., 1991; Naik & Mammone, 1992) and few-shot learning (Santoro et al., 2016;
Vinyals et al., 2016; Ravi &
Larochelle, 2017; Munkhdalai & Yu, 2017; Snell et al., 2017).
We apply two meta-learning algorithms, model-agnostic meta-learning
(Finn et al., 2017) and prototypical networks (Snell et al., 2017), to tasks constructed in an unsupervised manner.
Similar to our problem setting, some prior works have aimed to learn an unsupervised learning procedure with supervised data (Garg & Kalai, 2017;
Metz et al., 2018).
Instead, we consider a problem setting that is entirely unsupervised, aiming to learn efﬁcient learning algorithms using unlabeled datasets.
Our problem setting is similar to that considered by Gupta et al.
, but we develop an approach that is suitable for supervised downstream tasks, rather than reinforcement learning problems, and demonstrate our algorithm on problems with high-dimensional visual observations.
E XPERIMENTS
We begin the experimental section by presenting our research questions and how our experiments are designed to address them.
Links to code for the experiments can be found at https://sites.
google.com/view/unsupervised-via-meta .
Beneﬁt of meta-learning .
Is there any signiﬁcant beneﬁt to doing meta-learning on tasks derived from embeddings, or is the embedding function already sufﬁcient for downstream supervised learning of new tasks?
To investigate this, we run MAML and ProtoNets on tasks generated via CACTUs (CACTUs-MAML, CACTUs-ProtoNets).
We compare to ﬁve alternate algorithms, with four being supervised learning methods on top of the embedding function.
i)
Embedding k nn -nearest neighbors ﬁrst infers the embeddings of the downstream task images.
For a query test image, it predicts the plurality vote of the labels of the k nn training images that are closest in the embedding space to the query’s embedding.
ii)
Embedding linear classiﬁer also begins by inferring the embeddings of the downstream task images.
It then ﬁts a linear classiﬁer using the NK training embeddings and labels, and predicts labels for the query embeddings using the classiﬁer.
iii)
Embedding multilayer perceptron instead uses a network with one hidden layer of 128 units and tuned dropout (Srivastava et al., 2014).
iv)
To isolate the effect of meta-learning on images, we also compare to embedding cluster matching, i.e. directly using the meta-training clusters for classiﬁcation by labeling clusters with a task’s training data via plurality vote.
If a query datapoint maps to an unlabeled cluster, the closest labeled cluster is used.
v) As a baseline, we forgo any unsupervised pre-training and train a model with the MAML architecture from standard random network initialization via gradient descent separately for each evaluation task.
Different embedding spaces .
Does CACTUs result in successful meta-learning for many distinct task-generating embeddings?
To investigate this, we run unsupervised meta-learning using four embedding learning algorithms: ACAI (Berthelot et al., 2018), BiGAN (Donahue et al., 2017),
DeepCluster (Caron et al., 2018), and InfoGAN (Chen et al., 2016).
These four approaches collectively cover the following range of objectives and frameworks in the unsupervised learning literature: generative modeling, two-player games, reconstruction, representation interpolation, discriminative clustering, and information maximization.
We describe these methods in more detail in Appendix A.
Applicability to different tasks .
Can unsupervised meta-learning yield a good prior for a variety of task types?
In other words, can unsupervised meta-learning yield a good representation for tasks that assess the ability to distinguish between features on different scales, or tasks with various amounts of supervision signal?
To investigate this, we evaluate our procedure on tasks assessing recognition of character identity, object identity, and facial attributes.
For this purpose we choose to use the existing
Omniglot (Santoro et al., 2016) and miniImageNet (Ravi & Larochelle, 2017) datasets and few-shot classiﬁcation tasks and, inspired by Finn et al.
(2018), also construct a new few-shot classiﬁcation benchmark based on the CelebA dataset and its binary attribute annotations.
For miniImageNet, we consider both few-shot downstream tasks and tasks involving larger datasets (up to 50-shot).
Speciﬁcs on the datasets and human-designed tasks are presented in Appendix B.
Oracle .
How does the performance of our unsupervised meta-learning method compare to supervised meta-learning with a human-speciﬁed, near-optimal task distribution derived from a labeled dataset?
To investigate this, we use labeled versions of the meta-training datasets to run MAML and
ProtoNets as supervised meta-learning algorithms (Oracle-MAML, Oracle-ProtoNets).
To facilitate fair comparison with the unsupervised variants, we control for the relevant hyperparameters.
Task construction ablation .
How do the alternatives for constructing tasks from the embeddings compare?
To investigate this, we run MAML on tasks constructed via clustering (CACTUs-MAML) and MAML on tasks constructed via random hyperplane slices of the embedding space with varying margin (Hyperplanes-MAML).
The latter partitioning procedure is detailed in Appendix C. For the experiments where tasks are constructed via clustering, we also investigate the effect of sampling based on a single partition versus multiple partitions.
We additionally experiment with tasks based on random assignments of images to “clusters” (Random-MAML) and tasks based on pixel-space clusters (Pixels CACTUs-MAML) with the Omniglot dataset.
To investigate the limitations of our method, we also consider an easier version of our problem statement where the data distributions at meta-training and meta-test time perfectly overlap, i.e. the images share a common set of underlying labels (Appendix D).
Finally, we present results on miniImageNet after unsupervised meta-learning on most of ILSVRC 2012 (Appendix G).
[Figure]
train test
C e l e b
A m i n i
I m a g e
N e t
4. E XPERIMENTAL P ROTOCOL S UMMARY
As discussed by Oliver et al.
(2018), keeping proper experimental protocol is particularly important when evaluating unsupervised and semi-supervised learning algorithms.
Our foremost concern is to avoid falsely embellishing the capabilities of our approach by overﬁtting to the speciﬁc datasets and task types that we consider.
To this end, we adhere to two key principles.
We do not perform any architecture engineering: we use architectures from prior work as-is, or lightly adapt them to our needs if necessary.
We also keep hyperparameters related to the unsupervised meta-learning stage as constant as possible across all experiments, including the MAML and ProtoNets model architectures.
Details on hyperparameters and architectures are presented in
Appendix E. We assume knowledge of an upper bound on the number of classes N present in each downstream meta-testing task for each dataset.
However, regardless of the number of shots K , we do not assume knowledge of K during unsupervised meta-learning.
We use N -way 1-shot tasks during meta-training, but test on larger values of K during meta-testing.
We partition each dataset into meta-training, meta-validation, and meta-testing splits.
For Omniglot and miniImageNet, these splits contain disjoint sets of classes.
For all algorithms, we run unsupervised pre-training on the unlabeled meta-training split and report performance on downstream tasks dictated by the labeled data of the meta-testing split, generated using the procedure from prior work recounted in Section 2.2.
For the supervised meta-learning oracles, meta-training tasks are constructed in the same manner but from the dataset’s meta-training split.
See Figure 2 for illustrative examples of embedding-derived clusters and human-designed test tasks.
To facilitate analysis on meta-overﬁtting, we use the labels of the meta-validation split (instead of clustering embeddings) to construct tasks for meta-validation.
However, because our aim is to perform meta-learning without supervision, we do not tune hyperparameters on this labeled data.
We use a ﬁxed number of meta-training iterations, since there is no suitable criterion for early stopping.
When we experiment with the embedding-plus-supervised-learning methods used as fair comparisons to unsupervised meta-learning, we err on the side of providing more supervision and data than technically allowed.
Speciﬁcally, we separately tune the supervised learning hyperparameters for each dataset and each task difﬁculty on the labeled version of the meta-validation split.
With DeepCluster embeddings, we also use the entire meta-testing split’s statistics to perform dimensionality reduction (via PCA) and whitening, which is unfair as this shares information across tasks.
4. R ESULTS
Our primary results are summarized in Tables 1 and 2.
Task construction ablations are summarized in Tables 3 and 4.
Beneﬁt of meta-learning .
CACTUs-MAML consistently yields a learning procedure that results in more successful downstream task performance than all other unsupervised methods, including those that learn on top of the embedding that generated meta-training tasks for MAML.
We ﬁnd the same result for CACTUs-ProtoNets for 1-shot downstream tasks.
However, as noted by Snell et al.
(2017), ProtoNets perform best when meta-training shot and meta-testing shot are matched; this characteristic prevents ProtoNets from improving upon ACAI for 20-way 5-shot Omniglot and upon
DeepCluster for 50-shot miniImageNet.
We attribute the success of CACTUs-based meta-learning over the embedding-based methods to two factors: its practice in distinguishing between many distinct sets of clusters from modest amounts of signal, and the underlying classes of the meta-testing split data being out-of-distribution.
In principle, the latter factor is solely responsible for the success over embedding cluster matching, since this algorithm can be viewed as a meta-learner on embeddings that trivially obtains perfect accuracy (via memorization) on the meta-training tasks.
The same factor also helps explain why training from standard network initialization is, in general, competitive with directly using the task-generating embedding as a representation.
On the other hand, the
MNIST results (Table 7 in Appendix F) suggest that when the meta-training and meta-testing data distributions have perfect overlap and the embedding is well-suited enough that embedding cluster matching can already achieve high performance, CACTUs-MAML yields only a small beneﬁt, as we would expect.
[Table]
Different embedding spaces .
CACTUs is effective for a variety of embedding learning methods used for task generation.
The performance of unsupervised meta-learning can largely be predicted by the performance of the embedding-based non-meta-learning methods.
For example, the ACAI embedding does well with Omniglot, leading to the best unsupervised results with ACAI CACTUsMAML.
Likewise, on miniImageNet, the best performing prior embedding (DeepCluster) also corresponds to the best performing unsupervised meta-learner (DeepCluster CACTUs-MAML).
Applicability to different tasks .
CACTUs-MAML learns an effective prior for a variety of task types.
This can be attributed to the application-agnostic task-generation process and the expressive power of MAML (Finn & Levine, 2018).
We also observe that, despite all meta-learning models being trained for N -way 1 -shot classiﬁcation of unsupervised tasks, the models work well for a variety of M -way K -shot tasks, where M ≤ N and K ≤ 50 .
As mentioned previously, the representation that CACTUs-ProtoNets learns is best suited for downstream tasks which match the single shot used for meta-training.
Oracle .
The penalty for not having ground truth labels to construct near-optimal tasks ranges from substantial to severe, depending on the difﬁculty of the downstream task.
Easier downstream tasks (which have fewer classes and/or more supervision) incur less of a penalty.
We conjecture that with such tasks, the difference in the usefulness of the priors matters less since the downstream task-speciﬁc evidence has more power to shape the posterior.
[Table]
Task construction ablation .
As seen in Tables 3 and 4, CACTUs-MAML consistently outperforms
Hyperplanes-MAML with any margin.
We hypothesize that this is due to the issues with zero-margin
Hyperplanes-MAML pointed out in Appendix C, and the fact that nonzero-margin HyperplanesMAML is able to use less of the meta-training split to generate tasks than CACTUs-MAML is.
We ﬁnd that using multiple partitions for CACTUs-MAML, while beneﬁcial, is not strictly necessary.
Using non-zero margin with Hyperplanes-MAML is crucial for miniImageNet, but not for
Omniglot.
We conjecture that the enforced degree of separation between classes is needed for miniImageNet because of the dataset’s high diversity.
Meta-learning on random tasks or tasks derived from pixel-space clustering (Table 3) results in a prior that is much less useful than any other considered algorithm, including a random network initialization; evidently, practicing badly is worse than not practicing at all.
Note on overﬁtting .
Because of the combinatorially many unsupervised tasks we can create from multiple partitions of the dataset, we do not observe substantial overﬁtting to the unsupervised metatraining tasks.
However, we observe that meta-training performance is sometimes worse than metatest time performance, which is likely due to a portion of the automatically generated tasks being based on nonsensical clusters (for examples, see Figure 2).
Additionally, we ﬁnd that, with a few exceptions, using multiple partitions has a regularizing effect on the meta-learner: a diverse task set reduces overﬁtting to the meta-training tasks and increases the applicability of the learned prior.
[Table] [Table]
D ISCUSSION
We demonstrate that meta-learning on tasks produced using simple mechanisms based on embeddings improves upon the utility of these representations in learning downstream, human-speciﬁed tasks.
We empirically show that this holds across benchmark datasets and tasks in the few-shot classiﬁcation literature (Santoro et al., 2016; Ravi & Larochelle, 2017; Finn et al., 2018), task difﬁculties, and embedding learning methods while ﬁxing key hyperparameters across all experiments.
In a sense, CACTUs can be seen as a facilitating interface between an embedding learning method and a meta-learning algorithm.
As shown in the results, the meta-learner’s performance signiﬁcantly depends on the nature and quality of the task-generating embeddings.
We can expect our method to yield better performance as the methods that produce these embedding functions improve, becoming better suited for generating diverse yet distinctive clusterings of the data.
However, the gap between unsupervised and supervised meta-learning will likely persist because, with the latter, the meta-training task distribution is human-designed to mimic the expected evaluation task distribution as much as possible.
Indeed, to some extent, supervised meta-learning algorithms ofﬂoad the effort of designing and tuning algorithms onto the effort of designing and tuning task distributions.
With its evaluation-agnostic task generation, CACTUs-based meta-learning trades off performance in speciﬁc use-cases for broad applicability and the ability to train on unlabeled data.
In principle,
CACTUs-based meta-learning may outperform supervised meta-learning when the latter is trained on a misaligned task distribution.
We leave this investigation to future work.
While we have demonstrated that k -means is a broadly useful mechanism for constructing tasks from embeddings, it is unlikely that combinations of k -means clusters in learned embedding spaces are universal approximations of arbitrary class deﬁnitions.
An important direction for future work is to ﬁnd examples of datasets and human-designed tasks for which CACTUs-based meta-learning results in ineffective downstream learning.
This will result in better understanding of the practical scope of applicability for our method, and spur further development in automatic task construction mechanisms for unsupervised meta-learning.
A potential concern of our experimental evaluation is that MNIST, Omniglot, and miniImageNet exhibit particular structure in the underlying class distribution (i.e., perfectly balanced classes), since they were designed to be supervised learning benchmarks.
In more practical applications of machine learning, such structure would likely not exist.
Our CelebA results indicate that CACTUs is effective even in the case of a dataset without neatly balanced classes or attributes.
An interesting direction for future work is to better characterize the performance of CACTUs and other unsupervised pretraining methods with highly-unstructured, unlabeled datasets.
Since MAML and ProtoNets produce nothing more than a learned representation, our method can be viewed as deriving, from a previous unsupervised representation, a new representation particularly suited for learning downstream tasks.
Beyond visual classiﬁcation tasks, the notion of using unsupervised pre-training is generally applicable to a wide range of domains, including regression, speech (Oord et al., 2018), language (Howard & Ruder, 2018), and reinforcement learning (Shelhamer et al., 2017).
Hence, our unsupervised meta-learning approach has the potential to improve unsupervised representations for a variety of such domains, an exciting avenue for future work. A CKNOWLEDGMENTS
We thank Kelvin Xu, Richard Zhang, Brian Cheung, Ben Poole, A¨aron
van den Oord, Luke Metz,
Siddharth Reddy and the anonymous reviewers for feedback on an early draft of this paper
A PPENDIX A T HE E MBEDDING L EARNING Z OO
We evaluate four distinct methods from prior work for learning the task-generating embeddings.
In adversarially constrained autoencoder interpolation (ACAI), a convolutional autoencoder’s pixelwise L 2 loss is regularized with a term encouraging meaningful interpolations in the latent space (Berthelot et al., 2018).
Speciﬁcally, a critic network takes as input a synthetic image generated from a convex combination of the latents of two dataset samples, and regresses to the mixing factor.
The decoder of the autoencoder and the generator for the critic are one and the same.
The regularization term is minimized when the autoencoder fools the critic into predicting that the synthetic image is a real sample.
The bidirectional GAN (BiGAN) is an instance of a generative-adversarial framework in which the generator produces both synthetic image and embedding from real embedding and image, respectively (Donahue et al., 2017; Dumoulin et al., 2017).
Discrimination is done in joint imageembedding space.
The DeepCluster method does discriminative clustering by alternating between clustering the features of a convolutional neural network and using the clusters as labels to optimize the network weights via backpropagating a standard classiﬁcation loss (Caron et al., 2018).
The InfoGAN framework conceptually decomposes the generator’s input into a latent code and incompressible noise (Chen et al., 2016).
The structure of the latent code is hand-speciﬁed based on knowledge of the dataset.
The canonical GAN minimax objective is regularized with a mutual information term between the code and the generated image.
In practice, this term is optimized using variational inference, involving the approximation of the posterior with an auxiliary distribution
Q ( code | image ) parameterized by a recognition network.
Whereas ACAI explicitly optimizes pixel-wise reconstruction error, BiGAN only encourages the ﬁdelity of generated image and latent samples with respect to their respective prior distributions.
While InfoGAN also encourages the ﬁdelity of generated images, it leverages domain-speciﬁc knowledge to impose a favorable structure on the embedding space and information-theoretic methods for optimization.
DeepCluster departs from the aforementioned methods in that it is not concerned with generation or decoding, and only seeks to learn general-purpose visual features by way of end-to-end discriminative clustering.
A PPENDIX B D ATASET I NFORMATION
The Omniglot dataset consists of 1623 characters each with 20 hand-drawn examples.
Ignoring the alphabets from which the characters originate, we use 1100, 100, and 423 characters for our meta-training, meta-validation, and meta-testing splits.
The miniImageNet dataset consists ofclasses each with 600 examples.
The images are predominantly natural and realistic.
We use the same meta-training/meta-validation/meta-testing splits of 64/16/20 classes as proposed by Ravi &
Larochelle (2017).
The CelebA dataset includes 202,599 facial images of celebrities and 40 binary attributes that annotate every image.
We follow the prescribed 162,770/19,867/19,962 data split.
For Omniglot and miniImageNet, supervised meta-learning tasks and evaluation tasks are constructed exactly as detailed in Section 2.2: for an N -way K -shot task with Q queries per class, we sample N classes from the data split and K + Q datapoints per class, labeling the task’s data with a random permutation of N one-hot vectors.
For CelebA, we consider binary classiﬁcation tasks (i.e., 2-way), each deﬁned by 3 attributes and an ordering of 3 Booleans, one for each attribute.
Every image in a task-speciﬁc class shares all task-speciﬁc attributes with each other and none with images in the other class.
For example, the task illustrated in Figure 2 involves distinguishing between images whose subjects satisfy not Sideburns , Straight Hair , and not Young , and those whose subjects satisfy
Sideburns , not Straight Hair , and Young .
To keep with the idea of having distinct classes for meta-training and meta-testing, we split the task-deﬁning attributes.
For the supervised meta-learning oracle, we construct meta-training tasks from the ﬁrst 20 attributes (when alphabetically ordered), meta-validation tasks from the next 10, and meta-testing tasks from the last 10.
Discarding tasks with too few examples in either class, this results in 4287, 391, and 402 task prototypes (but many more possible tasks).
We use the same meta-test time tasks to evaluate the unsupervised methods.
We only consider assessment with 5-shot tasks because, given that there are multiple attributes other than the task-deﬁning ones, any 1-shot task is likely to be ill-deﬁned.
A PPENDIX C T ASK C ONSTRUCTION VIA R ANDOM H YPERPLANES
Given a set of embedding points { z i } in a space Z , a simple way of deﬁning a partition P = {C c } on { z i } is to use random hyperplanes to slice Z into subspaces and assign the embeddings that lie in the c -th subspace to subset C c .
However, a hyperplane slicing can group together two arbitrarily far embeddings, or separate two arbitrarily close ones; given our assumption that good embedding spaces have a semantically meaningful metric, this creates ill-deﬁned classes.
This problem can be partially alleviated by extending the hyperplane boundaries with a non-zero margin, as empirically shown in Section 4.2.
We now describe how to generate tasks via random hyperplanes in the embedding space.
We ﬁrst describe a procedure to generate a partition P of the set of embeddings { z i } for constructing metatraining tasks.
A given hyperplane slices the embedding space into two, so for an N -way task, we need H =
(cid:100) log 2 N (cid:101) hyperplanes to deﬁne sufﬁciently many subsets/classes for a task.
To randomly deﬁne a hyperplane in d -dimensional embedding space, we sample a normal vector n and a point on the plane z 0 , each with d elements.
For an embedding point z , the signed point-plane distance is given by n | n | 2 · ( z − z 0 ) .
Deﬁning H hyperplanes in this manner, we discard embeddings for which the signed point-plane distance to any of the H hyperplanes lies within ( − m, m ) , where m is a desired margin.
The H hyperplanes collectively deﬁne 2 H subspaces.
We assign embedding points in the c -th subspace to subset C c .
We deﬁne the partition as P =
{C c } .
We prune subsets that do not have at least R =
K m-tr + Q members, and check that the partition has at least N remaining subsets
; if not, we reject the partition and restart the procedure.
After obtaining partitions {P p } , meta-training tasks can be generated by following Algorithm 1 from Line 4.
In terms of practical implementation, we pre-compute 1000 hyperplanes and pruned pairs of subsets of { z i } .
We generate partitions by sampling combinations of the hyperplanes and taking intersections of their associated subsets to deﬁne the elements of the partition.
We determine the number of partitions needed for a given Hyperplanes-MAML run by the number of meta-training tasks desired for the meta-learner: we ﬁx 100 tasks per partition.
A PPENDIX D MNIST E XPERIMENTS
The MNIST dataset consists of 70,000 hand-drawn examples of the 10 numerical digits.
Our split respects the original MNIST 60,000/10,000 training/testing split.
We assess on 10-way classiﬁcation tasks.
This setup results in examples from all 10 digits being present for both meta-training and meta-testing, making the probem setting essentially equivalent to that of semi-supervised learning sans a ﬁxed permutation of the labels.
The MNIST scenario is thus a special case of the problem setting considered in the rest of the paper.
For MNIST, we only experiment with MAML as the meta-learning algorithm.
For ACAI and InfoGAN we constructed the meta-validation split from the last 5,000 examples of the meta-training split; for BiGAN this ﬁgure was 10,000.
After training the ACAI model and inferring embeddings, manually assigning labels to 10 clusters by inspection results in a classiﬁcation accuracy of 96.00% on the testing split.
As the ACAI authors observe, we found it important to whiten the ACAI embeddings before clustering.
The same metric for the InfoGAN embedding (taking an argmax over the categorical dimensions instead of actually running clustering) is 96.83%.
Note that these results are an upper-bound for embedding cluster matching.
To see this, consider the 10-way 1-shot scenario.
1 example sampled from each cluster is insufﬁcient to guarantee the optimal label for that cluster; 1 example sampled from each label is not guaranteed to each end up in the optimal category.
Aside from CACTUs-MAML, embedding k nn -nearest neighbors, embedding linear classiﬁer, and embedding direct clustering, we also ran CACTUs-MAML on embeddings instead of raw images, using a simple model with 2 hidden layers with 64 units each and ReLU activation, and all other
MAML hyperparameters being the same as in Table 5.
Departing from the ﬁxed k = 500 used for all other datasets, we deliberately use k = 10 to better understand the limitations of CACTUs-MAML.
The results can be seen in Table 7 in Appendix B.
In brief, with the better embeddings (ACAI and InfoGAN), there is only little beneﬁt of CACTUsMAML over embedding cluster matching.
Additionally, even in the best cases, CACTUs-MAML falls short of state-of-the-art semi-supervised learning methods.
A PPENDIX E H YPERPARAMETERS AND A RCHITECTURES E. MAML
[Table]
For MNIST and Omniglot we use the same 4-block convolutional architecture as used by Finn et al.
(2017) for their Omniglot experiments, but with 32 ﬁlters (instead of 64) for each convolutional layer for consistency with the model used for miniImageNet and CelebA, which is the same as what Finn et al.
(2017) used for their miniImageNet experiments.
When evaluating the meta-learned 20-way
Omniglot model with 5-way tasks, we prune the unused output dimensions.
The outer optimizer is Adam (Kingma & Ba, 2014), and the inner optimizer is SGD.
We build on the authors’ publicly available codebase found at https://github.com/cbfinn/maml .
When using batch normalization (Ioffe & Szegedy, 2015) to process a task’s training or query inputs, we observe that using only 1 query datapoint per class can allow the model to exploit batch statistics, learning a strategy analogous to a process of elimination that causes signiﬁcant, but spurious, improvement in accuracy.
To mitigate this, we ﬁx 5 queries per class for every task’s evaluation phase, meta-training or meta-testing.
E. P ROTO N ETS
[Table]
[Table] [Table]
[Table]
For the three considered datasets we use the same architecture as used by Snell et al.
(2017) for their
Omniglot and miniImageNet experiments.
This is a 4-block convolutional architecture with each block consisting of a convolutional layer with 64 3 × 3 ﬁlters, stride 1, and padding 1, followed by BatchNorm, ReLU activation, and 2 × 2 MaxPooling.
The ProtoNets embedding is simply the ﬂattened output of the last block.
We follow the authors and use the Adam optimizer, but do not use a learning rate scheduler.
We build upon the authors’ publicly available codebase found at https://github.com/jakesnell/prototypical-networks .
E. CACTU S
For Omniglot, miniImageNet, and CelebA we ﬁx the number of clusters k to be 500.
For Omniglot we choose the number of partitions P = 100 , but in the interest of keeping runtime manageable, choose P = 50 for miniImageNet and CelebA.
E. U SE OF U NSUPERVISED L EARNING M ETHODS
ACAI (Berthelot et al., 2018):
We run ACAI for MNIST and Omniglot.
We pad the images by 2 and use the authors’ architecture.
We use a 256-dimensional embedding for all datasets.
We build upon the authors’ publicly available codebase found at https://github.com/ brain-research/acai .
We unsuccessfully try running ACAI on 64 × 64 miniImageNet and CelebA. To facilitate this input size, we add one block consisting of two convolutional layers (512 ﬁlters each) and one downsampling/upsampling layer to the encoder and decoder.
However, because of ACAI’s pixel-wise reconstruction loss, for these datasets the ACAI embedding prioritizes information about the few “features” that dominate the reconstruction pixel count, resulting in clusters that only corresponded to a limited range of factors, such as background color and pose.
For curiosity’s sake, we tried running meta-learning on tasks derived from these uninteresting clusters anyways, and found that the meta-learner quickly produced a learning procedure that obtained high accuracy on the meta-training tasks.
However, this learned prior was not useful for solving downstream tasks.
BiGAN (Donahue et al., 2017):
For MNIST, we follow the BiGAN authors and specify a uniform 50-dimensional prior on the unit hypercube for the latent.
The BiGAN authors use a 200dimensional version of the same prior for their ImageNet experiments, so we follow suit for Omniglot, miniImageNet, and CelebA. For MNIST and Omniglot, we use the permutation-invariant architecture (i.e. fully connected layers only) used by the authors for their MNIST results; for miniImageNet and CelebA, we randomly crop to 64 × 64 and use the AlexNet-inspired architecture used by Donahue et al.
(2017) for their ImageNet results.
We build upon the authors’ publicly available codebase found at https://github.com/jeffdonahue/bigan .
DeepCluster (Caron et al., 2018):
We run DeepCluster for miniImageNet and CelebA, which we respectively randomly crop and resize to 64 × 64 .
We modify the ﬁrst layer of the AlexNet architecture used by the authors to accommodate this input size.
We follow the authors and use the input to the (linear) output layer as the embedding.
These are 4096-dimensional, so we follow the authors and apply PCA to reduce the dimensionality to 256, followed by whitening.
We build upon the authors’ publicly available codebase found at https://github.com/facebookresearch/ deepcluster .
InfoGAN (Chen et al., 2016):
We only run InfoGAN for MNIST.
We follow the InfoGAN authors and specify the product of a 10-way categorical distribution and a 2-dimensional uniform distribution as the latent code.
We use the authors’ architecture.
Given an image, we use the recognition network to obtain its embedding.
We build upon the authors’ publicly available codebase found at https: //github.com/openai/InfoGAN .
A PPENDIX F E XPERIMENTAL R ESULTS
This section containsfull experimental results for the MNIST, Omniglot, miniImageNet,
T a b l e 7 : M
N I S T d i g it c
l a ss i ﬁ ca ti on r e s u
lt s a v e r a g
e d o v e r t a s k s .
±
d e no t e s a 95 % c on ﬁ d e n ce
i n t e r v a l .
k : nu m b e r o f c l u s t e r s i n a p a r titi on , P : nu m b e r o f p a r titi on s u s e d du r i ng m e t a - l ea r n i ng
A l go r it h m ( w a y , s ho t ) ( 10 , 1 ) ( 10 , 5 ) ( 10 , 10 )
A C A I , d =
E m b e dd
i ng k nn - n ea r e s t n e i ghbo r s [Table] [Table]
[Table]
[Table]
l
[Table]
i
[Table]
e r
[Table]
[Table]
[Table]
[Table]
E m b e dd i
ng c l u s t e r m a t c h
i
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
C A C
T U sM A
M L on i m a g e s ( ou r s ) , P = 1 , k = 92 .± 0 .
% 96 .± 0 .2 % 96 .± 0 .
C A C
T U sM A
M L on e m b e dd i ng s ( ou r s ) , P = 1 , k = 94 .
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[-% 88 .± 
-][Table] B i G A N , d = E m b e dd
i ng k nn - n ea r e s t n e i ghbo r s 29 . [Table] [Table] [Table]
E m b e dd i ng li n ea r c l a s s i ﬁ e r
[Table]
[-% 92 .± 0 .
-][Table]
[Table]
E m b e dd i
ng c l u s t
e r m a t
c h i ng , k = 33 .
[Table]
[Table] [Table]
[Table]
[-% 96 .± 
-] [Table] [Table]
[Table]
C A C
T U sM A
M L on i m a g e s ( ou r s ) , P = 1 ,
[Table]
=
[Table]
[Table]
[Table]
[Table]
C A C
T U sM A
M L on i m a g e s
[Table]
r s ) , P = 100 , k = 49 .
[Table]
[Table]
[Table]
C A C
T U sM A
M L on e m b e dd i ng s ( ou r s ) , P = 1 , k = 36 .±
[Table]
C A C
T U sM A
M L on e m b e dd i
[Table]
s
[Table]
r s )
[Table]
=
[Table]
I n f o
G A N , d =
[Table]
E m b e dd i
[Table]
nn - n
[Table]
r e s t n e i
[Table]
s
[Table]
E m b e dd i
[Table]
li n
[Table]
r c l a s s i ﬁ e r
[Table]
E m b e dd i
ng c l u s t
e r m a t
c h
i ng ,
[Table]
=
[Table]
C A C
T U sM A
M L
[Table]
i m a g e s
[Table]
r s )
[Table]
P = 1 , [Table]
C A C
T U sM A
M L
[Table]
i m a g e s (
[Table]
r s )
[Table]
P
[Table]
C A C
T U sM A
M L
[Table]
e m
[Table]
e dd i
[Table]
T a b l e 8 : O m n i g l o t c h a r ac t e r c l a ss
i ﬁ ca ti on r e s u
lt s
a v e r a g
e d ov e r t a s k s .
±
d e no t e s a 95 % c on ﬁ d e n ce
i n t e r v a l .
d : d i m e n s i on a lit y
o f e m b e dd
i ng , h : nu m b e r
o
f
h
i dd e n un it s i n a f u ll y c onn ec t e d l a y e r , k : nu m b e r
o f c l u s t e r
s i n a p a r titi on , P : nu m b e r o f p a r titi on s u s e d du r
i ng m e t a - l ea r n i ng , m : m a r g i n on b ound a r y - d e ﬁ n i ng hyp e r p l a n e s .
A l go r it h m ( w a y , s ho t ) ( [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
B a s e li n e s
T r a i n i ng fr o m s c
r a t c
[Table]
[Table]
[Table]
ndo
[Table]
H yp e r p l a n e sM A
M L ( ou r s ) ,
P = , m = 1 .62 .± 0 .
% 83 .± 0 .
% 41 .± 0 .
% 65 .± 0 .
C A C
T U sM A
M L ( ou r s ) ,
P = 1 , k = 66 .± 0 .
% 85 .± 0 .
% 45 .± 0 .
% 69 .± 0 .
C A C
T U sM A
M L ( ou r s ) ,
P = 100 , k = 68 .± 0 .
% 87 .± 0 .
% 48 .± 0 .
% 73 .± 0 .
C A C
T U sP r o t o
N e t s ( ou r s ) , P = 100 , k = 68 .± 0 .
% 83 .± 0 .
[Table]
% 66 .± 0 .
B i G
A N , d =
E m b e dd
i ng k nn - n ea r e s t n e i ghbo r s 49 .± 1 .
% 68 .± 0 .
% 27 .± 0 .
% 46 .± 0 .
E m b e dd i ng li n ea r c l a ss i ﬁ e
r 48 .± 1 .
% 68 .± 0 .
% 27 .± 0 .
% 45 .± 0 .
E m b e dd i ng
M L P w [Table] d r op o u t
[Table]
m
[Table]
e dd i
[Table]
c l u s t e r m a t
[Table]
H
[Table]
r p l a n e sM A
M L (
[Table]
r s )
[Table]
P =
[Table]
H yp e r p l a n e sM A M L ( [Table] r s )
[Table]
C A C T U sM A M L (
[Table]
r s )
[Table]
P
[Table]
C A C
T U sM A
M L
[Table]
r s
[Table] C A C T U
[Table]
P r o t o
N e t s
[Table]
r s ) ,
[Table]
Sup e r
v
i s e d m e t a
[Table]
l e a
[Table]
n i ng
[Table]
ac
[Table]
e
[Table]
M
[Table]
M
[Table]
R e s u lt u s e d 64 ﬁ lt e
r
s p e r c onvo l u ti on a l l a y e r , 3 × d a t a
[Table]
a f t e r hyp e
[Table]
p a r a m e t e r t un i
[Table]
T a b l e 9 : m i n i
I m a g e N e t ob j ec t c l a ss i ﬁ ca ti on r e
s u lt s a v e r a g
e d ov e r t a s k s .
±
d e no t e s a 95 % c on ﬁ d e n ce
i n t e r v a l .
d : d i m e n s i on a lit y
o f e m b e dd
i ng , h : nu m b e r
o
f
h
i dd e n un it s i n a f u ll y c onn ec t e d l a y e r , k : nu m b e r
o f c l u s t e r
s i n a p a r titi on , P : nu m b e r o f p a r titi on s u s e d du r
i ng m e t a - l ea r n i ng , m : m a r g i n on b ound a r y - d e ﬁ n i ng hyp e r p l a n e s .
A l go r it h m ( w a y , s ho t ) ( 5 ,) ( 5 ,) ( 5 , 20 ) ( 5 , 50 ) B a s e li n e s T r a i n i ng fr o m s c r a t c h 27 .
[Table]
[Table]
[Table]
[Table]
[Table]
B i G A N , d = E m b e dd
i ng k nn - n ea r e s t n e i ghbo r s 25 .± 1 .
[Table] r [Table] l
[Table] t [Table] r p l a n e
[Table]
L
[Table]
r [Table] [Table]
e r p l a n e s [Table] M [Table]
M L (
[Table]
r s ) ,
P
[Table]
A C
T U [Table]
M [Table] L
[Table]
r s )
[Table]
% 67 .± 0 .
C A C
T U sM A M L ( ou r s ) ,
P = 50 , k = 36 .± 0 . % 51 .± 0 .
% 61 .± 0 .
% 66 .± 0 . C A C T U sP r o t o
N e t s ( ou r s ) , P = 50 , k = 36 .± 0 .
% 50 .± 0 . % 59 .± 0 .
% 63 .± 0 .
D ee p
C l u s t e r , d =
E m b e dd
i ng k nn - n ea r e s t n e i ghbo
r s 28 .± 1 .
% 42 .± 0 .
% 56 .± 0 .
[Table]
E m b e dd i ng li n ea r c l a ss i ﬁ e r 29 .± 1 . % 39 .± 0 . % 56 .± 0 .
% 65 .± 0 .
E m b e dd i ng
M L P w it h d r op o u t , h = 29 .± 0 .
% 39 .± 0 .
% 52 .± 0 .
% 60 .± 0 .
E m b e dd i
ng c l u s t
e r m a t
c h i ng , k = 22 .± 0 .
% 23 .± 0 .
% 24 .± 0 .
[Table]
yp e r p l a n e s
[Table]
M
[Table]
M L
[Table]
[Table]
yp
[Table]
r p l a n
[Table]
s
[Table]
M
[Table]
M L
[Table]
U [Table] [Table] P
[Table]
C A C
[Table]
M
[Table]
M L (
[Table]
r s )
[Table]
P =
[Table]
[Table]
A PPENDIX G I MAGE N ET E XPERIMENTS
We investigate unsupervised meta-learning in the context of a larger unsupervised meta-training dataset by using the ILSVRC 2012 dataset’s training split (Russakovsky et al., 2015), which is a superset of the miniImageNet dataset (including meta-validation and meta-testing data) consisting of 1000 classes and over 1,200,000 images.
To facilitate comparison to the previous miniImageNet experiments, for meta-validation and meta-test we use the miniImageNet meta-validation and metatest splits.
To avoid task leakage, we hold out all data from these 36 underlying classes from the rest of the data to construct the meta-training split.
For CACTUs, we use the best-performing unsupervised learning method from the previous experiments, DeepCluster, to obtain the embeddings.
Following Caron et al.
(2018), we run DeepCluster using the VGG-16 architecture with a 256-dimensional feature space and 10,000 clusters on the meta-training data until the normalized mutual information between the data-cluster mappings of two consecutive epochs converges.
To our knowledge, no prior works have yet been published on using MAML for ImageNet-sized meta-learning.
We extend the standard convolutional neural network model class with residual connections (He et al., 2016), validate hyperparameters with supervised meta-learning, then use it for unsupervised meta-learning without further tuning.
See Tablefor MAML hyperparameters.
The training from scratch, embedding k nn -nearest neighbors, and embedding linear classiﬁer algorithms are the same as they were in the previous sets of experiments.
For Oracle-MAML, we generated tasks using the ground-truth 964 ImageNet meta-training classes.
We also run semi-supervised MAML, with the meta-training tasks consisting of CACTUs-based tasks as well as tasks constructed from the 64 miniImageNet meta-training classes.
The unsupervised/supervised task proportion split was ﬁxed according to the ratio of the number of data available to each task proposal method.
As before, the meta-learning methods only meta-learned on 1-shot tasks.
[Table]
We ﬁnd that the vastly increased amount of unlabeled meta-training data (in comparison to miniImageNet) results in signiﬁcant increases for all methods over their counterparts in Table 9 (other than training from scratch, which does not use this data).
We ﬁnd that CACTUs-MAML slightly outperforms embedding linear classiﬁer for the 1-shot test tasks, but that the linear classiﬁer on top of the unsupervised embedding becomes better as the amount of test time supervision increases.
Augmenting the unsupervised tasks with (a small number of) supervised tasks during meta-training results in slight improvement for the 1-shot test tasks.
The lackluster performance of CACTUs-MAML is unsurprising insofar as meta-learning with large task spaces is still an open problem: higher shot
Oracle-MAML only marginally stays ahead of the embedding linear classiﬁer, which is not the case in the other, smaller-scale experiments.
We expect that using a larger architecture in conjunction with MAML (such as Kim et al.
(2018)) would result in increased performance for all methods based on MAML.
Further, given the extensive degree to which unsupervised learning methods have been studied, we suspect that unsupervised task construction coupled with better meta-learning algorithms and architectures will result in improved performance on the entire unsupervised learning problem.
We leave such investigation to future work.
T a b l e 12 : m i n i
I m a g e
N e t ob j ec t c l a ss i ﬁ ca ti on r e s u lt s a v e r a g
e d ov e r t
a s k s , w it h
I m a g e N e t - s ca
l e m e t a - t r a i n i ng .
± d e n o t e s a%
[Table]
on
[Table]
d e n ce
i n t e r v a l .
d : d i m e n s i on
[Table]
e m b e dd
i
ng , k : nu m b e r o f c l u s t e r s i n a p a r titi on , P : n u m b e r o f p a r titi on s u s e d du r i ng m e t a - l ea r n i ng .
A l go r it h m ( w a y , s ho t ) ( 5 , 1 ) ( 5 ,
[Table]
[Table]
a s
[Table]
li n e s T r a i n i ng fr o m s c r a t c h
[Table]
± 0 .
[Table]
±
[Table]
[Table]
±
[Table]
[Table]
±
[Table]
D ee p C l u s t e r , d =
E m b e dd
i ng k nn - n ea r e s t n e i ghbo
r s
[Table]
± 0 .
[Table]
± 0 .
[Table]
± 0 .
[Table]
± 0 .
E m b e dd i ng li n ea
[Table]