M ETA -GMVAE: M IXTURE OF G AUSSIAN VAE S FOR U NSUPERVISED M ETA -L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Unsupervised learning aims to learn meaningful representations from unlabeled data which can capture its intrinsic structure, that can be transferred to downstream tasks.
Meta-learning, whose objective is to learn to generalize across tasks such that the learned model can rapidly adapt to a novel task, shares the spirit of unsupervised learning in that the both seek to learn more effective and efﬁcient learning procedure than learning from scratch.
The fundamental difference of the two is that the most meta-learning approaches are supervised, assuming full access to the labels.
However, acquiring labeled dataset for meta-training not only is costly as it requires human efforts in labeling but also limits its applications to pre-deﬁned task distributions.
In this paper, we propose a principled unsupervised meta-learning model, namely Meta-GMVAE, based on Variational Autoencoder (VAE) and set-level variational inference.
Moreover, we introduce a mixture of Gaussian (GMM) prior, assuming that each modality represents each class-concept in a randomly sampled episode, which we optimize with
Expectation-Maximization (EM).
Then, the learned model can be used for downstream few-shot classiﬁcation tasks, where we obtain task-speciﬁc parameters by performing semi-supervised EM on the latent representations of the support and query set, and predict labels of the query set by computing aggregated posteriors.
We validate our model on Omniglot and Mini-ImageNet datasets by evaluating its performance on downstream few-shot classiﬁcation tasks.
The results show that our model obtain impressive performance gains over existing unsupervised metalearning baselines, even outperforming supervised MAML on a certain setting.
I NTRODUCTION
Unsupervised learning is one of the most fundamental and challenging problems in machine learning, due to the absence of target labels to guide the learning process.
Thanks to the enormous research efforts, there now exist many unsupervised learning methods that have shown promising results on real-world domains, including image recognition (Le, 2013) and natural language understanding (Ramachandran et al., 2017).
The essential goal of unsupervised learning is obtaining meaningful feature representations that best characterize the data, which can be later utilized to improve the performance of the downstream tasks, by training a supervised task-speciﬁc model on the top of the learned representations (Reed et al., 2014; Cheung et al., 2015; Chen et al., 2016) or ﬁne-tuning the entire pre-trained models (Erhan et al., 2010).
Meta-learning , whose objective is to learn general knowledge across diverse tasks, such that the learned model can rapidly adapt to novel tasks, shares the spirit of unsupervised learning in that both seek more efﬁcient and effective learning procedure over learning from scratch.
However, the essential difference between the two is that most meta-learning approaches have been built on the supervised learning scheme, and require human-crafted task distributions to be applied in fewshot classiﬁcation.
Acquiring labeled dataset for meta-training may require a massive amount of human efforts, and more importantly, meta-learning limits its applications to the pre-deﬁned task distributions (e.g. classiﬁcation of speciﬁc set of classes); thus applying it to a completely new downstream task from an unknown distribution is not straightforward.
To tackle such obstacles in the widespread applications of meta-learning, two recent works have proposed unsupervised meta-learning that can bridge the gap between unsupervised learning and meta-learning by focusing on constructing supervised tasks with pseudo-labels from the unlabeled data.
To do so, CACTUs (Hsu et al., 2019) clusters data in the embedding space learned with several unsupervised learning methods, while UMTRA (Khodadadeh et al., 2019) assumed that each randomly drawn sample represents a different class and augmented each pseudo-class with data augmentation (Cubuk et al., 2018).
After constructing the meta-training dataset with such heuristics, they simply apply supervised meta-learning algorithms as usual.
Despite the success of the existing unsupervised meta-learning methods, they are fundamentally limited, since 1) they only consider unsupervised learning for heuristic pseudo-labeling of unlabeled data, and 2) the two-stage approach makes it impossible to recover from incorrect pseudo-class assignment when learning the unsupervised representation space.
In this paper, we propose a principled, end-to-end unsupervised meta-learning model based on Variational Autoencoder (VAE) (Kingma & Welling, 2014) and set-level variational inference using selfattention (Vaswani et al., 2017).
Moreover, we introduce multi-modal prior distributions, a mixture of Gaussians (GMM), assuming that each modality represents each class-concept in any given tasks.
Then the parameter of GMM is optimized by running Expectation-Maximization (EM) on the observations sampled from the task-dependent variational posterior, which is inferred by set-encoding a given task (dataset) on to latent space.
In this framework, however, there is no guarantee that each modality obtained from EM algorithm corresponds to a label we are interested in.
To realize modality as label, we deploy semi-supervised EM algorithm at meta-test time, considering the support set and query set as labeled and unlabeled observations, respectively.
We refer to our method as Meta-Gaussian Mixture Variational Autoencoders (Meta-GMVAE)
(See Figure 1 for high-level concept).
While our method can be used as a full generative model for generating the samples (images), the ability to generalize samples may not be necessary for capturing the the meta-knowledge for non-generative downstream tasks.
Thus, we propose another version of Meta-GMVAE that reconstructs high-level features instead of generating samples, in the embedding space learned with other powerful unsupervised representation learning approaches (e.g. SimCLR, Chen et al.
To investigate the effectiveness of our framework, we run experiments it on two benchmark fewshot image classiﬁcation datasets, namely Omiglot (Lake et al., 2011) and Mini-Imagenet (Ravi &
Larochelle, 2017).
The experimental results show that our Meta-GMVAE obtains impressive performance gains over the relevant unsupervised meta-learning baselines on both datasets, obtaining even better accuracy than fully supervised MAML (Finn et al., 2017) while utilizing as small as 0 .
1% of the labeled data on one-shot settings in Omniglot dataset.
Moreover, our model can generalize to classiﬁcation tasks with different number of ways (classes) without loss of accuracy.
Our contribution is threefold:
• We propose a novel unsupervised meta-learning model, namely Meta-GMVAE, which metalearns the set-conditioned prior and posterior network for a VAE.
Our Meta-GMVAE is a principled unsupervised meta-learning method, unlike existing methods on unsupervised meta-learning that combines heuristic pseudo-labeling with supervised meta-learning.
• We propose to learn the multi-modal structure of a given dataset with the Gaussian mixture prior, such that it can adapt to a novel dataset via the EM algorithm.
This ﬂexible adaptation to a new task, is
not possible with existing methods that propose VAEs with Gaussian mixture priors for single task learning.
• We show that Meta-GMVAE largely outperforms relevant unsupervised meta-learning baselines on two benchmark datasets, while obtaining even better performance than a supervised metalearning model under a speciﬁc setting.
We further show that Meta-GMVAE can generalize to classiﬁcation tasks with different number of ways (classes).
R ELATED W ORK
Unsupervised learning The framework we propose is in the line with the unsupervised learning (Hastie et al., 2019; Le, 2013), which aims at obtaining transferable feature representations from the given data without any supervision.
To obtain better feature representations for downstream tasks, many prior unsupervised learning methods have developed proxy objectives which is either based on reconstruction (Vincent et al., 2010; Higgins et al., 2017), adversarially obtained image ﬁdelity (Radford et al., 2016; Salimans et al., 2016; Donahue et al., 2017; Dumoulin et al., 2017), disentanglement (Bengio et al., 2013;
Reed et al., 2014; Cheung et al., 2015; Chen et al., 2016; Mathieu et al., 2016; Denton & Birodkar, 2017; Kim & Mnih, 2018; Ding et al., 2020), clustering (Coates &
Ng, 2012; Kr¨ahenb¨uhl et al., 2016; Bojanowski & Joulin, 2017; Caron et al., 2018), or contrastive learning (Chen et al., 2020).
In the unsupervised learning literature, the most relevant work to ours are (Dilokthanakul et al., 2016; Jiang et al., 2017), which propose to use Gaussian Mixture priors for variational autoencoders.
However, they consider single task learning and therefore, the learned parameter of the GMM is ﬁxed after training, and thus cannot adapt to new tasks.
Contrarily, we model the GMM prior in a set-dependent manner and thus it can ﬂexibly adapt to new tasks.
Meta-learning Meta-learning (Thrun & Pratt, 1998) shares the intuition of unsupervised learning in that it aims to improve the model performance on an unseen task by leveraging prior knowledge, rather than learning from scratch.
While the literature on meta-learning is vast, we only discuss relevant existing works for few-shot image classiﬁcation.
Metric-based meta-learning (Koch et al., 2015; Vinyals et al., 2016;
Snell et al., 2017; Oreshkin et al., 2018;
Mishra et al., 2018) is one of the most popular approaches, where it learns to embed the data instances of the same class to be closer in the shared embedding space.
One can measure the distance in the embedding space by cosine similarity (Vinyals et al., 2016), or Euclidean distance
(Snell et al., 2017).
On the other hand, gradient-based meta-learning (Finn et al., 2017; 2018; Li et al., 2017; Lee & Choi, 2018; Ravi & Beatson, 2019; Flennerhag et al., 2020) aims at learning a global initialization of parameters, which can rapidly adapt to a novel task with only a few gradient steps.
Moreover, some previous works (Hewitt et al., 2018; Edwards & Storkey, 2017; Garnelo et al., 2018) tackle meta-learning by modeling the set-dependent variational posterior with a single global latent variable, however, we model the variational posterior conditioned on each data instances.
Moreover, while all of these works assume supervised learning scenarios where one has access to full labels in meta-training stage, we focus on unsupervised setting in this paper.
Unsupervised meta-learning One of the main limitations of conventional meta-learning methods is that their application is strictly limited to the tasks from a pre-deﬁned task distribution.
A few works (Hsu et al., 2019; Khodadadeh et al., 2019) have been proposed to resolve this issue by combining unsupervised learning with meta-learning.
The main idea is to construct meta-training dataset in an unsupervised manner by leveraging existing supervised meta-learning models.
CACTUs (Hsu et al., 2019) deploy several deep metric learning (Berthelot et al., 2019; Donahue et al., 2017; Caron et al., 2018; Chen et al., 2016) to episodically cluster the unlabeled dataset, and then train MAML (Finn et al., 2017) and Prototypical Networks (Snell et al., 2017) on the constructed data.
UMTRA (Khodadadeh et al., 2019) assumes that each randomly drawn sample is from a different class from others, and use data augmentation (Cubuk et al., 2018) to construct synthetic task distribution for meta-training.
Instead of only deploying unsupervised learning for constructing meta-training task distributions, we propose an unsupervised meta-learning model that meta-learns set-level variational posterior by matching the multi-modal prior distribution representing latent classes.
U NSUPERVISED M ETA -L EARNING WITH M ETA -GMVAE S
In this section, we describe our problem setting with respect to unsupervised meta-learning, and demonstrate our approach.
The graphical illustration of our model for unsupervised meta-training and supervised meta-test is depicted in Figure 2.
[Figure]
3. P ROBLEM S TATEMENT
Our goal is to learn unsupervised feature representations which can be transferred to wide range of downstream few-shot classiﬁcation tasks.
As suggested by Hsu et al.
; Khodadadeh et al.
(2019), we only assume an unlabeled dataset D u
= { x u } Uu =1 in the meta-training stage.
We aim toward applying the knowledge learned during unsupervised meta-training stage to novel tasks in meta-test stage, which comes with a modest amount of labeled data (or as few as a single example per class) for each task.
As with most meta-learning methods, we further assume that the labeled data are drawn from the same distribution as that of the unlabeled data, with a different set of classes.
Speciﬁcally, the goal of a K -way S -shot classiﬁcation task T is to correctly predict the labels of query data points Q = { x q } Qq =1 , using S support data points and labels S
= { ( x s , y s ) }
Ss =1 per class, where S is relatively small (i.e. between 1 and 50).
3. M ETA -L EVEL G AUSSIAN M IXTURE VAE
Unsupervised meta-training We now describe the meta-learning framework for learning unsupervised latent representations that can be transferred to human-designed few-shot imageclassiﬁcation tasks.
In particular, we aim toward learning multi-modal latent spaces for Variational
Autoencoder (VAE) in an episodic manner.
We use the Gaussian mixture for the prior distribution p ψ ( z )
=
(cid:80)
Kk =1 p ψ ( y = k ) p ψ ( z | y = k ) , where ψ is the parameter of the prior network.
Then the generative process can be described as follows:
• y ∼ p ψ ( y ) , where y corresponds to the categorical L.V. for a single mode.
• z ∼ p ψ ( z | y ) , where z corresponds to the Gaussian L.V. responsible for data generation.
• x ∼ p θ ( x | z ) , where θ is the parameter of the generative model.
The above generative process is similar to those from the previous works (Dilokthanakul et al., 2016;
Jiang et al., 2017) on modeling the VAE prior with Gaussian mixtures.
However, they target singletask learning and the parameter of the prior network is ﬁxed after training such as equation 1c in Dilokthanakul et al. and equation 5 in Jiang et al. (2017), which is suboptimal since a model should be able to adapt and generalize to a novel task.
To learn the set-dependent multi-modalities, we further assume that there exists a parameter ψ i for each episodic dataset D
i
= { x j } Mj =1 , which is randomly drawn from the unlabeled dataset D u .
Then we derive the variational lower bound for the marginal log-likelihood of D i as follows:
[Equation]
Here the lower bound for each datapoint is approximated by Monte Carlo estimation with the sample size N .
Following the convention of the VAE literature, we assume that the variational posterior follows an isotropic Gaussian distribution.
Our derivation of the evidence lower bound in Eq 4 is similar to that of the hierarchical VAE framework, such as equation 3 in Edwards & Storkey (2017) and equation 4 in Hewitt et al.
(2018), in that we use the i.i.d assumption that the log likelihood of a dataset at each episode equals the sum over the log-likelihoods of each individual data point.
Yet, previous works assume that each input set consists of data instances from a single concept (e.g. a class), therefore, they encode the dataset into a single global latent variable (e.g. q φ ( z |D ) ).
However, this is suboptimal for unsupervised meta-learning where labels are unavailable.
Thus we learn a set-conditioned variational posterior q φ ( z j | x j , D i ) , which models a latent variable to encode each data x j within the given dataset D i into the latent space.
Speciﬁcally, we model the variational posterior q φ ( z j | x j , D i ) using the selfattention mechanism (Vaswani et al., 2017) as follows:
[Equation]
where TransformerEncoder ( · ) is a neural network based on the multi-head self-attention mechanism proposed by Vaswani et al.
(2017), and f is either a convolutional neural network (for the Omniglot) or an identity function (for the Mini-ImageNet).
Further, W µ and W σ 2 are projection weight matrices, b µ and b σ 2 are bias vectors, and exp is an element-wise exponential function.
We deploy the TransformerEncoder layer to model the dependency between data instances.
For more details on TransformerEncoder layer, please refer to Vaswani et al.
Moreover, we use the reparameterization trick (Kingma & Welling, 2014) to train the model with backpropagation since the stochastic sampling process z ( n ) j
[Equation]
The parameter ψ i of the prior Gaussian Mixture then is a task-speciﬁc parameter that characterizes the given dataset D i .
To obtain task-speciﬁc parameter that optimally explain the given dataset, we propose to locally maximize the lower bound in Eq 4 for the given episode dataset D i , with respect to the task-speciﬁc prior parameter ψ i .
We can obtain the optimal parameter ψ ∗ i by solving the following optimization problem:
[Equation]
where we only consider the term related to the task-speciﬁc parameter ψ i , and eliminate the normalization term 1 N since it does not change the solution of the optimization problem.
The above formula implies that the optimal parameter maximizes the log-likelihood of observations which can be drawn from the variational posterior distribution.
However, we do not have an analytic solution for Maximum Likelihood Estimation (MLE) of a GMM.
The most prevalent approach for estimating the parameters for the mixture of Gaussian is solving it with Expectation Maximization (EM) algorithm.
To this end, we propose to optimize the taskspeciﬁc parameter of GMM prior distribution using EM algorithm as follows:
[Equation]
where π k , µ k , and N ( · ) denote the mixing probability of k -th component, mean parameter, and normal distribution, respectively.
We assume that the covariance matrix of Gaussian distribution is ﬁxed with the identity matrix I , following the assumption of original VAE on the prior distribution.
We initialize π k and µ k as 1 K and randomly drawn K different points.
We can obtain MLE solution for the parameters of GMM, by iteratively performing E-step and M-step until the log-likelihood converges.
However, we ﬁnd that using a ﬁxed number of iterations for the EM algorithm does not degrade the performance, and consider it as a hyperparameter of our framework.
Note that we want to maximize the variational lower bound of the marginal log-likelihood over all the episode datasets D i that can be sampled from D u .
We use stochastic gradient ascent with respect to the variational parameter φ and the generative parameter θ , to maximize the following objective:
[Equation]
Here we use B mini-batch of episode datasets, where each dataset consists of M datapoints.
The task-speciﬁc parameter ψ ∗ i for each episode dataset D i is obtained by EM algorithm in Eq 7.
Supervised meta-test By introducing the multi-modal prior distribution into a generative learning framework, our model learns pseudo-class concepts by clustering latent features with EM algorithm.
However, there is no guarantee that each modality obtained by EM algorithm corresponds to the label we are interested in at the meta-test stage.
To realize modality as label in downstream few-shot image classiﬁcation tasks, we deploy semi-supervised EM algorithm instead.
Given a task T that consists of support set S
= { ( x s , y s ) }
Ss =1 and query set Q = { x q } Qq =1 , we use both the support set and query set as an episode dataset D
= { x s } Ss =1 ∪ { x q } Qq =1 and draw latent variables from the variational posterior q φ ( z j | x j , D ) .
Note that we abbreviate the index i since we consider a single task for now.
We then perform semi-supervised EM algorithm as follows:
[Equation]
where 1 denotes an indicator function.
We ﬁx the mixing probability as 1 K since the labels in each task T are uniformly distributed.
Moreover, we utilize diagonal covariance σ 2 k to obtain more accurate statistics for the inference.
We initialize µ k as the average value of support latent representation and σ 2 k as I .
We then obtain the MLE solution for the parameters of GMM, by iteratively performing E-step and M-step in Eq 10.
Finally, we compute the conditional probability of p ( y q | x q , D ) using the obtained parameters ψ ∗ as follows:
[Equation]
Here we compute p ψ ∗ ( y q | z ( n ) q ) with Bayes rule, and we reuse N different Monte Carlo samples that is drawn for Eq 10, where y q = arg max k p ( y q = k | x q , D ) .
For the pseudo-code of the algorithm for training and inference of Meta-GMVAE, please see the Algorithm 1 and 2 in the Appendix.
Visual feature reconstruction While our method is a generative model that can generate samples from output distribution, the ability to generate samples may not be necessary for discriminative downstream tasks (Chen et al., 2020).
Moreover, it is known to be challenging to train generative models in high-dimensional data (e.g. Mini-ImageNet).
In preliminary experiments, we found that
VAE almost fails to learn in Mini-ImageNet dataset with the architecturally limited constraints of the meta-learning literature.
Thus, we propose a high-level feature reconstruction objective instead for Mini-ImageNet dataset.
We experimentally ﬁnd that the recently proposed constrastive learning framework, namely SimCLR (Chen et al., 2020), is the most effective for our settings.
Speciﬁcally,
SimCLR learns high-level representation by performing a constrastive prediction task on pairs of augmented examples derived from a minibatch.
We train SimCLR on the unsupervised dataset
D u = { x u } Uu =1 , and use high-level features extracted by SimCLR as an input for our framework.
We also refer the reader to Chen et al.
(2020) for more details.
E XPERIMENT
In this section, we now validate the effectiveness of our Meta-GMVAE on several downstream fewshot classiﬁcation tasks.
4. E XPERIMENTAL S ETUPS
Baselines and ours We now describe two supervised meta-learning approaches which we consider as “oracles”, unsupervised meta-learning baselines, and the proposed Meta-GMVAE.
1) MAML (oracle) :
Model Agnostic Meta Learning by Finn et al.
We compare against its performance reported in Hsu et al.
2) ProtoNets (oracle) :
Euclidean distance-based meta-learning approach by Snell et al.
We also compare against it using its performance reported in Hsu et al.
3) CACTUs : Clustering to Automatically Construct Tasks for Unsupervised meta-learning by Hsu et al.
It automatically constructs tasks by clustering the unsupervised dataset in embedding space learned by ACAI (Berthelot et al., 2019), BiGAN (Donahue et al., 2017), and DeepCluster (Caron et al., 2018).
Then they train either MAML or ProtoNets using the cluster indices as pseudo-labels.
4) UMTRA :
Unsupervised Meta-learning with Tasks constructed by Random sampling and Augmentation by Khodadadeh et al.
For constructing a K-way 1-shot task, it randomly samples K-way datapoints from unsupervised dataset and augments each datapoint.
Then
MAML is trained on the constructed tasks.
Meta-GMVAE :
Our proposed Meta-level Gaussian
Mixture VAE.
It learns a latent representation by matching set-level amortized variational posterior and task-speciﬁc multimodal prior optimized by EM algorithm.
Datasets We validate all models on two benchmark datasets for few-shot classiﬁcation.
Omniglot : This is a collection of 28 × 28 gray-scale hand-written characters that describe 1623 different alphabets, each of which contains 20 instances.
Following the experimental setup of Hsu et al.
(2019), we use 1200 classes for unsupervised meta-training, 100 classes for meta-validation and the remaining 323 classes for meta-test.
We further augment each class by rotating the images 90, 180, and 270 degrees, such that the total number of classes is 1623 × 4, following the convention.
Mini-ImageNet : This is a subset of ILSVRC-2012 (Deng et al., 2009) introduced by
Ravi & Larochelle (2017), consisting of 100 classes that comes with 600 images of size 84 ×that describe different instances.
We use 64 classes for unsupervised meta-training, 16 classes for meta-validation, and the remaining 20 classes for meta-test, following the standard protocol.
Implementation details We now introduce the speciﬁc implementation details of Meta-GMVAE on the two benchmark datasets.
Variational posterior network q φ ( z | x , D i ) : we use the standard
Conv4 architecture on Omniglot dataset for a fair comparison against relevant baselines.
On top of the Conv4 architecture, we stack two TransformerEncoder layers and an afﬁne transformation layer to predict the mean and log-variance of Gaussian distribution.
For Mini-ImageNet dataset, we only utilize two Transformer encoder layers and an afﬁne transformation layer since the input used for Mini-ImageNet is already a high-level visual representation extracted from the Conv5 architecture trained with SimCLR.
For both datasets, we set the dimensionality of the latent variable to 64 . 2) Generative network p θ ( x | z ) :
For Omniglot dataset, the architecture of generative network is symmetric to the Conv4 architecture of variational posterior network.
The last layer predicts the parameter of output Bernoulli distribution.
For Mini-ImageNet dataset, we use 3-layer MLP with ReLU activation to predict the mean of output Gaussian distribution.
The variance of output
Gaussian distribution is obtained as suggested in Rybkin et al.
Other details : we utilize
Adam optimizer (Kingma & Ba, 2015) with a constant learning rate of 0.0001 to optimize the generative and variational parameters.
We set the number of iterations for EM algorithm as 5 for all the experiments.
For the more details, please see the Appendix.
[Table]
[Figure] [Figure]
[Figure] [Figure]
[Figure]
4. E XPERIMENTAL R ESULTS
Table 1 shows the few-shot classiﬁcation results obtained by supervised meta-learning baselines (oracle), the two unsupervised meta-learning baselines, and our Meta-GMVAE.
For the Omniglot dataset, the Meta-GMVAE outperforms all the baselines that only utilize unsupervised-learning for constructing meta-training tasks, except for the UMTRA on the Omniglot dataset 20-shot 5-shot classiﬁcation.
Meta-GMVAE also outperforms baselines on Mini-ImageNet 1-shot, and 5-shot settings which are the most widely used settings, while it matches the performance of baselines in 20-shot, and 50-shot settings.
This shows that meta-learning the posterior network can capture the multi-modal distribution of any given tasks with Meta-GMVAE, is indeed more effective over unsupervised meta-learning baselines which simply trains supervised meta-learning models with pseudolabels obtained from unlabeled data.
Moreover, our Meta-GMVAE obtains better performance than supervised MAML on Omniglot 5-way 1-shot classiﬁcation, while utilizing as small as 0 .
1% of the labeled data.
This matches the observation in Chen et al.
(2020) that well-calibrated unsupervised learning approaches with a modest amount of labels can obtain a performance comparable to or even better than supervised approaches.
To better understand how our Meta-GMVAE learns and realizes class-concepts in few-shot classiﬁcation tasks, we visualize the actual samples in an episode and ones generated by generative network p θ ( x | z ) , during unsupervised meta-training and supervised meta-testing.
Speciﬁcally, for a given episode dataset D i , we obtain the task-speciﬁc parameter ψ ∗ i using either Eq 7 or Eq 10.
Then we classify the actual samples using Eq 11, and generate samples by feeding each latent representation drawn from p ψ ∗
i ( z | y = k ) into generative network p θ ( x | z ) .
Then we visualize the actual samples and generated ones that have a same modality in a same row.
In Figure
3- Left , we can observe that our Meta-GMVAE captures the similar visual structure in each modality during meta-training, but the modalities are not the class-concepts.
However, as shown Figure
3- Right , our Meta-GMVAE easily realizes each modality as each class-concept with only few examples at meta-test time.
Furthermore, we compare the performance of our model variants by eliminating each of the most important components for our model.
We describe the each variant as follows: 1) LR (SimCLR) :
This performs the logistic regression using support set on top of features pretrained by SimCLR.
blue2) Vanilla VAE : We train Vanilla VAE on D u and predict labels using semi-supervised EM with ﬁxed identity covariance I . 3) Vanilla VAE (SimCLR) :
This is same as ) Vanilla VAE
[Table]
except that it is trained on features pretrained by SimCLR. 4) Ep : Meta-GMVAE with an episodic training with task speciﬁc parameter ψ ∗ i obtained by EM. 5) Set : Meta-GMM whether having setlevel variational posterior (i.e. q φ ( z | x , D i ) ) or not (i.e. q φ ( z | x ) ).
6) σ 2 : Meta-GMM performs semi-supervised EM algorithm whether using diagonal covariance matrix or ﬁxing it with identity matrix I .
Table
2- Left shows that all the components we consider are critical for the performance on the few-shot classiﬁcation tasks as expected.
The best performance gain comes from Ep , which supports our proposal on meta-learning the set-level variational posterior by matching it with the multi-modal prior, where the task-speciﬁc parameter is obtained with EM. [Figure]
[Figure]
We then experiment our Meta-GMVAE by varying the number of way (between 2 and 50) and ﬁxing the number of shot as 1.
In particular, we set the number of component k as the Test way for the meta-test and perform semi-supervised EM algorithm in Eq 10.
Table
2- Right shows that the difference in the number of way used for training and test does not signiﬁcantly affect the performance, which demonstrates the robustness of Meta-GMVAE on varying number of way at test time.
We also visualize the latent space for the cross-shot experiment using t-SNE (Rauber et al., 2016), in Figure 4, which shows that Meta-GMVAE trained with 20-way can cluster 5-way meta-test task.
C ONCLUSION
We proposed a novel unsupervised meta-learning model, namely Meta-GMVAE, which can generate a task-dependent posterior for a given unseen task with multi-modal Gaussian Mixture priors.
Given a random episode that consists of samples from diverse classes, we optimize the task-speciﬁc parameter of the mixture of Gaussian prior with Expectation-Maximization algorithm, such that each mode can capture intrinsic groupings in the given data.
We meta-train the variational posterior network over such data-driven prior obtained over large number of episodes.
Then, at the metatest step, we realize each modality with a label by deploying semi-supervised EM algorithm with both the support and the query set.
We validate our method on two few-shot image classiﬁcation benchmark datasets, and show that Meta-GMVAE largely outperforms the relevant unsupervised meta-learning baselines, even achieving better performance than supervised MAML on Omniglot 5-way 1-shot experiments.
We further show that Meta-GMVAE can generalize across classiﬁcation tasks with different number of ways (classes).
As future work, we plan to extend our model to the zero-shot generation to unseen class-concepts.
A T HE P SEUDO - CODE OF THE A LGORITHM FOR M ETA -GMVAE
We provide the pseudo-code of the algorithm for our Meta-GMVAE at meta-training and meta-test step in Algorithm 1 and Algorithm 2.
Algorithm 1 Meta-training
B O MNIGLOT E XPERIMENTS B. 95% C ONFIDENCE I NTERVAL
We provide the standard errors of our model’s performance at 95% conﬁdence interval overepisodes on the Omniglot dataset in Table 3. Omniglot (5,1) (5,5) (
[Table]
B. T RAINING P ROCEDURE
Omniglot is a collection of 28 × 28 gray-scale hand-written characters that describe 1623 different alphabets, each of which contains 20 instances.
Following the experimental setup of Hsu et al.
(2019), we use 1200 classes for unsupervised meta-training, 100 classes for meta-validation and the remaining 323 classes for meta-test.
We further augment each class by rotating the images 90, 180, and 270 degrees, such that the total number of classes is 1623 × 4, following the convention.
We evaluate the trained model uisng 1000 randomly selected tasks from test set.
During evaluation,
K × S data instances are used as support inputs and K × 15 data instances are used as query inputs.
We use the Adam (Kingma & Ba, 2015) optimizer with a constant learning rate of 0.0001 to train all models.
All models are trained for 60,000 iterations.
For the 5-way experiments (i.e. K = 5 ), we set the mini-batch size, the number of datapoints, and Monte Carlo sample size as 4, 200, and 32, respectively (i.e. B = 4 , M = 200 , and N = 32 ).
For the 20-way experiments (i.e. K = 20 ), we set them as 4, 300, and 32 (i.e. B = 4 , M = 300 , and N = 32 ).
We set the number of EM iterations as 5.
B. N ETWORK A RCHITECTURE
We summarize the network architecture in the following Table 4, and 5.
We assume that the output follows Bernoulli distribution, therefore, the output of generative network p θ ( x | z ) is the mean parameter. [Table] [Table]
[Table]
2, padding
[Table]
), Sigmoid
C M INI -I MAGE N ET E XPERIMENTS C. 95% C ONFIDENCE I NTERVAL
We provide the standard errors at 95% conﬁdence interval over 1000 episodes on the Mini-ImageNet dataset in Table 6. [Table]
C. A DDITIONAL C OMPARISON USING S IM CLR
To further understand where the improvement of Meta-GMVAE on the Mini-ImageNet dataset
[Table]
we ran experiment on CACTUs clustered in the embedding space pretrained by SimCLR.
Moreover, we ﬁx the pretrained SimCLR features as the setting of Meta-GMVAE.
Table 7 shows that MetaGMVAE outperforms CACTUs-MAML baseline with SimCLR
[Table]
which supports the effectiveness of
Meta-GMVAE combined with SimCLR pretrained features
[Table]
C. T RAINING P ROCEDURE
Mini-ImageNet is a subset of ILSVRC-2012 (Deng et al., 2009) introduced by Ravi & Larochelle (2017), consisting of 100 classes that comes with 600 images of size 84 × 84 that describe different instances.
We ﬁrst train Conv5 feature extractor using SimCLR objective with temperature term τ = 0 .
5 , on Mini-ImageNet unsupervised meta-training dataset.
We train the feature extractor using
Adam optimizer with learning rate of 0.0001 for 400 epochs.
We use 64 classes for unsupervised meta-training, 16 classes for meta-validation, and the remaining 20 classes for meta-test, following the standard protocol.
We evaluate the trained model using 1000 randomly selected tasks from test set.
During evaluation, 5 × S data instances are used as support inputs and 5 × 15 data instances are used as query inputs.
For all the experiments, we use the Adam (Kingma & Ba, 2015) optimizer with a constant learning rate of 0.0001, and set the mini-batch size, the number of datapoints, and
Monte Carlo sample size as 4, 5, and 32, respectively (i.e. B
= 4 , M = 5 , and N = 32 ).
For the 1, 5, 20, 50-shot experiments (i.e. S = 1 , 5 ), we train the models for 3K, 20K, 25K, and 30K, respectively. We set the number of EM iterations as 5.
C. N ETWORK A RCHITECTURE
We summarize the network architecture in the following Table 8, 9, and 10.
We assume that the output follows Gaussian distribution, therefore, the output of generative network p θ ( x | z ) is the mean parameter.
Moreover, the variance of output Gaussian distribution is obtained as suggested in
Rybkin et al.
[Table]
[Table] [Table]