Original: U NSUPERVISED M ETA -L EARNING THROUGH L ATENTS PACE I NTERPOLATION IN G ENERATIVE M ODELS
Revised: U NSUPERVISED M ETA -L EARNING THROUGH L ATENTS PACE I NTERPOLATION IN G ENERATIVE M ODELS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Several recently proposed unsupervised meta-learning approaches rely on synthetic meta-tasks created using techniques such as random selection, clustering and/or augmentation.
Revised: Several recently proposed unsupervised meta-learning approaches rely on synthetic meta-tasks created using techniques such as random selection, clustering and/or augmentation.

Original: In this work, we describe a novel approach that generates metatasks using generative models.
Revised: In this work, we describe a novel approach that generates metatasks using generative models.

Original: The proposed family of algorithms generate pairs of in-class and out-of-class samples from the latent space in a principled way, allowing us to create synthetic classes forming the training and validation data of a meta-task.
Revised: The proposed family of algorithms generate pairs of in-class and out-of-class samples from the latent space in a principled way, allowing us to create synthetic classes forming the training and validation data of a meta-task.

Original: We ﬁnd that the proposed approach, LAtent Space Interpolation
Revised: We ﬁnd that the proposed approach, LAtent Space Interpolation

Original: Unsupervised Meta-learning (LASIUM), outperforms or is competitive with current unsupervised learning baselines on few-shot classiﬁcation tasks on the most widely used benchmark datasets
Revised: Unsupervised Meta-learning (LASIUM), outperforms or is competitive with current unsupervised learning baselines on few-shot classiﬁcation tasks on the most widely used benchmark datasets

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Few-shot meta-learning algorithms for neural networks such as Mishra et al.
Revised: Few-shot meta-learning algorithms for neural networks such as Mishra et al.

Original: Finn et al.
Revised: Finn et al.

Original: Snell et al.
Revised: Snell et al.

Original: (2017) prepare networks to quickly adapt to unseen tasks.
Revised: (2017) prepare networks to quickly adapt to unseen tasks.

Original: This is done in a meta-training phase that typically involves a large number of supervised learning tasks.
Revised: This is done in a meta-training phase that typically involves a large number of supervised learning tasks.

Original: Generating supervised tasks depends on large, labeled datasets and hand-speciﬁed task distribution.
Revised: Generating supervised tasks depends on large, labeled datasets and hand-speciﬁed task distribution.

Original: Very recently, several approaches had been proposed that perform the meta-training by generating synthetic training tasks from an unsupervised dataset.
Revised: Very recently, several approaches had been proposed that perform the meta-training by generating synthetic training tasks from an unsupervised dataset.

Original: This requires us to generate samples with speciﬁc pairwise information: in-class pairs of samples that are with high likelihood in the same class, and out-of-class pairs that are with high likelihood not in the same class.
Revised: This requires us to generate samples with speciﬁc pairwise information: in-class pairs of samples that are with high likelihood in the same class, and out-of-class pairs that are with high likelihood not in the same class.

Original: For instance, UMTRA by Khodadadeh et al.
Revised: For instance, UMTRA by Khodadadeh et al.

Original: (2019) and AAL by Antoniou & Storkey (2019) achieve this through random selection from a domain with many classes for out-of-class pairs and by augmentation for in-class pairs.
Revised: (2019) and AAL by Antoniou & Storkey (2019) achieve this through random selection from a domain with many classes for out-of-class pairs and by augmentation for in-class pairs.

Original: Hsu et al.
Revised: Hsu et al.

Original: (2019) propose CACTUs that creates synthetic labels through unsupervised clustering of the domain.
Revised: (2019) propose CACTUs that creates synthetic labels through unsupervised clustering of the domain.

Original: In this paper, we rely on recent advances in the ﬁeld of generative models, such as the variants of generative adversarial networks (GANs) and variational autoencoders (VAEs), to generate the in-class and out-of-class pairs of meta-training data.
Revised: In this paper, we rely on recent advances in the ﬁeld of generative models, such as the variants of generative adversarial networks (GANs) and variational autoencoders (VAEs), to generate the in-class and out-of-class pairs of meta-training data.

Original: The fundamental idea of our approach is that in-class pairs are close while out-of-class pairs are far away in the latent space representation of the generative model.
Revised: The fundamental idea of our approach is that in-class pairs are close while out-of-class pairs are far away in the latent space representation of the generative model.

Original: Thus, we can generate in-class pairs by interpolating between two out-of-class samples in the latent space and choosing interpolation ratios that put the new sample close to one of the objects.
Revised: Thus, we can generate in-class pairs by interpolating between two out-of-class samples in the latent space and choosing interpolation ratios that put the new sample close to one of the objects.

Original: From this latent sample, the generative model creates the new in-class object.
Revised: From this latent sample, the generative model creates the new in-class object.

Original: Our approach requires little domain-speciﬁc tweaking, and the necessary tweaks are human-comprehensible.
Revised: Our approach requires little domain-speciﬁc tweaking, and the necessary tweaks are human-comprehensible.

Original: For instance, we need to choose thresholds for latent space distance that ensure that classes are in different domains, as well as interpolation ratio thresholds that ensure that the sample is in the same class as the nearest edge.
Revised: For instance, we need to choose thresholds for latent space distance that ensure that classes are in different domains, as well as interpolation ratio thresholds that ensure that the sample is in the same class as the nearest edge.

Original: Another advantage of the approach is that we can utilize off-the-shelf, pre-trained generative models.
Revised: Another advantage of the approach is that we can utilize off-the-shelf, pre-trained generative models.

Original: The main contributions of this paper can be summarized as follows:
Revised: The main contributions of this paper can be summarized as follows:

Original: • We describe an algorithm, LAtent Space Interpolation Unsupervised Meta-learning (LASIUM), that creates training data for a downstream meta-learning algorithm starting from an unlabeled dataset by leveraging interpolation in the latent space of a generative model.
Revised: • We describe an algorithm, LAtent Space Interpolation Unsupervised Meta-learning (LASIUM), that creates training data for a downstream meta-learning algorithm starting from an unlabeled dataset by leveraging interpolation in the latent space of a generative model.

Original: • We show that on the most widely used few-shot learning datasets, LASIUM outperforms or performs competitively with other unsupervised meta-learning algorithms, signiﬁcantly outperforms transfer learning in all cases, and in a number of cases approaches the performance of supervised meta-learning algorithms.
Revised: • We show that on the most widely used few-shot learning datasets, LASIUM outperforms or performs competitively with other unsupervised meta-learning algorithms, signiﬁcantly outperforms transfer learning in all cases, and in a number of cases approaches the performance of supervised meta-learning algorithms.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Meta-learning or “learning to learn” in the ﬁeld of neural networks is an umbrella term that covers a variety of techniques that involve training a neural network over the course of a meta-training phase, such that when presented with the target task, the network is able to learn it much more efﬁciently than a randomly initialized network would.
Revised: Meta-learning or “learning to learn” in the ﬁeld of neural networks is an umbrella term that covers a variety of techniques that involve training a neural network over the course of a meta-training phase, such that when presented with the target task, the network is able to learn it much more efﬁciently than a randomly initialized network would.

Original: Such techniques had been proposed since the 1980s (Schmidhuber (1987); Bengio et al.
Revised: Such techniques had been proposed since the 1980s (Schmidhuber (1987); Bengio et al.

Original: ; Naik & Mammone (1992); Thrun & Pratt (1998)).
Revised: ; Naik & Mammone (1992); Thrun & Pratt (1998)).

Original: In recent years, meta-learning has gained a resurgence, through approaches that either “learn to optimize” (Finn et al.
Revised: In recent years, meta-learning has gained a resurgence, through approaches that either “learn to optimize” (Finn et al.

Original: Ravi & Larochelle (2016); Mishra et al.
Revised: Ravi & Larochelle (2016); Mishra et al.

Original: ; Nichol et al.
Revised: ; Nichol et al.

Original: Rusu et al.
Revised: Rusu et al.

Original: ; Rajeswaran et al.
Revised: ; Rajeswaran et al.

Original: or learn embedding functions in a non-parametric setting (Snell et al.
Revised: or learn embedding functions in a non-parametric setting (Snell et al.

Original: (2017); Vinyals et al.
Revised: (2017); Vinyals et al.

Original: (2016a)

Original: ; Ren et al.
Revised: ; Ren et al.

Original: Liu et al.
Revised: Liu et al.

Original: Hybrids between these two approaches had also been proposed (Triantaﬁllou et al.
Revised: Hybrids between these two approaches had also been proposed (Triantaﬁllou et al.

Original: ; Wang et al.
Revised: ; Wang et al.

Original: Most approaches use labeled data during the meta-learning phase.
Revised: Most approaches use labeled data during the meta-learning phase.

Original: While in some domains there is an abundance of labeled datasets, in many domains such labeled data is difﬁcult to acquire.
Revised: While in some domains there is an abundance of labeled datasets, in many domains such labeled data is difﬁcult to acquire.

Original: Unsupervised meta-learning approaches aim to learn from an unsupervised dataset from a domain similar from that of the target task.
Revised: Unsupervised meta-learning approaches aim to learn from an unsupervised dataset from a domain similar from that of the target task.

Original: Typically these approaches generate synthetic few-shot learning tasks for the meta-learning phase through a variety of techniques.
Revised: Typically these approaches generate synthetic few-shot learning tasks for the meta-learning phase through a variety of techniques.

Original: CACTUs (Hsu et al.
Revised: CACTUs (Hsu et al.

Original: (2019)) uses a progressive clustering method.
Revised: (2019)) uses a progressive clustering method.

Original: UMTRA (Khodadadeh et al.
Revised: UMTRA (Khodadadeh et al.

Original: (2019)) utilizes the statistical diversity properties and domain-speciﬁc augmentations to generate synthetic training and validation data.
Revised: (2019)) utilizes the statistical diversity properties and domain-speciﬁc augmentations to generate synthetic training and validation data.

Original: AAL (Antoniou & Storkey (2019)) uses augmentation of the unlabeled training set to generate the validation data.
Revised: AAL (Antoniou & Storkey (2019)) uses augmentation of the unlabeled training set to generate the validation data.

Original: The accuracy of these approaches was shown to be comparable with but lower than supervised meta-learning approaches, though with the advantage of requiring orders of magnitude less labeled training data.
Revised: The accuracy of these approaches was shown to be comparable with but lower than supervised meta-learning approaches, though with the advantage of requiring orders of magnitude less labeled training data.

Original: A common weakness of these approaches is that the techniques used to generate the synthetic tasks (clustering, augmentation, random sampling) are highly domain dependent.
Revised: A common weakness of these approaches is that the techniques used to generate the synthetic tasks (clustering, augmentation, random sampling) are highly domain dependent.

Original: Our proposed approach, LASIUM, takes advantage of generative models trained on the speciﬁc domain to create the in-class and out-of-class pairs of meta-training data.
Revised: Our proposed approach, LASIUM, takes advantage of generative models trained on the speciﬁc domain to create the in-class and out-of-class pairs of meta-training data.

Revised: By creating new training data through interpolation between training samples, LASIUM-OC has similarities with mixup Zhang et al.

Original: The most successful neural-network based generative models in recent years are variational autoencoders (VAE) (Diederik & Welling (2014)) and generative adversarial networks (GANs) (Goodfellow et al.
Revised: The most successful neural-network based generative models in recent years are variational autoencoders (VAE) (Diederik & Welling (2014)) and generative adversarial networks (GANs) (Goodfellow et al.

Original: The implementation variants of the LASIUM algorithm described in this paper rely on the original VAE model and on two speciﬁc variations of the GAN concept, respectively.
Revised: The implementation variants of the LASIUM algorithm described in this paper rely on the original VAE model and on two speciﬁc variations of the GAN concept, respectively.

Original: MSGAN (aka MissGAN) (Mao et al.
Revised: MSGAN (aka Miss-GAN) (Mao et al.

Original: (2019)) aims to solve the missing mode problem of conditional GANs through a regularization term that maximizes the distance between the generated images with respect to the distance between their corresponding input latent codes.
Revised: (2019)) aims to solve the missing mode problem of conditional GANs through a regularization term that maximizes the distance between the generated images with respect to the distance between their corresponding input latent codes.

Original: Progressive GANs (Karras et al.
Revised: Progressive GAfNs (Karras et al.

Original: (2018)) are growing both the generator and discriminator progressively, and approach resembling the layer-wise training of autoencoders.
Revised: (2018)) are growing both the generator and discriminator progressively, and approach resembling the layer-wise training of autoencoders.

Original:  L ATENT SPACE INTERPOLATION UNSUPERVISED META - LEARNING
Revised:  L ATENT SPACE INTERPOLATION UNSUPERVISED META - LEARNING

Original: Preliminaries : We deﬁne an N -way, K ( tr ) -shot supervised classiﬁcation task, T , as a set D ( tr ) T composed of i ∈ { 1 , . . .
Revised: Preliminaries : We deﬁne an N -way, K ( tr ) -shot supervised classiﬁcation task, T , as a set D ( tr ) T composed of i ∈ { 1 , . . .

Original: , N × K ( tr ) } data points ( x i , y i ) such that there are exactly K ( tr ) samples for each categorical label y i ∈ { 1 , . . . , N } .
Revised: , N × K ( tr ) } data points ( x i , y i ) such that there are exactly K ( tr ) samples for each categorical label y i ∈ { 1 , . . . , N } .

Original: During meta-learning, an additional set , D ( val ) T , is attached to each task that contains another N × K ( val ) data points separate from the ones in D ( tr ) T .
Revised: During meta-learning, an additional set , D ( val ) T , is attached to each task that contains another N × K ( val ) data points separate from the ones in D ( tr ) T .

Original: We have exactly K ( val ) samples for each class in D ( val ) T as well.
Revised: We have exactly K ( val ) samples for each class in D ( val ) T as well.

Original: It is straightforward to package N -way, K ( tr ) -shot tasks with D ( tr ) T and D ( val ) T from a labeled dataset.
Revised: It is straightforward to package N -way, K ( tr ) -shot tasks with D ( tr ) T and D ( val ) T from a labeled dataset.

Original: However, in unsupervised meta-learning setting, a key challenge is how to automatically construct tasks from the unlabeled dataset U = { x i } .
Revised: However, in unsupervised meta-learning setting, a key challenge is how to automatically construct tasks from the unlabeled dataset U = { x i } .

Original: 3. G ENERATING META - TASKS USING GENERATIVE MODELS
Revised: 3. G ENERATING META - TASKS USING GENERATIVE MODELS

Original: We have seen that in order to generate the training data for the meta-learning phase, we need to generate N -way training tasks with K ( tr ) training and K ( val ) validation samples.
Revised: We have seen that in order to generate the training data for the meta-learning phase, we need to generate N -way training tasks with K ( tr ) training and K ( val ) validation samples.

Original: The label associated with the classes in these tasks is not relevant, as it will be discarded after the metalearning phase.
Revised: The label associated with the classes in these tasks is not relevant, as it will be discarded after the metalearning phase.

Original: Our objective is simply to generate samples of the type x i,j with i ∈ { 1 . . .
Revised: Our objective is simply to generate samples of the type x i,j with i ∈ { 1 . . .

Original: N } and j ∈ { 1 . . .
Revised: N } and j ∈ { 1 . . .

Original: K ( tr ) + K ( val ) } with the following properties: (a) all the samples x i,j are different (b) anytwo samples with the same i index are in-class samples and (c) any two samples with different i index
Revised: K ( tr ) + K ( val ) } with the following properties: (a) all the samples x i,j are different (b) any

Original: [Figure]
Revised: [Figure]

Original: generation 𝒢 ...
Revised: generation 𝒢 ...

Original: K (tr) ...
Revised: K (tr) ...

Original: K (val) (b) In-class sampling zz ′
Revised: K (val) (b) In-class sampling zz ′

Original: z ′
Revised: z ′

Original: z ′
Revised: z ′

Original: 𝓖 (z i ) 𝓖 (z ′i ) 𝓖 (z ″i ) z
Revised: 𝓖 (z i ) 𝓖 (z ′i ) 𝓖 (z ″i ) z

Original: ″ z ″ z ″ ...
Revised: ″ z ″ z ″ ...

Original: N  are out-of-class samples.
Revised: N indicates two samples with the same i index are in-class samples and (c) any two samples with different i index are out-of-class samples.

Original: In the absence of human provided labels, the class structure of the domain is deﬁned only implicitly by the sample selection procedure.
Revised: In the absence of human provided labels, the class structure of the domain is deﬁned only implicitly by the sample selection procedure.

Original: Previous approaches to unsupervised meta-learning chose samples directly from the training data x i,j ∈ U , or created new samples through augmentation.
Revised: Previous approaches to unsupervised meta-learning chose samples directly from the training data x i,j ∈ U , or created new samples through augmentation.

Original: For instance, we can deﬁne the class structure of the domain by assuming that certain types of augmentations keep the samples in-class with the original sample.
Revised: For instance, we can deﬁne the class structure of the domain by assuming that certain types of augmentations keep the samples in-class with the original sample.

Original: One challenge of such approaches is that the choice of the augmentation is domain dependent, and the augmentation itself can be a complex mathematical operation.
Revised: One challenge of such approaches is that the choice of the augmentation is domain dependent, and the augmentation itself can be a complex mathematical operation.

Original: In this paper we approach the sample selection problem differently.
Revised: In this paper, we approach the sample selection problem differently.

Original: Instead of sampling x i,j from U , we use the unsupervised dataset to train a generative model p ( x ) .
Revised: Instead of sampling x i,j from U , we use the unsupervised dataset to train a generative model p ( x ) .

Original: Generative models represent the full probability distribution of a model and allow us to sample new instances from the distribution.
Revised: Generative models represent the full probability distribution of a model, and allow us to sample new instances from the distribution.

Original: For many models, this sampling process can be computationally expensive iterative process.
Revised: For many models, this sampling process can be computationally expensive iterative process.

Original: Many successful neural network based generative models use the reparametrization trick for the training and sampling which concentrate the random component of the model in a latent representation z .
Revised: Many successful neural network based generative models use the reparametrization trick for the training and sampling which concentrate the random component of the model in a latent representation z .

Original: By choosing the latent representation z from a simple (uniform or normal) distribution, we can obtain a sample from the complex distribution p ( x ) by passing z through a deterministic generator
Revised: By choosing the latent representation z from a simple (uniform or normal) distribution, we can obtain a sample from the complex distribution p ( x ) by passing z through a deterministic generator

Original: G ( Z ) → X .
Revised: G ( Z ) → X .

Original: Two of the most popular generative models, variational autoencoders (VAEs) and generative adversarial networks (GANs) follow this model.
Revised: Two of the most popular generative models, variational autoencoders (VAEs) and generative adversarial networks (GANs) follow this model.

Original: The idea of the LASIUM algorithm is that given a generator G ( . ) , nearby latent space values z 1 and z 2 map to in-class samples x 1 and x 2 that belong to the same class.
Revised: The idea of the LASIUM algorithm is that given a generator G ( . ) , nearby latent space values z 1 and z 2 map to in-class samples x 1 and x 2 that belong to the same class.

Original: Conversely, z 1 and z 2 values that are far away from each other, map to out of class samples that belong to different classes.
Revised: Conversely, z 1 and z 2 values that are far away from each other, map to out of class samples that belong to different classes.

Original: Naturally, we still need to deﬁne what we mean by “near” and “far” in the latent space and how to choose the corresponding z values.
Revised: Naturally, we still need to deﬁne what we mean by “near” and “far” in the latent space and how to choose the corresponding z values.

Original: However, this is a signiﬁcantly simpler task than, for instance, deﬁning the set of complex augmentations that might retain class membership.
Revised: However, this is a signiﬁcantly simpler task than, for instance, deﬁning the set of complex augmentations that might retain class membership.

Original: Training a generative model Our method for generating meta-tasks is agnostic to the choice of training algorithm for the generative model and can use either a VAE or a GAN with the only constraint of having appropriately structured latent space.
Revised: Training a generative model Our method for generating meta-tasks is agnostic to the choice of training algorithm for the generative model and can use either a VAE or a GAN with the only constraint of having appropriately structured latent space.

Original: In our VAE experiments, we used a network trained with the standard VAE training algorithm (Diederik & Welling (2014)).
Revised: In our VAE experiments, we used a network trained with the standard VAE training algorithm (Diederik & Welling (2014)).

Original: For the experiments with GANs we used two different methods mode seeking GANs (MSGAN)
Revised: For the experiments with GANs we used two different methods mode seeking GANs (MSGAN)

Original: (Mao et al.
Revised: (Mao et al.

Original: and progressive growing of GANs (proGAN) (Karras et al.
Revised: and progressive growing of GANs (ProGAN) (Karras et al.

Original: MSGAN is trained for
Revised: MSGAN is trained for

Original: Omniglot and proGAN is trained for CelebA.
Revised: Omniglot and ProGAN is trained for CelebA.

Original: Algorithm 1 describes the steps of our method.
Revised: Algorithm 1 describes the steps of our method.

Original: We will delve into each step in the following parts of this section.
Revised: We will delve into each step in the following parts of this section.

Original: Sampling out of class instances from the latent space representation: Our sampling techniques differ slightly whether we are using a GAN or VAE.
Revised: Sampling out of class instances from the latent space representation: Our sampling techniques differ slightly whether we are using a GAN or VAE.

Original: For GAN, we use rejection sampling to ﬁnd
Revised: For GAN, we use rejection sampling to ﬁnd

Original: N latent space vectors that are at a pairwise distance of at least threshold (cid:15) - see Figure 1(a).
Revised: N latent space vectors that are at a pairwise distance of at least threshold (cid:15) - see Figure 1(a).

Original: When using a VAE, we also have an encoder network that allows us to map from the domain to the latentspace.
Revised: When using a VAE, we also have an encoder network that allows us to map from the domain to the latent

Original: Taking advantage of this, we can additionally sample data points from our unlabeled dataset

Original: [Figure]
Revised: [Figure]

Original:  require : Unlabeled dataset U = { x i } , Pre-trained generatorG require :
Revised: Algorithm require : Unlabeled dataset U = { x i } , Pre-trained generator

Original:  Sampling Strategy S : could be N, RO, or OC require : K ( tr ) , K ( val ) : number of samples for train and validation during meta-learningrequire : N : class-count, N MB : meta-batch size 1 B = {} ; // meta-batch of tasks 2 for i in 1 , . . .
Revised: G require : N : class-count, N MB : meta-batch size, (cid:15) : minimum threshold, σ or α : sampling hyperparameters require : Sampling Strategy S : could be N( σ 2 ), RO( α ), or OC( α ) require : K ( tr ) , K ( val ) : number of samples for train and validation during meta-learning 1 B = {} ; // meta-batch of tasks 2 for i in 1 , . . .

Original: , N MB do
Revised: , N MB do

Original: Sample N class-vectors in latent space of G and add them to task-vectors  for ω in 1 , . . 
Revised: Sample N class-vectors in latent space of G and add them to task-vectors until they are at least (cid:15) units away from each other in euclidean for ω in 1 , . . .

Original: , K ( tr ) + K ( val ) − 1 do
Revised: , K ( tr ) + K ( val ) − 1 do

Original: Generate new-vectors =
Revised: Generate new-vectors =

Original: S (class-vectors, ω ) and add them to task-vectors end
Revised: S (class-vectors, ω ) and add them to task-vectors end

Original: Generate N × ( K ( tr ) + K ( val ) ) images by feeding task-vectors to generator G
Revised: Generate N × ( K ( tr ) + K ( val ) ) images by feeding task-vectors to generator G

Original: Construct task T i by putting the ﬁrst N × K ( tr ) images in task train set and the last
Revised: Construct task T i by putting the ﬁrst N × K ( tr ) images in task train set and the last

Original: N × K ( val ) images in task validation set
Revised: N × K ( val ) images in task validation set

Original: B ← B ∪ T
Revised: B ← B ∪ T

Original: i 10 end 11 return B 
Revised: i 10 end 11 return B space.

Revised: Taking advantage of this, we can additionally sample data points from our unlabeled dataset

Original: U and embed them into a latent space.
Revised: U and embed them into a latent space.

Original: If the latent space representation of these N images are too close to each other, we re-sample, otherwise we can use the N images and their representations and continue the following steps exactly the same as GANs - see Figure 2(a) and (b).
Revised: If the latent space representation of these N images are too close to each other, we re-sample, otherwise we can use the N images and their representations and continue the following steps exactly the same as GANs - see Figure 2(a) and (b).

Original: We will refer to the vectors selected here as anchor vectors.
Revised: We will refer to the vectors selected here as anchor vectors.

Original: Generating in-class latent space vectors Next, having N sampled anchor vectors { z 1 , . .
Revised: Generating in-class latent space vectors Next, having N sampled anchor vectors { z 1 , . .

Original: . , z N } from the latent space representation, we aim to generate N new vectors { z (cid:48) 1 , . . .
Revised: . , z N } from the latent space representation, we aim to generate N new vectors { z (cid:48) 1 , . . .

Original: , z (cid:48) N } from the latent space representation such that the generated image G ( z i ) belongs to the same class as the one of
Revised: , z (cid:48) N } from the latent space representation such that the generated image G ( z i ) belongs to the same class as the one of

Original: G ( z (cid:48) i ) for i ∈ 1 , . . .
Revised: G ( z (cid:48) i ) for i ∈ 1 , . . .

Original: , N .
Revised: , N .

Original: This process needs to be repeated for K ( tr ) + K ( val ) − 1 times.
Revised: This process needs to be repeated for K ( tr ) + K ( val ) − 1 times.

Original: The sampling strategy takes as input the sampled vectors and a number ω ∈ { 1 . . .
Revised: The sampling strategy takes as input the sampled vectors and a number ω ∈ { 1 . . .

Original: K ( tr ) + K ( val ) − 1 } and returns N new vectors such that z
Revised: K ( tr ) + K ( val ) − 1 } and returns N new vectors such that z

Original: i
Revised: i

Original: and z (cid:48)
Revised: and z (cid:48)

Original: i are an in-class pair for i ∈ { 1 . . .
Revised: i are an in-class pair for i ∈ { 1 . . .

Original: N } .
Revised: N } .

Original: This ensures that no two z (cid:48)
Revised: This ensures that no two z (cid:48)

Original: i belong to the same class and creates N groups of ( K ( tr ) + K ( val ) ) vectors in our latent space.
Revised: i belong to the same class and creates N groups of ( K ( tr ) + K ( val ) ) vectors in our latent space.

Original: We feed these vectors to our generator to get N groups of ( K ( tr ) + K ( val ) ) images.
Revised: We feed these vectors to our generator to get N groups of ( K ( tr ) + K ( val ) ) images.

Original: From each group we pick the ﬁrst K ( tr ) for D ( tr ) T and the last K ( val ) for D ( val ) T .
Revised: From each group we pick the ﬁrst K ( tr ) for D ( tr ) T and the last K ( val ) for D ( val ) T .

Original: What remains is to deﬁne the strategy to sample the individual in-class vectors.
Revised: What remains is to deﬁne the strategy to sample the individual in-class vectors.

Original: We propose three different sampling strategies, all of which can be seen as variations of the idea of latent space  interpolation sampling.
Revised: We propose three different sampling strategies, all of which can be seen as variations of the idea of latent space [Figure] interpolation sampling.

Original: This motivates the name of the algorithm LAtent Space Interpolation
Revised: This motivates the name of the algorithm LAtent Space Interpolation

Original: Unsupervised Meta-learning (LASIUM).
Revised: Unsupervised Meta-learning (LASIUM).

Original: LASIUM-N (adding Noise) :
Revised: LASIUM-N (adding Noise) :

Original: This technique generates in-class samples by adding Gaussian noise to the anchor vector z (cid:48)
Revised: This technique generates in-class samples by adding Gaussian noise to the anchor vector z (cid:48)

Original: i =
Revised: i =

Original: z i + (cid:15) where (cid:15) ∼ N (0 , σ 2 ) (see Figure 3-Left).
Revised: z i + (cid:15) where (cid:15) ∼ N (0 , σ 2 ) (see Figure 3-Left).

Original: In the context of LASIUM, we[Figure] can see this as an interpolation between the anchor vector and a noise vector, with the interpolation factor determined by σ .
Revised: In the context of LASIUM, we can see this as an interpolation between the anchor vector and a noise vector, with the interpolation factor determined by σ .

Original: For the impact of different choices of σ see the ablation study in the supplemental material.
Revised: For the impact of different choices of σ see the ablation study in the supplemental material.

Original: LASIUM-RO (with Random Out-of-class samples)
Revised: LASIUM-RO (with Random Out-of-class samples)

Original: To generate a new in-class sample to anchor vector
Revised: To generate a new in-class sample to anchor vector

Original: z i we
Revised: z i we

Original: ﬁrst ﬁnd a random out-of-class sample v i , and choose an interpolated version closer to the anchor: z (cid:48)
Revised: ﬁrst ﬁnd a random out-of-class sample v i , and choose an interpolated version closer to the anchor: z (cid:48)

Original: i =
Revised: i =

Original: z i + α × ( v i − z i ) (see Figure 3-Middle).
Revised: z i + α × ( v i − z i ) (see Figure 3-Middle).

Original: Here, α is a hyperparameter, which can be tuned to deﬁne the size of the class.
Revised: Here, α is a hyperparameter, which can be tuned to deﬁne the size of the class.

Original: As we are in a comparatively high-dimensional latent space (in our case, 512 dimensions), we need relatively large values of α , such as α = 0 .
Revised: As we are in a comparatively high-dimensional latent space (in our case, 512 dimensions), we need relatively large values of α , such as α = 0 .

Original: 4 to deﬁne classes of reasonable size.
Revised: 4 to deﬁne classes of reasonable size.

Original: This model effectively allows us to deﬁne complex augmentations (such as a person seen without glasses, or in a changed lighting) with only one scalar hyperparameter to tune.
Revised: This model effectively allows us to deﬁne complex augmentations (such as a person seen without glasses, or in a changed lighting) with only one scalar hyperparameter to tune.

Original: By interpolating towards another sample we ensure that we are staying on the manifold that deﬁnes the dataset (in the case of Figure 3, this being human faces).
Revised: By interpolating towards another sample we ensure that we are staying on the manifold that deﬁnes the dataset (in the case of Figure 3, this being human faces).

Original: LASIUM-OC (with Other Classes’ samples)
Revised: LASIUM-OC (with Other Classes’ samples)

Original: This technique is similar to LASIUM-RO, but instead of using a randomly generated out-of-class vector, we are interpolating towards vectors already chosen from the other classes in the same task (see Figure 3-Right).
Revised: This technique is similar to LASIUM-RO, but instead of using a randomly generated out-of-class vector, we are interpolating towards vectors already chosen from the other classes in the same task (see Figure 3-Right).

Original: This limits the selection of the samples to be conﬁned to subspace of the convex hull containing the initial anchor points.
Revised: This limits the selection of the samples to be conﬁned to subspace of the convex hull containing the initial anchor points.

Original: The intuition behind this approach is that choosing the samples this way focuses the attention of the meta-learner towards the hard to distinguish samples that are between the classes in the few shot learning class (eg.
Revised: The intuition behind this approach is that choosing the samples this way focuses the attention of the meta-learner towards the hard to distinguish samples that are between the classes in the few shot learning class (eg.

Original: they share certain attributes).
Revised: they share certain attributes).

Original: Finally, failure cases exist for these sampling strategies.
Revised: Finally, failure cases exist for these sampling strategies.

Original: For example, if z 1 , z 2 , and z 3 lie on a line, we end up generating poor out of class samples that belong to other classes in our task.
Revised: For example, if z 1 , z 2 , and z 3 lie on a line, we end up generating poor out of class samples that belong to other classes in our task.

Original: However, as we are in high dimensional latent space(e.g. 512), the likelihood for three points being approximately colliniar is very low (as the volume fraction in which the middle point would have to be shrinks exponentially with the dimensionality).
Revised: However, as we are in high dimensional latent space(e.g. 512), the likelihood for three points being approximately colliniar is very low (as the volume fraction in which the middle point would have to be shrinks exponentially with the dimensionality).

Original: Note that meta-learning algorithms are not sensitive to a small fraction of bad training examples in the meta-training phase (e.g. see Hsu et al.
Revised: Note that meta-learning algorithms are not sensitive to a small fraction of bad training examples in the meta-training phase (e.g. see Hsu et al.

Original: ; Khodadadeh et al.
Revised: ; Khodadadeh et al.

Original:  E XPERIMENTAL VALIDATION
Revised:  E XPERIMENTAL VALIDATION

Original: We tested the proposed algorithms on three few-shot learning benchmarks: (a) the 5 -way Omniglot
Revised: We tested the proposed algorithms on three few-shot learning benchmarks: (a) the 5 -way Omniglot

Original: (Lake et al.
Revised: (Lake et al.

Original: (2011)), a benchmark for few-shot handwritten character recognition, (b) the 5 -way CelebA few-shot identity recognition, and (c) the CelebA attributes dataset (Liu et al.
Revised: (2011)), a benchmark for few-shot handwritten character recognition, (b) the 5 -way CelebA few-shot identity recognition, and (c) the CelebA attributes dataset (Liu et al.

Original: proposed as a few-shot learning benchmark by (Hsu et al.
Revised: proposed as a few-shot learning benchmark by (Hsu et al.

Original: The latter benchmark comprises binary classiﬁcation ( 2 -way) tasks in which each task is deﬁned by selecting 3 different attributes and 3 boolean values corresponding to each attribute.
Revised: The latter benchmark comprises binary classiﬁcation ( 2 -way) tasks in which each task is deﬁned by selecting 3 different attributes and 3 boolean values corresponding to each attribute.

Original: Every image in a certain task-speciﬁc class has the same attributes with each other while does not share any of these attributes with images in the other class.
Revised: Every image in a certain task-speciﬁc class has the same attributes with each other while does not share any of these attributes with images in the other class.

Original: We partition each dataset into meta-training, meta-validation, and meta-testing splits between classes.
Revised: We partition each dataset into meta-training, meta-validation, and meta-testing splits between classes.

Original: To evaluate our method, we use the classes in the test set to generate 1000 tasks as described in section 3.
Revised: To evaluate our method, we use the classes in the test set to generate 1000 tasks as described in section 3.

Original: We set K ( val ) to be 15.
Revised: We set K ( val ) to be 15.

Original: Note that unlike meta-training, where we use synthetic generated tasks, for evaluation we use 1000 tasks from real unseen labeled data.
Revised: Note that unlike meta-training, where we use synthetic generated tasks, for evaluation we use 1000 tasks from real unseen labeled data.

Original: Furthermore, we ﬁx the random seed to make sure we compare on the exact same tasks with other baselines.
Revised: Furthermore, we ﬁx the random seed to make sure we compare on the exact same tasks with other baselines.

Original: We average the accuracy on all tasks and report a 95% conﬁdence interval.
Revised: We average the accuracy on all tasks and report a 95% conﬁdence interval.

Original: To ensure that comparisons are fair, we use the same random seed in the whole task generation process.
Revised: To ensure that comparisons are fair, we use the same random seed in the whole task generation process.

Original: For the Omniglot dataset, we report the results for K ( tr ) ∈ { 1 , 5 } , and K ( val ) = 15 .
Revised: For the Omniglot dataset, we report the results for K ( tr ) ∈ { 1 , 5 } , and K ( val ) = 15 .

Original: For CelebA identity recognition, we report our results for K ( tr ) ∈ { 1 , 5 , 15 } and K ( val ) = 15 .
Revised: For CelebA identity recognition, we report our results for K ( tr ) ∈ { 1 , 5 , 15 } and K ( val ) = 15 .

Original: For CelebA attributes, we follow the
Revised: For CelebA attributes, we follow the

Original: K ( tr ) = 5 and K ( val )
Revised: K ( tr ) = 5 and K ( val )

Original: = 5 tasks as proposed by Hsu et al.
Revised: = 5 tasks as proposed by Hsu et al.

Original: More ablation studies over the hyperparameters and result visualizations are provided in the supplemental material.
Revised: More ablation studies over the hyperparameters and result visualizations are provided in the supplemental material.

Original: Since excessive tuning of hyperparameters can lead to the overestimation of the performance of a model (Oliver et al.
Revised: Since excessive tuning of hyperparameters can lead to the overestimation of the performance of a model (Oliver et al.

Original: , we keep the hyperparameters of the unsupervised meta-learning as constant as possible (including the MAML, and ProtoNets model architectures) in all experiments.
Revised: , we keep the hyperparameters of the unsupervised meta-learning as constant as possible (including the MAML, and ProtoNets model architectures) in all experiments.

Original: Our model architecture consists of four stacked convolutional blocks.
Revised: Our model architecture consists of four stacked convolutional blocks.

Original: Each block comprises 64 ﬁlters that carry out 3 ×convolutions, followed by batch normalization, a ReLU non-linearity, and 2 × 2 max-pooling.
Revised: Each block comprises 64 ﬁlters that carry out 3 ×convolutions, followed by batch normalization, a ReLU non-linearity, and 2 × 2 max-pooling.

Original: For the
Revised: For the

Original: MAML experiments, classiﬁcation is performed by a fully connected layer, whereas for the ProtoNets model we compute distances based on the feature vectors produced by the last convolution module without any dense layers.
Revised: MAML experiments, classiﬁcation is performed by a fully connected layer, whereas for the ProtoNets model we compute distances based on the feature vectors produced by the last convolution module without any dense layers.

Original: The input size to our model is 84 × 84 × 3 for CelebA and 28 × 28 × 1 for
Revised: The input size to our model is 84 × 84 × 3 for CelebA and 28 × 28 × 1 for

Original: Omniglot.
Revised: Omniglot.

Original: The detail of the neural networks architectures for each experiment is described in the supplemental material.
Revised: The detail of the neural networks architectures for each experiment is described in the supplemental material.

Original: 4. B ASELINES
Revised: 4. B ASELINES

Original: As baseline algorithms for our approach we follow the practice of recent papers in the unsupervised meta-learning literature.
Revised: As baseline algorithms for our approach we follow the practice of recent papers in the unsupervised meta-learning literature.

Original: The simplest baseline is to train the same network architecture from scratch with N × K ( tr ) images.
Revised: The simplest baseline is to train the same network architecture from scratch with N × K ( tr ) images.

Original: More advanced baselines can be obtained by learning an unsupervised embedding on U and use it for downstream task training.
Revised: More advanced baselines can be obtained by learning an unsupervised embedding on U and use it for downstream task training.

Original: We used the ACAI (Berthelot et al.
Revised: We used the ACAI (Berthelot et al.

Original: BiGAN (Donahue et al.
Revised: BiGAN (Donahue et al.

Original: ; Dumoulin et al.
Revised: ; Dumoulin et al.

Original: (2017)), and DeepCluster (Caron et al.
Revised: (2017)), and DeepCluster (Caron et al.

Original: as representative of the unsupervised learning literature.
Revised: as representative of the unsupervised learning literature.

Original: On top of these embeddings, we report accuracy for K nn -nearest neighbors, linear classiﬁer, multi layer perceptron (MLP) with dropout, and cluster matching.
Revised: On top of these embeddings, we report accuracy for K nn -nearest neighbors, linear classiﬁer, multi layer perceptron (MLP) with dropout, and cluster matching.

Original: The direct competition for our approach are the current state-of-the-art algorithms in unsupervised meta-learning.
Revised: The direct competition for our approach are the current state-of-the-art algorithms in unsupervised meta-learning.

Original: We compare our results with CACTUs-MAML, CACTUs-ProtoNets (Hsu et al.
Revised: We compare our results with CACTUs-MAML, CACTUs-ProtoNets (Hsu et al.

Original: and UMTRA Khodadadeh et al.
Revised: and UMTRA Khodadadeh et al.

Original: Finally, it is useful to compare our approach with algorithms that require supervised data.
Revised: Finally, it is useful to compare our approach with algorithms that require supervised data.

Original: We include results for supervised standard transfer learning from VGG19 pre-trained on ImageNet (Simonyan & Zisserman (2015)) and two supervised meta-learning algorithms, MAML (Finn et al.
Revised: We include results for supervised standard transfer learning from VGG19 pre-trained on ImageNet (Simonyan & Zisserman (2015)) and two supervised meta-learning algorithms, MAML (Finn et al.

Original: (2017)), and ProtoNets (Snell et al.
Revised: (2017)), and ProtoNets (Snell et al.

Original: 4. R ESULTS ON O MNIGLOT
Revised: 4. R ESULTS ON O MNIGLOT

Original: Table 1 shows the results on the Omniglot dataset.
Revised: Table 1 shows the results on the Omniglot dataset.

Original: We ﬁnd that the LASIUM-RO-GAN-MAML conﬁguration outperforms all the unsupervised approaches, including the meta-learning based ones like CACTUs and UMTRA.
Revised: We ﬁnd that the LASIUM-RO-GAN-MAML conﬁguration outperforms all the unsupervised approaches, including the meta-learning based ones like CACTUs and UMTRA.

Original: Beyond the increase in performance, we must note that the competing approaches use more domain speciﬁc knowledge (in case of UMTRA augmentations, in case of

Original: CACTUs, learned clustering).

Original: Besides, when it comes to augmentations, the (one-dimensional) search for the intra-class shift was much cheaper than ﬁne-tuning the currently popular augmentation strategies.
Revised: Beyond the increase in performance, we must note that the (onedimensional) search for the intra-class shift was much cheaper than ﬁne-tuning the currently popular augmentation strategies.

Original: We also ﬁnd that on this benchmark, LASIUM outperforms transfer learning using the much larger VGG-19 network. 
Revised: We also ﬁnd that on this benchmark, LASIUM outperforms transfer learning using the much larger VGG-19 network. [Table] [Table]

Original: As expected even the best LASIUM result is worse than the supervised meta-learning models.
Revised: As expected even the best LASIUM result is worse than the supervised meta-learning models.

Original: However, we need to consider that the unsupervised meta-learning approaches use several orders[Table] [Table] of magnitude less labels.
Revised: However, we need to consider that the unsupervised meta-learning approaches use several orders of magnitude less labels.

Original: For instance, the 95.29% accuracy of LASIUM-RO-GAN-MAML was obtained with only 25 labels, while the supervised approaches used 25,000.
Revised: For instance, the 95.29% accuracy of LASIUM-RO-GAN-MAML was obtained with only 25 labels, while the supervised approaches used 25,000.

Original: 4. R ESULTS ON C ELEB A
Revised: 4. R ESULTS ON C ELEB A

Original: Table 2 shows our results on the CelebA identity recognition tasks where the objective is to recognize
Revised: Table 2 shows our results on the CelebA identity recognition tasks where the objective is to recognize

Original: N different people given K ( tr ) images for each.
Revised: N different people given K ( tr ) images for each.

Original: We ﬁnd that on this benchmark as well, the LASIUMRO-GAN-MAML conﬁguration performs better than other unsupervised meta-learning models as well as transfer learning with VGG-19 - it only falls slightly behind LASIUM-RO-GAN-ProtoNets on the one-shot case.
Revised: We ﬁnd that on this benchmark as well, the LASIUMRO-GAN-MAML conﬁguration performs better than other unsupervised meta-learning models as well as transfer learning with VGG-19 - it only falls slightly behind LASIUM-RO-GAN-ProtoNets on the one-shot case.

Original: As we have discussed in the case of Omniglot results, the performance remainslower then the supervised meta-learning approaches which use several orders of magnitude more labeled data.
Revised: As we have discussed in the case of Omniglot results, the performance remains

Original: [Table]
Revised: [Table]

Original: Finally
Revised: lower then the supervised meta-learning approaches which use several orders of magnitude more labeled data

Original: [Table]
Revised: [Table]

Original:  Table 3 shows our results for CelebA attributes benchmark introduced in (Hsu et al.
Revised: Finally, Table 3 shows our results for CelebA attributes benchmark introduced in (Hsu et al.

Original: A peculiarity of this dataset is that the way in which classes are deﬁned based on the attributes, the classes are unbalanced in the dataset, making the job of synthetic task selection more difﬁcult.
Revised: A peculiarity of this dataset is that the way in which classes are deﬁned based on the attributes, the classes are unbalanced in the dataset, making the job of synthetic task selection more difﬁcult.

Original: We ﬁnd that LASIUM-N-GAN-MAML obtainsthe second best on this test with a performance of 74 .
Revised: We ﬁnd that LASIUM-N-GAN-MAML obtains a performance of 75 .

Original: 79 ± 1 . 01 , within the conﬁdence interval of the winner, CACTUs MAML with BiGAN 74 .
Revised: 07 ± 1 . 08 , within the conﬁdence interval of the second best, CACTUs MAML with BiGAN 74 .

Original: 98 ± 1 . 02 .
Revised: 98 ± 1 . 02 .

Original: In this benchmark, transfer learning with the VGG-19 network performed better than all unsupervised meta-learning approaches, possibly due to existing representations of the discriminating attributes in that much more complex network.
Revised: In this benchmark, transfer learning with the VGG-19 network performed better than all unsupervised meta-learning approaches, possibly due to existing representations of the discriminating attributes in that much more complex network.

Original: D ISCUSSION
Revised: 4. R ESULTS ON M INI -I MAGE N ET

Revised: Table 4 shows the comparison results on the Mini-ImageNet benchmark. On this benchmark, we used as the GAN a pre-trained BigBiGAN 1 . Examples of meta-training tasks constructed by LASIUM-N with σ 2 = 1 . 0 are shown in Figure 4.

Original: We described LASIUM, an unsupervised meta-learning algorithm for few-shot classiﬁcation.
Revised: We notice that LASIUM-N-GAN-MAML outperforms all other unsupervised learning algorithms for K=1, 5 and 20. For K=50, it is in the second place behind

Original: The algorithm is based on interpolation in the latent space of a generative model to create synthetic meta-tasks.
Revised: CACTUs MAML with DeepCluster, the accuracy difference being within the margin of error. Train

Original: In our experiments, we saw that augmentation based methods signiﬁcantly boost the performance in comparison with clustering based methods on simple tasks that have different instances of an equivalence class (e.g. class is a given handwritten letter).
Revised: Validation ... [Table]

Original:  However, the performance gets worse on more natural datasets where we face with more varied domains like CelebA dataset that has the same person but with a lot of variation in clothing, hairstyle, background, and so on.
Revised:  D ISCUSSION

Original: Clustering based approaches could also be seen as generating tasks based on notion of distance in  latent space to create tasks. A conceptual difference between clustering and LASIUM is that clustering relies on encoding whereas LASIUM depends on decoding. LASIUM is limited by generative models to produce high-ﬁdelity samples and appropriately structured latent space.
Revised: We described LASIUM, an unsupervised meta-learning algorithm for few-shot classiﬁcation. The algorithm creates synthetic meta-tasks using interpolation in the latent space of a generative model with the general idea that points that are close in the latent space will likely generate in-class samples, while points far in the latent space will likely generate out-of-class samples.

Original: The former is shared with clustering based unsupervised learning since they also rely on learning good unsupervisedfeature extractors. Interestingly, training of these feature extractors is based on augmentations like random crop and random ﬂips (Caron et al.
Revised: In this, LASIUM differs from techniques such as UMTRA and AAL that generate in-class samples through pixelspace augmentation and techniques such as CACTUs which uses clustering to separate unsupervised training data into in-class and out-of-class samples.

Original: or auto encoders (Berthelot et al. Finally, we found that LASIUM outperforms  state-of-the-art unsupervised algorithms on the Omniglot and
Revised: We found that LASIUM outperforms or comes within the margin of error to state-of-the-art unsupervised algorithms on Omniglot, Mini-ImageNet,

Original: CelebA identity recognition benchmarks andcompetes very closely with CACTUs on the CelebA attributes learning benchmark.
Revised: CelebA attributes learning benchmark and the CelebA identity recognition benchmarks.