Diversity vs. Recognizability: Human-like generalization in one-shot generative models
AfﬁliationAddress
Abstract
Robust generalization to new concepts has long remained a distinctive feature ofhuman intelligence.
However, recent progress in deep generative models has nowled to neural architectures capable of synthesizing novel instances of unknownvisual concepts from a single training example.
Yet, a more precise comparisonbetween these models and humans is not possible because existing performancemetrics for generative models (i.e., FID, IS, likelihood) are not appropriate forthe one-shot generation scenario.
Here, we propose a new framework to evaluateone-shot generative models along two axes: sample recognizability vs. diversity(i.e.
, intra-class variability).
Using this framework, we perform a systematic evaluation of representative one-shot generative models on the Omniglot handwrittendataset.
We ﬁrst show that GAN-like and VAE-like models fall on opposite endsof the diversity-recognizability space.
Extensive analyses of the effect of keymodel parameters further revealed that spatial attention and context integrationhave a linear contribution to the diversity-recognizability trade-off.
In contrast,disentanglement transports the model along a parabolic curve that could be used tomaximize recognizability.
Using the diversity-recognizability framework, we wereable to identify models and parameters that closely approximate human data.
Introduction
Our ability to learn and generalize from a limited number of samples is a hallmark of human cognition.
In language, scientists have long highlighted how little training data children need in comparison tothe richness and complexity of the language they learn so efﬁciently to master [12, 40].
Similarly,children and adults alike are able to learn novel object categories from as little as a single trainingexample [17, 8].
From a computational point of view, such feats are remarkable because they suggestthat learners must be relying on inductive biases to overcome such challenges [33] – from an abilityto detect suspicious coincidences or ’non-accidental’ features [43, 52] to exploiting the principle ofcompositionality [32, 33].
While a common criticism of modern AI approaches is their reliance on large training datasets,progress in one-shot categorization has been signiﬁcant.
One-shot categorization involves predictingan image category based on a unique training sample per class.
Multiple algorithms have beenproposed including meta-learning algorithms [18, 46, 14, 37] or metric-learning algorithms [47, 31,48] that are now starting to approach human accuracy.
Perhaps a less studied problem is the one-shotgeneration problem – aimed at creating new variations of a prototypical shape seen only once.
Sincethe seminal work of Lake et al.
[32] who introduced the Bayesian Program Learning algorithm, onlya handful of promising one-shot generative algorithms have been proposed [42, 15, 1] (see section 2.2for a more exhaustive description of prior work).
Why have so few algorithms for one-shot image generation vs. image categorization been proposed?
We argue that one of the main reasons for this lack of progress is the absence of an adequate evaluationmetric.
As of today, one-shot generative models are evaluated using methods initially developed formodels producing samples that belong to the training categories and trained on large datasets.
Thosemetrics include the likelihood, the FID (Frechet Inception Distance), or the IS (Inception Score).
In the one-shot image generation scenario in which training images are scarce and the generatedsamples represent new visual concepts, the likelihood, the FID, and the IS are biased [3, 13, 38](see 2.1 for more details).
These limitations urge us to look for new metrics tailored for one-shotimage generation.
Recent psychophysics work [52] has characterizedhumans’ ability for one-shot generation along twomain axes: samples diversity (i.e. intra-class vari
ability) and samples recognizability (i.e., how easy orhard they are to classify).
According to this frame
work, ideal generalization corresponds to a combination of high recognizability and high diversity.
Asillustrated in Fig.
1, an ideal model should be ableto generate samples that span the entire space withinthe decision boundary of a classiﬁer (Box 1 ).
In comparison, the model of Box 2 has learned to makeidentical copies of the prototype (i.e., low diversitybut high accuracy).
Such a model has failed to generalize the visual concept exempliﬁed by the prototype.
Similarly, if the model’s samples are so diverse thatthey cannot be recognized accurately as shown in the
Box 3 of Fig.
1, then the generated samples won’tlook like the prototype.
Here, we borrow from this work and adapt it to createthe ﬁrst framework to evaluate and compare humansand one-shot generative models.
Using this framework, we systematically evaluate an array of representative one-shot generative models on the Omniglotdataset [32].
We show that GAN-like and VAE-likeone-shot generative models fall on opposite ends ofthe diversity-recognizability space: GAN-like modelsfall on the high recognizability — low diversity endof the space while VAE-like models fall on the lowrecognizability — high diversity end of the space.
We further study some key model parameters thatmodulate spatial attention, context integration, and disentanglement.
Our results suggest that spatialattention and context have an (almost) linear effect on the diversity vs. recognizability trade-off.
Incontrast, varying the disentanglement moves the models on a parabolic curve that could be used tomaximize the recognizability.
Last but not least, we have leveraged the diversity vs. recognizabilityspace to identify models and parameters that best approximate the human data on the Omniglothandwritten dataset.
Related work2.1 Metrics to evaluate generative models and their limitations for one-shot generation tasks
Different types of generative models are typically evaluated using different metrics.
On the onehand, likelihood-based algorithms (e.g., VAE [29], PixelCNN [39], GLOW [30], etc.) are evaluatedusing their own objective function applied on a testing set.
Likelihood provides a direct estimate ofthe KL divergence between the data points and the model’s samples.
On the other hand, implicitgenerative models such as Generative Adversarial Networks (GANs) [20] for which the loss functioncannot be used, are typically evaluated using other scores such as the Inception Score (IS) [45] or
Frechet Inception Distance (FID) [22].
IS and FID are heuristic measures used to aggregate both thesample quality and diversity in one single score.
The IS scores a sample quality according to the conﬁdence with which an Inception v3 Net [49] assigns the correct class label to it.
The FID score isthe Wasserstein-2 distance between two Gaussian distributions: one ﬁtted on the features of the datadistribution and the other on the features of the model distribution (the features are also extractedfrom an Inception v3 Network).
All these metrics are problematic for the one-shot generation scenario for 2 main reasons that areintrinsically related to the task: the low number of samples per class, and the dissimilarity betweentraining and testing visual concepts.
IS and FID rely on statistical distances (either KL divergencefor IS or Wasserstein-2 distance for FID) that require a high number of data points N to produce anunbiased estimator of the distance.
Even when used in the traditional settings (i.e., N = 50000 ), ithas been demonstrated that both scores are biased [13].
This is to be compared with the N = 20samples typically available in popular few-shot learning datasets such as Omniglot [32]. Another [Table]
problem caused by the limited number of samples per class in the training set is the overﬁtting of [Table] the Inception Net used to extract the features to compute the IS and FID [7]. [Table]
Such [Table] phenomenon
[Table] is [Table] The second limitation of these metrics appears because the training and the testing samples are toodissimilar.
Likelihood scores are known to yield higher scores for out-of-domain data compared toin-domain data [38].
Therefore, the evaluation of the novel visual concepts generated by one-shotgenerative models will be biased toward higher scores.
In addition, both FID and IS rely on distancebetween features extracted by an Inception Net which comes with no guarantee that it will producemeaningful features for novel categories.
For example, class misalignment has been reported when the
Inception Net was trained on ImageNet and tested on CIFAR10 [3].
Because of all the aforementionedlimitations, it is pretty clear that new procedures are needed to evaluate the performance of few-shotgenerative algorithms.
2.2 One-shot generative models
One can distinguish between two broad classes of one-shot generative models: structured modelsand statistical models [16].
Structured models have strong inductive biases and rigid parametricassumptions based on a priori knowledge such as for example a given hierarchy of features, a knowngrammar or program [44].
A prominent example of a structured model includes the very ﬁrst algorithmfor one-shot image generation, the Bayesian Program Learning (BPL) model [32].
Statistical modelslearn visual concepts by learning statistical regularities between observed patterns [42, 15, 19].
Here, we focus on representative architectures of one-shot generative statistical models, which wesummarize below.
• VAE with Spatial Transformer Network ( VAE-STN ) [42].
The VAE-STN is a sequential andconditional Variational Auto-Encoder (VAE) constructing images iteratively.
The VAE-STNalgorithm uses a recurrent neural network (i
e., an LSTM) to encode the sequence of localpatches extracted by an attentional module.
A key ingredient of the VAE-STN is an attentionmodule composed of a Spatial Transformer Network (STN) [26] to learn to shift attention todifferent locations of the input image.
The STN is a trainable module to learn all possibleafﬁne transformations (i.e., translation, scaling, rotation, shearing) of an input image (see S8 forsamples and details of the VAE-STN ).
• Neural statistician ( VAE-NS ) [15]: The Neural Statistician is an extension of the conditional
VAE model including contextual information.
Therefore, in addition to learning an approximateinference network over latent variables for every image in the set (as done in a VAE), the
approximate inference is also implemented over another latent variable, called the context
variable, that is speciﬁc to the considered visual concept.
The context inference network is fed
with a small set of images representing variations of a given visual concept.
The VAE-NS has
been extended to include attention and hierarchical factorization of the generative process [19]
(see S9 for samples and details of the VAE-NS ).
• Data-Augmentation GAN (DA-GAN) [1]: Data-Augmentation GAN is a generative adversarial
network conditioned on a prototype image.
The DA-GAN generator is fed with a concatenation
of a vector drawn from a normal distribution and a compressed representation of the prototype.
The discriminator is trained to differentiate images produced by the generator from images
of the dataset
, while the generator has to fool the discriminator.
We have trained 2 different
DA-GAN, one is based on the U-Net architecture ( DA-GAN-UN ) and the other one on the
ResNet architecture ( DA-GAN-RN ) (see S10 for samples and details of the DA-GAN-UN andS11 for samples and details of the DA-GAN-RN ).
All these models are generative models conditioned by an image prototype extracted from the trainingor the test set.
The way we have selected the prototypes is detailed in Eq.
To the best of ourknowledge, these models offer a representative set of one-shot generative models.
We have reproducedall these models (sometimes with our own implementation when it was not available online).
Our codecould be found at https://anonymous.4open.science/r/diversity_recognizability/ .
The diversity vs. accuracy framework
Let { x ji } be a dataset composed of K concepts (i.e., classes) with N samples each ( i ∈ [[1 , N ]] andj ∈ [[1 , K ]] ).
The framework we propose aims at evaluating the performance of a generative modelp θ , parameterized by θ , that produces new images v ji based on a single sample (or prototype) of aconcept given to the generator ˜ x
j (i.e., v ji ∼ p θ ( ·| ˜ x j ) ).
For each concept j , we deﬁne a prototype asthe sample closest to the center of mass for the concept j : [Equation] i [Equation]
[Equation]
In Eq.
1, f denotes a function that projects the input image from the pixel space to a feature space.
We will detail the feature extractor f shortly.
Note that this deﬁnition of a prototype is not unique(one could also select the prototype randomly within individual classes j ), nevertheless this selectionmechanism is a guarantee that the selected sample will be representative of the concept.
Dataset.
In this article, we use the Omniglot dataset [32] with a weak generalization split [42].
Omniglot is composed of binary images representing 1 , 623 classes of handwritten letters and symbols(extracted from 50 different alphabets) with just 20 samples per class.
We have downsampled theoriginal dataset to be 50 × 50 pixels.
The weak generalization split consists of a training set composedof all available symbols minus 3 symbols per alphabet which are left aside for the test set.
It is said tobe weak because all the alphabets were shown during the training process (albeit not all symbols inthose alphabets).
As the Omniglot dataset is hand-written by humans, we consider that these samplesreﬂect a human generative process and we refer to this later as the human model.
Diversity.
In the proposed framework, diversity refers to the intra-class variability of the samplesproduced by a generative model p θ ( ·| ˜ x j ) .
For a given prototype ˜ x j , we compute the diversity as thestandard deviation of the generated samples in the feature space f :
[Equation]
We use the Bessel-corrected standard deviation to keep a good estimate of the data dispersion despitethe relatively small number of samples (e.g., N = 20 for the Omniglot dataset used here).
To verifythat this diversity measure is robust to the speciﬁc choice of the feature extractor f , we explored twodifferent settings: features learned with class supervision by a Prototypical Net [47] and featureslearned with self-supervision by a SimCLR network [10].
In both cases, we extracted the featuresfrom the ﬁrst fully-connected layer following the last convolutional layer.
The Prototypical Netwas optimized so that images that belong to the same category share similar latent representationsas measured by the (cid:96) 2 -norm.
Similarly, SimCLR leverages a contrastive loss to deﬁne a latentrepresentation such that a sample is more similar to its augmented version than to other imagesamples.
In SimCLR, this similarity is computed with cosine similarity.
These two approachesrepresent two ends of a continuum of methods to learn suitable representational spaces without theneed to explicitly learn to classify images and are thus more suitable for few-shot learning tasks [35].
For the sake of comparison, we have used the exact same network architecture for both featureextractors (see sections S1 and S2 for more details on Prototypical Net and SimCLR, respectively).
We computed the samples diversity for all 150 categories of the Omniglot test set (i.e., v ji = x ji in thisexperiment) using both the supervised and unsupervised settings.
We found a high linear correlation( ρ = 0 .
86 , p-value < 10 − 5 ) and a high rank-order Spearman correlation ( ρ = 0 .
85 , p-value < 10 − 5 )between the two settings (see section S3.1).
Hence, the two feature extraction methods producecomparable diversity measures and henceforth, we will report results using the unsupervised setting.
As an additional control, we have also veriﬁed that the SimCLR metric is robust to changes to theaugmentation method used (see section S3.3) and to the speciﬁc choice of the dispersion metric (seesection S3.2 for more details on this comparison).
We have compared the feature space of the Prototypical Net and SimCLR using a t-SNE analysis (see section S3.4).
We observed a strong clusteringof samples belonging to the same category for both networks.
It suggests that the augmentationmethods used by the SimCLR contrastive loss are sufﬁcient to disentangle the class information.
Fig.
2 shows the 10 concepts from the Omniglot test set with the lowest and highest samplesdiversity, respectively (for more diversity-ranked concepts with unsupervised or supervised setting,see sections S4 and S5, respectively).
One can see that the proposed diversity metric is qualitativelysimilar to human judgment.
Concepts with low diversity are composed of very few relatively basicstrokes (e.g., lines, dots, etc) with little room for any kind of “creativity” in the generation processwhile more diverse concepts are composed of more numerous and more complex stroke combinationswith many more opportunities for creativity.
[Figure]
Recognizability.
We evaluate the recognizability of the samples produced by the one-shot generative models by leveraging one-shot classiﬁcation models.
(cid:58)(cid:58)(cid:58) Note (cid:58)(cid:58)(cid:58)(cid:58) that (cid:58)(cid:58)(cid:58)(cid:58) one (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) could (cid:58)(cid:58)(cid:58)(cid:58) [Table] proposed [Table]
in [Table] (cid:58)(cid:58) In order to make sure our classiﬁcation accuracy measure is robustto the choice of the model, we test different models which belong to the two main approaches used inmachine learning for one-shot classiﬁcation: metric learning and meta-learning [35].
We selected the Prototypical Net [47] as a representative metric-learning approach and the Model-Agnostic MetaLearning (MAML) [18] model as a representative meta-learning approach.
Both models were trainedand tested in a 1-shot 20-ways setting (see section S6 for more details on the MAML architectureand training details).
We report a high Pearson (negative) correlation between the logits produced by
Prototypical Net and MAML ( ρ = − 0 .
60 , p-value < 10 − 5 ) as well as a strong Spearman rank-ordercorrelation between the classiﬁcation accuracy of both networks ( ρ = 0 .
62 , p-value < 10 − 5 ).
Seesection S7 for more details about this control experiment.
Hence, our recognizability metric is robustto the choice of the one-shot classiﬁcation model (even when those models are leveraging differentapproaches) and henceforth, we will report results using the Prototypical Net model.
Results
4.1 GAN-like vs. VAE-like models
For all algorithms listed in section 2.2 we have explored different hyper-parameters (see section 4.2for more details), leading to various models represented in the diversity vs. recognizability plotin Fig. 3a.
In this ﬁgure, we have reduced each model to a single point by averaging the diversityand recognizability over all classes of the Omniglot testing set.
The black star corresponds tothe human model, and colored data
We observe that the GAN-like models (i.e., DA-GAN-UN and DA-GAN-RN ) tend to be located atthe upper left side of the graph while VAE-like models (i.e., VAE-NS and VAE-STN ) spread on theright side of the graph.
Therefore, the GAN-like models produce very recognizable samples that arehighly similar to each other (high recognizability and low diversity).
In contrast, VAE-like modelsgenerate more diverse but less recognizable samples.
The samples in Fig.
3b illustrate this observation.
The difference between GAN and VAE-like samples could be explained by their loss functions [36].
The GANs’ adversarial loss tends to drop some of the modes of the training distribution.
In general,the distribution learned by GANs put excessive mass on the more likely modes but discards secondarymodes [2].
This phenomenon leads to sharp and recognizable generations at the cost of reducedsamples diversity.
On the other hand, VAEs (and likelihood-based models in general) are sufferingfrom over-generalization: they cover all the modes of the training distribution and put mass inspurious regions [4].
We refer the reader to Fig.
4 of Lucas et al.
[36] for an illustration of modedropping in GANs and over-generalization in VAEs.
Our diversity vs. recognizability plot in Fig.
3ashows that this phenomenon is holding even when the testing distribution is different from the trainingdistribution as in the case of the one-shot generation scenario.
[Figure]
base architectures (corresponding to the bigger circles in Fig. 3a).
Prototypes for individual concepts are shown within a red box next to actual class samples.
4.2 VAE-NS vs. VAE-STN
In this section we compare some key hyper-parameters of the VAE-NS and VAE-STN .
The coreidea of the VAE-NS is to integrate context information to the sample generation process.
Duringtraining, the context is composed of several samples that all represent variations of the same visualconcept.
During the testing phase, the VAE-NS infers the context statistics using a single image(i.e.
, the prototype).
We evaluate the effect of the context on the position of the VAE-NS models onthe diversity-recognizability space by varying the number of samples used to compute the contextstatistics during the training phase (from 2 to 20 samples).
Importantly, varying the number ofcontext samples do not change the number of parameters of the network.
For all tested runs, weobserve a monotonic decrease of the samples diversity (see Fig. S13a) and a monotonic increase ofthe samples recognizability (see Fig.
S13b) when the number of context samples is increased.
Inthe diversity-recognizability space, the resulting curve is monotonically transporting models fromthe lower-right side to the upper-left side of the plot (see Fig.
4a, dark blue curve).
The effect of thenumber of context samples is large: the diversity is almost divided by 2 (from 2 .
4 to 1 . 2 )
and theclassiﬁcation accuracy is increased by 80% (from 53% to 96% ).
[Figure]
[Table] In contrast to the VAE-NS , the VAE-STN uses spatial attention to sequentially attend to sub-partsof the image to decompose it into simpler elements.
These sub-parts are then easier to encode andsynthesize.
In the VAE-STN , one can vary the number of attentional steps (i.e., the number ofattended locations) to modulate spatial attention.
Importantly, varying the number of attentional stepsdoes not change the number of parameters.
We have varied the number of attentional steps from 20 to90 .
The relationship between the number of attentional steps, the samples diversity, and the samplesrecognizability is non-monotonic.
We have used a parametric curve ﬁtting method (i.e., the leastcurve ﬁtting method from [21]) to parameterize the curve while maintaining the order of the datapoint (see S13 for more details on the ﬁtting procedure).
We report a convex parabolic relationshipbetween the number of attentional steps and the samples diversity (see Fig. S14a).
This curve isminimal at 60 steps.
We observe a concave parabolic relationship between the number of attentionalsteps and the recognizability of samples.
This curve is maximal at 60 attentional steps (see Fig. S14b).
In Fig.
4b we have plotted the parametric ﬁt illustrating the position of the VAE-STN models in thediversity-recognizability space when one increases the number of attentional steps (the light bluecurve).
This curve follows a quasi-linear trend with a sharp turn-around (at 60 attentional steps).
Theeffect of the number of attentional steps on the diversity-recognizability is limited compared to theeffect of the number of context samples.
Both VAE-NS and VAE-STN are trained to maximize the Evidence Lower Bound (ELBO), it is thenpossible to tune the weight of the prior in the loss function.
One can operate such a modulationby changing the β coefﬁcient in the ELBO loss function [24].
We refer the reader to S15 for moremathematical details about the ELBO.
A high β value enforces the latent variable to be closer toa normal distribution and increases the information bottleneck in the latent space.
Increasing β isknown to force the disentanglement of the generative factors [9].
We observe a monotonic decreasingrelationship between the value of β and the samples diversity for both the VAE-STN and the VAENS (see Fig.
S15a and Fig.
S16a, respectively).
We report a concave parabolic relationship betweenβ and the samples recognizability.
We use the least curve ﬁtting method to ﬁnd the optimal paraboliccurves [21].
This curve is maximal at β = 2 . 5 for the VAE-STN and at β = 3 for the VAE-NS (see
Fig.
S15b and Fig.
S16b, respectively).
The overall effect of β on the position of the VAE-likemodels on the diversity-recognizability space is relatively similar for both the VAE-STN and the
NS and follows a clear parabolic trend.
These curves demonstrate that one could modulate [Table] the value of β to maximize the recognizability [Table]
on [Table] the [Table]
[Table] space [Table] We have also varied the size of the latent space of the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) architecture (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58) the (cid:58) DA-GAN-UN , DA-GANRN , VAE-NS , and VAE-STN (cid:58)(cid:58) by
[Table] changing [Table] the
[Table] size [Table] of (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) latent
[Table] space. We did not ﬁnd any commontrend between the size of the latent variable, the diversity, and the recognizability (see S16 for moredetails). We observe that the DA-GAN-UN tends to produce slightly more recognizable but lessdiverse samples than the DA-GAN-RN while both architectures have the same number of parameters.
It suggests that the extra skip connections included in the U-Net architecture, in between the encoderand the decoder of the DA-GAN-UN , allow to trade diversity for recognizability. [Figure]
4.3 Comparison with humans
[Figure]
We now compare the tested models with the human data in the diversity-recognizability space.
Toperform such a comparison, we ﬁrst normalize all the model’s diversity and recognizability (includinghumans) using the z-score such that both axes are scaled and centered similarly.
Then, for all models,we compute the (cid:96) 2 -distance between models and humans in the diversity-recognizability space.
Weremind that the human data point is computed using the samples of the Omniglot test set.
Distancesto humans as well as their distributions are reported for all models in Fig.
5a.
The median of VAENS models is closer to humans, followed by DA-GAN-UN , DA-GAN-RN and VAE-STN (mediansare indicated in Fig.
5a with horizontal bars).
The VAE-NS model showing the smallest distance isalmost at the human level (see dark blue square Fig.
5a).
It has a context size of 20 samples (thehighest possible context size), and a β = 2 . 5 .
The VAE-STN model that best approximates humanhas a β = 2 .
25 and 60 attentional steps (see light blue square in Fig.
5a).
So far, we have reduced all models to single points by averaging the diversity and recognizabilityvalues over all classes.
We now study distances to humans for individual classes and for the VAENS and the VAE-STN models showing the shortest distance to humans (indicated by blue squares in Fig. 5a).
In Fig.
5b, we report distances to human for these 2 models and for 16 visual concepts.
The visual concepts 1 to 8 and 9 to 16 are selected so that they minimize the distance to humans withthe VAE-STN and the VAE-NS , respectively.
We observe that these visual concepts are differentfor the VAE-NS and the VAE-STN model.
Therefore, both models are well approximating humandata for some visual concepts but not for others.
Interestingly, we qualitatively observe that thevisual concepts 1 to 8 look simpler (i.e., made with fewer strokes) than the visual concepts 9 to16 .
It suggests that the spatial attention mechanism used by VAE-STN provides a better humanapproximation for simple visual concepts, while the context integration method leveraged by the
VAE-NS is more relevant to mimic human data on more complex visual concepts.
Discussion
In this article, we have described a novel framework for comparing computational models withhuman participants on the one-shot generation task.
The framework measures the diversity and therecognizability of the produced samples using metrics compatible with the one-shot scenario.
Tothe best of our knowledge, this is the ﬁrst and only framework speciﬁcally tailored to evaluate andcompare one-shot image generation models.
Among all tested algorithms, the VAE-NS is the best human approximator on Omniglot (see Fig.
5a).
It suggests that the context integration mechanism of the VAE-NS is an important component toreach human-like generalization.
Interestingly, motor learning experiments have demonstrated thathuman generalization performances are also strongly related to contextual information [50].
In (cid:58)(cid:58)(cid:58)(cid:58) [51] (cid:58) ,(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) author (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) [Table]
observer
In addition, we demonstrate that one can tune β so that the model becomes closer to humandata (see Fig. 4b).
This is consistent with a prior computational neuroscience study that has shown thatdisentangled VAEs (with β > 1 ) provide a good model of face-tuned neurons in the inferotemporalcortex [23].
Our comparison between the VAE-NS and the VAE-STN suggests that a model whichuses a spatial attention mechanism better ﬁts human data for simple visual concepts.
In contrast, thecontext integration mechanism of the VAE-NS appears to be a better human approximator for morecomplex visual concepts.
One could thus try to combine both mechanisms towards improving thesimilarity with human data independent of the complexity of the visual concept.
We have also foundthat GAN-like models ( DA-GAN-RN and DA-GAN-UN ) better account for human recognizabilitywhile [Table]
contrast, VAE-like models 345 ( VAE-NS and VAE-STN ) better account for human diversity.
An interesting approach would be toleverage a hybrid architecture (such as the VAE-GAN [34]) to try to better match human data.
Other candidate ingredients include the ability to harness compositionality [32] or the recurrentprocesses thought to be crucial for human generalization [54].
Compositionality could be introducedin one-shot generative algorithms by quantizing the latent space (as in the VQVAE [53]).
As a result,each coordinate of the latent variable represents an address in a codebook, and the role of the prioris then to combine simpler concepts to generate more complex samples.
One promising way toinclude recurrent processing into generative models is through the predictive coding framework [41].
Predictive Coding suggests that each processing step is part of an inference scheme that minimizesthe prediction error [6].
Previous work has demonstrated that such networks are more robust andexhibit improved generalization abilities [5, 11].
All these ingredients could be tested and comparedagainst human abilities using the diversity/recognizability framework we have proposed in this paper.
In the current version of the Omniglot dataset, the intra-class variability does not reﬂect the humanlevel of creativity.
It is mainly due to the experimental protocol in which one asks human participantsto copy a given visual concept.
The Omniglot dataset could be enriched with more diverse samples, byexplicitly asking human participants to be as creative as possible.
Other drawing databases with morecomplex symbols such as Quick Draw!
[27] could also be considered to strengthen the comparisonwith humans.
By decomposing the performance of the one-shot generation task along the recognizability vs.diversity axes we wanted to shed light on the relationship between generalization and creativity(quantiﬁed by the samples diversity in our framework).
We hope one can make use of our frameworkto validate key hypotheses about human generalization abilities so that we can better understandthe brain.
We argue that the best way to reach human-like generalization abilities is to unleash thealgorithms’ creativity.
Acknowledgement
This work was funded by ANITI (Artiﬁcial and Natural Intelligence Toulouse Institute) and the French
National Research Agency, under the grant agreement number : ANR-19-PI3A-0004.
Additionalfunding to TS was provided by ONR (N00014-19-1-2029) and NSF (IIS-1912280 and EAR-1925481).
Computing hardware supported by NIH Ofﬁce of the Director grant S10OD025181 via the Center for
Computation and Visualization (CCV).
We thanks Roland W. Fleming and his team for the insightfulfeedback and discussion about the diversity vs. recognizability framework. Checklist [Table] For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s [Table] contributions and scope? [Yes]
[Table]
(b) Did you describe the limitations of your work?
[Yes] Described in the conclusion.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] We havebrieﬂy discussed societal impact in S18.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
No theoretical
results
(b) Did you include complete proofs of all theoretical results?
[N/A]
No theoretical results
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] All the codes
are available on github ( link given in section 2.2)
(b) Did you specify all the training details (e.g., data splits, hyper-parameters,
how they
were chosen)?
[Yes] All the training details, as well as the architectural details aregiven in the supplementary materials.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] Error bars have been reported on 3 different runs
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
In the Supplementary Informationat section S
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a)
If your work uses existing assets, did you cite the creators?
[Yes] All the github linksare disclosed in the
Supplementary information, and the contributing authors are citedin the main article.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
The
github
link of our
code
is included in the article
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A] All assets we are using are publicly available
and do not require
any consent
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
No personal information in Omniglot
If
you
used crowdsourcing or
conducted
research with human subjects
(a) Did you include the full text of
instructions given to participants and screenshots
, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional
Review Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A] Supplementary InformationS1 More details on Prototypical NetArchitecture Table S1 describes the architecture of the Propotypical Net [47] we are using in this article.
We usethe Pytorch convention to describe the layers of the network.
[Table]
The overall number of parameters of the Prototypical Net we are using is around 292 K parameters.
The loss of the Prototypical Net is applied on the output of the last fully connected layers (of size128 ).
For the computation of the samples diversity, we extract the features on the ﬁrst fully-connectedlayer after the last convolutional layer (i.e., of size 256 ).
Training details
The Prototypical Net is trained in a 1-shot 60-ways setting and tested on a 1 -shot 20 -ways setting.
Thesize of the query set is always 1 for both training and testing phase.
The model is trained during 80epochs, with a batch size of 128 .
For training, we are using an Adam optimizer [28] with a learningrate of 1 × 10 − 3 (all other parameters of the Adam optimizer are the default ones).
We are schedulingthe learning rate such that it is divided by 2 every 20 epochs.
At the end of the training, the training accuracy (evaluated on 1000 episodes) has reached 100% andthe testing accuracy reaches a plateau at 96 .
S2 More details on SimCLRS2.1 Architecture and Data Augmentation
The architecture we are using for SimCLR [10] is the exact same than the one used for Prototypical
Net (see Table S1).
In SimCLR, we also extract the features on the ﬁrst fully-connected layer afterthe last convolutional layer (i.e., of size 256 ).
The augmentations we use are randomly chosen amongthe 3 following transformations
• Random resized crop: it crops random portion of the image and resizes it to a given size.
2 sets of parameters are used for this transformation: the scale and the ratio.
The scale
parameter speciﬁes the lower and upper bounds for the random area of the crop.
The ratioparameter speciﬁes the lower and upper bounds for the random aspect ratio of the crop.
Ourscale range is ( 0 . 1 , 0 . 9 ) and our ratio range is ( 0 . 8 , 1 . 2 ).
• Random afﬁne transformation: it applies a random afﬁne transformation of the image
while keeping the center invariant.
The afﬁne transformation is a combination of a rotation
(from − 15 ◦ to 15 ◦ ), a translation (from − 5 pixels to 5 pixels), a zoom (with a ratio from
75 to 1 . 25 )
and a shearing (from − 10 ◦ to 10 ◦ ).
• Random perspective transformation: apply a scale distortion with a certain probability to simulate 3D transformations.
The scale distortion we have chosen is 0 . 5 , and it is appliedto the image with a probability of 50%
Please see the site https://pytorch.org/vision/main/auto_examples/plot_transforms.html for illustration of the transformations.
Note that we have tried different settings for theaugmentations (varying the parameters of the augmentations), and we have observed a very limitedimpact of those settings on the computation of the samples diversity (see S3.3 for more details).
S2.2 Training details
Our SimCLR network is trained for 100 epochs with a batch size of 128 .
We used an RMSpropoptimizer [25], with a learning rate of 10 − 3 (all other parameters of the RMSprop are the defaultones).
S3 Control experiments for the samples diversity computationS3.1 Comparing the supervised and the unsupervised settings for the computation of thesamples diversity
To compare the unsupervised with the supervised setting, we have computed for all of the 150 classesof the Omniglot testing set the samples diversity.
We plot the samples diversity values for eachcategory and for both settings in Fig.
S1.
We report a linear correlation coefﬁcient R 2 = 0 .
74 and a
Spearman rank order correlation ρ = 0 . 85 (see Table S2, ﬁrst line).
It does mean that the samplesdiversity, as computed with one of the setting, is strongly correlated both in terms of rank order andexplained variance, with the samples diversity as computed with the other setting.
samples diversity is computed applying the standard deviation (see Eq. 2) on SimCLR features (for x-axis) or on the features of Prototypical Net (for y-axis)
S3.2 More control experiments on the effect of the dispersion measure
To make our analysis more robust we have conducted additional control experiments with differentmeasures of dispersion.
In Eq.
2 we have presented a classical measure of dispersion that is thestandard deviation.
Another measure of data dispersion is the pair-wise cosine distance among the
[Table]
samples belonging to the same class:
[Equation]
v
[Equation]
In Eq.
3, C denotes the cosine similarity.
In Fig.
S2a, we plot the samples diversity for both featureextraction networks but with a dispersion measure based on the pairwise cosine distance as formulatedin Eq. 3.
We report a linear correlation of R 2 = 0 .
57 and a Spearman rank order correlation ofρ = 0 .
71 (see second line of Table S2).
This control experiment suggests that even by using adifferent dispersion metric (i.e., the pairwise cosine distance), the 2 feature extraction networksproduce samples diversity values that are heavily correlated.
This strengthen our observation madein S3.1: the representations produced by the SimCLR and Prototypical Net are similar.
Anotherinteresting control experiment is to compare the impact of the dispersion measure on the samplesdiversity metric.
To do so, we have compared the samples diversity computed with one featureextractor (either Prototypical Net in Fig.
S2b or SimCLR in Fig.
S2c)
but for 2 different dispersionmetrics (i.e., the standard deviation as formulated in Eq.
2 and the pairwise cosine distance as deﬁnedin Eq. 3).
In both cases, we have a non negligible linear correlation (i.e., R 2 > 0 . 44 ) and a strong
Spearman rank order correlation (i.e., ρ > 0 .
63 , see third and fourth lines of Table S2).
All thesecontrol experiments conﬁrm that our computation of the samples diversity is robust to 1) the type ofapproach we used to extract the features and 2) the measure of dispersion we are using to computethe intraclass variability.
[Figure]
S3.3 Impact of the image augmentation on the diversity measure
To test the impact of the image augmentations on the SimCLR network we have trained 3 SimCLRnetworks with different augmentation levels.
• With moderate level of image augmentation.
All the augmentations here are those describedin section S2.• With a low level of image augmentation.
Here the scale of the random resized crop isvaried from 0 .
05 to 0 .
95 and the crop ratio is ranging from 0 .
9 to 1 . 1 .
The rotation of theafﬁne transformation is ranging from − 7 deg to 7 deg , the translation from − 3 pixels to 3pixels, the zoom from 0 .
9 to 1 . 1 and the shearing from − 5 deg to 5 deg .
The scale distortionapplied to the image is 0 .
25 (with a probability of 50 %).•
With a high level of image augmentation.
In this setting, the scale of the random resizedcrop is varied from 0 .
2 to 0 .
8 and the crop ratio is ranging from 0 .
6 to 1 . 4 .
The rotationof the afﬁne transformation is ranging from − 30 deg to 30 deg , the translation from − 10pixels to 10 pixels, the zoom from 0 .
5 to 1 . 5 and the shearing from − 20 deg to 20 deg .
Thescale distortion applied to the image is 0 .
75 (with a probability of 50 %).
In Fig.
S3, we compare the samples diversity obtains for each category of the Omniglot testing setwhen we train the SimCLR network with moderate level of image augmentation and with a low levelof image augmentation (see Fig.
S3a), or with a high level of image augmentation (see Fig.
S3b).
Wealso report the Spearman correlation in Table S3.
[Figure]
[Table]
We observe a high linear correlation as well as a high Spearman rank order correlation between thetested settings.
It suggests that the samples diversity is relatively independent to the level of imageaugmentations used during the SimCLR training.
S3.4 T-SNE of the SimCLR and Prototypical Net latent space
In Fig.
S4a and Fig.
S4b, we show a t-SNE analysis of the feature space of Prototypical Net and
SimCLR respectively.
In Fig.
S4a, the t-SNE analysis of the Prototypical Net feature space reveals
[Figure]
a strong clustering of the samples belonging to the same class.
Note that this phenomenon is notsurprising as the loss of the Prototypical Net forces the samples belonging to the same class to beclose in the feature space.
More surprisingly, we also observe a clustering effect in the SimCLRt-SNE analysis (see Fig.
S4b).
Note that SimCLR is a fully unsupervised algorithm: there is no classinformation given to the algorithm.
Consequently, the strong clustering effect we observe suggeststhat forcing the proximity between a sample and its augmented version is enough to retrieve the classinformation.
This observation might explain why contrastive learning algorithms are in general soefﬁcient in semi-supervised (or even unsupervised) classiﬁcation tasks.
S6 MAML architecture and training details
The architecture we have used for the MAML classiﬁer is exactly the same used for the Prototypical
Net (see Table S1).
The only difference is the last fully-connected layer that is : Linear(256, 20).
Indeed, as the MAML network is directly predicting the logits (and not a distance metric), the lastlayer needs to have the same dimension than the number of class of the experiment.
In a 1 -shot20 -way classiﬁcation experiment, the number of classes is 20 .
We have used a 2 nd order meta-learning scheme [18].
The outer-loop optimizer is an Adam optimizerwith a learning rate of 10 − 3 , and the inner-loop optimizer is a simple Stochastic Gradient Decent
witha learning rate of 10 − 2 .
The number of inner loops is set to 5 during the training and to 10 during thetesting.
The number of tasks for each outer-loop is set to 4 .
S7 Control experiments: Comparing Prototypical Net and MAML
To rigorously compare MAML and Prototypical Net, we have conducted 2 types of control experiments.
First we have veriﬁed whether the classiﬁcation accuracy obtained for each class were rankedin the same order for both MAML and Prototypical Net.
To do so, we have presented the same seriesof categorization tasks to both algorithms.
The high Spearman rank coefﬁcient ( ρ = 0 . 60 ) indicatesthat both classiﬁers rank each category’ classiﬁcation accuracy similarly (see section S4).
To conﬁrm this result, we have computed the correlation between the logits generated by both models.
In the case of the MAML model, extracting the logits is straightforward.
For Prototypical Net, weuse the distance to prototypes as logits.
This explain why both model’s logits are anti-correlated:the MAML logits are the (un-normalized) probability of belonging to a given classes whereas the
Prototypical Net logits correspond to the distance to the category (so the lower the distance, thehigher the probability).
We report a strong negative correlation ( r = − 0 . 62 ) between the logits of the
MAML network and those of Prototypical Net (see section S4).
[Table]
S8 Architecture and training details of
the VAE-STNS8.1 Architecture of the VAE-STN
The VAE-STN is a sequential VAE that allows for the iterative construction of a complex image [42].
A pseudo-code of the algorithm is described in Algo 1.
At each iteration, the algorithm focuses itsattention on a speciﬁc part of the image ( x ), the prototype ( ˜ x ) and the residual image ( ˆ x ) using the
Reading Spatial Transformer Network (STN r ).
Then the extracted patch is passed to an encodingnetwork (EncBlock) to transform it into a latent variable.
This latent variable is concatenated to apatch extracted from the prototype and then passed to the RecBlock network.
The produced hiddenstate is ﬁrst passed to DecBlock to recover the original patch, and then to the STN w to replace andrescale the patch into the original image.
The LocNet network is used to learn the parameter of theafﬁne transformation we used in the STN.
Note that the afﬁne parameters used in STN w are simplythe inverse of those used in STN r .
Algorithm
Input: image: x , prototype: ˜ x c ←θ 1 ← [[1 , 0 , 0] , [0 , 1 , 0]] h 1 ←for i = 1 to N steps do ˆ x = x − sigmoid ( c ) r , ˆ r , ˜ r
=
STN r ( θ t , x ) , STN r ( θ t , ˆ x ) , STN r ( θ t , ˜ x ) r ← [ r , ˆ r , ˜ r , h t ] µ , σ = EncBlock ( r ) z
= µ + (cid:15)σ with (cid:15) ∼ N (0 , 1)
z ← [ z , ˜ r ] p =
DecBlock ( h t ) c
← c + STN w ( θ − 1 t , p ) h t +1 ← RecBlock ( z , h t ) θ t + 1 ← LocNet ( h t +1 ) end for
The STN modules take 2 variables in input: an image (or a patch in the case to the STN w ) and amatrix (3 × 2) describing the parameters of the afﬁne transformation to apply to the input image [26].
All other modules are made with MLPs networks, and are described in Table S5.
In the Table S5 weuse the following notations:
• s z : This the size of the latent space.
In the base architecture, we set s z = 80 .
• s LSTM :
This is the size of the output of the Long-Short Term Memory (LSTM) unit.
In thebase architecture, we set s LSTM =
• s r : This is the resolution of the patches extracted by the Spatial Transformer Net (STN)during the reading operation.
In the base architecture we set s r = 15 .
• s loc :
This is the number of neurons used at the input of the localization network.
In the basearchitecture, we set s loc = 100•
s w : This is the resolution of the patch passed to the the STN network for the writingoperation.
In the base architecture s w = 15 .
For the base architecture we used N steps = 60 .
The base architecture of the VAE-STN has 6 .
2millions parameters
For more details on the loss function, please refer to [42].
S8.2 Training details of the VAE-STN
The VAE-STN is trained for 500 epochs, with batches of size 128 .
We use an Adam optimizer with alearning rate of 1 × 10 − 3 and β 1 = 0 .
All other parameters are the default Pytorch parameters.
Toavoid training instabilities we clip the norm of the gradient to 5 .
The learning rate was divided by 2when the evaluation loss has not decreased for 10 epochs (reduce on plateau strategy).
[Table]
EncBlock(s r , s LSTM , s z )
[Table]
DecBlock(s LSTM , s loc , s w )
[Table]
RecBlock(s z , s
[Table]
, s LSTM )
[Table] [Table]
[Table]
S8.3 VAE-STN samples
[Figure]
[Table] [Table]
Table S6 describes the base architecture of the Neural Statistician model adopted from [19] which isa close approximation of [15].
We make minor changes in the network architecture to accommodatethe higher input image size of 50 × 50 of the Omniglot dataset.
The Neural Statistician model iscomposed of the following sub-networks:
• Shared encoder x (cid:55)→ h :
An instance encoder E that takes each individual datapoint x i to a feature representation h i
=
E ( x i ) .
• Statistic network q ( c | D, φ ) : h 1 , ..., h k (cid:55)→ µ c , σ 2 c : A pooling layer that aggregates thematrix ( h 1 , ..., h k ) to a single pre-statistic vector v .
[15] uses sample mean for theirexperiments
Which is followed by a post-pooling network that takes v to a parametrization of a Gaussian.
• Inference network q ( z | x, c, φ ) : h, c (cid:55)→ µ z , σ 2 z : Inference network gives an approximate posterior over latent variables.
• Latent decoder network p ( z | c ; θ ) : c (cid:55)→ µ z , σ 2 z• Observation decoder network p ( x | c, z ; θ ) : c, z (cid:55)→ µ x
The overall number of parameters of the base model (which has the same architecture as used in[15]) for the Neural Statistician we are using is around 7.48M parameters.
Training details
The Neural Statistician is trained for 300 epochs, with batch size of 32 and learning rate of 1 × 10 − 3 .
We adopt the same setting of the Neural Statistician as used in [15] for the omniglot dataset.
We constructed context sets by splitting each class into datasets of size 5 while training, anduse a single out-of-distribution exemplar while testing.
As discussed in the paper, we createnew classes by reﬂecting and rotating characters.
We based our implementation from https://github.com/georgosgeorgos/hierarchical-few-shot-generative-models and https://github.com/comRamona/Neural-Statistician .
[Table]
(cid:58)(cid:58)
[Table]
context (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) correspond (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) samples (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) used (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) during (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) training, (cid:58)(cid:58) to (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)
evaluate(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) statistics (cid:58)(cid:58) of (cid:58)(cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) speciﬁc (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) category (cid:58)(cid:58)(cid:58)(cid:58) (i.e. (cid:58)(cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) concept).
(cid:58)(cid:58)(cid:58)
In (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) practice, (cid:58)(cid:58)(cid:58) we (cid:58)(cid:58)(cid:58)(cid:58) pass (cid:58)(cid:58)(cid:58) to (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) network (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) different(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) samples (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) representing (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) same (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) concept (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58) we (cid:58)(cid:58)(cid:58)(cid:58) vary (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) number (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) these (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)
[Table]
ConvBlock
[Table]
ConvBlock
[Table]
ConvBlock
[Table]
ConvBlock
[Table]
ConvBlock
[Table]
ConvBlock
[Table] [Table] [Table] [Table]
ConvBlock
[Table]
ConvBlock
[Table]
[Table]
BatchNorm1d
[Table]
(cid:55)→
[Table]
[Table]
[Table]
{3× FcBlock(
[Table]
Linear(
[Table]
32), BatchNorm1d(
[Table]
to µ z
[Table]
σ
[Table]
Linear(512, 256) (cid:55)→ c , ELU
[Table]
{3× FcBlock(
[Table]
Linear(256, 32), BatchNorm1d(1) to µ z , log σ 2 z
Observation decoder network
FcBlock(512, 256) (cid:55)→ z 3,324,
FcBlock(512, 256) (cid:55)→ c combine z and c, ELU
FcBlock(256, 256*4*4)
ConvBlock(256, 256, 1) ConvBlock(256, 256, 1)
DeConvBlock(256, 256)
ConvBlock(256, 128, 1)
ConvBlock(128, 128, 1)
DeConvBlock(128, 128)
ConvBlock(128, 64, 1)
Conv2d(64, 64, 4, 1, 0)
DeConvBlock(64, 64)
ConvBlock(64, 32, 1)
Conv2d(32, 32, 2, 1, 0)
DeConvBlock(32, 32)
Conv2d(32, 1, 1)
[Table]
[Figure] [Table] Table S7 describes the base architecture of the DA-GAN-UN’s Generator model adopted from [1].
We have modiﬁed the architecture of the DA-GAN-UN model such that it can accommodate a higherinput image size 50 × 50 .
Also, we reduced the number of trainable parameters in the original
DA-GAN-UN architecture to have a fair comparison with other few-shot models.
Following are thenotations used in Table S7:
• s z : This is the size of the latent space.
In the base architecture, we set s z =
128• Generator G ( x, z ) :
A generator network that takes data points and Gaussian noise as input,and generate new samples.
The base architecture of the DAGAN model we are using in our experiments has around 6.8 million parameters
Training details
The DA-GAN-UN model was trained for 30 epochs, with batches of size 32.
We use an Adamoptimizer with a learning rate of 1 × 10 − 4 and β 1 = 0.9.
We update our generator after every 5updates of discriminator.
We based our implementation from https://github.com/amurthy1/dagan_torch [Table]
ConvBlock(In c , Out c , s l )
Conv2d
[Table]
LeakyReLU
[Table]
), BatchNorm2d
[Table]
DeConvBlock(In c , Out c , s l ) ConvTranspose2d(In c , Out c , 3, stride= s l , padding=1) Out
[Table]
× (In
[Table]
×
[Table]
LeakyReLU(0.2), BatchNorm2d [Table] Out
[Table]
[Table]
EncoderBlock(In p , In c , Out c ) ConvBlock(In p , In p ) ConvBlock(In c + In p , Out c )
{+Conv2d(In c + Out c , Out c ) Conv2d(In c + 
 × Out c , Out c ) DecoderBlock(In p , In c , Out c )+}
DeConvBlock(In p , In p , 1) ConvBlock(In c +In p , Out c , 1) DeConvBlock(In p , In p , 1)
ConvBlock(In c + In p + Out c , Out c , 1) DeConvBlock(In c + 2 × Out c , Out c , 1)
Generator(s z ) ConvBlock(1, 64, 2) 6,813, EncoderBlock(1, 64, 64) EncoderBlock(64, 64, 128)
EncoderBlock(128, 128, 128) Linear(s z , 4 × 4 × 8) DecoderBlock(0, 136, 64) Linear(s z , 7 × 7 × 4)
DecoderBlock(128, 260, 64)
Linear(s z , 13 × 13 × 2)
DecoderBlock(128, 194, 64)
DecoderBlock(64, 128, 64)
DecoderBlock(64, 65, 64)
ConvBlock(64, 64, 1)
ConvBlock
[Table]
training  Architecture
We use the same base architecture of DA-GAN-UN , except we remove the skip connections betweenthe contracting path (encoder) and the expansive path (decoder).
[1] used a combination of UNetand ResNet in their results, in DA-GAN-RN we consider only a ResNet type architecture.
Training details
Refer S10 for training details.
[Figure]
We observe a monotonic decrease of the diversity and a monotonic increase of the recognizabilitywhen the number of context samples increases.
We vary the number of context samples from 2 to20 .
This experiment has been conducted with 3 different seeds (i.e., different network initialization),represented with red, green and blue data points, respectively.
For each seed, we report 19 datapoints.
To highlight the trend in the diversity-recognizability space, we have smoothed the curves in
Fig.
S13a and Fig.
S13b, using a Savitzky-Golay ﬁlter (second order, window size of 7). S
of the number of attentional steps on the diversity/recognizability framework
[Figure]
when one varies the number of attentional steps from 20 to
In this experiment, we have varied the number of attentional steps from 20 to 90 .
Note that wecould not go below 20 attentional steps to make sure the attentional process is fully covering theentire image.
We did not go over 90 attentional steps because we faced some training instabilitiesbeyond this point.
We observe a non-monotonic evolution of the diversity and the recognizabilitywith the increase of the number of attentional steps.
This experiment has been conducted with 3different seeds (i.e., different network initialization), represented with red, green and blue data points,respectively.
For each seed we report 8 data points.
In order to properly assess the type of parametriccurves that govern the evolution of the diversity-recognizability space when one varies the number ofattentional steps, we have used a least curve ﬁtting method [21].
This method involves ﬁnding the best polynomial ﬁt (second order in our case) for the 3 curves (Fig.
S14a, b and c) simultaneously.
This method is iteratively reﬁning all the ﬁts to minimize the sum of all least square error.
S14 Mathematical formulation of the ELBO
Let us consider a dataset X
= { x ( i ) }
Ni =1 composed of N i.i.d samples of a random variable x .
Weassume that x is generated by some random process involving an unobserved random variable z .
Thelatent variable z is sampled from a Gaussian distribution (see Eq. 5).
The mean of the likelihood isparametrized by µ θ (in which θ denotes the parameters) and its variance is considered constant.
[Equation]
The Variational Auto Encoder is optimized by maximizing the Evidence Lower Bound (ELBO), asformalized in its simplest form in Eq. 6: [Equation]
[Equation]
One could observe that the β coefﬁcient is tuning the importance of the prior (through the KL).
Ifβ > 1 , then the latent space will be forced to be closer to the prior distribution but will attenuate theweight of the reconstruction loss.
Such a scenario tends to improve the disentanglement of the latentspace [24].
On the contrary, if β is low, then the reconstruction loss (i.e., E q φ ( z | x ) [log p θ ( x | z )] )will take over, and then the latent space will be less regularized.
Note that in the extreme case whereβ = 0 , the VAE becomes an auto-encoder.
The ELBO loss can be updated to include a latent variable encoding for the context c as in the
VAE-NS .
In this formulation, the context corresponds to a dataset D (see Eq. 7):
[Equation]
The ELBO could also be extended to include a sequential generative process as in the VAE-STN .
Inthis case, the latent variable z is time-indexed and is now a sequence of random variables denoted( z 1 , .., z T ) .
In Eq.
8, z <k indicates the collection of all latent variables from step t = 1 to t = k .
[Equation]
S15 Effect of the beta coefﬁcient on the diversity/recognizability framework
In this experiment, we have varied the value of the β coefﬁcient from 0 .
25 to 4 for the VAE-STN andfrom 0 .
25 to 5 for VAE-NS model.
This experiment has been conducted with 3 different seeds(i.e.
, different network initialization), represented with red, green and blue data points, respectively.
For the VAE-STN and for each seed, we have collected 16 data points (see Fig.
S15), and 20 forthe VAE-NS (see Fig.
S16).
We use a similar method than in S13 to ﬁnd a polynomial ﬁt (secondorder in our case) of the curves shown in Fig.
S15a, b, and c and Fig.
S16a, b, and c.
We reporta quasi-monotonic decline of the diversity when the beta value is increased (see Fig.
S15a and
Fig. S16a).
In contrast, the recognizability follows a parabolic relationship when the beta value isincreased.
For the VAE-STN , the maximum recognizability ( ≈ 80% ) is reached for a β value of 2 .
25(see Fig.
S15b).
For the VAE-NS , the maximum recognizability ( ≈ 91% ) is reached for a β value of3 (see Fig.
S16b).
Even if the change of amplitude in recognizability and in diversity is larger for the
VAE-STN than for VAE-NS , the shapes of the curves are very similar.
[Figure]
[Figure] [Figure]
Effect of the size of the latent space on the diversity/recognizability framework
[Figure] [Figure]
[Figure]
S17 (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Overﬁtting (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) standard (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) classiﬁer (cid:58)(cid:58) in (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) low-data (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) regime(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)
Omniglot (cid:58)(cid:58) is (cid:58) [Table] [Table] from (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) other (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) datasets (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)
(e.g. (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) MNIST, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) CIFAR10...).(cid:58)(cid:58)
In (cid:58)(cid:58)(cid:58)(cid:58) such (cid:58)(cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) low-data (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) regime, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) standard (cid:58)(cid:58)(cid:58)(cid:58) deep (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) learning (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) classiﬁers (cid:58)(cid:58)(cid:58) are (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) known (cid:58)(cid:58) [Table]
experimental(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) conﬁrmation (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) such (cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) phenomenon.
(cid:58)(cid:58)(cid:58)(cid:58) We (cid:58)(cid:58)(cid:58)(cid:58) have (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trained (cid:58) 3 (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) different (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) classiﬁers, (cid:58)(cid:58)(cid:58) all (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) having (cid:58) a (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) similar (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) architecture (cid:58)(cid:58)(cid:58)(cid:58) (the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)
[Table] [Table]
(cid:58)(cid:58)(cid:58)(cid:58) have(cid:58)(cid:58) an (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) output (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) activation (cid:58)(cid:58)(cid:58) of (cid:58)(cid:58)(cid:58) size (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) 1623 .
(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Said (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) differently, (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) layer (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) entitled (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) "Linear(256, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) 128)"(cid:58)(cid:58) in (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Table (cid:58)(cid:58)(cid:58) S1 (cid:58)(cid:58)(cid:58) has (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) been (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) replaced (cid:58)(cid:58)(cid:58) by (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) "Linear(256, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) 1623)".
(cid:58)(cid:58)(cid:58)(cid:58) We (cid:58)(cid:58)(cid:58)(cid:58) have (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trained (cid:58)(cid:58)(cid:58) this (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) classiﬁer(cid:58)(cid:58)(cid:58)(cid:58) using (cid:58)(cid:58)(cid:58) 18 (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) samples (cid:58)(cid:58)(cid:58) per (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) class (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Omniglot (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) dataset. (cid:58)(cid:58)(cid:58)(cid:58) The (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) testing (cid:58)(cid:58)(cid:58) set (cid:58)(cid:58) is (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) composed (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58) the(cid:58) 2 (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)
remaining (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) samples (cid:58)(cid:58)(cid:58) per (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) class. (cid:58)(cid:58)(cid:58) To (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) summarize, (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) training (cid:58)(cid:58)(cid:58) set (cid:58)(cid:58) is (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) composed (cid:58)(cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) 29 , 214(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) samples (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ( 1623 × 18 ) (cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) training (cid:58)(cid:58)(cid:58) set (cid:58)(cid:58) is (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) composed (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) 3246 (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) samples (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) ( 1623 × 2 ). (cid:58)(cid:58)(cid:58)(cid:58) This(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) classiﬁer (cid:58)(cid:58) is (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) trained (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)
[Table]
[Figure] [Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure]
[Table] [Table] [Table]
[Table] [Table]
[Table] set).
(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Train/test (cid:58)(cid:58)(cid:58) loss (cid:58)(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) classiﬁcation (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) accuracy (cid:58)(cid:58)(cid:58) are (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) reported (cid:58)(cid:58)(cid:58) for (cid:58)(cid:58) all (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) training (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) epochs
(cid:58)(cid:58) in (cid:58)(cid:58)(cid:58) Fig (cid:58)(cid:58)(cid:58)(cid:58) S21b (cid:58)(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58) Fig (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) S21e, (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)
[Table] [Table] [Table] [Table]
epochs (cid:58)(cid:58)(cid:58) in (cid:58)(cid:58)(cid:58) Fig (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) S21c (cid:58)(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58) Fig (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) S21f,(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) respectively.
(cid:58)(cid:58)(cid:58)(cid:58) For (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) standard (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) classiﬁer, (cid:58)(cid:58)(cid:58) we (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) observe (cid:58)(cid:58)(cid:58) an (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) increase (cid:58)(cid:58) of (cid:58)(cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58) test (cid:58)(cid:58)(cid:58)(cid:58) loss (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) [Table]
increasing), (cid:58)(cid:58)(cid:58) see(cid:58)(cid:58)(cid:58) Fig (cid:58)(cid:58)(cid:58)(cid:58) S21a (cid:58)(cid:58)(cid:58)(cid:58) and (cid:58)(cid:58)(cid:58)(cid:58) Fig (cid:58)(cid:58)(cid:58)(cid:58)(cid:58) S21d.
(cid:58)(cid:58)(cid:58) It (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) suggests (cid:58)(cid:58)(cid:58)(cid:58) that (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) network (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) becomes (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) better (cid:58)(cid:58) at (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) classifying (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) training(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) samples (cid:58)(cid:58)(cid:58)(cid:58) but (cid:58)(cid:58)(cid:58)(cid:58) worst (cid:58)(cid:58) at (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) dealing (cid:58)(cid:58)(cid:58)(cid:58) with (cid:58)(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) testing (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) samples. (cid:58)(cid:58)(cid:58)(cid:58)
The (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) standard (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) classiﬁer (cid:58)(cid:58) is (cid:58)(cid:58)(cid:58)(cid:58) then (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) overﬁtting (cid:58)(cid:58) on(cid:58)(cid:58) the (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) training (cid:58)(cid:58)(cid:58) set.
(cid:58)(cid:58)(cid:58)(cid:58)(cid:58) Note (cid:58)(cid:58)(cid:58)(cid:58) that (cid:58)(cid:58)(cid:58) the {+(cid:58) 
 (cid:58)(cid:58)(cid:58)(cid:58)(cid:58)+}
[Table] [Table] [Table]
S18 Computational Resources
All the experiments of this paper have been performed using Tesla V100 with 16 Gb memory.
Thetraining time is dependent on the hyper-parameters, but varies between 4h to 24h per simulation.
S19 Broader Impact
This work does not present any foreseeable negative societal consequences.
We think the societalimpact of this work is positive.
It might help the neuroscience community to evaluate the differentmechanisms that allow human-level generalization, and then better understand the brain.
[Figure]
[Table]